Image processing method and electronic device

By performing multiple image drawings between the first Vsync signal and the second Vsync signal in the electronic device, and using the map buffer queue priority configuration and wake-up indication mechanism, the lag problem caused by excessive image rendering is solved. It achieves timely completion of image drawing and improvement of user experience.

WO2024244595A1PCT designated stage expired Publication Date: 2024-12-05HONOR DEVICE CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/080782
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-31
Filing Date
2024-03-08
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

When the electronic device is sliding, the lag caused by the long-term image rendering process affects the user experience.

Method used

By performing at least two new images drawings between the first Vsync signal and the second Vsync signal and completing the drawing before the second Vsync signal arrives, display lags due to excessively long image drawing time are avoided. The specific method includes performing a first drawing operation after the first Vsync signal arrives, performing a second drawing operation before the second Vsync signal arrives, and performing a synthetic display after the third Vsync signal arrives, and using the map buffer queue priority configuration and wake-up instructions. The mechanism performs drawing operations in advance.

Benefits of technology

It effectively avoids display lag due to the long image drawing time, ensures that the image has been drawn when the specified Vsync signal arrives, and ensures the continuity and fluency of the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024080782_05122024_PF_FP_ABST
    Figure CN2024080782_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Embodiments of the present application relate to the field of image processing, and provide an image processing method and an electronic device, capable of effectively solving the problems of stuttering and the like caused by long time consumption in an image drawing process. The method may comprise: receiving a first operation of a user; in response to the first operation, running a first application, and presenting an interface of the first application; receiving a second operation of the user on the interface of the first application, the second operation being a sliding operation; in response to the second operation, performing a first drawing operation within a first synchronization Vsync period to obtain a first image; and in response to the second operation, performing a second drawing operation within the first synchronization Vsync period to obtain a second image.
Need to check novelty before this filing date? Find Prior Art

Description

Image processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 31, 2023, with application number 202310639151.0 and invention name “An Image Processing Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method and electronic device. Background Art

[0003] When an electronic device provides a display function to a user, it often switches and displays consecutive frame images under the user's instruction.

[0004] For example, after the user inputs a sliding operation to the electronic device, the electronic device can continuously render frame images and update the current display interface according to the user's sliding operation.

[0005] In this way, if one or more frames in the continuous frame images cannot be rendered in time, different degrees of frame loss or frame skipping will occur, thereby affecting the user experience.

[0006] Summary of the Invention

[0007] The present application provides an image processing method and electronic device, which can effectively avoid problems such as freezes caused by a long image drawing process.

[0008] To achieve the above technical objectives, this application adopts the following technical solutions:

[0009] In a first aspect, an image processing method is provided, which is applied to an electronic device and includes: receiving a first operation from a user; in response to the first operation, running a first application and presenting an interface of the first application; receiving a second operation from the user on the interface of the first application, the second operation being a sliding operation; in response to the second operation, performing a first drawing operation within a first synchronous Vsync cycle to obtain a first image; and in response to the second operation, performing a second drawing operation within the first synchronous Vsync cycle to obtain a second image, wherein the first Vsync cycle starts at the time when a first Vsync signal is received, the first Vsync signal ends at the time when a second Vsync signal is received, the first Vsync signal and the second Vsync signal are adjacent Vsync signals, and the start time of the second drawing operation is later than the start time of the first drawing operation. The first image and the second image are images displayed sequentially by the electronic device in response to the second operation.

[0010] It should be noted that in this application, the first drawing operation and / or the second drawing operation are specific implementations of the drawing operation. The drawing operation may include the complete process of waiting for resources and rendering processing after the flutter framework is awakened.

[0011] In this way, after the first Vsync signal arrives and before the second Vsync signal arrives, the electronic device can execute at least two renderings of the new image. This ensures that, even if the rendering process of the second image takes a long time, the rendering of the second image is completed before the Vsync signal that is required for the second image to be displayed arrives. This avoids the problem of display lag caused by the first image being continuously displayed for multiple Vsync signal cycles due to the long rendering time of the second image not being completed when the corresponding Vsync signal arrives.

[0012] Optionally, the method further includes: after the second Vsync signal is generated, the electronic device performing a first display operation, the first display operation being used to control the display screen of the electronic device to display the first image. After the third Vsync signal is generated, the electronic device performing a second display operation, the second display operation being used to control the display screen of the electronic device to display the second image. The third Vsync signal is later than the second Vsync signal, and the third Vsync signal is adjacent to the second Vsync signal.

[0013] In this way, after the first image is drawn, it can be synthesized and displayed in the period after the second Vsync signal arrives. Correspondingly, after the second image is drawn, it can be synthesized and displayed in the period after the third Vsync signal arrives. It can be understood that because the second image is drawn before the second Vsync signal, the second image can be completed before the third Vsync signal arrives, thereby ensuring that the second image can be correctly synthesized and displayed after the third Vsync signal arrives.

[0014] Optionally, the duration between the first Vsync signal and the second Vsync signal, and the duration between the second Vsync signal and the third Vsync signal, all correspond to the resolution of the display screen. For example, the duration between the first Vsync signal and the second Vsync signal, and the duration between the second Vsync signal and the third Vsync signal, can be the inverse of the resolution of the display screen.

[0015] Optionally, the electronic device is configured with a texture buffer queue TextureBufferQueue, which

[0016] TextureBufferQueue includes at least one valid buffer, each of which stores a completed image. In this example, the valid buffer field length of the TextureBufferQueue in the electronic device can be the length of one buffer or the length of multiple buffers. When the valid buffer field length of the TextureBufferQueue is the length of multiple buffers, it corresponds to that the TextureBufferQueue includes multiple valid buffers. It can be understood that when the TextureBufferQueue includes multiple valid buffers, the TextureBufferQueue can simultaneously store multiple images that have been completed and are waiting to be synthesized and displayed.

[0017] Optionally, the at least one valid buffer includes a first buffer. The performing a first drawing operation within a first synchronous Vsync period to obtain a first image includes: performing the first drawing operation within the first Vsync period to obtain the first image, and when the first buffer is empty, storing the first image in the first buffer.

[0018] Optionally, the at least one valid buffer includes a second buffer. The performing a second drawing operation within the first synchronous Vsync period to obtain a second image includes: performing the second drawing operation within the first Vsync period to obtain the second image, and storing the second image in the second buffer if the first buffer is not empty.

[0019] In this example, when the second image is finished drawing, the first image has not yet been consumed. In this way, the first image can continue to be stored in the first buffer, and the second image is stored in the second buffer.

[0020] Optionally, the display priority of the first buffer is higher than that of the second buffer. The electronic device performs a first display operation, including: acquiring the first image from the first buffer with the higher display priority, so as to synthesize and display the first image.

[0021] Optionally, after the electronic device executes the first display operation, the method further includes: releasing the first image in the first buffer, and the first buffer is empty after the release.

[0022] Optionally, after executing the first display operation, the method further includes: storing the second image in the second buffer into the first buffer, and releasing the second image in the second buffer.

[0023] Optionally, the electronic device performs a second display operation, including: acquiring the second image from the first buffer, so as to synthesize and display the second image.

[0024] In this way, by configuring the priorities of different buffers, no matter how many valid buffers are included in the TextureBufferQueue, the electronic device can obtain the image from the first buffer for synthesis and display when synthesis and display are needed.

[0025] Optionally, the electronic device is configured with a callback queue, which is used to store display callbacks and / or drawing callbacks. Before the electronic device performs the first drawing operation, the method also includes: after the first Vsync signal is generated, the electronic device determines that the callback queue includes a first drawing callback. The first drawing callback includes first image information. The first image information is used to draw the first image. The electronic device performs the first drawing operation, including: the electronic device generates a first wake-up indication based on the first drawing callback. The first wake-up indication carries the second image information. The first wake-up indication is used to wake up the flutter rendering module configured in the electronic device to perform a drawing operation based on the second image information. According to the first wake-up indication, the flutter rendering module performs the first drawing operation to obtain the second image.

[0026] Optionally, the electronic device performs a second drawing operation, including: the electronic device generates a second wake-up instruction. The second wake-up instruction carries second image information. The second wake-up instruction is used to wake up the flutter rendering module to perform a drawing operation based on the second image information. According to the second wake-up instruction, the flutter rendering module performs the second drawing operation to obtain the second image.

[0027] Optionally, the method further includes: after the first Vsync signal is generated, the electronic device determines that a second image needs to be continued to be drawn. The electronic device generates a second wake-up indication, including: when the electronic device determines that the second image needs to be continued to be drawn, triggering the generation of the second wake-up indication after a preset time period.

[0028] Optionally, the preset duration corresponds to 2 / 3 of the first Vsync period.

[0029] In this way, a solution for executing the drawing operation of the second image in advance is provided. It can be understood that in some other implementations, the preset duration can also be flexibly configured according to actual conditions. For example, the electronic device can configure a preset duration greater than or equal to the average drawing duration based on the average drawing duration of the completed images, and the average drawing duration can be less than the first Vsync cycle. In this way, the flutter framework can start triggering the execution of the drawing of the second image after the first image is completed, even if the next Vsync signal has not arrived, and the electronic device can also start triggering the execution of the drawing of the second image according to the end of the preset duration.

[0030] Optionally, the method also includes: the electronic device encapsulates the wake-up indication generated in the first encapsulation interval and the wake-up indication generated in the second encapsulation interval, so that the type of the wake-up indication related to the encapsulation processing is adjusted, and the type of the adjusted wake-up indication is a non-flutter type. The starting point of the first encapsulation interval corresponds to the electronic device determining that the callback queue includes a drawing callback, and the end point of the first encapsulation interval corresponds to generating the wake-up indication corresponding to the drawing callback. The starting point of the second encapsulation interval corresponds to the electronic device generating the wake-up indication, and the end point of the second encapsulation interval corresponds to the electronic device determining whether the callback queue includes a drawing callback after the next Vsync signal is generated. This ensures that two or more wake-up indications will not appear in the same Vsync cycle, thereby avoiding multiple invalid wake-ups of the flutter framework.

[0031] Optionally, before the electronic device performs the second display operation, the method further includes: the electronic device obtaining the number of valid buffers in the TextureBufferQueue. The electronic device performs the second display operation, including: when the number of valid buffers is greater than or equal to 1, the electronic device performs the second display operation. Therefore, when there are multiple valid buffer configurations, the electronic device can achieve accurate consumption of subsequent buffers based on the number of remaining valid buffers before executing buffer consumption.

[0032] Optionally, the drawing callback is the postAnimation callback, and the display callback is the postTraversal callback.

[0033] In a second aspect, the present application further provides an electronic device comprising: a display screen, a memory, and one or more processors. The memory and the processor are coupled. The memory is configured to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device executes the technical solution provided in the first aspect and any possible implementation thereof.

[0034] In a third aspect, the present application further provides a chip system, which is applied to an electronic device. The chip system may include one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected via a circuit. The interface circuit is configured to receive a signal from the memory of the electronic device and send the signal to the processor. The signal includes computer instructions stored in the memory. When the processor executes the computer instructions, the electronic device implements the technical solution provided in the first aspect and any possible implementation thereof.

[0035] In a fourth aspect, the present application also provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the technical solution provided in the above-mentioned first aspect and any possible implementation thereof.

[0036] In a fifth aspect, the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the technical solution provided in the above-mentioned first aspect and any possible implementation thereof.

[0037] It can be understood that the solutions provided in the second to fifth aspects of the present application can respectively correspond to the first aspect and any possible design thereof, so the beneficial effects that can be achieved are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a schematic diagram of an interface interaction for a user to input a page sliding operation;

[0039] FIG2 is a schematic diagram of the logical composition of a rendering processing framework;

[0040] FIG3 is a logical diagram of the production and consumption of buffers in a texture buffer queue;

[0041] Figure 4 is a schematic diagram of the module interaction between buffer production and consumption;

[0042] Figure 5 is a timing diagram of buffer production and consumption in one case;

[0043] Figure 6 is a timing diagram of buffer production and consumption in another case;

[0044] FIG7 is a schematic diagram of software composition of an electronic device provided in an embodiment of the present application;

[0045] FIG8 is a schematic diagram of software composition of another electronic device provided in an embodiment of the present application;

[0046] FIG9 is a schematic diagram of module interaction of an image processing method provided in an embodiment of the present application;

[0047] FIG10 is a schematic diagram of a software implementation for triggering early wake-up of a flutter framework provided in an embodiment of the present application;

[0048] FIG11 is a schematic diagram of a software implementation for triggering early wake-up of a flutter framework provided in an embodiment of the present application;

[0049] FIG12 is a schematic diagram of a software implementation for triggering early wake-up of the Flutter framework provided in an embodiment of the present application;

[0050] FIG13 is a timing diagram of buffer production and consumption after adopting the solution provided in an embodiment of the present application;

[0051] FIG14 is a logic diagram of a command encapsulation provided in an embodiment of the present application;

[0052] FIG15 is a logic diagram of a multi-buffer control provided in an embodiment of the present application;

[0053] FIG16 is a logic diagram of a software internal implementation of a multi-buffer control according to an embodiment of the present application;

[0054] FIG17 is a schematic diagram of module interaction of an image processing method;

[0055] FIG18 is a schematic diagram of module interaction of an image processing method;

[0056] Figure 19 is a timing diagram of buffer production and consumption in one case;

[0057] FIG20 is a logical diagram of the production and consumption of buffers in a texture buffer queue provided by an embodiment of the present application;

[0058] FIG21 is a schematic diagram of module interaction of an image processing method provided in an embodiment of the present application;

[0059] FIG22 is a timing diagram of buffer production and consumption after adopting the solution provided in an embodiment of the present application;

[0060] FIG23 is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0061] FIG24 is a schematic diagram showing the composition of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0063] One or more application programs may be installed in an electronic device. By running the application programs, the electronic device may provide a variety of different functions to the user.

[0064] Taking a mobile phone as an example, the electronic device may have installed applications including shopping applications, calendar applications, photo galleries, memos, and the like.

[0065] 1 , icons of various application programs may be displayed on the mobile phone interface 12. For example, the mobile phone interface 12 may display an icon 11 corresponding to a shopping application.

[0066] When the shopping application needs to be used, the user can input a corresponding operation on the icon 11 to instruct the electronic device to start running the shopping application. The corresponding operation input on the icon 11 may include a click operation, etc. Exemplarily, the operation on the icon 11 may be a first operation.

[0067] In response to a click operation input by the user, the mobile phone may run a shopping application and switch to display an interface provided by the shopping application on the display screen.

[0068] For example, after the user inputs an operation on the icon 11 , the mobile phone may switch to display the interface 13 as shown in FIG. 1 .

[0069] As an example, the interface 13 may include display windows for multiple different products provided by the shopping application. For example, the interface 13 may include a display frame 14 corresponding to product 1 and a display frame 15 corresponding to product 2.

[0070] In this example, the shopping app also provides display windows corresponding to more products.

[0071] The user can input a switching instruction on the interface 13 to instruct the mobile phone to refresh the interface and display the display window of other products. For example, the switching instruction can include a sliding operation corresponding to the operation 16 input on the interface 13. Exemplarily, the switching instruction can also be a second operation.

[0072] In response to user input operation 31, the mobile phone may refresh the interface and display interface 17. Interface 17 may include display windows for more products other than product 1 and product 2. For example, interface 17 may include display box 18 corresponding to product 3 and display box 19 corresponding to product 4.

[0073] In this way, during the interaction with the user, the shopping application in the mobile phone can provide the user with more product information.

[0074] In the scenario shown in FIG. 1 , the electronic device can implement interface display including interface 12 , interface 13 , and interface 17 through image rendering and display.

[0075] The image rendering may be performed by an operating system of the electronic device.

[0076] It's understandable that different electronic devices may have different operating systems. Image rendering mechanisms also vary across operating systems. Currently, the Flutter framework can be configured in electronic devices. This framework can be used to render all interface elements, and then perform operations such as display through the device's native framework. This allows for a certain degree of uniformity in the image rendering process across different operating systems, improving code compatibility.

[0077] In this application, in conjunction with Figure 2, the frameworks involved in rendering processing include the native framework and the flutter framework as an example. Among them, the threads related to image rendering in the native framework can be collectively referred to as the main thread. Correspondingly, the threads related to image rendering in the flutter framework can be collectively referred to as flutter threads.

[0078] Exemplarily, the main thread may include a main drawing thread, or a main UI thread. The main thread may also include a rendering thread M for performing image rendering processing. The flutter thread may include a flutter UI thread and a rendering thread F for performing image rendering processing.

[0079] Figure 3 shows the logical implementation of image rendering and display using the Flutter framework in conjunction with the native framework.

[0080] In this example, one application may correspond to one texture buffer queue (TextureBufferQueue). Currently, the texture buffer queue may include a buffer for storing display content that has been rendered.

[0081] In the logical implementation shown in Figure 3, the Flutter framework can act as a buffer content producer. For example, the Flutter UI thread can request a free buffer from the TextureBufferQueue using the dequeueBuffer() function. The Flutter framework's rendering thread F can render a new frame and obtain the display content for the next frame. The Flutter UI thread can then use the euqueueBuffer() function to fill the buffer with the display content for the next frame and return it to the queue.

[0082] Correspondingly, the native framework can act as a consumer of buffer content. Exemplarily, the native framework can obtain the buffer area from TextureBufferQueue through the acquireBuffer() function. The buffer area can store the display content of the frame image filled by the flutter framework. Thus, the native framework can send the content in the acquired buffer through SurfaceFlinger for display, so as to display the frame image on the display screen of the electronic device. In addition, the native framework can also release the buffer through the releaseBuffer() function after sending it for display.

[0083] In some implementations, after the native framework retrieves a buffer from the TextureBufferQueue and displays it, the buffer in the TextureBufferQueue becomes empty. Accordingly, the Flutter framework can continue to generate new buffers using the logic shown in Figure 3 for subsequent consumption by the native framework. This cycle allows for the rendering and display of continuous image frames.

[0084] It should be noted that, as shown in Figure 3, in the existing implementation, when the euqueueBuffer() function executes, if a buffer already exists in the queue, the new buffer replaces the existing buffer. This corresponds to configuring a maximum of one buffer in the TextureBufferQueue. When the acquireBuffer() function executes, it drops all buffers in the queue and acquires a new buffer. The Drop operation discards the contents of all buffers in the queue, allowing the rendering of a new frame to be performed on the new buffer.

[0085] As shown in the buffer production and consumption logic in Figure 3, the native framework, as a buffer consumer, needs to establish a connection with the Flutter framework, which is the buffer producer. This connection ensures that there are available buffers to retrieve from the TextureBufferQueue for display consumption. This connection also allows the native framework to learn from the Flutter framework that it needs to wake up the Flutter framework after the next Vsync signal arrives to draw the subsequent frame image, thereby producing the buffers required for subsequent consumption.

[0086] It can be understood that the Vsync signal involved in the embodiments of the present application can be an indication signal. When an application needs to display or draw an image, it can apply for a Vsync signal from an electronic device (such as surfaceflinger). Correspondingly, surfaceflinger can send a Vsync signal to the application according to a certain period. For example, the certain period can correspond to the refresh period of the display screen. The refresh period can refer to the market between two adjacent frames of images displayed corresponding to the refresh rate. For example, corresponding to the display screen, after the Nth frame image is displayed for a length of TL1, the N+1th frame image is switched to be displayed. Then the refresh period (or refresh interval) can be a duration of TL1.

[0087] In the application, combined with the example of Figure 1, take the case where the user inputs a page sliding operation (such as operation 16) on the main interface of the shopping application as an example. Then, the shopping application can receive multiple Vsync signals such as the first Vsync signal and the second Vsync signal from the surfaceflinger, and the shopping application can send the received Vsync signal to the native framework to trigger the display consumption of the corresponding image and the drawing production of the new image. For example, after the user inputs the page sliding operation, the shopping application can instruct the electronic device to produce and consume multiple frame images (such as the Nth frame image, the N+1th frame image, etc.) through multiple Vsync signals such as the first Vsync signal and the second Vsync signal. From another perspective, the multiple frame images can be images displayed on the display screen in sequence by the electronic device in response to the page sliding operation input by the user.

[0088] As an example, the Flutter framework can achieve the connection between the native framework and the Flutter framework by registering one or more callbacks with the native framework.

[0089] For example, the one or more callbacks may include a postTraversal callback and a postAnimation callback.

[0090] The postTraversal callback can be used to indicate to the native framework that a new buffer has been produced and placed in the TextureBufferQueue. Correspondingly, when the postTraversal callback has been received, the native framework can execute the Traversal method after a new Vsync signal arrives to obtain the buffer from the TextureBufferQueue for consumption.

[0091] The postAnimation callback indicates to the native framework that subsequent frames need to be rendered and displayed after the next Vsync signal is sent. Accordingly, if the native framework has received the postAnimation callback, it can execute the Animation method after the next Vsync signal arrives and send a wake-up instruction to the Flutter framework. This allows the Flutter framework to generate the buffers corresponding to the subsequent frames based on the wake-up instruction.

[0092] The following describes the rendering switching process of interface 13 to interface 17 in the scenario shown in FIG1 with reference to the example of FIG4 .

[0093] It can be understood that, during the switching process from interface 13 to interface 17 as shown in Figure 1, the electronic device can draw multiple frames of images continuously under the instruction of the user's operation 16. Correspondingly, the electronic device can perform continuous sliding display on the interface. The starting point of the continuous sliding display can be the interface 13 corresponding to when the operation 16 is received (such as pressing the sliding operation). The end point of the continuous sliding display can be the interface 17 at the end of the operation 16 (such as the release of the sliding operation). In the example of Figure 4, the drawing and display process of any frame image (such as the Nth frame image) in the interface 13-interface 17 is taken as an example. The display process from the Nth frame image to the N+1th frame image is shown in the example of Figure 4. Among them, the display image of the Nth frame image can be rendered and obtained during the processing of the previous frame image. Correspondingly, during the processing of the N+1th frame image, the electronic device can also render the next frame N+2 frame image.

[0094] As shown in FIG4 , the drawing and displaying of the Nth frame image may include:

[0095] S401: The main drawing thread receives an operation 16 input by the user and transparently transmits the operation information to the Flutter UI thread.

[0096] Exemplarily, in conjunction with the example of FIG1 , after the electronic device runs an application, it may display an interface corresponding to the application.

[0097] The user can input various operations on the interface of the application. For example, the user can input an operation 16 (such as a sliding operation) on the interface (such as interface 13) of the running shopping application to instruct the electronic device to slide the page.

[0098] Correspondingly, the electronic device can transparently transmit the operation information input by the user to the flutter UI thread for processing.

[0099] In this example, the sliding operation corresponding to the operation 16 input by the user may include multiple consecutive sub-operations.

[0100] For example, the sliding operation may include: pressing at position 1, continuously pressing and sliding along track 2, releasing (ie, releasing the hand) at position 3 corresponding to the end point of track 2, and other sub-operations.

[0101] In this way, the main drawing thread can send the operation information corresponding to each sub-operation to the Flutter UI thread for processing. Among them, an operation information can include at least one of the following: the operation type corresponding to the current moment (such as press, slide, release, etc.) and the coordinate information on the display screen.

[0102] That is to say, when the user inputs continuous operations as shown in operation 16, the main drawing thread can correspondingly and continuously transmit the operation information of the user input operation to the flutter UI thread, so that the flutter UI thread can determine whether there are multiple consecutive frame images to be drawn, and perform rendering processing on each frame image.

[0103] It should be noted that in some implementations, the Flutter framework can register with the native framework when determining that it is necessary to perform a subsequent rendering operation of at least one frame image based on the operation information, in combination with the description in FIG3 .

[0104] The postTraversal callback and the postAnimation callback are used so that the native framework can perform the corresponding consumption and wake-up operations after the next Vsync signal arrives.

[0105] S402: The main drawing thread receives a Vsync signal V0, triggering execution of a DoFrame operation.

[0106] During image rendering, the electronic device may generate multiple Vsync signals corresponding to the screen refresh rate. Each Vsync signal may be used to instruct the rendering and display of a frame of image.

[0107] In this example, the Vsync signal V0 may be included in the multiple Vsync signals. The Vsync signal V0 may be used to indicate the display of the Nth frame image and the rendering of the N+1th frame image. The display content of the Nth frame image may be drawn by the Flutter framework and stored in the TextureBufferQueue after the previous Vsync signal before the Vsync signal V0 arrives.

[0108] Correspondingly, the main drawing thread can trigger the execution of the doFrame operation after receiving the Vsync signal V0.

[0109] Exemplarily, the doFrame operation may include an animation operation for performing animation-related processing, and a traversal operation for performing LayerTree traversal-related processing.

[0110] Next, the electronic device may display the Nth frame image through the following S403-S404. The electronic device may also render the N+1th frame image through the following S405-S406. It should be noted that the implementation of the scheme shown in FIG4 does not constitute a limitation on the order of displaying the Nth frame image and rendering the N+1th frame image. In different implementations, the displaying of the Nth frame image and the rendering of the N+1th frame image may be performed serially, or the displaying of the Nth frame image and the rendering of the N+1th frame image may be performed simultaneously.

[0111] It is understandable that, for the main drawing thread, before executing the issuing of the rendering instruction 21 corresponding to S403 and executing the issuing of the wake-up instruction 22 corresponding to S405 , a traversal search process on the traversal queue may be executed.

[0112] The traversal queue, also known as the callback queue, can contain one or more callbacks received by the main drawing thread. Each callback can correspond to one or more functions. When the main thread encounters a callback, it can execute the function configured in the callback to perform the corresponding operation.

[0113] For example, in combination with the above description, in this example, when the main drawing thread executes the DoFrame operation corresponding to S402 , the traversal queue may include a postTraversal callback and a postAnimation callback.

[0114] Then, when traversing to the postTraversal callback, the main drawing thread may generate the corresponding rendering instruction 21 according to one or more functions configured in the postTraversal callback, and then perform the corresponding issuing process in S403.

[0115] When the postAnimation callback is traversed, the main drawing thread may generate a corresponding wake-up instruction 22 according to one or more functions configured in the postAnimation callback, and then execute the corresponding sending process in S405.

[0116] In the embodiment of the present application, the sending of the wake-up instruction 22 before the sending of the rendering instruction 21 is taken as an example.

[0117] The following describes the buffer consumption (i.e., the display consumption of the new image) based on the postTraversal callback and the buffer production (i.e., the rendering of the next frame) based on the postAnimation callback.

[0118] S403: The main drawing thread sends a rendering instruction 21 to the rendering thread M.

[0119] The rendering instruction 21 may be used to instruct the rendering thread M to perform display-related processing of the Nth frame image through a drawing command.

[0120] S404: Rendering thread M obtains a buffer from the texture buffer queue for consumption.

[0121] In this example, the electronic device can create Buffer0 in the TextureBufferQueue of the shopping application corresponding to the current operation before the current Vsync signal V0 arrives, and store the display content P0 in Buffer0. This process can be performed by the flutter framework after the previous Vsync signal arrives.

[0122] Correspondingly, in S404, after receiving the rendering instruction 21, the rendering thread M can obtain Buffer0 from TextureBufferQueue and consume it. For example, the rendering thread M can obtain buffer0 and the display content P0 stored therein through the acquireBuffer() function. The rendering thread M can transmit the display content P0 to SurfaceFlinger so that SurfaceFlinger can perform synthesis processing based on the display content P0, thereby displaying the Nth frame on the display screen. For another example, the rendering thread M can release the consumed buffer0 through the releaseBuffer() function.

[0123] Thus, after completing S404, the available buffer in TextureBufferQueue is empty.

[0124] S405. The main drawing thread sends a wake-up instruction 22 to the Flutter UI thread.

[0125] The wake-up instruction 22 can be used to instruct the Flutter framework to render a new frame image. In conjunction with the description in S402, in this example, as shown in FIG4 , S405 can be executed before S403. Generally, since the production process of a new buffer is time-consuming, in this example, although buffer consumption is triggered after buffer production, it is also possible to complete the consumption of the existing buffer before the new buffer is completed.

[0126] In combination with the description in S401, the main drawing thread may execute S405 after receiving the postAnimation callback sent by the flutter framework.

[0127] Exemplarily, the wake-up indication 22 may carry a timestamp and corresponding rendering information.

[0128] The timestamp corresponds to a moment, and the timestamp is used to indicate that the image rendering process is performed after the corresponding moment.

[0129] Correspondingly, the rendering information carried in the wake-up indication 22 may correspond to rendering-related information of the content to be displayed after the time corresponding to the timestamp. For example, the rendering information may include vertex information and color information required for rendering the interface after the time corresponding to the timestamp arrives.

[0130] S406. The Flutter UI thread instructs the rendering thread F to draw a new frame image and store it in the texture buffer queue.

[0131] Exemplarily, in some embodiments, the flutter UI thread may execute a response after the corresponding moment according to the timestamp carried by the wake-up indication 22 .

[0132] For example, the timestamp carried by the wake-up instruction 22 is time A. Then, after receiving the wake-up instruction 22, the flutter UI thread can instruct the rendering thread F to render the new image according to the rendering information carried in the wake-up instruction 22 when time A arrives.

[0133] In this way, when time A arrives, the flutter UI thread can send a drawing instruction 23 to the rendering thread F, so as to instruct the rendering thread F to perform rendering of the N+1th frame image according to the rendering information carried in the wake-up instruction 22.

[0134] In response to the drawing instruction 23, the rendering thread F may render the N+1th frame image. For example, by rendering the N+1th frame image, the corresponding display content P1 may be obtained. In this application, the N+1th frame image corresponding to the display content P1 may also be referred to as the first image.

[0135] It is understandable that when drawing a frame image, the rendering thread F needs to use a corresponding buffer to store intermediate data and drawing results.

[0136] 3 , after receiving the drawing instruction 23 , the rendering thread F can obtain a buffer area (e.g., buffer 1 ) from the TextureBufferQueue for rendering the N+1th frame image. After the rendering thread F completes rendering the N+1th frame image, the rendering result corresponding to the N+1th frame image can be stored in buffer 1 , i.e., the display content P1 is stored in buffer 1 .

[0137] The rendering thread F or the flutter UI thread may also put the buffer 1 storing the display content P1 into the TextureBufferQueue for the next consumption by the native framework.

[0138] In some implementations, as shown in Figure 4, after generating a new buffer (such as buffer1) and putting it into TextureBufferQueue, the flutter UI thread can execute S407, that is, registering the postTraversal callback and the postAnimation callback with the main drawing thread, so that after the next Vsync signal arrives, the native framework and the flutter framework can continue a new round of buffer display consumption and new buffer production.

[0139] Thus, through the operations of S402-S406, the electronic device has achieved the display of the Nth frame image and the rendering of the N+1th frame image. In addition, through the operation of S407, the electronic device can continue to consume and produce subsequent frame images.

[0140] Exemplarily, as shown in FIG4 , after receiving the Vsync signal V1, the solution may further include:

[0141] S408: The main drawing thread executes a DoFrame operation.

[0142] The operation of S408 may refer to S402 .

[0143] In this example, combined with the description of S407, as a response to the postTraversal callback, the DoFrame operation may include executing the Traversal method to consume the buffer, which may specifically trigger the following S409-S410.

[0144] In addition, as a response to the postAnimation callback, the DoFrame operation may also include executing the Animation method to generate a buffer, which may specifically trigger the following S411-S412.

[0145] S409: The main drawing thread sends a rendering instruction 24 to the rendering thread M.

[0146] Exemplarily, the rendering instruction 24 may be used to instruct the rendering thread M to draw and display the N+1th frame image through a Draw command.

[0147] With reference to the description in S406 , the newly generated buffer 1 may be stored in the current TextureBufferQueue, and the buffer 1 may store the display content P1 corresponding to the N+1th frame image.

[0148] Then, the rendering instruction 24 can also be used to instruct the rendering thread M to consume the buffer in the current TextureBufferQueue, thereby realizing the display consumption of the N+1th frame image.

[0149] S410: Rendering thread M obtains a buffer from a texture buffer queue (ie, TextureBufferQueue) for consumption.

[0150] Thus, the rendering thread M can obtain the currently stored buffer 1 from the TextureBufferQueue and send the display content P1 in buffer 1 to surfaceflinger for display. Similar to the description of S404, after consuming the buffer 1, the buffer 1 can be released and the corresponding buffer in the TextureBufferQueue is empty.

[0151] Correspondingly, surfaceflinger can perform synthesis processing according to the display content P1 so as to display the image of the N+1th frame on the display screen.

[0152] During the execution of the DoFrame operation, in addition to completing the consumption of the above buffer1, the electronic device can also execute the production of the buffer corresponding to the next frame image through the following S411-S412.

[0153] S411. The main drawing thread sends a wake-up instruction 25 to the flutter UI thread.

[0154] Exemplarily, the execution of this step may refer to the description in S405.

[0155] The corresponding timestamp and rendering information may also be carried in the wake-up indication 25. This allows the flutter framework to render the next frame (such as the N+2th frame) according to the timestamp and rendering information.

[0156] S412. The Flutter UI thread instructs the rendering thread F to draw a new frame image and store it in the texture buffer queue.

[0157] Exemplarily, the flutter UI thread may send a drawing instruction 26 to the rendering thread F. In some implementations, the drawing instruction 26 may be issued after the time corresponding to the timestamp carried by the wake-up instruction 25.

[0158] In response to the drawing instruction 26, the rendering thread F may draw the display content P2 corresponding to the N+2 frame image according to the rendering information carried by the wake-up instruction 25. The rendering thread F may also store the display content P2 in a new buffer (e.g., buffer2) of the TextureBufferQueue. In this application, the N+2 frame image corresponding to the display content P2 may also be referred to as the second image.

[0159] 3 , the rendering thread F can obtain buffer 2 from TextureBufferQueue and perform rendering of the display content P2 on buffer 2. After completing the rendering, the rendering thread F can also put the buffer 2 back into the TextureBufferQueue.

[0160] In this way, the rendering of the N+2th frame image can be completed.

[0161] Similar to the operation in S407, after completing the generation of the buffer corresponding to the N+2th frame image, the Flutter framework can execute S413 to initiate the postTraversal callback and postAnimation callback to the main drawing thread. This allows the electronic device to perform the corresponding display and / or rendering of the new image according to a similar process from S402 to S407 or S408 to S413 when the Vsync signal arrives.

[0162] In order to more clearly illustrate the process shown in FIG4 , FIG5 shows a schematic diagram of the timing logic in the image sending and rendering process.

[0163] As shown in Figure 5, before the arrival of the Vsync signal V0, the TextureBufferQueue may contain buffer0 corresponding to the already produced Nth frame image. Before the arrival of the Vsync signal V0, the main drawing thread can receive the postTraversal callback and postAnimation callback from the Flutter framework. Therefore, after receiving the Vsync signal V0, it can trigger DoFrame to consume buffer0 and produce buffer1 corresponding to the new N+1th frame image.

[0164] For example, after receiving the Vsync signal V0, the native framework can instruct the corresponding rendering thread M to consume the current buffer0 through a rendering instruction. For example, at time T0, the native framework completes the display consumption of buffer0.

[0165] After receiving the Vsync signal V0, the native framework can also wake up the flutter framework through a wake-up instruction to produce the buffer corresponding to the new frame image. For example, before the next Vsync signal V1 arrives, the flutter framework can complete the waiting resources and rendering processing, thereby storing the buffer1 including the N+1 frame image obtained by rendering in the TextureBufferQueue. Among them, waiting resources (also called Runnable) can correspond to the process of waiting for the electronic device to allocate corresponding processing resources (such as computing power resources, storage resources, etc.) after the flutter UI thread is awakened. After the waiting resources are allocated, the flutter UI thread can instruct the rendering process F to perform the rendering processing of the new frame image, which corresponds to the Running stage.

[0166] It is understood that, in conjunction with the description of S407 in Figure 4, in this example, the Flutter framework can also send callback instructions to the native framework, which can include postTraversal callbacks and postAnimation callbacks. This allows the main drawing thread to consume buffer1 based on the received postTraversal callback after the next Vsync signal arrives, and wake up the Flutter framework based on the postAnimation callback to produce a new buffer.

[0167] In the example of Figure 5, after receiving the Vsync signal V1, the native framework can cooperate with the flutter framework to perform buffer consumption (such as the consumption of buffer1) and the production of new buffers (such as buffer2) similar to the above examples. The flutter framework can also send a postTraversal callback to the native framework after putting buffer2 into the TextureBufferQueue, so that it can be displayed and consumed in the next cycle (that is, after the next Vsync signal arrives). In addition, when the flutter framework determines that there is a new image that needs to be rendered, it can send a postAnimation callback to the native framework before the next cycle arrives (that is, before the next Vsync signal arrives), so that after the next cycle arrives, the native framework wakes up the flutter framework to render the new buffer.

[0168] In this example, the native framework completes the display consumption of buffer 1 corresponding to the N+1th frame image at time T1.

[0169] Similarly, at time T2, the native framework can complete the display consumption of buffer 2 corresponding to the N+2th frame image. This buffer 2 can be generated by the Flutter framework during the period between the Vsync signal V1 and the Vsync signal V2.

[0170] At time T3, the native framework can complete the display consumption of buffer 3 corresponding to the N+3th frame image. This buffer 3 can be generated by the Flutter framework during the period between the Vsync signals V2 and Vsync signals V3.

[0171] In this way, between the frames of T0 and T1, the electronic device can display the Nth frame image. Between the frames of T1 and T2, the electronic device can display the N+1th frame image. Between the frames of T2 and T3, the electronic device can display the N+2th frame image. In the display frames after T3, the electronic device can correspondingly display the N+3th frame image. Thus, by implementing the scheme shown in Figure 4, the electronic device can continuously and timely display the page update corresponding to the user sliding operation (such as operation 16). Among them, the display frame can correspond to the refresh rate of the current display screen. The duration of a display frame is based on the display duration of 1 frame image corresponding to the refresh rate.

[0172] It can be seen that in order to enable each display frame to display each frame image continuously and accurately, it is necessary to ensure that the frame image currently to be displayed can be successfully produced and put into the TextureBufferQueue during the display process of the previous frame image.

[0173] However, even for continuous frame images, there may be situations where the frame images that need to be displayed cannot be produced and put into the TextureBufferQueue in time.

[0174] For example, a new frame might wait a long time for resources before being rendered. This could be due to a high load on the device or a low priority command for allocating resources. This can delay rendering, and consequently prevent the buffer being rendered and placed into the TextureBufferQueue in a timely manner.

[0175] For example, during the rendering process of a new frame image, if the current electronic device is heavily loaded, the computing power is insufficient, or the elements of the current frame image are complex or numerous, rendering time will be long. This will prevent the buffer of the frame image from being generated and placed in the TextureBufferQueue in time.

[0176] This will affect the display of continuous frame images, causing problems such as freezes, frame drops, and frame skips.

[0177] As an example, it is assumed that the rendering time of the N+2th frame image is longer.

[0178] Referring to Figure 6, buffer0 can be consumed normally at time T0, and buffer1 can be consumed normally at time T1. Thus, between time T0 and time T1, the electronic device can display the Nth frame image normally. The electronic device can then display the N+1th frame image starting at time T1.

[0179] In this example, the rendering time corresponding to buffer2 is longer, which results in buffer2 not being completed before the arrival of the Vsync signal V2. In this way, before the arrival of the Vsync signal V2, since the new buffer has not been completed, the native framework will not receive the corresponding postTraversal callback. Then the native framework will not trigger the consumption of the new buffer according to the arrival of the Vsync signal V2. As a result, the switching display of the N+2th frame image that should have been performed at time T2 will not be performed. Correspondingly, between time T2 and time T3, the electronic device can only continue to display the N+1th frame corresponding to the previous frame.

[0180] In this example, buffer2 can be generated within the period between Vsync signals V2 and Vsync signals V3. Therefore, when Vsync signal V3 arrives, the native framework can already receive the postTraversal callback. Therefore, at time T3 after Vsync signal V3 arrives, the native framework can consume buffer2 to display the N+2th frame image.

[0181] That is, in the example shown in FIG6 , the electronic device will continuously display the N+1th frame from time T1 to time T2 to time T3 , which causes a freeze.

[0182] In order to solve the above problems, an embodiment of the present application provides an image processing method that can flexibly adjust the wake-up timing of the flutter UI thread, so that the flutter framework in S406 as shown in Figure 4 can perform the buffer production operation in advance. Therefore, even if waiting for resources and / or rendering processing (i.e., the drawing process) takes a long time, the production of the new buffer can be effectively completed before the next Vysnc signal arrives. Then, after the next Vysnc signal arrives, the native framework can smoothly obtain the new buffer from the TextureBufferQueue for display consumption. In this way, it is possible to avoid problems such as jamming caused by the inability to produce the buffer in time as shown in Figure 6 or other reasons.

[0183] The technical solutions provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0184] It should be noted that the technical solutions provided in the embodiments of the present application can be applied to the user's electronic devices. The electronic devices may include mobile phones, foldable electronic devices, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, vehicle-mounted devices, smart home devices, or smart city devices. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device.

[0185] In different embodiments of the present application, the electronic device may have different compositions.

[0186] For example, from the perspective of hardware composition, an embodiment of the present application provides an example of the composition of an electronic device.

[0187] In this example, the electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) connector, a charging management module, a power management module, a battery, antenna 1, antenna 2, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, an earphone interface, a sensor module, a button, a motor, an indicator, a camera module, a display, and a subscriber identification module (SIM) card interface, etc. The sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0188] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0189] In some implementations of this example, the processor may include one or more processing units, for example, the processor may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0190] The processor can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0191] The processor may also include a memory for storing instructions and data. In some embodiments, the memory in the processor may be a cache memory. This memory can store instructions or data that have been used by the processor or that are frequently used. When the processor needs to use the instruction or data, it can directly access it from the memory. This avoids duplicate accesses, reduces processor latency, and thus improves system efficiency.

[0192] Electronic devices can implement display functions through GPUs, displays, and application processors. A microprocessor for image processing connects the display and application processor. The GPU performs mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0193] A display screen is used to display images, videos, and the like. The display screen includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, an electronic device may include one or more display screens.

[0194] It is understandable that the above is only a schematic diagram of the composition of an electronic device provided in this application. In other embodiments, the electronic device may also have other components.

[0195] For example, refer to FIG7 , which is a schematic diagram of the software composition of an electronic device provided in an embodiment of the present application.

[0196] In this example, the software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the electronic device is illustrated.

[0197] In the example shown in Figure 7, the layered architecture of an electronic device divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers: from top to bottom: the application layer, the application framework layer, the Android runtime (ART) and native C / C++ libraries, the hardware abstraction layer (HAL), and the kernel layer.

[0198] The application layer, also known as the application layer, can include a series of application packages.

[0199] As shown in FIG7 , the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. In some embodiments, the application package may also include a shopping application. Specifically, the shopping application may be At least one of the common shopping applications such as .

[0200] For example, consider a user using a shopping app. While the app is running, the electronic device can receive a swipe action from the user. In response, the electronic device can implement the solution shown in Figure 4 to render new images and display existing images, thereby presenting the user with a continuous scrolling effect.

[0201] The application framework layer, also known as the Application Framework layer (or simply the Framework layer), provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0202] As shown in Figure 7, the application framework layer may include a window manager, content provider, view system, resource manager, notification manager, activity manager, and input manager. The window manager provides the Window Manager Service (WMS), which can be used for window management, window animation management, surface management, and as a relay station for the input system. The content provider is used to store and retrieve data and make it accessible to applications. This data may include video, images, audio, dialed and received calls, browsing history and bookmarks, and the phone book. The view system includes visual controls, such as those that display text and images. The view system is used to build applications. The display interface can be composed of one or more views. For example, the display interface of a text message notification icon may include a view that displays text and a view that displays images. The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on. The notification manager enables applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the notification manager is used to announce the completion of downloads and message reminders. The notification manager can also be a notification that appears in the form of an icon or scrolling text in the status bar at the top of the system, such as a notification from an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, it can prompt a text message in the status bar, emit a prompt sound, vibrate the electronic device, flash the indicator light, etc. The activity manager can provide the activity management service (AMS), which can be used to start, switch, and schedule system components (such as activities, services, content providers, and broadcast receivers) as well as manage and schedule application processes. The input manager can provide the input management service (IMS), which can be used to manage system input, such as touch screen input, key input, sensor input, etc. The IMS extracts events from the input device node and distributes the events to the appropriate window through interaction with the WMS.

[0203] In this example, the framework layer supports the rendering processing framework shown in Figure 2. For example, the framework layer can include a main rendering module and a Flutter rendering module. The main rendering module corresponds to the capabilities of the native framework and can be used to deliver and consume buffers through the main drawing unit and the main rendering unit. The Flutter rendering unit corresponds to the capabilities of the Flutter framework and can be used to produce buffers by running the Flutter drawing unit and the Flutter rendering unit.

[0204] Among them, the main drawing unit may correspond to the main drawing thread as shown in Figure 4. The main rendering unit may correspond to the rendering thread M as shown in Figure 4. The flutter drawing unit may correspond to the flutter UI thread as shown in Figure 4. The flutter rendering unit may correspond to the rendering thread F as shown in Figure 4.

[0205] Based on the solution provided in the embodiment of the present application, the main rendering module can send a wake-up instruction to the flutter rendering module in advance according to a preset strategy, so that even if the rendering time is long as shown in Figure 6, or the buffer production (i.e., the drawing process) takes a long time due to other reasons (such as inability to obtain resources in time), the flutter rendering module can have sufficient time to produce a new buffer.

[0206] The Android runtime consists of the core libraries and the Android runtime. The Android runtime is responsible for converting source code into machine code. It primarily utilizes ahead-of-time (AOT) and just-in-time (JIT) compilation technologies.

[0207] The core library is mainly used to provide basic Java class library functions, such as basic data structures, mathematics, IO, tools, databases, networks and other libraries. The core library provides an API for users to develop Android applications. The native C / C++ library can include multiple functional modules. For example: surface manager, media framework, libc, OpenGLES, SQLite, Webkit, etc. Among them, the surface manager is used to manage the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media framework supports a variety of commonly used audio and video formats for playback and recording, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGLES provides the drawing and operation of 2D graphics and 3D graphics in applications. SQLite provides a lightweight relational database for the application of the electronic device 100.

[0208] The hardware abstraction layer (HAL) runs in user space, encapsulating kernel-layer drivers and providing a calling interface to upper layers. The kernel layer is the layer between hardware and software. It includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0209] FIG. 7 above shows the software partitioning of an electronic device, and is not sufficient to constitute an actual restriction on the electronic device.

[0210] Refer to FIG8 , which is a schematic diagram of software composition of another electronic device provided in an embodiment of the present application.

[0211] As shown in Figure 8, a shopping application may be configured in the application layer of the electronic device. The user may instruct the electronic device to run the shopping application, and correspondingly, the electronic device may display the interface of the shopping application to the user on the display screen.

[0212] It is understandable that when the user wants to view more shopping interfaces, he or she can input instructions on the interface of the shopping application. For example, the user can input a sliding operation on the interface of the shopping application to instruct the electronic device to slide and update the display.

[0213] The process can be referred to the description shown in Figure 1. No further details will be given here.

[0214] In the example shown in FIG8 , the application framework layer of the electronic device may be configured with multiple modules, and each module may further include a corresponding unit for supporting the electronic device to perform a sliding update display operation according to a user's sliding operation.

[0215] Exemplarily, the application framework layer may include a main rendering module, a flutter rendering module, and a display module.

[0216] The main rendering module is used to implement native framework capabilities. For example, as shown in Figure 4, the main drawing unit in the main rendering module can start the main drawing thread and, upon receiving the postTraversal callback, trigger the display consumption of the buffer already in the TextureBufferQueue based on the arrival of the Vysnc signal.

[0217] For example, the main rendering unit can read an existing buffer from the TextureBufferQueue in the memory through a rendering instruction, and can also transmit the existing buffer to the display module so that the display module can update the display screen according to the buffer.

[0218] The display module may correspond to the SurfaceFlinger configured in the electronic device. After receiving the display consumption buffer sent by the main rendering module, SurfaceFlinger can transfer the data in the buffer to the buffer used for composite display. SurfaceFlinger can also instruct the display to update the display based on the data in the buffer used for composite display.

[0219] The main drawing unit in the main rendering module can also trigger the wake-up of the flutter rendering module to produce a new buffer based on the arrival of the Vysnc signal when receiving the postAnimation callback.

[0220] The flutter rendering unit in the flutter rendering module can start the flutter UI thread and, upon receiving a wake-up instruction from the main rendering unit, instruct the flutter rendering unit to obtain a new buffer from the TextureBufferQueue according to the rendering thread F as shown in Figure 4, and put the newly rendered frame image into the buffer. The flutter rendering unit can also put the buffer including the newly rendered frame image into the TextureBufferQueue to facilitate subsequent display consumption by the main rendering module. For example, after completing the production of the new buffer and putting it into the TextureBufferQueue, the flutter rendering module can send a postTraversal callback to the main rendering module so that after the next Vsync signal arrives, the main rendering module continues to display consumption and realizes display update.

[0221] In the embodiment of the present application, the memory of the electronic device may include a TextureBufferQueue. The TextureBufferQueue may include one or more buffers.

[0222] For example, consider a TextureBufferQueue containing one buffer. After the main rendering module finishes consuming the buffer, the buffer in the TextureBufferQueue becomes empty. Conversely, after the Flutter rendering module finishes producing the buffer, the number of buffers in the TextureBufferQueue is 1. If the Flutter rendering module produces a new buffer even if the buffer in the TextureBufferQueue has not been consumed, the new buffer can overwrite the existing buffer stored in the TextureBufferQueue because the number of buffers in the TextureBufferQueue is limited to 1.

[0223] Take the example of two buffers in TextureBufferQueue. When TextureBufferQueue is empty, the Flutter rendering module can put the buffer into the head of TextureBufferQueue each time it completes the production of a buffer. If the buffer has not been consumed, the Flutter rendering module completes the production of a new buffer, and the new buffer can be configured in the TextureBufferQueue in sequence. That is, in this example, the TextureBufferQueue can include the earlier produced buffer at the head, and another buffer produced subsequently. In this way, when the main rendering module is sending and consuming, the main rendering module can sequentially obtain the buffers in the TextureBufferQueue starting from the head. This realizes the display consumption of the earlier produced buffers. After the head buffer is consumed, the subsequent buffers can be progressively moved to the head of TextureBufferQueue, so that after the next Vysnc signal arrives, the main rendering module can continue to consume the buffer at the head of the TextureBufferQueue.

[0224] The following examples use a TextureBufferQueue with two buffers. As you can see, with two buffers in a TextureBufferQueue, the situation described above, where a new buffer overwrites an older buffer, will not occur. Accordingly, by implementing the TextureBufferQueue header, frame skipping and other display issues will not occur.

[0225] The solution for configuring multiple buffers in the TextureBufferQueue will be described in detail later.

[0226] In this example, in order to solve the problem of display jamming caused by the failure to complete any buffer production between two adjacent Vsync signals in the scenario shown in Figure 6, an embodiment of the present application provides an image processing method, based on the electronic device composition as shown in Figure 8, so that the main rendering module (or the main drawing unit therein) can wake up the flutter rendering module in advance to produce a new buffer when it is determined that the postAnimation callback has been received. In this way, buffers with longer waiting resource time and / or rendering processing time can be processed in advance, thereby significantly improving the display jamming problem in the scenario shown in Figure 6.

[0227] In the following description, each execution action of the native framework can be implemented by the main rendering module as shown in Figure 8. The operations of the main drawing thread in the native framework can be executed by the main drawing unit, and the operations of the rendering thread M in the native framework can be executed by the main rendering unit. Similarly, the operations of the Flutter UI thread in the Flutter framework can be executed by the Flutter drawing unit, and the operations of the rendering thread F in the Flutter framework can be executed by the Flutter rendering unit.

[0228] Refer to Figure 9, which is a flowchart of an image processing method provided in an embodiment of the present application. The implementation of the solution shown in Figure 9 can be considered an improvement to the existing process shown in Figure 4. By waking up the Flutter framework in advance, new buffers can be processed in advance, thus avoiding display freezes caused by excessive buffer production time.

[0229] As shown in Figure 9, the solution may include:

[0230] S901. The main drawing thread receives an operation 16 input by the user and transparently transmits the operation information to the Flutter UI thread.

[0231] For the implementation of S901 , reference may be made to S401 in FIG. 4 .

[0232] In this example, operation 16 may correspond to a sliding operation input by the user on the shopping application interface. In response to operation 16, the electronic device may perform a sliding switching display on the current page.

[0233] It is understood that during this sliding switching display process, the electronic device will rapidly switch and display multiple consecutive frame images. For example, each display frame can display a new frame image. In this way, the electronic device will generate multiple consecutive Vsync instructions to trigger buffer consumption and new buffer production. In scenarios where continuous and rapid frame image display is required, it is more likely to experience lag as shown in Figure 6.

[0234] In the implementation scheme shown in Figure 9, combined with the description in Figure 4, the FlutterUI thread can determine the need for continuous buffer production based on the multiple operation information transmitted by the main drawing thread to support the subsequent buffer production. Then, the flutterUI thread can send the above-mentioned postAnimation callback to the main drawing thread before the next Vsync signal arrives. In the case where the display interface needs to be switched later, the flutterUI thread can also send the above-mentioned postTraversal callback to the main drawing thread before the next Vsync signal arrives to support the switching display of the next frame image.

[0235] The solution example shown in FIG9 illustrates the execution of the postAnimation callback and the postTraversal callback.

[0236] S902: The main drawing thread receives a Vsync signal V0, triggering execution of a DoFrame operation D1.

[0237] The implementation of S902 can refer to S402 in Figure 4. The Vsync signal V0 can also be called the first Vsync signal. The subsequent Vsync signal V1 can also be called the second Vsync signal, and the Vsync signal V2 can also be called the third Vsync signal.

[0238] The Vsync signal V0 may trigger the switching display of the currently displayed N-1th frame image. For example, the Vsync signal V0 may trigger the switching display of the Nth frame image.

[0239] The Vsync signal V0 may also trigger the generation of a new buffer. For example, the Vsync signal V0 may trigger the generation of buffer 1 for the N+1th frame image.

[0240] Correspondingly, the execution of the DoFrame operation D1 may include executing the Traversal method to realize the display consumption of buffer0 corresponding to the Nth frame image. Before executing the Traversal method, the main drawing thread has already received the postTraversal callback. For details, please refer to S903-S904 below.

[0241] The execution of the DoFrame operation D1 may include executing the Animation method to generate buffer 1 corresponding to the N+1 frame image. Before executing the Animation method, the main drawing thread has received the postAnimation callback. For details, please refer to S905-S906 below.

[0242] Similar to the description in Figure 4 above, in this example, the main drawing thread can send a wake-up instruction 92 (such as executing S905) before sending a rendering instruction 91 (such as executing S903). After the rendering thread M receives the rendering instruction 91 and the flutter UI thread receives the wake-up instruction 92, the two units can each asynchronously and independently perform the corresponding operations. The following are explained separately.

[0243] S903: The main drawing thread sends a rendering instruction 91 to the rendering thread M.

[0244] The implementation of S903 may refer to S403 in Figure 4. The rendering instruction 91 may correspond to the rendering instruction 21 in S403.

[0245] For example, the rendering instruction 91 may include a Draw command, which is used to instruct the rendering thread M to consume the current buffer.

[0246] It is understandable that before the Vsync signal V0 arrives, the TextureBufferQueue may include buffer0 corresponding to the Nth frame image that has been produced. In this way, the rendering instruction 91 can be used to instruct the rendering thread M to send buffer0 for display consumption.

[0247] S904: Rendering thread M obtains a buffer from the texture buffer queue (TextureBufferQueue) for consumption.

[0248] For the implementation of S904 , reference may be made to S404 in FIG. 4 .

[0249] Exemplarily, in conjunction with the example in Figure 3, the rendering thread M can obtain the currently stored buffer0 for consumption through acquireBuffer(). For example, the rendering thread M can send the storage address of the acquired buffer0 or the display content P0 stored in buffer0 to the display module (such as SurfaceFlinger). Correspondingly, SurfaceFlinger can store the display content P0 in the display buffer for display on the display screen. This allows SurfaceFlinger to perform relevant processing (such as synthesis processing) on ​​the display content P0 entering the display buffer, and then update the display interface according to the processing results. This also realizes the switching of the current display interface from the N-1 frame image to the N frame image.

[0250] 3 and 4 , in some implementations, the rendering thread M can also release the consumed buffer 0 through the releaseBuffer() function. In this way, the number of buffers in the TextureBufferQueue can be 0 before a new buffer is produced.

[0251] S905. The main drawing thread sends a wake-up instruction 92 to the Flutter UI thread.

[0252] S906, the Flutter UI thread instructs the rendering thread F to draw a new frame image and store it in the texture buffer queue. For example, the flutter UI thread can instruct the rendering thread F to produce a new buffer through the drawing instruction 93. Correspondingly, the rendering thread F can apply for a new buffer (such as buffer1) in the TextureBufferQueue and render the N+1 frame image on buffer1. After the display content P1 corresponding to the N+1 frame image is placed in buffer1, buffer1 is placed in the TextureBufferQueue.

[0253] The operations of S905-S906 may refer to S405-S406 shown in FIG4 , and will not be described in detail here.

[0254] It is understandable that, through the above steps S903 to S906, the electronic device can complete the consumption of buffer 0 and the production of buffer 1 in TextureBufferQueue. The order of the two operations of consuming buffer 0 and producing buffer 1 is not limited.

[0255] It's important to note that in existing solutions, the number of buffers in a TextureBufferQueue cannot exceed 1. Therefore, if buffer1 is produced before buffer0 is consumed, buffer1 will overwrite buffer0 in the TextureBufferQueue. This may prevent buffer0 from being properly displayed, resulting in lag or frame skipping when the Nth frame is not displayed.

[0256] Correspondingly, in the present application, based on the aforementioned description of the number of buffers in the TextureBufferQueue in the memory, the number of buffers can be 1 or more. For example, the number of buffers can be 2. At the same time, the present application also provides a consumption mechanism when the number of buffers is greater than 1, that is, taking out the buffers from the head in sequence for display consumption. As a result, even if buffer1 is produced before buffer0 is consumed, buffer1 and buffer0 can be stored in the TextureBufferQueue at the same time. Among them, the buffer0 produced earlier can be located at the head of the TextureBufferQueue, and correspondingly, buffer1 can be located after buffer0. In this way, when consuming buffers, buffer0 can be taken out first for display consumption. That is, it can support the flexible sequential implementation of the above-mentioned two operations of buffer0 consumption and buffer1 production to avoid the situation where the Nth frame image cannot be displayed.

[0257] In this way, due to the flexible configuration of the number of buffers in TextureBufferQueue, the buffer consumption logic of the native framework can be relatively isolated from the buffer production logic of the Flutter framework, and the problem of frame image loss due to different completion orders will not be caused.

[0258] In different implementations of the present application, the number of buffers in the TextureBufferQueue is configured to be 1 or 2 or other values, which can be flexibly determined by the electronic device based on the current rendering situation. For example, the electronic device can determine that buffer consumption is earlier than new buffer production in the first few Vysnc cycles of the current application. In this way, the electronic device can configure the number of buffers in the TextureBufferQueue to 1, thereby saving the storage space overhead when configuring multiple buffers. For another example, the electronic device may find that buffer consumption is later than buffer production in the first few Vysnc cycles of the current application. In this way, the electronic device can configure the number of buffers in the TextureBufferQueue to 2 or a larger value, thereby ensuring that frame images are not lost. For another example, the electronic device can configure at least 2 buffers for the TextureBufferQueue by default. In this way, after the production of a new buffer is completed in the current Vysnc cycle, the existing unconsumed buffers can be stored in the TextureBufferQueue without overwriting. In the present application, the Vysnc cycle can be the time period between two adjacent Vsync signals. For example, the period between the Vsync signal V0 and the Vsync signal V1 may correspond to a Vysnc cycle (such as called Vsync cycle 0-1, or called the first Vsync cycle). For another example, the period between the Vsync signal V1 and the Vsync signal V2 may correspond to a Vysnc cycle (such as called Vsync cycle 1-2).

[0259] In the following example, two buffers are configured in a TextureBufferQueue. The two buffers can correspond to a first buffer and a second buffer, respectively. The first buffer can be positioned before the second buffer in the TextureBufferQueue. For example, the first buffer can be at the head of the TextureBufferQueue, and the second buffer can be positioned after the first buffer in the TextureBufferQueue (e.g., at the tail).

[0260] Take the Vsync cycle 0-1 as an example. Before the buffer is consumed, buffer0 can be stored in the first buffer, and the second buffer is empty. Then, in the Vsync cycle 0-1, the native framework can obtain buffer0 in the first buffer for display consumption. After display consumption, the first buffer and the second buffer in TextureBufferQueue are both empty. Correspondingly, in the Vsync cycle 0-1, after completing the production of the new buffer (buffer1), the flutter framework can put the buffer1 into the first buffer and send a postTraversal callback to the native framework to facilitate the consumption of buffer1 in the first buffer in the next Vysnc cycle.

[0261] It is understood that in the example shown in Figure 9, assuming that buffer0 is consumed before buffer1 is produced, after buffer0 is consumed, both the first and second buffers are empty. After buffer1 is produced, the first buffer corresponds to buffer1 and may store display content P1. The second buffer remains empty.

[0262] In other implementations, buffer1 can be produced before buffer0 is consumed in Vsync cycle 0-1. Therefore, before buffer1 is produced, the first buffer can store buffer0, and the second buffer can be empty. After buffer1 is produced, the first buffer can store buffer0, and the second buffer can store buffer1. Thus, during this Vsync cycle 0-1, when consuming buffers, the native framework can consume buffer0 from the first buffer in the header. After buffer0 is consumed, buffer1 can enter the first buffer, and the corresponding second buffer is released to be empty.

[0263] In the example shown in Figure 9, after completing the production of buffer1, the Flutter framework can make corresponding callbacks to the native framework to facilitate the smooth progress of subsequent production and consumption.

[0264] For example, as shown in S907, the flutter UI thread can send a postTraversal callback CB1 and a postAnimation callback CB2 to the main drawing thread.

[0265] In some embodiments, the main drawing thread may store the received callback information in a callback queue. For example, the postTraversal callback CB1 and the postAnimation callback CB2 may be stored in the callback queue. The postTraversal callback CB1 and the postAnimation callback CB2 may each be marked with "flutter," indicating that the two callbacks are from the flutter framework. As an example, the main drawing thread may identify the type of the postTraversal callback CB1 and the postAnimation callback CB2 as flutter.

[0266] This way, before executing DoFrame, the main drawing thread can check the callback queue and, if the callback queue contains a Flutter-type postTraversal callback, execute the Traversal method to consume the buffer. Furthermore, before executing DoFrame, the main drawing thread can check the callback queue and, if the callback queue contains a Flutter-type postAnimation callback, execute the Animation method to wake up the Flutter framework for buffer production.

[0267] It's understandable that after executing the Traversal method, the main drawing thread can also delete the postTraversal callback from the callback queue, preventing the same postTraversal callback from triggering multiple Traversal methods for buffer consumption. Similarly, after executing the Animation method, the main drawing thread can also delete the postAnimation callback from the callback queue, preventing the same postAnimation callback from triggering multiple Animation methods to wake up the Flutter framework for buffer production.

[0268] Therefore, through S901-S907 shown in FIG9 , the electronic device can complete the transmission and display consumption of the Nth frame image and the production of the N+1th frame image in the Vsync period 0-1.

[0269] As shown in Figure 9, during Vsync cycle 0-1, the native framework can also initiate the production of more new buffers. In conjunction with the descriptions in S905-S906, the Flutter framework can execute the production of buffer 1 after being awakened by the native framework. Subsequently, during Vsync cycle 0-1, the native framework can also wake up the Flutter framework to execute the subsequent production of buffer 2. This buffer 2 can be the buffer storing display content P2, which corresponds to the N+2th frame image.

[0270] In this way, by executing the production of buffer2 in the Vsync cycle 0-1, even if the production of buffer2 takes a long time (such as the runnable process and / or running process of buffer2 takes a long time), buffer2 can also be produced and placed in the TextureBufferQueue before it needs to be sent to the display for consumption.

[0271] As an example, as shown in FIG9 , after S907 , the method further includes:

[0272] S908: The main drawing thread determines that there is a wake-up requirement.

[0273] For example, the main drawing thread may check the callback queue and determine that there is a wake-up requirement when a flutter-type postAnimation callback exists in the callback queue.

[0274] It is understandable that, combined with the above description of the postAnimation callback, the flutter framework can register the postAnimation callback with the native framework when it determines that subsequent image rendering is required.

[0275] For example, in S907, the flutter framework may register a postAnimation callback CB2 with the native framework when determining that a subsequent image needs to be rendered (eg, an N+2th frame image corresponding to a new buffer2 exists after the generated buffer1).

[0276] Correspondingly, the postAnimation callback CB2 can be placed in a callback queue, and its type can be a flutter type.

[0277] In S908 , after completing the creation and enqueueing of buffer1 (i.e., entering the TextureBufferQueue), the main rendering thread checks the callback queue. The main rendering thread can determine that the callback queue includes a flutter-type postAnimation callback CB2, thereby determining that there is a need to draw a subsequent new frame image. Since drawing a new image can be achieved by waking up the flutter framework, the main rendering thread can determine that there is a need to wake up if the callback queue includes a flutter-type postAnimation callback CB2.

[0278] S909. The main drawing thread sends a wake-up instruction 95 to the flutter UI thread.

[0279] The wake-up indication 95 can be used to wake up the flutter framework again to produce a new buffer within the Vsync period 0-1.

[0280] It can be understood that, through the execution of S908-S909, the main drawing thread can send a wake-up instruction 95 to the flutter UI thread in advance, so as to advance the production of buffer2 corresponding to the wake-up instruction 95 to within the Vsync cycle 0-1.

[0281] As a specific implementation, Figures 10 to 12 provide a specific execution logic of S908-S909. This solution can be applied to the arrival of the Vsync signal V0, by setting a delay task so that S908-S909 is executed after S901-S907.

[0282] As shown in Figure 10, after the Vsync signal V0 arrives, the main drawing thread can start to perform corresponding operations. For example, triggering the execution of run. Then, the main drawing thread can execute the DoFrame operation D1. This process can correspond to the implementation in the aforementioned S901-S907. In the example of Figure 10, the main drawing thread can also determine whether the advance FlutterAnim is effective.

[0283] The following example illustrates how to determine whether the early FlutterAnim is effective.

[0284] In this example, the condition for determining whether the early FlutterAnim is effective may also include at least one of the following: the current thread is the main thread; a hand-off action of a sliding operation is received; the current application is not in a multi-window display; the current application uses a single textureview.

[0285] Take the condition for determining whether the early FlutterAnim is effective, including the current thread being the main thread, as an example.

[0286] If the current thread is the main thread, it is determined that the advance FlutterAnim takes effect. Otherwise, if the current thread is not the main thread, it is determined that the advance FlutterAnim does not take effect.

[0287] Take the condition for determining whether the early FlutterAnim is effective including receiving the hand-off of the sliding operation as an example.

[0288] If there is no release action for the sliding operation and the sliding operation is currently in progress, it is determined that the early FlutterAnim takes effect. On the contrary, if the release action has been received, it is determined that the early FlutterAnim does not take effect.

[0289] Take the condition for determining whether the early FlutterAnim is effective, including the current non-multi-window display, as an example.

[0290] Multi-window display corresponds to two or more Draw instructions appearing at the same time for window drawing. This situation may occur when the user inputs window switching. In this example, if the current display is not multi-window, it is determined that FlutterAnim takes effect in advance.

[0291] Take the condition for determining whether the early FlutterAnim is effective as an example, including the current application using a single textureview.

[0292] A single texture view corresponds to rendering processing using the Flutter framework. Therefore, if the current View class is fluttertextureView, FlutterAnim must be enabled in advance.

[0293] The above example provides multiple solutions for determining whether early FlutterAnim is effective. In other embodiments, two or more of these solutions can be used simultaneously. If each condition determines that early FlutterAnim is effective, it is considered that the current scene can produce buffers in advance, triggering the generation of a delayed task.

[0294] In the present application, the delay task may correspond to a first duration. In some embodiments, the first duration may be 2 / 3 of the Vsync period 0-1. That is, after receiving the Vsync signal V0, the main rendering thread may configure a delay task. The duration of the delay task may be 2 / 3 of the Vsync period 0-1. Thus, after the delay task ends, the main rendering thread may begin executing related actions.

[0295] Referring to Figure 11, when the delayed task corresponding to the first duration ends, the main rendering thread can check whether the current callback queue includes the PostAnimation callback. Before the first duration ends, the electronic device can complete the production of buffer1 and execute S907, resulting in the inclusion of the postTraversal callback CB1 in the callback queue. This determines the need to preemptively execute the animation method.

[0296] As shown in Figure 11, after the first duration following the Vsync signal V0, if it is determined that a PostAnimation callback has occurred, the main drawing thread can obtain the timestamp corresponding to the execution of the animation method in the next frame. For example, this timestamp can be the timestamp carried in the wake-up indication 25 in Figure 4. Take this timestamp as time TB as an example.

[0297] In this way, the main drawing thread can execute doCallbacks according to the time TB corresponding to the determined timestamp.

[0298] For example, the main drawing thread can execute doCallbacks at a first moment. This first moment can be time TB minus delta. Delta can be a preset second duration. Because delta is advanced based on time TB, the animation method in doCallbacks is executed earlier by delta duration. This also advances the processing of buffer2 by the second duration.

[0299] As an example, as shown in FIG12 , the main drawing thread may configure the timestamp of the animation method to be currently executed based on the time TB, so that the animation method can be executed before the preset second duration.

[0300] Specifically corresponding to the example in FIG9 , after the second time period is advanced, the flutter framework may trigger the execution of the following S910 .

[0301] S910. The Flutter UI thread instructs the rendering thread F to draw a new frame image and store it in the texture buffer queue.

[0302] In conjunction with the description in S909, the flutter UI thread may receive a wake-up instruction 95 carrying a timestamp of (TA-delta). Thus, the flutter UI thread may send a drawing instruction 96 to the rendering thread F after the (TA-delta) duration.

[0303] Rendering thread F can obtain a new buffer 2 from TextureBufferQueue based on the rendering information corresponding to the N+2 frame image carried in drawing instruction 96, and draw the display content P2 corresponding to the N+2 frame image on buffer 2. After the drawing is completed, the Flutter framework can put buffer 2 including the display content P2 into TextureBufferQueue.

[0304] In conjunction with the above description of setting the first buffer and the second buffer in TextureBufferQueue, in this example, the first buffer may store buffer1 that has been produced but not consumed, and the produced buffer2 may be stored in the second buffer.

[0305] In the example shown in Figure 9, the production of buffer2 can be completed before the arrival of the Vsync signal V1. In other embodiments, the production of buffer2 may also be completed after the arrival of the Vsync signal V1. It is understandable that due to the early production of buffer2, as long as buffer2 enters the TextureBufferQueue before the arrival of VsyncV2, buffer2 can be normally displayed and consumed.

[0306] S911, the Flutter UI thread sends a callback to the main thread.

[0307] Similar to the description in S907 above, after completing the production of buffer2, the flutter framework can register a callback with the main drawing thread so that buffer2 can be consumed later and a new buffer can be produced.

[0308] For example, the postTraversal callback CB3 can be used to register the consumption of buffer2. Another example is the postAnimation callback CB4 can be used to register the production of buffer3.

[0309] It should be noted that in the example shown in Figure 9, buffer2 and buffer1 can be produced in the same Vsync cycle 0-1. Correspondingly, this application provides a TextureBufferQueue configured with two buffers for storing buffer1 and buffer2 respectively, avoiding buffer2 covering buffer1 and causing frame loss.

[0310] In addition, the callback registration of S911 and S907 may also be performed within the same Vsync cycle 0-1.

[0311] In this application, the flutter framework can encapsulate the callback before executing S911 to avoid two postTraversal callbacks or postAnimation callbacks in the same Vsync cycle.

[0312] For example, the Flutter framework can encapsulate the postTraversal callback CB3 and the postAnimation callback CB4 into callbacks of different Flutter types. This prevents the main drawing thread from directly consuming buffer2 according to the postTraversal callback CB3 or producing buffer3 according to the postAnimation callback CB4 after the Vsync signal V1 arrives.

[0313] In some implementations, the encapsulated postTraversal callback CB3 and postAnimation callback CB4 can have a delay attribute. The delay attribute corresponds to the duration of a Vsync cycle. After the delay attribute ends, the type of the postTraversal callback CB3 and postAnimation callback CB4 can automatically change to the flutter type.

[0314] In this way, during Vsync cycle 0-1, the main drawing thread can receive the flutter-type postTraversal callback CB1. As a result, in the next cycle (such as Vsync cycle 1-2), the main drawing thread can consume buffer1 according to the postTraversal callback CB1.

[0315] In addition, during the Vsync cycle 0-1, the main drawing thread can also receive the encapsulated postTraversal callback CB3 and postAnimation callback CB4. Because the types of postTraversal callback CB3 and postAnimation callback CB4 are not Flutter types, the main drawing thread will not trigger the corresponding Traversal method or Animation method after the Vsync signal V1 arrives.

[0316] After receiving the encapsulated postTraversal callback CB3 and postAnimation callback CB4 for a Vsync cycle, the postTraversal callback CB3 and postAnimation callback CB4 types can be changed to the flutter type. This is equivalent to the main drawing thread receiving the flutter type postTraversal callback CB3 and postAnimation callback CB4 during Vsync cycles 1-2. Therefore, when the next Vsync signal V2 arrives, the main drawing thread can consume buffer2 based on the postTraversal callback CB3 and wake up the flutter framework based on the postAnimation callback CB4 to produce buffer3.

[0317] It can be understood that, in the above example as shown in FIG9 , the flutter framework is taken as an example to encapsulate the postTraversal callback CB3 and the postAnimation callback CB4 and then send them.

[0318] In other embodiments, the encapsulated operation may also be performed by the main rendering thread.

[0319] For example, the flutter framework can also directly register the postTraversal callback CB3 and the postAnimation callback CB4 to the main drawing thread according to the existing scheme. Correspondingly, the main drawing thread can check the current callback queue after receiving the postTraversal callback CB3 and the postAnimation callback CB4. When it is determined that there is an unresponsive postTraversal callback CB1, the main drawing thread can actively encapsulate the postTraversal callback CB3 and the postAnimation callback CB4, so that after the duration of a Vsync cycle, the type of the postTraversal callback CB3 and the postAnimation callback CB4 is automatically changed to the flutter type, so that the corresponding response can be made after the next Vsync signal V2 arrives.

[0320] In this way, the production of buffer2 is advanced through S908-S911 in the Vsync cycle 0-1, thereby avoiding the problem of failure to display normally in the Vsync cycle 2-3 caused by failure to complete the production of buffer2 in time in the Vsync cycle 1-2.

[0321] It is understandable that after completing the above operations in the Vsync cycle 0-1, the electronic device can consume the N+1th frame image according to the arrival of the next Vsync signal V1. Exemplarily, the method further includes the following S912-S914.

[0322] S912: The main drawing thread executes the DoFrame operation D2.

[0323] Among them, the main drawing thread can trigger the execution of DoFrameD2 after the Vsync signal V1 arrives.

[0324] Similar to the description of the aforementioned DoFrame operation D1, the main drawing thread may determine to include the postTraversal callback CB1 in the callback queue. Correspondingly, the DoFrame operation D2 may include a response to the postTraversal callback CB1.

[0325] That is, the buffer in the first buffer in the current TextureBufferQueue is consumed. The first buffer may store display content P1 corresponding to buffer1, corresponding to the N+1th frame image.

[0326] S913: The main drawing thread sends a rendering instruction 94 to the rendering thread M. The rendering instruction 94 is used to instruct the rendering thread M to consume the buffer in the TextureBufferQueue.

[0327] S914. Rendering thread M obtains a buffer from the texture buffer queue for consumption.

[0328] Specifically, the rendering thread M may obtain the content in the buffer (such as the first buffer) stored at the head of the TextureBufferQueue for consumption. The first buffer may store display content P1.

[0329] After obtaining the display content P1, the rendering thread M may transmit the display content P1 to the surfaceflinger, so that the surfaceflinger displays the N+1th frame on the display screen according to the display content P1.

[0330] It is understandable that after the Vsync signal V1 arrives, according to the existing solution (as shown in FIG4 ), the main drawing thread can wake up the flutter framework to produce buffer2.

[0331] In this example, the generation of buffer2 and its entry into the TextureBufferQueue have been completed in Vsync cycle 0-1 through S908-S910. Therefore, there is no need to perform the generation process of buffer2 again in Vsync cycle 1-2.

[0332] Thus, based on the description of Figures 9-12 above, in the solution provided by the embodiment of the present application, the main drawing thread can, through the judgment mechanism shown in Figures 10-12, advance the animation method triggered by the arrival of the next Vsync signal to the current Vsync cycle when a wake-up demand is confirmed, thereby achieving rapid subsequent buffer production. This avoids the failure of subsequent display consumption caused by excessive buffer production time, thereby resolving the corresponding display lag problem.

[0333] As an example, Figure 13 provides an example of the timing of buffer consumption and production when implementing the solution of this application. This example, based on the problem scenario shown in Figure 6, solves the problem in Figure 6 where buffer 2 cannot be produced in time, resulting in the inability to display the N+1 frame image normally between time T2 and T3.

[0334] Exemplarily, as shown in Figure 13, after the arrival of the Vsync signal V0, the flutter framework can, according to the wake-up instruction, wait for resources and perform rendering processing, and produce buffer1 and put it into the TextureBufferQueue. In the example of Figure 13, the first buffer and the second buffer are still configured in the TextureBufferQueue as an example. The buffer1 can be placed in the first buffer of the TextureBufferQueue. Based on the solution implementation shown in Figure 9, after the arrival of the Vsync signal V0, the electronic device can determine that the DoAnimation processing needs to be triggered in advance. In this way, before the arrival of the Vsync signal V1, the electronic device can wake up the flutter framework again to perform the drawing processing of the new buffer (such as buffer2). At 2 / 3 of the Vsync cycle 0-1, the electronic device can start executing the DoAnimation processing corresponding to buffer2.

[0335] This allows the DoAnimation processing corresponding to buffer2 to be advanced. As shown in Figure 13, by advancing the drawing process of buffer2, buffer2 can be completed and enter the TextureBufferQueue before the Vsync signal V2 arrives. As a result, after the Vsync signal V2 arrives, the electronic device can smoothly consume the buffer and switch to display the N+2 frame image at time T2. This can avoid the situation shown in Figure 6 where the drawing process of buffer2 is too long to be produced in time, affecting the normal display. This also avoids the display freeze problem caused by this.

[0336] In addition, in conjunction with the description of S911 in FIG. 9 , in order to avoid two postAnimation callbacks in the same Vsync cycle, the main drawing thread may proactively encapsulate the received multiple callback instructions.

[0337] In the embodiment of the present application, during the specific execution of S911, the main drawing thread can perform the corresponding callback encapsulation operation according to the preset first encapsulation interval and second encapsulation interval, temporarily encapsulating the flutter type into other types. This avoids the display problem caused by two or more postAnimation callbacks in the same Vsync cycle.

[0338] As an example, referring to FIG14 , the encapsulation mechanism provided in this application is described in combination with the timing logic of FIG13 .

[0339] Combined with the aforementioned native framework (such as the main drawing thread therein), DoFrame can be executed after a new Vsync signal is received.

[0340] Exemplarily, the DoFrame operation may include traversing the callbacks received in the callback queue.

[0341] Take Vsync cycle 0-1 as an example. Before the Vsync signal V0 arrives, the postAnimation callback and the postTraversal callback have entered the callback queue.

[0342] Then, as shown in FIG14 , after the Vsync signal V0 arrives, the main drawing thread of the native frame can execute DoFrame . When traversing to the postAnimation callback, the main drawing thread can execute DoAnimation , which corresponds to executing the animation method configured in the postAnimation callback, thereby generating and issuing a wake-up instruction 92 .

[0343] Similarly, when traversing to the postTraversal callback, the main drawing thread can execute DoTraversal, and correspondingly execute the Traversal method configured in the postTraversal callback, thereby generating and issuing a rendering instruction 91.

[0344] Based on the existing solution, the wake-up instruction 95 for waking up the Flutter framework to draw the next frame (such as the N+2 frame) should be issued in the Vsync cycle 1-2. In this way, a freeze may occur due to the long rendering time of the N+2 frame image.

[0345] Based on the solution implementation shown in FIG9 , the wake-up instruction 95 can be issued in advance in the Vysnc cycle 0-1, thereby enabling early processing of the N+2th frame image.

[0346] During the specific execution process, as shown in Figure 14, in the Vsync cycle 0-1, after the main drawing thread performs the DoFrame operation in the existing solution, it can execute DoAnimation again before the Vsync signal V1 arrives, and generate and issue a corresponding wake-up instruction 95, thereby realizing the execution of S909 in Figure 9.

[0347] In this example, to prevent the main drawing thread from receiving multiple postAnimation callbacks in the same Vsync period, the main drawing thread can encapsulate the callbacks received in the first encapsulation interval and the second encapsulation interval as shown in FIG14 .

[0348] The callback encapsulation process may include packaging the encapsulated callback, where the callback instruction class after packaging may be different from the callback instruction class before packaging. For example, if the encapsulated callback is of the flutter type, then after packaging, the type of the encapsulated callback may be different from the flutter type.

[0349] In some implementations, the encapsulation process may further include: configuring a delay mechanism for the callback instruction after the encapsulation process. The delay mechanism is used to unpack the callback instruction after the encapsulation process after a certain period of time. The type of the callback instruction after the unpacking process is the same as the callback instruction before the encapsulation process. For example, the callback instruction to be encapsulated is of the flutter type before the encapsulation process. Then, after the encapsulation process, the type of the callback instruction to be encapsulated is different from the flutter type. And after the unpacking process, the type of the callback instruction to be encapsulated is of the flutter type. In some implementations, the duration corresponding to the delay mechanism may be the duration of one Sync cycle. As a result, after one Sync cycle, the main drawing thread can respond normally to the callback instruction.

[0350] As a specific implementation, in the example of Figure 14, the start time of the first encapsulation interval can correspond to the moment when the main rendering thread starts executing DoAnimation for the second time in any Vsync cycle (such as Vsync cycle 0-1). Correspondingly, the end time of the first encapsulation interval can correspond to the moment when the main rendering thread ends executing DoAnimation for the second time in any Vsync cycle (such as Vsync cycle 0-1), that is, the moment when the main rendering thread generates and issues a wake-up instruction (such as wake-up instruction 95). Accordingly, the main rendering thread can encapsulate all types of postanimation callbacks in the first encapsulation interval.

[0351] The start time of the second encapsulation interval can be the same as the end time of the first encapsulation interval. For example, the start time of the second encapsulation interval can correspond to the end time of the second execution of DoAnimation by the main drawing thread in any Vsync cycle (such as Vsync cycle 0-1), that is, the time when the main drawing thread generates and issues a wake-up instruction (such as wake-up instruction 95). Accordingly, the main drawing thread can encapsulate the flutter type callback (that is, the postanimation callback sent by the flutter framework) in the second encapsulation interval.

[0352] Therefore, the effect of accurately encapsulating the postAnimation callback can be achieved through the solution shown in Figure 14. It is understandable that in some scenarios, the main rendering thread can also encapsulate the postTraversal callback based on a similar solution, and the specific implementation is not repeated here.

[0353] It should be noted that, in conjunction with the effect examples of Figures 9 and 14, in some embodiments of the present application, a TextureBufferQueue may include two or more buffers. For example, in the above example, the TextureBufferQueue includes a first buffer and a second buffer.

[0354] As shown in FIG15 , reference is made to the description of the existing solution in FIG4 .

[0355] In the current solution implementation, the length of TextureBufferQueue can be 1 buffer. That is to say, the data of 1 buffer can be stored in TextureBufferQueue at the same time. For example, take an electronic device configured with 3 buffers for data caching as an example. The three buffers can be: Buffer_B1, Buffer_B2, Buffer_B3. Then, the data of one of the three buffers can be stored in TextureBufferQueue at the same time. When the data of other buffers arrives, the buffer data in the TextureBufferQueue is discarded, and the data of the new buffer is stored in TextureBufferQueue. For example, as shown in Figure 15, the data of Buffer_B3 can be stored in the current TextureBufferQueue. Then, when the data of Buffer_B1 arrives, the data of Buffer_B3 can be discarded in TextureBufferQueue, and the data of Buffer_B1 can be stored.

[0356] Correspondingly, in the TextureBufferQueue configuration provided in the embodiment of the present application, the length of the TextureBufferQueue can be 2 buffers or larger. Taking the example of a TextureBufferQueue having a length of 2 buffers, the data of two buffers can be stored in the TextureBufferQueue at the same time.

[0357] For example, as shown in Figure 15, the data of Buffer_B2 can be stored in the first buffer, and the data of Buffer_B3 can be stored in the second buffer. In this way, when the data of Buffer_B3 is not consumed, if new data of Buffer_B2 arrives, the data in Buffer_B3 will not be discarded. As a result, the data of Buffer_B3 will not be lost when the buffer is consumed next time, and normal display consumption can be achieved.

[0358] In this application, combined with the description of S910, the above problem is solved by configuring the first and second buffers, so that the newly generated buffer does not overwrite the existing buffer. This avoids frame loss caused by buffer overwriting. In this way, after the Vsync signal V1 arrives, the native framework can consume buffer1 in the first buffer and correctly display the N+1 frame image.

[0359] In an embodiment of the present application, referring to Figure 16, based on the setting method of multiple buffers in the TextureBufferQueue, a command control mechanism corresponding to multiple buffers is also provided, thereby enabling the consumption and production of each buffer to be successfully completed when there are multiple buffers in the TextureBufferQueue.

[0360] As shown in Figure 16, the buffer management process can involve multiple software modules. These multiple software modules or units (such as buffer queue producer (BufferQueueProducer.cpp), Android graphics surface texture (Android_graphics_SurfaceTexture.cpp), surface texture (surfaceTexture.java), structure view (TextureView.java), general view or view (View.java), etc.) can implement the corresponding functions shown in Figure 16 under the control of the native framework. Therefore, from another logical point of view, the buffer management scheme shown in Figure 16 can be implemented by the main rendering module shown in Figure 8. Specifically, in some implementations, the scheme implementation shown in Figure 16 can be implemented by the main drawing unit.

[0361] In this example, BufferQueueProducer.cpp is the server-side implementation of IGraphicBufferProducer and is responsible for buffer application. Correspondingly, BufferQueueConsumer.cpp is the server-side implementation of IGraphicBufferProducer and is the buffer consumer. In this application, when new buffer production is required (such as when the Flutter framework receives a wake-up instruction), a new buffer can be applied through BufferQueueProducer.cpp to render new content on the newly applied buffer.

[0362] When the data frame is available (i.e., when onFrameAvailable is called), GLSurfaceView.requestRender() is called to explicitly request rendering, triggering the renderer's onDrawFrame(). Furthermore, as shown in Figure 16, Android_graphics_SurfaceTexture.cpp can also send a native callback event (postEventFramNative) to surfaceTexture.java. This allows surfaceTexture.java to send an empty message (e.g., sendEmptyMessage) to TextureView.java, triggering TextureView.java to send an invalidate message (invalidate) to View.java, thus consuming the current buffer.

[0363] It is understandable that after SurfaceTexture is initialized, a JNISurfaceTextureContext listener is set to GLConsumer, which calls back to the Java layer SurfaceTexture.postEventFromNative method, and further calls back to the OnFrameAvailableListener listener registered in SurfaceTexture to notify the business layer that a new GraphicBuffer has been queued.

[0364] In this application, since there are multiple buffer configurations, surfaceTexture.java can reasonably configure the consumption of each buffer by triggering the operation of obtaining the current buffer number in S1601 and confirming the current valid buffer number.

[0365] As shown in S1601 in Figure 16 , surfaceTexture.java executes getBufferCount , which retrieves the number of currently valid (consumable) buffers. For example, if existing buffers are unconsumed and new buffers are being generated, the number of valid buffers is 2. Correspondingly, after a buffer is consumed, the number of valid buffers decreases by 1, and so on.

[0366] Next, surfaceTexture.java sends a nativeGetBufferCount instruction to Android_graphics_SurfaceTexture.cpp, thereby sharing the number of available buffers with the surfacetexture in the native framework. In some implementations of this application, if the number of available buffers is greater than or equal to 1 (i.e., there are available buffers), Android_graphics_SurfaceTexture.cpp can continue to perform subsequent operations.

[0367] In this example, Android_graphics_SurfaceTexture.cpp can instruct BufferQueueProducer.cpp to obtain the current buffer size by executing getQueueBufferSize. The buffer size can correspond to the size of the texture stored in the buffer. It is understood that if the texture size matches the interface size of the currently running application, the corresponding buffer can be the buffer displayed by the current application (such as a shopping app).

[0368] In the example shown in Figure 16, the invalidate corresponding to step 1603a can realize the consumption of the first buffer. Through step S1601, the main rendering unit (ie, corresponding to the main rendering thread) can determine that there are other buffers in the TextureBufferQueue waiting to be consumed.

[0369] BufferQueueProducer.cpp can send the mCore->mQueue.size() message to surfaceTexture.java through Android_graphics_SurfaceTexture.cpp to transfer the buffer size, thereby instructing surfaceTexture.java to trigger the consumption of the unconsumed buffer content.

[0370] It should be noted that in this example, combined with the description of S910 above, after the display content P1 in the first buffer is consumed as shown in 1603a in Figure 16, the display content P2 in the second buffer can be progressively transferred to the first buffer. This allows that when a new buffer (such as display content P3) is subsequently generated, if the display content P2 has not been consumed, the display content P3 can enter the second buffer and be consumed after the display content P2.

[0371] Continuing with Figure 16, as shown in S1602, surfaceTexture.java can delay sending a sendEmptyMessage to TextureView.java (i.e., sendEmptyMessageDelayed). The delay corresponding to this delayed sending can be the length of one Vsync cycle. This allows the sendEmptyMessage to be sent during the next Vsync cycle, triggering 1603b to consume the display content P2.

[0372] It is understandable that through the internal implementation shown in Figure 16, the solution of determining the number of valid buffers in S1601 and the solution of delaying the sending of the consumption buffer indication in S1602 can be implemented so that TextureView.java will not receive two consumption buffer indications within a Vsync cycle. This ensures that buffer consumption proceeds normally when there are multiple buffers in the TextureBufferQueue.

[0373] As shown in Figures 9-16 above, in this example, the native framework wakes up the Flutter framework in advance, allowing buffers produced during the next Vsync cycle to begin production during the current Vsync cycle. This allows buffers with long production times to be completed and queued before the corresponding Vsync signal arrives. This prevents display failures caused by delayed buffer production and resolves the corresponding display lag issues.

[0374] Combined with the description of Figures 9 and 16, in the above example, a multi-buffer configuration scheme for TextureBufferQueue is still used. This ensures that the buffer produced in advance will not overwrite the buffer produced earlier but not yet consumed. In the above example, in this multi-buffer configuration, the new buffer may be caused by the corresponding wake-up instruction being initiated in advance. It is understandable that in other scenarios, even if the buffer is not produced in advance, the two buffers may overlap.

[0375] Thus, if the existing single buffer solution as shown in FIG. 4 or FIG. 15 is still used, a new buffer may overwrite an existing unconsumed buffer, thereby causing frame loss or frame skipping.

[0376] The following examples illustrate the implementation of this multi-buffer configuration and the corresponding scenarios. As shown in the above examples, regardless of whether the technical solution S908-S911 shown in Figure 9 is used for pre-buffer generation, this multi-buffer configuration can solve the frame loss and frame skipping problems caused by buffer coverage loss in existing scenarios.

[0377] As an example, refer to FIG17 , which is an example of a buffer production and consumption scenario.

[0378] Figure 17 shows a specific implementation of normal buffer production and consumption. The execution of each step in Figure 17 can correspond to steps S401-S407 shown in Figure 4.

[0379] The following is a brief description. As shown in Figure 17, the solution may include:

[0380] S1701. After receiving operation 16 input by the user, the main drawing thread transparently transmits the operation information to the flutter UI thread.

[0381] The execution of this step can refer to S401 shown in Figure 4. In this way, the flutter UI thread can determine whether the sliding operation corresponding to operation 16 has ended based on the obtained operation information. If the sliding operation has not ended, the flutter UI thread can determine based on the operation information whether to continue to perform operations such as waking up the corresponding buffer production.

[0382] The Flutter UI thread can determine the subsequent production and consumption of at least the Nth frame image to the N+3th frame image based on the operation information.

[0383] In this example, similar to the previous scenario, Vsync signal V0 is used to trigger the consumption of the Nth frame image and the production of the N+1th frame image. Similarly, Vsync signal V1 is used to trigger the consumption of the N+1th frame image and the production of the N+2th frame image. Vsync signal V2 is used to trigger the consumption of the N+2th frame image and the production of the N+3th frame image.

[0384] The Nth frame image corresponds to the display content P0, the N+1th frame image corresponds to the display content P1, the N+2th frame image corresponds to the display content P2, and the N+3th frame image corresponds to the display content P3.

[0385] In the example shown in FIG17 , the electronic device can produce a new buffer according to S1702 a to S1702 e and consume an existing buffer according to S1703 a to S1703 c.

[0386] S1702a. After the Vsync signal V0 arrives, the main drawing thread sends a wake-up instruction 22 to the flutter framework.

[0387] For the execution of this step, reference may be made to S405 shown in FIG. 4 .

[0388] The wake-up instruction 22 may be generated and sent when the main drawing thread receives the postAnimation callback before the Vsync signal V0 and executes the Animation method in the DoFrame operation after receiving the Vsync signal V0. For specific implementation, please refer to the above description.

[0389] In some implementations, the wake-up indication 22 may include a timestamp and rendering information. The timestamp may be used to instruct the Flutter framework to start producing a new buffer at the time corresponding to the timestamp. The rendering information may be used by the Flutter framework to draw the new buffer map accordingly.

[0390] S1702b. The Flutter UI thread sends a drawing instruction 23 to the rendering thread F.

[0391] The execution of this step may refer to the relevant description in S406 shown in Figure 4. The drawing instruction 23 may be used to draw and queue a new buffer according to the rendering information.

[0392] In some embodiments, the flutterUI thread may send a drawing instruction 23 to the rendering thread F when the time corresponding to the timestamp corresponding to the wake-up instruction 22 arrives.

[0393] In other embodiments, the flutter UI thread may directly send a drawing instruction 23 to the rendering thread F after receiving the wake-up instruction 22. In this implementation, the rendering thread F may start producing a new buffer when the time stamp of the wake-up instruction 22 arrives based on the time information carried in the drawing instruction 23.

[0394] S1702c. Rendering thread F draws and displays content P1.

[0395] The drawing process may include waiting for resources (runnable) and rendering processing (running) as shown in Figure 5 or Figure 6. In the example of Figure 17, the rendering thread F may complete the drawing of the display content P1 after a period of time.

[0396] S1702d. The rendering thread F sends the display content P1 to the texture buffer queue (TextureBufferQueue).

[0397] S1702e: The texture buffer queue stores display content P1 in Buffer1.

[0398] In this way, the production corresponding to buffer 1 can be completed. The buffer 1 may include the display content P1 corresponding to the N+1th frame image.

[0399] In addition, in parallel with the process from S1702a to S1702e, the electronic device may execute S1703a-S1703b to consume the existing buffer (eg, buffer0).

[0400] S1703a: The main drawing thread sends a rendering instruction 21 to the rendering thread M.

[0401] For the execution of S1703a-S1703c, reference may be made to the relevant description in S403 shown in FIG. 4 .

[0402] The rendering instruction 21 may include a Draw instruction for consuming the existing buffer.

[0403] S1703b. Rendering thread M obtains the buffer from the texture buffer queue for consumption.

[0404] For example, the rendering thread M may obtain an existing buffer from the current TextureBufferQueue, which may be buffer0 storing the display content P0.

[0405] The rendering thread M can also transmit the display content P0 in the buffer0 to SurfaceFlinger so that SurfaceFlinger can perform subsequent processing.

[0406] S1703c, SurfaceFlinger synthesizes the display content P0 and sends it for display.

[0407] In this way, the Nth frame image can be sent and displayed for consumption through the implementation of S1703a-S1703c.

[0408] It should be noted that the execution process of S1702a-S1702e and the execution process of S1703a-S1703c are independent of each other, and there is no fixed order between the two.

[0409] S1704. The Flutter UI thread sends the postTraversal callback CB1 and the postAnimation callback CB2 to the main drawing thread.

[0410] For example, after completing the production of buffer1 in S1702a-S1702e, the Flutter UI thread can send a postTraversal callback CB1 to the main rendering thread. This allows the main rendering thread to repeat a similar solution in S1703a-S1703c after the next Vsync signal V1 arrives to produce buffer1.

[0411] The Flutter UI thread can also send a postAnimation callback CB2 to the main drawing thread when it determines that it needs to continue waking up the Flutter framework to produce a new buffer.

[0412] In this example, the postTraversal callback CB1 and the postAnimation callback CB2 are sent adjacently or simultaneously. In other implementations, the postTraversal callback CB1 and the postAnimation callback CB2 can be sent independently of each other, and the order and timing of their sending are not specifically limited.

[0413] In this way, through S1701-S1704 shown in Figure 17, the consumption of buffer0 in the Vsync cycle 0-1 is completed, the TextureBufferQueue is cleared, and the buffer1 to be produced thereafter can re-enter the TextureBufferQueue and wait for the next consumption.

[0414] After buffer0 is consumed, the Nth frame of image can be displayed on the display screen of the electronic device.

[0415] Under normal circumstances, if the electronic device can complete the consumption of buffer 1 and the production of buffer 2 as shown in FIG17 within the next Vsync cycle 1-2, it can switch to display the N+1th frame image.

[0416] However, due to the significant differences in rendering time for different buffers, it is difficult to ensure that each Vsync cycle can complete the production and consumption of a buffer in the order shown in Figure 17. This can lead to display stuttering, frame drops, and frame skipping.

[0417] For example, FIG18 shows an example of buffer production and consumption in an abnormal scenario.

[0418] The solution shown in FIG18 may be an example of processing corresponding to a subsequent Vsync signal received after S1704 shown in FIG17 . For example, after executing S1704, S1705a shown in FIG18 may be executed. And / or, after executing S1704, S1706a shown in FIG18 may be executed.

[0419] S1705a. After the Vsync signal V1 arrives, the main drawing thread sends a wake-up instruction 25 to the flutter framework.

[0420] S1705b, the Flutter UI thread sends a drawing instruction 26 to the rendering thread F.

[0421] S1705c. Rendering thread F draws and displays content P2.

[0422] S1705d. Rendering thread F sends the display content P2 to TextureBufferQueue.

[0423] S1705e, TextureBufferQueue stores display content P2 in Buffer2.

[0424] S1705f, the Flutter UI thread sends a postTraversal callback CB4 to the main drawing thread.

[0425] In this example, the display content P2 takes a long time to produce. For example, before starting to render, there is a long waiting time for resources. And / or after starting to render, the rendering process takes a long time. As a result, the display content P2 may not be rendered and queued until the next Vsync signal V2 arrives.

[0426] Correspondingly, after completing the queueing of the new buffer2 (corresponding to the display content P2), the Flutter UI thread executes S1705f. This means that the main drawing thread receives the postTraversal callback CB4 only after the Vsync signal V2 arrives.

[0427] S1706a: The main drawing thread sends a rendering instruction 24 to the rendering thread M.

[0428] S1706b. Rendering thread M obtains the buffer from the texture buffer queue for consumption.

[0429] S1706c: SurfaceFlinger synthesizes the display content P1 and sends it for display.

[0430] In the Vsync period 1-2, the electronic device can implement switching display of the N+1th frame image based on S1706a-S1706c.

[0431] In this example, since there is a rendering requirement for subsequent frame images (such as the N+3th frame image corresponding to buffer3), the flutter UI thread can execute S1707 before the Vsync signal V2 and send the postAnimation callback CB3 to the main drawing thread.

[0432] Thus, during Vsync cycle 1-2, the electronic device can complete the consumption of the N+1th frame image (i.e., buffer 1), and the corresponding TextureBufferQueue is cleared to 0. However, during Vsync cycle 0-1, the production of the new buffer 2 is not completed. Therefore, when the next Vsync signal V2 arrives, the main drawing thread does not receive the postTraversal callback CB4 to consume the new buffer.

[0433] This will prevent the main drawing thread from consuming the buffer during the Vsync period 2-3 corresponding to the arrival of the Vsync signal V2, and will not update the currently displayed content.

[0434] Exemplarily, within the Vsync period 2-3, the electronic device may further execute S1708a-S1708f and S1709.

[0435] S1708a. The main drawing thread sends a wake-up instruction 27 to the Flutter framework.

[0436] S1708b. The Flutter UI thread sends a drawing instruction 28 to the rendering thread F.

[0437] S1708c. Rendering thread F draws and displays content P3.

[0438] S1708d. Rendering thread F sends the display content P3 to TextureBufferQueue.

[0439] S1708e, TextureBufferQueue stores display content P3 in Buffer3.

[0440] S1708f, the Flutter UI thread sends a postTraversal callback CB5 to the main drawing thread.

[0441] S1708a-S1708f can be used to generate new buffers during Vsync cycles 2-3. It can be understood, combined with the description of S1707, that during Vsync cycles 1-2, the Flutter UI thread can determine that there is a need for subsequent new buffer generation and issue the corresponding postAnimation callback CB3. Then, after the arrival of Vsync signal V2, the main drawing thread can execute the Animation method in the DoFrame operation to generate buffer3 corresponding to the N+3 frame image.

[0442] It should be noted that due to the delay in the production of the previous buffer2, no buffer2 was consumed during the Vsync period 2-3. Therefore, when the production of buffer3 was completed, buffer2 was still stored in the TextureBufferQueue.

[0443] Then, due to the current single buffer configuration, after buffer 3 is produced and queued, it will overwrite the original unconsumed buffer 2. In other words, after buffer 3 completes production and is queued, it will overwrite the unconsumed buffer 2.

[0444] In the Vsync cycle 2-3, the flutter UI framework may determine that there is a subsequent demand for new buffer production, and execute S1709, that is, send a postAnimation callback CB6 to the main drawing thread.

[0445] It is understandable that at the end of the Vsync cycle 2-3, the electronic device can still display the N+1th frame image corresponding to buffer1, and the TextureBufferQueue stores the display content P3 corresponding to the N+3th frame image.

[0446] Next, after the Vsync signal V3 arrives, the electronic device may execute the following S1710a-S1710c to consume the new buffer.

[0447] S1710a: The main drawing thread sends a rendering instruction 29 to the rendering thread M.

[0448] S1710b: Rendering thread M obtains the buffer from the texture buffer queue for consumption.

[0449] S1710c and SurfaceFlinger synthesize and process the display content P3 and then send it for display.

[0450] In this way, after the Vsync signal V3 arrives, the electronic device can execute the Traversal method in DoFrame based on the postTraversal callback CB5 and the triggering of the Vsync signal V3, and read the buffer from the current TextureBufferQueue for consumption.

[0451] Combined with the above description of Vsync period 2-3, when the Vsync signal V3 arrives, buffer 3 may be stored in TextureBufferQueue. Thus, through S1710a-S1710c, the electronic device can switch to display the corresponding N+3 frame image on the display screen.

[0452] In the scenarios shown in Figures 17 and 18, the corresponding buffer production and consumption logic is shown in Figure 19.

[0453] As shown in Figure 19, after the arrival of the Vsync signal V0, the electronic device can complete the consumption of the existing buffer0 at time T0 and begin displaying the Nth frame of the image. Correspondingly, before the arrival of the Vsync signal V1, the production of buffer1 is completed. Therefore, before the arrival of the Vsync signal V1, the main drawing thread can receive the callback corresponding to the consumption of buffer1.

[0454] After the Vsync signal V1 arrives, the electronic device can consume buffer 1 and switch to display the N+1th frame image at time T1. After the Vsync signal V1 arrives, the electronic device can also start to produce buffer 2.

[0455] As shown in FIG18 , the buffer2 can be completed and queued after the Vsync signal V3 arrives.

[0456] In this way, within the Vsync cycle 1-2, the main drawing thread will not receive a callback for buffer consumption after the Vsync signal V2 arrives.

[0457] Therefore, after the Vsync signal V2 arrives, the consumption of the buffer will not be triggered. Then in the Vsync period 2-3, the electronic device will not update the display image.

[0458] During Vsync cycles 2-3, the electronic device can complete the production and enqueue of buffer 2 and buffer 3. However, due to the single buffer setting, when buffer 3 is produced and put into TextureBufferQueue, it will overwrite the produced but unconsumed buffer 2.

[0459] In this way, when the Vsync signal 3 arrives, the electronic device can continue to display the (N+1)th frame image, and the TextureBufferQueue can include buffer3 but not buffer2.

[0460] Therefore, after the Vsync signal 3 arrives, the electronic device can obtain buffer 3 from TextureBufferQueue for consumption display, so that the display interface switches from the N+1th frame image to the N+3th frame image.

[0461] This results in the loss of the N+2th frame image, which corresponds to frame loss and frame skipping during the display process.

[0462] In this embodiment, the above problems can be effectively avoided by configuring multiple buffers as shown in FIG15 .

[0463] It should be noted that, in some embodiments, the electronic device may configure multiple buffers for TextureBufferQueue in a flexible manner according to the current scenario.

[0464] For example, the electronic device can determine the need to set multiple buffers in a similar manner by combining any one or more of the various methods provided in S908 in Figure 9. The specific implementation can be referred to the above description and will not be repeated here.

[0465] In the following example, a TextureBufferQueue is configured with a first buffer and a second buffer. Different buffers can be used to store different display contents.

[0466] In this application, when scheduling multiple buffers, the processing logic of different producers and consumers for different buffers can be controlled to achieve normal consumption and production logic.

[0467] For example, refer to FIG20 in combination with FIG3.

[0468] When the first and second buffers are configured in the TextureBufferQueue, the Flutter framework, as the buffer producer, can request a free buffer from the TextureBufferQueue using dequeueBuffer(), similar to the current solution shown in Figure 3. It can fill the buffer using euqueueBuffer() and return it to the queue.

[0469] It should be noted that in this embodiment, euqueueBuffer() does not drop the existing buffer when placing the buffer into the queue. In this embodiment, if there is a buffer in the queue, the new buffer will continue to be stored in TextureBufferQueue.

[0470] For example, if data already exists in the first buffer, the Flutter framework can put the newly generated data into the second buffer while retaining the data stored in the first buffer, thereby avoiding the discarding of unused buffers.

[0471] When a TextureBufferQueue contains both a first and a second buffer, the native framework, as a buffer consumer, can obtain the buffer from the TextureBufferQueue using acquireBuffer(), similar to the current solution shown in Figure 3. It can release the buffer after consumption using releaseBuffer(), thus avoiding duplicate buffer consumption.

[0472] It should be noted that, in this embodiment, since there are multiple buffers, acquireBuffer() can be used to acquire one of the multiple buffers without performing drop processing on other buffers.

[0473] For example, acquireBuffer() can be used to acquire the content in the first buffer for consumption. If the second buffer also contains other content, the content in the second buffer is retained.

[0474] In some embodiments of the present application, after the content in the first buffer is consumed and released, the content in the second buffer can be progressively added to the first buffer. This allows the native framework to continue to use acquireBuffer() to acquire the unconsumed content for display consumption the next time. This implementation of first-buffer priority consumption can also be described as: the first buffer has a higher priority than the second buffer, so when consuming buffers, the native framework can prioritize consuming the content in the first buffer with a higher priority.

[0475] In other embodiments of the present application, each buffer in a TextureBufferQueue can be configured with a timestamp in addition to storing the corresponding display content. For example, if the first buffer stores display content P2, the timestamp of the first buffer can be the time when display content P2 enters the first buffer. For another example, if the second buffer stores display content P3, the timestamp of the second buffer can be the time when display content P3 enters the second buffer.

[0476] In this way, when the native framework consumes buffers, it can consume them one at a time according to the order of the timestamps of the buffers, from earliest to latest. For example, if the TextureBufferQueue contains a first buffer storing display content P2 and a second buffer storing display content P3, the native framework can prioritize consuming display content P2 in the first buffer because the timestamp of the first buffer is earlier than that of the second buffer.

[0477] It is understood that in this implementation, after the display content P2 in the first buffer is consumed, the first buffer is released and becomes empty. The display content P3 in the second buffer can continue to remain in the second buffer. The timestamp of the second buffer can be the time when the display content P3 enters the second buffer. After the new buffer 4 is produced and queued, it can be stored in the idle first buffer. At this time, the timestamp of the first buffer is changed to the time when buffer 4 enters the queue.

[0478] In this way, when the native framework consumes the buffer, it can obtain the display content P3 from the second buffer for consumption based on the time stamp of the second buffer being earlier than that of the first buffer.

[0479] In the following example, the priority of the first buffer is higher than that of the second buffer. After the content of the first buffer is consumed, the content of the second buffer is progressively entered into the first buffer.

[0480] Refer to Figure 21, which is a flowchart of an image processing method provided in an embodiment of the present application.

[0481] In the scenario shown in Figure 21, the electronic device can run an application under the user's instruction. For example, the application is a shopping application. Similar to the descriptions of the aforementioned scenarios, in this example, the user can input a sliding operation on the shopping application interface. For example, this sliding operation may correspond to operation 16 in the aforementioned scenario.

[0482] In this example, a first buffer and a second buffer may be set in the TextureBufferQueue of the electronic device, and the first buffer has a higher priority than the second buffer.

[0483] In conjunction with the description of Figures 17 and 18 , before the Vsync signal V1 shown in Figure 21 arrives, the electronic device can implement the solution shown in Figure 17 , consuming the buffer corresponding to the Nth frame image and producing the buffer corresponding to the N+1th frame image. The display content P1 of the N+1th frame image can be stored in the first buffer. When the Vsync signal V1 arrives, the second buffer is empty.

[0484] As shown in Figure 21, the solution may include:

[0485] S2101a: After the Vsync signal V1 arrives, the main drawing thread sends a wake-up instruction 25 to the flutter framework.

[0486] S2101b, the Flutter UI thread sends a drawing instruction 26 to the rendering thread F.

[0487] S2101c. Rendering thread F draws and displays content P2.

[0488] S2101d. Rendering thread F sends the display content P2 to TextureBufferQueue.

[0489] S2101e, TextureBufferQueue stores display content P2 in Buffer2.

[0490] S2101f, the Flutter UI thread sends a postTraversal callback CB4 to the main drawing thread.

[0491] In this example, the execution of S2101a-S2101f may refer to S1705a-S1705f shown in FIG18 .

[0492] It should be noted that in this example, when buffer2 is finished producing, the previous buffer (such as buffer1) has already finished sending it to the display. Therefore, after the Vsync signal V2 arrives, when buffer2 is finished producing, the first and second buffers in the TextureBufferQueue are both empty.

[0493] Therefore, in S2101d-S2101e, after completing the production of buffer2, the rendering thread F can put the display content P2 in buffer2 into the first buffer of TextureBufferQueue, so that when it is needed later, buffer2 can be consumed first.

[0494] In this way, after buffer2 is queued (ie, in Vsync cycle 2-3), the display content P2 corresponding to buffer2 can be stored in the first buffer of TextureBufferQueue, and the second buffer is empty.

[0495] Similar to the description in FIG. 18 , in the Vsync period 1 - 2 , the electronic device may also consume the existing buffer.

[0496] For example, as shown in FIG21 , the existing buffer may be buffer1, and the consumption of buffer1 may be performed through S2102a-S2102c.

[0497] S2102a: The main drawing thread sends a rendering instruction 24 to the rendering thread M.

[0498] S2102b. Rendering thread M obtains the buffer from the texture buffer queue for consumption.

[0499] S2102c, SurfaceFlinger synthesizes the display content P1 and sends it for display.

[0500] In this example, the execution process of S2102a-S2102c can refer to S1706a-S1706c in Figure 18.

[0501] In this way, the N+1th image can be displayed and consumed in the Vsync period 1-2.

[0502] It is understandable that in Vsync cycle 1-2, the TextureBufferQueue can be described according to the timing: when the Vsync signal V1 arrives, the first buffer stores display content P1. After executing S2102c to complete consumption, the first and second buffers are both empty. In this Vsync cycle 1-2, since the new buffer has not been produced, until the Vsync signal V2 arrives, the first and second buffers are both empty.

[0503] As shown in Figure 21, during Vsync cycle 1-2, the Flutter UI thread can also determine that it needs to wake up again in the next Vsync cycle to produce a new buffer. Accordingly, the Flutter UI thread can execute S2103 during Vsync cycle 1-2 and send a postAnimation callback CB3 to the main drawing thread.

[0504] In this way, the electronic device can start processing the next Vsync cycle after the Vsync signal V2 arrives.

[0505] The processing in the Vsync period 2-3 may include the following S2104a-S2104f, and S2105, etc.

[0506] S2104a. The main drawing thread sends a wake-up instruction 27 to the Flutter framework.

[0507] S2104b, the Flutter UI thread sends a drawing instruction 28 to the rendering thread F.

[0508] S2104c. Rendering thread F draws and displays content P3.

[0509] S2104d. Rendering thread F sends display content P3 to TextureBufferQueue.

[0510] S2104e, TextureBufferQueue stores display content P3 in Buffer3.

[0511] S2104f, the Flutter UI thread sends a postTraversal callback CB5 to the main drawing thread.

[0512] In this example, through S2104a-S2104f, the electronic device can generate a new buffer based on the postAnimation callback CB3 received in S2103 and after the Vsync signal V2 arrives, the new buffer can be buffer3.

[0513] In conjunction with the above description of buffer 2 production, during Vsync cycle 2-3, the electronic device can complete the production of two buffers. Buffer 2 is the first to be produced and queued. This buffer 2 can be stored in the first buffer. Buffer 3 is the next to be produced and queued. This buffer 3 can be stored in the second buffer.

[0514] In addition, if there is a new buffer (such as buffer4 corresponding to the N+4 frame image) that needs to be produced, in the Vsync cycle 2-3, the corresponding S2105 is executed, and the flutter UI thread also sends a postAnimation callback CB6 to the main drawing thread.

[0515] In this way, at the end of the Vsync cycle 2-3, the first buffer in the TextureBufferQueue can store the display content P2, and the second buffer can store the display content P3.

[0516] The electronic device may execute the following S2106a-S2106c after the Vsync signal V3 arrives.

[0517] S2106a: The main drawing thread sends a rendering instruction 29 to the rendering thread M.

[0518] S2106b. Rendering thread M obtains the buffer from the texture buffer queue for consumption.

[0519] S2106c, SurfaceFlinger synthesizes the display content P2 and sends it for display.

[0520] 20 , in this example, the main drawing thread may prioritize the consumption of the first buffer when performing buffer consumption. The first buffer may store display content P2 that has not yet been consumed.

[0521] A logical example of buffer consumption and production during the execution of the solution described in Figure 21 is provided in conjunction with Figure 22.

[0522] Comparing this with the existing solution shown in Figure 19 , we can see that although buffer 2 is not displayed during Vsync cycle 2-3 due to incomplete buffer production, the configuration of the first and second buffers shown in Figure 21 allows the electronic device to display the N+2th frame image during Vsync cycle 3-4, rather than directly skipping the N+2th frame image and displaying the N+3th frame image. This solves the buffer loss issue associated with a single buffer configuration.

[0523] It is understandable that Figures 15-22 illustrate the configuration and use of multiple (e.g., at least two) buffers in TextureBufferQueue, using the example of not generating buffers in advance, thereby solving the common problem of display freezes and frame skipping caused by buffer loss.

[0524] Thus, the embodiments of this application provide an implementation of a pre-buffered buffer generation scheme through Figures 9-17, and also provide an implementation of a dual-buffer configuration scheme through Figures 15-22. In different scenarios, either of these two implementations can be used to optimize the issue of image display freezes after a user's swipe operation. Alternatively, the two solutions can be configured together to optimize image display freezes after a user's swipe operation.

[0525] It is understandable that the electronic device provided in the embodiment of the present application includes a hardware structure and / or software module for performing each function in order to realize the above functions. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiment of the present application.

[0526] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0527] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of each functional module. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0528] The above-mentioned integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. In actual implementation, there may be other division methods.

[0529] For example, FIG23 shows a schematic diagram of the composition of an electronic device 2300. As shown in FIG23, the electronic device 2300 may include: a processor 2301 and a memory 2302. The memory 2302 is used to store computer execution instructions. The electronic device may also be provided with a display screen 2303. The display screen 2303 may display the interface of the application under the user's instructions. The display screen 2303 may also receive a sliding operation of the user to trigger the execution of any of the methods provided in the embodiments of the present application. For example, in some embodiments, when the processor 2301 executes the instructions stored in the memory 2302, the electronic device 2300 may execute any of the methods shown in the above embodiments.

[0530] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0531] Figure 24 shows a schematic diagram of the composition of a chip system 2400. The chip system 2400 may include: a processor 2401 and a communication interface 2402, which are used to support related devices to implement the functions involved in the above embodiments. In one possible design, the chip system also includes a memory for storing program instructions and data necessary for electronic devices. The chip system can be composed of chips, or it can include chips and other discrete devices. It should be noted that in some implementations of the present application, the communication interface 2402 may also be referred to as an interface circuit.

[0532] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0533] The functions, actions, operations, steps, etc. in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, they can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that can be integrated with one or more media. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0534] Although the present application has been described with reference to specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely illustrative of the present application as defined by the appended claims and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, the present application is intended to include such modifications and variations as fall within the scope of the claims of the present application and their equivalents.

Claims

1. An image processing method, characterized in that: The method is applied to an electronic device, and the method comprises: Receiving a first operation from a user; In response to the first operation, running a first application and presenting an interface of the first application; receiving a second operation performed by a user on an interface of the first application, where the second operation is a sliding operation; In response to the second operation, performing a first drawing operation in a first synchronous Vsync period to obtain a first image; In response to the second operation, a second drawing operation is performed within the first synchronous Vsync cycle to obtain a second image, wherein the moment when the first Vsync cycle starts is the moment when the first Vsync signal is received, the moment when the first Vsync signal cycle ends is the moment when the second Vsync signal is received, and the first Vsync signal and the second Vsync signal are two adjacent Vsync signals, and the start time of the second drawing operation is later than the start time of the first drawing operation; the first image and the second image are images displayed sequentially by the electronic device in response to the second operation.

2. The method according to claim 1 or 2, characterized in that: The method further comprises: After the second Vsync signal is generated, the electronic device performs a first display sending operation, where the first display sending operation is used to control the display screen of the electronic device to display the first image; After the third Vsync signal is generated, the electronic device performs a second display sending operation, where the second display sending operation is used to control the display screen of the electronic device to display the second image; The third Vsync signal is later than the second Vsync signal, and the third Vsync signal is adjacent to the second Vsync signal.

3. The method according to claim 2, characterized in that The electronic device is provided with a texture buffer queue TextureBufferQueue, and the TextureBufferQueue includes at least one valid buffer, and each of the at least one valid buffer stores a drawn image.

4. The method according to claim 3, characterized in that The at least one valid buffer includes a first buffer; The performing of a first drawing operation within a first synchronous Vsync period to obtain a first image includes: The first drawing operation is performed within the first Vsync period to obtain the first image, and when the first buffer is empty, the first image is stored in the first buffer.

5. The method according to claim 4, characterized in that The at least one valid buffer includes a second buffer; The performing a second drawing operation within the first synchronous Vsync period to obtain a second image includes: The second drawing operation is performed within the first Vsync period to obtain the second image, and when the first buffer is not empty, the second image is stored in the second buffer.

6. The method according to claim 5, characterized in that The display priority of the first buffer is higher than that of the second buffer; The electronic device performs a first display operation, including: The first image is obtained from the first buffer having a higher display priority, so as to synthesize and display the first image.

7. The method according to claim 6, characterized in that After the electronic device performs the first display sending operation, the method further includes: The first image in the first buffer is released, and the first buffer is empty after being released.

8. The method according to claim 6 or 7, characterized in that: After performing the first display sending operation, the method further includes: The second image in the second buffer is stored in the first buffer; and the second image in the second buffer is released.

9. The method according to claim 8, characterized in that The electronic device performs a second display sending operation, including: The second image is obtained from the first buffer so as to synthesize and display the second image.

10. The method according to any one of claims 1 to 9, characterized in that The electronic device is provided with a callback queue, and the callback queue is used to store a display callback and / or a drawing callback; Before the electronic device performs the first drawing operation, the method further includes: After the first Vsync signal is generated, the electronic device determines that the callback queue includes a first drawing callback; the first drawing callback includes first image information; the first image information is used to draw the first image; The electronic device performs a first drawing operation, including: The electronic device generates a first wake-up instruction according to the first drawing callback; the first wake-up instruction carries the second image information; the first wake-up instruction is used to wake up the flutter rendering module configured in the electronic device to perform a drawing operation according to the second image information; According to the first wake-up indication, the flutter rendering module performs the first drawing operation to obtain the second image.

11. The method according to claim 10, characterized in that The electronic device performs a second drawing operation, including: The electronic device generates a second wake-up instruction; the second wake-up instruction carries second image information; the second wake-up instruction is used to wake up the flutter rendering module to perform a drawing operation according to the second image information; According to the second wake-up indication, the flutter rendering module performs the second drawing operation to obtain the second image.

12. The method according to claim 11, characterized in that The method further comprises: After the first Vsync signal is generated, the electronic device determines that there is a second image that needs to be continuously drawn; The electronic device generates a second wake-up instruction, including: When the electronic device determines that there is a second image that needs to be continuously drawn, the electronic device triggers the generation of the second wake-up indication after a preset time period.

13. The method according to claim 12, characterized in that The preset duration corresponds to 2 / 3 of the first Vsync period.

14. The method according to any one of claims 10 to 13, characterized in that: The method further comprises: The electronic device encapsulates the wake-up indication generated in the first encapsulation interval and the wake-up indication generated in the second encapsulation interval, so that the type of the wake-up indication related to the encapsulation process is adjusted, and the adjusted type of the wake-up indication is a non-flutter type; The starting point of the first encapsulation interval corresponds to the electronic device determining that the callback queue includes a drawing callback, and the end point of the first encapsulation interval corresponds to generating a wake-up indication corresponding to the drawing callback; The starting point of the second encapsulation interval corresponds to the electronic device generating the wake-up indication, and the end point of the second encapsulation interval corresponds to the electronic device determining whether the callback queue includes a drawing callback after the next Vsync signal is generated.

15. The method according to any one of claims 2 to 14, characterized in that Before the electronic device performs the second display sending operation, the method further includes: The electronic device obtains the number of valid buffers in TextureBufferQueue; The electronic device performs a second display sending operation, including: When the number of valid buffers is greater than or equal to 1, the electronic device performs the second display operation.

16. The method according to any one of claims 10 to 14, characterized in that: The drawing callback is a postAnimation callback; The display callback is a postTraversal callback.

17. An electronic device, characterized in that: The electronic device comprises: a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; The memory is used to store computer program codes, and the computer program codes include computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1 to 16.

18. A chip system, characterized in that: The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method as described in any one of claims 1-16.

19. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 16.

Citation Information

Patent Citations

  • Image processing method based on vertical synchronization signal and electronic equipment

    CN110503708A

  • Image processing method and electronic equipment

    CN116075808A

  • Method To Improve Draw Performance Of Electronic Device By Accelerating UI Traversal

    IN201941027068A

  • Image processing device, and non-transitory computer-readable storage medium storing image processing program

    US20150161756A1