Image processing method and electronic equipment
By performing multiple image drawings in an electronic device in advance after the first Vsync signal arrives, and completing the drawing before the second Vsync signal arrives, the lag problem caused by the long image drawing time in continuous frame image rendering is solved, and the problem of lag caused by the image drawing time is realized. A smoother user experience.
Patent Information
- Application Number
- CN202310639151.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
In electronic devices, if a certain frame of image cannot be completed in time during the rendering of continuous frame images, it will lead to different degrees of frame drop or skipping, affecting the user experience.
After the first Vsync signal arrives, the drawing of the new image is performed at least twice in advance and the drawing is completed before the second Vsync signal arrives, the lag caused by the long image drawing time is avoided. The specific method includes performing a first drawing operation after the first Vsync signal arrives, then performing a second drawing operation within the second Vsync signal, and completing the display operation when the third Vsync signal arrives, ensuring that the image is displayed at a suitable moment. .
It effectively avoids display lag caused by the long image drawing time, ensures the continuity and fluency of the user experience, and avoids frame drops and frame skipping problems caused by the incomplete image drawing.
Smart Images

Figure CN119071561A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of image processing technology, and in particular, to an image processing method and electronic device. Background Art
[0002] When an electronic device provides a display function to a user, it often switches and displays consecutive frame images under the instruction of the user.
[0003] For example, after the user inputs a sliding operation to the electronic device, the electronic device can continuously render frame images and update the current display interface along with the user's sliding operation.
[0004] In this way, if one or more frames of continuous frame images cannot be rendered in time, frame loss or frame skipping will occur to varying degrees, thereby affecting the user experience. Summary of the invention
[0005] The present application provides an image processing method and an electronic device, which can effectively avoid problems such as freezing caused by a long image drawing process.
[0006] In order to achieve the above technical objectives, this application adopts the following technical solutions:
[0007] In a first aspect, an image processing method is provided, which is applied to an electronic device, and the method includes: receiving a first operation of a user. In response to the first operation, running a first application and presenting an interface of the first application. Receive a second operation of the user on the interface of the first application, the second operation being a sliding operation. In response to the second operation, performing a first drawing operation within a first synchronous Vsync cycle to obtain a first image. In response to the second operation, performing a second drawing operation within the first synchronous Vsync cycle to obtain a second image, wherein the moment when the first Vsync cycle starts is the moment when the first Vsync signal is received, the moment when the first Vsync signal cycle ends is the moment when the second Vsync signal is received, and the first Vsync signal and the second Vsync signal are two adjacent Vsync signals, and the start time of the second drawing operation is later than the start time of the first drawing operation. The first image and the second image are images displayed sequentially by the electronic device in response to the second operation.
[0008] It should be noted that in the present application, the first drawing operation and / or the second drawing operation are specific implementations of the drawing operation. The drawing operation may include the complete process of waiting for resources and rendering processing after the flutter framework is awakened.
[0009] In this way, after the first Vsync signal arrives and before the second Vsync signal arrives, the electronic device can execute at least two new image drawing operations, so that even if the drawing process of the second image takes a long time, it can effectively ensure that the drawing of the second image is completed before the Vsync signal that needs to be sent for display of the second image arrives. This also avoids the problem of display freeze caused by the first image being continuously displayed in multiple Vsync signal cycles due to the long drawing time of the second image, which makes the second image still not completed when the corresponding Vsync signal arrives.
[0010] Optionally, the method further includes: after the second Vsync signal is generated, the electronic device performs a first display operation, the first display operation is used to control the display screen of the electronic device to display the first image. After the third Vsync signal is generated, the electronic device performs a second display operation, the second display operation is used to control the display screen of the electronic device to display the second image. The third Vsync signal is later than the second Vsync signal, and the third Vsync signal is adjacent to the second Vsync signal.
[0011] In this way, after the first image is drawn, it can be synthesized and displayed in the period after the second Vsync signal arrives. Correspondingly, after the second image is drawn, it can be synthesized and displayed in the period after the third Vsync signal arrives. It can be understood that since the second image is drawn before the second Vsync signal, the second image can be drawn before the third Vsync signal arrives, thereby ensuring that the second image can be correctly synthesized and displayed after the third Vsync signal arrives.
[0012] Optionally, the duration between the first Vsync signal and the second Vsync signal, and the duration between the second Vsync signal and the third Vsync signal all correspond to the resolution of the display screen. For example, the duration between the first Vsync signal and the second Vsync signal, and the duration between the second Vsync signal and the third Vsync signal can be the inverse of the resolution of the display screen.
[0013] Optionally, the electronic device is configured with a texture buffer queue TextureBufferQueue, which includes at least one valid buffer, each of which stores a completed image. In this example, the effective buffer field length of the TextureBufferQueue in the electronic device can be the length of one buffer or the length of multiple buffers. When the effective buffer field length of the TextureBufferQueue is the length of multiple buffers, it corresponds to that the TextureBufferQueue includes multiple valid buffers. It can be understood that when the TextureBufferQueue includes multiple valid buffers, the TextureBufferQueue can simultaneously store multiple images that have been completed and are waiting to be synthesized and displayed.
[0014] Optionally, the at least one valid buffer includes a first buffer. The performing a first drawing operation in a first synchronous Vsync cycle to obtain a first image includes: performing the first drawing operation in the first Vsync cycle to obtain the first image, and storing the first image in the first buffer when the first buffer is empty.
[0015] Optionally, the at least one valid buffer includes a second buffer. The performing of the second drawing operation within the first synchronous Vsync cycle to obtain the second image includes: performing the second drawing operation within the first Vsync cycle to obtain the second image, and storing the second image in the second buffer when the first buffer is not empty.
[0016] In this example, when the second image is finished drawing, the first image has not yet been consumed. In this way, the first image can continue to be stored in the first buffer, and the second image is stored in the second buffer.
[0017] Optionally, the display priority of the first buffer is higher than that of the second buffer. The electronic device performs a first display operation, including: acquiring the first image from the first buffer with a higher display priority, so as to perform synthesis display according to the first image.
[0018] Optionally, after the electronic device executes the first display operation, the method further includes: releasing the first image in the first buffer, and the first buffer is empty after the release.
[0019] Optionally, after executing the first display operation, the method further includes: storing the second image in the second buffer into the first buffer, and releasing the second image in the second buffer.
[0020] Optionally, the electronic device performs a second display operation, including: acquiring the second image from the first buffer to synthesize and display the second image.
[0021] In this way, by configuring the priorities of different buffers, no matter how many valid buffers are included in the TextureBufferQueue, the electronic device can obtain the image from the first buffer for synthesis and display when synthesis and display are needed.
[0022] Optionally, a callback queue is configured in the electronic device, and the callback queue is used to store a display callback and / or a drawing callback. Before the electronic device performs the first drawing operation, the method also includes: after the first Vsync signal is generated, the electronic device determines that the callback queue includes a first drawing callback. The first drawing callback includes first image information. The first image information is used to draw the first image. The electronic device performs a first drawing operation, including: the electronic device generates a first wake-up indication according to the first drawing callback. The first wake-up indication carries the second image information. The first wake-up indication is used to wake up the flutter rendering module configured in the electronic device to perform a drawing operation according to the second image information. According to the first wake-up indication, the flutter rendering module performs the first drawing operation to obtain the second image.
[0023] Optionally, the electronic device performs a second drawing operation, including: the electronic device generates a second wake-up indication. The second wake-up indication carries second image information. The second wake-up indication is used to wake up the flutter rendering module to perform a drawing operation according to the second image information. According to the second wake-up indication, the flutter rendering module performs the second drawing operation to obtain the second image.
[0024] Optionally, the method further includes: after the first Vsync signal is generated, the electronic device determines that there is a second image that needs to be continued to be drawn. The electronic device generates a second wake-up indication, including: when the electronic device determines that there is a second image that needs to be continued to be drawn, after a preset time has passed, triggering the generation of the second wake-up indication.
[0025] Optionally, the preset duration corresponds to 2 / 3 of the first Vsync period.
[0026] In this way, a solution for executing the drawing operation of the second image in advance is provided. It can be understood that in some other implementations, the preset duration can also be flexibly configured according to actual conditions. For example, the electronic device can configure a preset duration greater than or equal to the average drawing duration based on the average drawing duration of the completed images, and the average drawing duration can be less than the first Vsync cycle. In this way, the flutter framework can start triggering the drawing of the second image after the first image is drawn, even if the next Vsync signal has not arrived, and the electronic device can also start triggering the execution of the drawing of the second image according to the end of the preset duration.
[0027] Optionally, the method also includes: the electronic device encapsulates the wake-up indication generated in the first encapsulation interval and the wake-up indication generated in the second encapsulation interval, so that the type of the wake-up indication related to the encapsulation processing is adjusted, and the adjusted type of the wake-up indication is a non-flutter type. The starting point of the first encapsulation interval corresponds to the electronic device determining that the callback queue includes a drawing callback, and the end point of the first encapsulation interval corresponds to generating the wake-up indication corresponding to the drawing callback. The starting point of the second encapsulation interval corresponds to the electronic device generating the wake-up indication, and the end point of the second encapsulation interval corresponds to the electronic device determining whether the callback queue includes a drawing callback after the next Vsync signal is generated. This ensures that two or more wake-up indications will not appear in the same Vsync cycle, thereby avoiding multiple invalid wake-ups of the flutter framework.
[0028] Optionally, before the electronic device performs the second display operation, the method further includes: the electronic device obtains the number of valid buffers in TextureBufferQueue. The electronic device performs the second display operation, including: when the number of valid buffers is greater than or equal to 1, the electronic device performs the second display operation. Thus, in the case of multiple valid buffer configurations, the electronic device can achieve accurate consumption of subsequent buffers based on the number of remaining valid buffers before executing buffer consumption.
[0029] Optionally, the drawing callback is a postAnimation callback, and the display callback is a postTraversal callback.
[0030] In a second aspect, the present application further provides an electronic device, the electronic device comprising: a display screen, a memory and one or more processors. The memory and the processor are coupled. The memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the technical solution provided in the above-mentioned first aspect and any possible implementation thereof.
[0031] In a third aspect, the present application also provides a chip system, which is applied to an electronic device; the chip system may include one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by a line, and the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, and the signal includes a computer instruction stored in the memory. When the processor executes the above-mentioned computer instruction, the electronic device executes the technical solution provided in the above-mentioned first aspect and any possible implementation thereof.
[0032] In a fourth aspect, the present application also provides a computer-readable storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the technical solution provided in the above-mentioned first aspect and any possible implementation thereof.
[0033] In a fifth aspect, the present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the technical solution provided in the above-mentioned first aspect and any possible implementation thereof.
[0034] It can be understood that the solutions provided in the second aspect to the fifth aspect of the present application can respectively correspond to the first aspect and any possible design thereof, so the beneficial effects that can be achieved are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A schematic diagram of an interface interaction for a user to input a page sliding operation;
[0036] Figure 2 A schematic diagram of the logical composition of a rendering processing framework;
[0037] Figure 3 A logical diagram of the production and consumption of buffers in a texture buffer queue;
[0038] Figure 4 A schematic diagram of the module interaction of buffer production and consumption;
[0039] Figure 5 This is a timing diagram of buffer production and consumption in one situation;
[0040] Figure 6 This is a timing diagram of buffer production and consumption in another case;
[0041] Figure 7 A schematic diagram of software composition of an electronic device provided in an embodiment of the present application;
[0042] Figure 8 A schematic diagram of software composition of another electronic device provided in an embodiment of the present application;
[0043] Fig. 9 A schematic diagram of module interaction of an image processing method provided in an embodiment of the present application;
[0044] Fig.10 A schematic diagram of software implementation for triggering early wake-up of the flutter framework provided in an embodiment of the present application;
[0045] Fig.11 A schematic diagram of software implementation for triggering early wake-up of the flutter framework provided in an embodiment of the present application;
[0046] Fig.12 A schematic diagram of software implementation for triggering early wake-up of the flutter framework provided in an embodiment of the present application;
[0047] Fig.13 A timing diagram of buffer production and consumption after adopting the solution provided in the embodiment of the present application;
[0048] Fig.14 A logical diagram of a command package provided in an embodiment of the present application;
[0049] Fig.15 A logical schematic diagram of multi-buffer control provided in an embodiment of the present application;
[0050] Fig.16 A logic diagram of the internal implementation of software for multi-buffer control provided in an embodiment of the present application;
[0051] Fig.17 A schematic diagram of module interaction of an image processing method;
[0052] Fig.18 A schematic diagram of module interaction of an image processing method;
[0053] Fig.19 This is a timing diagram of buffer production and consumption in one situation;
[0054] Fig. 20 A logical schematic diagram of the production and consumption of buffers in a texture buffer queue provided in an embodiment of the present application;
[0055] Fig.21 A schematic diagram of module interaction of an image processing method provided in an embodiment of the present application;
[0056] Fig. 22 A timing diagram of buffer production and consumption after adopting the solution provided in the embodiment of the present application;
[0057] Fig.23A schematic diagram of the composition of an electronic device provided in an embodiment of the present application;
[0058] Fig.24 A schematic diagram of the composition of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0059] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.
[0060] One or more application programs may be installed in an electronic device. By running the application programs, the electronic device may provide a variety of different functions to the user.
[0061] Taking a mobile phone as an example, the electronic device may include shopping applications, calendar applications, photo galleries, memos, etc.
[0062] Exemplary, reference Figure 1 , icons of various application programs may be displayed in the interface 12 of the mobile phone. For example, the mobile phone may display an icon 11 corresponding to a shopping application on the interface 12.
[0063] When the shopping application needs to be used, the user can input a corresponding operation to the icon 11 to instruct the electronic device to start running the shopping application. The corresponding operation input to the icon 11 may include a click operation, etc. Exemplarily, the operation to the icon 11 may be a first operation.
[0064] In response to a click operation input by the user, the mobile phone can run a shopping application and switch to display an interface provided by the shopping application on the display screen.
[0065] For example, after the user inputs an operation on icon 11, the mobile phone can switch to display Figure 1 Interface 13 is shown.
[0066] As an example, the interface 13 may include display windows of multiple different commodities provided by the shopping application. For example, the interface 13 may include a display frame 14 corresponding to commodity 1 and a display frame 15 corresponding to commodity 2.
[0067] In this example, the shopping application also provides display windows corresponding to more products.
[0068] The user can input a switching instruction on the interface 13 to instruct the mobile phone to refresh the interface and display the display window of other products. For example, the switching instruction can include a sliding operation corresponding to the operation 16 input on the interface 13. Exemplarily, the switching instruction can also be a second operation.
[0069] In response to the operation 31 input by the user, the mobile phone can refresh the interface and display the interface 17. In the interface 17, more display windows of commodities other than commodity 1 and commodity 2 can be included. For example, the interface 17 can include a display frame 18 corresponding to commodity 3 and a display frame 19 corresponding to commodity 4.
[0070] In this way, during the interaction with the user, the shopping application in the mobile phone can provide the user with more product information.
[0071] In such Figure 1 In the scenario shown, the electronic device can implement interface displays including interface 12, interface 13, and interface 17 through image rendering and display.
[0072] The image rendering may be performed by an operating system of the electronic device.
[0073] It is understandable that different electronic devices may have different operating systems. Different operating systems also have different image rendering mechanisms. Currently, the flutter framework can be configured in electronic devices. The flutter framework can be used to render the complete interface elements, and then perform operations such as display through the native framework of the electronic device. This can unify the image rendering process in different operating systems to a certain extent and improve code compatibility.
[0074] In this application, combined Figure 2 , taking the frameworks involved in rendering processing including native framework and flutter framework as an example. Among them, the threads related to image rendering in the native framework can be collectively referred to as main threads. Correspondingly, the threads related to image rendering in the flutter framework can be collectively referred to as flutter threads.
[0075] Exemplarily, the main thread may include a main drawing thread, or a main UI thread. The main thread may also include a rendering thread M for performing image rendering processing. The flutter thread may include a flutter UI thread and a rendering thread F for performing image rendering processing.
[0076] Figure 3 The following is a logical implementation of image rendering and display using a flutter framework in conjunction with a native framework.
[0077] In this example, one application may correspond to one texture buffer queue (TextureBufferQueue). Currently, the texture buffer queue may include a buffer (buffer) for storing display content that has been rendered.
[0078] In this Figure 3 In the logical implementation shown, the flutter framework can act as a producer of buffer content. Exemplarily, the flutter UI thread can request a free buffer area from TextureBufferQueue through the dequeueBuffer() function. The rendering thread F of the flutter framework can be used to render a new frame image and obtain the display content of the next frame image. Then, the flutter UI thread can fill the buffer area with the display content of the next frame image through the euqueueBuffer() function and return it to the queue.
[0079] Correspondingly, the native framework can act as a consumer of buffer content. Exemplarily, the native framework can obtain the buffer area from TextureBufferQueue through the acquireBuffer() function. The buffer area can store the display content of the frame image filled by the flutter framework. Thus, the native framework can send the content in the acquired buffer through SurfaceFlinger to display the frame image on the display screen of the electronic device. In addition, the native framework can also release the buffer through the releaseBuffer() function after sending it for display.
[0080] In some implementations, after the native framework obtains the buffer from TextureBufferQueue and sends it to the display, the buffer in TextureBufferQueue is empty. Correspondingly, the Flutter framework can do this as follows: Figure 3 The logic shown in the figure continues to generate new buffers for subsequent native framework consumption. This cycle can achieve continuous frame rendering and display.
[0081] It should be noted that if Figure 3As shown, in the existing solution implementation, when the euqueueBuffer() function is executed, if there is a buffer in the queue, the new buffer will replace the existing buffer. This corresponds to configuring at most one buffer in the TextureBufferQueue. When the acquireBuffer() function is executed, all buffers in the queue will be dropped and a new buffer will be acquired. The Drop operation corresponds to discarding the contents of all buffers in the queue. This enables the rendering of a new frame image on the new buffer.
[0082] like Figure 3 As shown in the buffer production and consumption logic, the native framework, as a consumer of the buffer, needs to establish a connection with the flutter framework between the buffer producers, so as to obtain the buffer from the TextureBufferQueue for display consumption when there is a buffer that can be consumed. Based on the connection between the native framework and the flutter framework, the native framework can also know from the flutter framework that after the next Vsync signal arrives, the flutter framework needs to be awakened to draw the subsequent frame image, that is, to produce the buffer that needs to be consumed later.
[0083] It can be understood that the Vsync signal involved in the embodiments of the present application can be an indication signal. When an application needs to display or draw an image, it can apply for a Vsync signal from an electronic device (such as surfaceflinger). Correspondingly, surfaceflinger can send a Vsync signal to the application at a certain period. For example, the certain period may correspond to the refresh period of the display screen. The refresh period may refer to the market between two adjacent frames of images displayed corresponding to the refresh rate. For example, corresponding to the display screen, after the Nth frame image is displayed for a duration of TL1, the N+1th frame image is switched to be displayed. Then the refresh period (or refresh interval) can be a duration of TL1.
[0084] In the application, Figure 1Take the example of a user inputting a page sliding operation (such as operation 16) on the main interface of a shopping application. Then, the shopping application can receive multiple Vsync signals such as the first Vsync signal and the second Vsync signal from the surfaceflinger, and the shopping application can send the received Vsync signal to the native framework to trigger the corresponding image display consumption and the drawing production of the new image. For example, after the user inputs the page sliding operation, the shopping application can instruct the electronic device to produce and consume multiple frame images (such as the Nth frame image, the N+1th frame image, etc.) through multiple Vsync signals such as the first Vsync signal and the second Vsync signal. From another perspective, the multiple frame images can be images displayed on the display screen in sequence by the electronic device in response to the page sliding operation input by the user.
[0085] As an example, the flutter framework can realize the connection between the native framework and the flutter framework by registering one or more callbacks with the native framework.
[0086] For example, the one or more callbacks may include a postTraversal callback, and a postAnimation callback.
[0087] Among them, the postTraversal callback can be used to indicate to the native framework that a new buffer has been produced and put into the TextureBufferQueue. Correspondingly, when the postTraversal callback has been received, the native framework can execute the Traversal method after the new Vsync signal arrives and obtain the buffer from the TextureBufferQueue for consumption.
[0088] The postAnimation callback can be used to indicate to the native framework that there are subsequent frames that need to be rendered and displayed after the next Vsync signal is sent to the display. Correspondingly, when the postAnimation callback has been received, the native framework can execute the Animation method after the new Vsync signal arrives and send a wake-up instruction to the flutter framework. This allows the flutter framework to generate the buffer corresponding to the subsequent frame image according to the wake-up instruction.
[0089] The following combination Figure 4 For example, Figure 1 The rendering switching process of interface 13-interface 17 in the shown scenario is illustrated by way of example.
[0090] It is understandable that in Figure 1During the switching process from interface 13 to interface 17, the electronic device can draw multiple frames of images continuously under the instruction of the user's operation 16. Correspondingly, the electronic device can perform continuous sliding display on the interface. The starting point of the continuous sliding display can be the interface 13 corresponding to when the operation 16 is received (such as pressing the sliding operation). The end point of the continuous sliding display can be the interface 17 when the operation 16 ends (such as releasing the sliding operation). Figure 4 In the example, the drawing and display process of any frame image (such as the Nth frame image) in the interface 13-interface 17 is taken as an example. Figure 4 The example of shows the display process of the Nth frame image to the N+1th frame image. The display image of the Nth frame image can be rendered and obtained during the processing of the previous frame image. Correspondingly, during the processing of the N+1th frame image, the electronic device can also render the next frame N+2th frame image.
[0091] like Figure 4 As shown, the drawing and displaying of the Nth frame image may include:
[0092] S401, the main drawing thread receives the operation 16 input by the user and transparently transmits the operation information to the flutter UI thread.
[0093] Exemplary, combined Figure 1 For example, after the electronic device runs the application, the interface corresponding to the application can be displayed.
[0094] The user can input various operations on the interface of the application program. For example, the user can input an operation 16 (such as a sliding operation) on the interface (such as interface 13) of the running shopping application to instruct the electronic device to slide the page.
[0095] Correspondingly, the electronic device can transparently transmit the operation information input by the user to the flutter UI thread for processing.
[0096] In this example, the sliding operation corresponding to the operation 16 input by the user may include multiple consecutive sub-operations.
[0097] For example, the sliding operation may include: pressing at position 1, continuously pressing and sliding along track 2, releasing at position 3 corresponding to the end point of track 2 (ie, releasing the hand), and other sub-operations.
[0098] In this way, the main drawing thread can send the operation information corresponding to each sub-operation to the flutter UI thread for processing. Among them, an operation information may include at least one of the following: the operation type corresponding to the current moment (such as pressing, sliding, leaving the hand, etc.), and the coordinate information on the display screen.
[0099] That is to say, when the user inputs continuous operations as shown in operation 16, the main drawing thread can correspondingly and continuously transmit the operation information of the operation input by the user to the flutter UI thread, so that the flutter UI thread can determine whether there are multiple continuous frame images to be drawn, and perform rendering processing on each frame image.
[0100] It should be noted that in some implementations, the flutter framework may combine the following steps to determine that a subsequent rendering operation of at least one frame image needs to be performed according to the operation information: Figure 3 Register the postTraversal callback and postAnimation callback with the native framework as described in , so that the native framework can perform the corresponding consumption and wake-up operations after the next Vsync signal arrives.
[0101] S402: The main drawing thread receives a Vsync signal V0, triggering execution of a DoFrame operation.
[0102] During the image rendering process, the electronic device can generate multiple Vsync signals corresponding to the screen refresh rate. Each Vsync signal can be used to instruct the rendering and display of a frame image.
[0103] In this example, the Vsync signal V0 may be included in the multiple Vsync signals. The Vsync signal V0 may be used to indicate the display of the Nth frame image and the rendering of the N+1th frame image. The display content of the Nth frame image may be drawn by the flutter framework and stored in the TextureBufferQueue after the arrival of the previous Vsync signal before the Vsync signal V0.
[0104] Correspondingly, the main drawing thread can trigger the execution of the doFrame operation after receiving the Vsync signal V0.
[0105] Exemplarily, the doFrame operation may include an animation operation for performing animation-related processing, and a traversal operation for performing LayerTree traversal-related processing.
[0106] Next, the electronic device may send and display the Nth frame image through the following steps S403-S404. The electronic device may also render the N+1th frame image through the following steps S405-S406. Figure 4The implementation of the scheme shown does not constitute a limitation on the order of sending and displaying the Nth frame image and rendering the N+1th frame image. In different implementations, the processing of sending and displaying the Nth frame image and rendering the N+1th frame image can be performed in series, or the processing of sending and displaying the Nth frame image and rendering the N+1th frame image can be performed simultaneously.
[0107] It is understandable that, for the main drawing thread, before executing the sending of the rendering instruction 21 corresponding to S403 and the sending of the wake-up instruction 22 corresponding to S405, a traversal search process of the traversal queue may be performed.
[0108] The traversal queue may also be referred to as a callback queue. The traversal queue may include one or more callbacks received by the main drawing thread, and each callback may correspond to one or more functions. When the main thread traverses to the callback, the function configured in the callback may be executed to perform corresponding operations.
[0109] Exemplarily, in combination with the above description, in this example, when the main drawing thread executes the DoFrame operation corresponding to S402, the traversal queue may include a postTraversal callback and a postAnimation callback.
[0110] Then, when traversing to the postTraversal callback, the main drawing thread can generate the corresponding rendering instruction 21 according to one or more functions configured in the postTraversal callback, and then perform the corresponding sending process in S403.
[0111] When traversing to the postAnimation callback, the main drawing thread may generate a corresponding wake-up instruction 22 according to one or more functions configured in the postAnimation callback, and then execute the corresponding sending process in S405.
[0112] In the embodiment of the present application, the sending of the wake-up instruction 22 before the sending of the rendering instruction 21 is taken as an example.
[0113] The following describes the buffer consumption based on the postTraversal callback (i.e., the display consumption of the new image) and the buffer production based on the postAnimation callback (i.e., the rendering of the next frame of the image).
[0114] S403: The main drawing thread sends a rendering instruction 21 to the rendering thread M.
[0115] The rendering instruction 21 may be used to instruct the rendering thread M to perform display-related processing of the Nth frame image through a drawing command.
[0116] S404, rendering thread M obtains buffer from the texture buffer queue for consumption.
[0117] In this example, the electronic device may create Buffer0 in the TextureBufferQueue of the shopping application corresponding to the current operation before the current Vsync signal V0 arrives, and store the display content P0 in Buffer0. This process may be performed by the flutter framework after the last Vsync signal arrives.
[0118] Correspondingly, in S404, the rendering thread M can obtain Buffer0 from TextureBufferQueue and consume it after receiving the rendering instruction 21. For example, the rendering thread M can obtain buffer0 and the display content P0 stored therein through the acquireBuffer() function. The rendering thread M can transmit the display content P0 to SurfaceFlinger so that SurfaceFlinger can perform synthesis processing based on the display content P0, thereby displaying the Nth frame on the display screen. For another example, the rendering thread M can release the consumed buffer0 through the releaseBuffer() function.
[0119] Thus, after completing S404, the available buffer in TextureBufferQueue is empty.
[0120] S405. The main drawing thread sends a wake-up instruction 22 to the Flutter UI thread.
[0121] The wake-up instruction 22 may be used to instruct the flutter framework to render a new frame image. Figure 4 As shown, the execution of S405 can be before S403. Generally, since the production process of a new buffer takes a long time, in this example, although the triggering of buffer consumption is after the buffer production, the consumption of the existing buffer can be completed before the production of the new buffer is completed.
[0122] In combination with the description in S401, the main drawing thread may execute S405 when the postAnimation callback sent by the flutter framework has been received.
[0123] Exemplarily, the wake-up indication 22 may carry a timestamp and corresponding rendering information.
[0124] The timestamp corresponds to a moment, and the timestamp is used to indicate that the image rendering process is performed after the corresponding moment.
[0125] Correspondingly, the rendering information carried in the wake-up indication 22 may correspond to the rendering related information of the content to be displayed after the corresponding time of the timestamp. For example, the rendering information may include vertex information, color information, etc. required for rendering the interface after the corresponding time of the timestamp arrives.
[0126] S406. The Flutter UI thread instructs the rendering thread F to draw a new frame image and store it in the texture buffer queue.
[0127] Exemplarily, in some embodiments, the flutter UI thread may execute a response after the corresponding time according to the timestamp carried by the wake-up indication 22 .
[0128] For example, the timestamp carried by the wake-up instruction 22 is taken as time A. Then, after receiving the wake-up instruction 22, the flutter UI thread can instruct the rendering thread F to render the new image according to the rendering information carried in the wake-up instruction 22 when time A arrives.
[0129] In this way, when timeA arrives, the flutter UI thread can send a drawing instruction 23 to the rendering thread F, so as to instruct the rendering thread F to perform rendering of the N+1th frame image according to the rendering information carried in the wake-up instruction 22.
[0130] In response to the drawing instruction 23, the rendering thread F may render the N+1th frame image. For example, by rendering the N+1th frame image, the corresponding display content P1 may be obtained. In the present application, the N+1th frame image corresponding to the display content P1 may also be referred to as the first image.
[0131] It is understandable that when drawing a frame image, the rendering thread F needs to use a corresponding buffer to store intermediate data and drawing results.
[0132] In this way, combined Figure 3 , after receiving the drawing instruction 23, the rendering thread F can obtain the cache area (such as buffer1) from the TextureBufferQueue for rendering the N+1th frame image. In this way, after the rendering thread F completes the rendering of the N+1th frame image, the rendering result corresponding to the N+1th frame image can be stored on the buffer1, that is, the display content P1 is stored on the buffer1.
[0133] The rendering thread F or the flutter UI thread may also put the buffer 1 storing the display content P1 into the TextureBufferQueue for the next consumption by the native framework.
[0134] In some implementations, such as Figure 4 As shown, after producing a new buffer (such as buffer1) and putting it into TextureBufferQueue, the flutter UI thread can execute S407, that is, register the postTraversal callback and the postAnimation callback with the main drawing thread, so that after the next Vsync signal arrives, the native framework and the flutter framework can continue to perform a new round of buffer display consumption and the production of new buffers.
[0135] Thus, through the operations of S402-S406, the electronic device realizes the sending and display of the Nth frame image and the rendering of the N+1th frame image. In addition, through the operation of S407, the electronic device can continue to consume and produce subsequent frame images.
[0136] For example, Figure 4 As shown, after receiving the Vsync signal V1, the solution may also include:
[0137] S408: The main drawing thread executes a DoFrame operation.
[0138] The operation of S408 may refer to S402.
[0139] In this example, combined with the description of S407, as a response to the postTraversal callback, the DoFrame operation may include executing the Traversal method to consume the buffer, which may specifically trigger the following S409-S410.
[0140] In addition, as a response to the postAnimation callback, the DoFrame operation may also include executing the Animation method to produce a buffer, which may specifically trigger the following S411-S412.
[0141] S409: The main drawing thread sends a rendering instruction 24 to the rendering thread M.
[0142] Exemplarily, the rendering instruction 24 may be used to instruct the rendering thread M to draw and display the N+1th frame image through a Draw command.
[0143] In combination with the description in S406 , the newly generated buffer 1 may be stored in the current TextureBufferQueue, and the buffer 1 may store the display content P1 corresponding to the N+1th frame image.
[0144] Then, the rendering instruction 24 can also be used to instruct the rendering thread M to consume the buffer in the current TextureBufferQueue, so as to realize the display consumption of the N+1th frame image.
[0145] S410, rendering thread M obtains a buffer from a texture buffer queue (ie, TextureBufferQueue) for consumption.
[0146] Thus, the rendering thread M can obtain the currently stored buffer 1 from the TextureBufferQueue, and send the display content P1 in the buffer 1 to the surfaceflinger for display. Similar to the description of S404, after the consumption of the buffer 1, the buffer 1 can be released, and the corresponding buffer in the TextureBufferQueue is empty.
[0147] Correspondingly, surfaceflinger can perform synthesis processing according to the display content P1 so as to display the image of the N+1th frame on the display screen.
[0148] In the process of executing the DoFrame operation, in addition to completing the consumption of the above buffer1, the electronic device can also execute the production of the buffer corresponding to the next frame image through the following S411-S412.
[0149] S411. The main drawing thread sends a wake-up instruction 25 to the flutter UI thread.
[0150] Exemplarily, the execution of this step may refer to the description in S405.
[0151] The corresponding timestamp and rendering information may also be carried in the wake-up indication 25. Thus, the flutter framework can render the next frame (such as the N+2th) frame image according to the timestamp and rendering information.
[0152] S412. The Flutter UI thread instructs the rendering thread F to draw a new frame image and store it in the texture buffer queue.
[0153] Exemplarily, the flutter UI thread may send a drawing instruction 26 to the rendering thread F. In some implementations, the drawing instruction 26 may be issued after the time corresponding to the timestamp carried by the wake-up instruction 25.
[0154] In response to the drawing instruction 26, the rendering thread F may draw the display content P2 corresponding to the N+2 frame image according to the rendering information carried by the wake-up instruction 25. The rendering thread F may also store the display content P2 in a new buffer (such as buffer2) of TextureBufferQueue. In this application, the N+2 frame image corresponding to the display content P2 may also be referred to as the second image.
[0155] Combination Figure 3 , the rendering thread F can obtain buffer2 from TextureBufferQueue and perform rendering of the display content P2 on buffer2. After completing the rendering, the rendering thread F can also put the buffer2 back into TextureBufferQueue.
[0156] In this way, the rendering of the N+2th frame image can be completed.
[0157] Similar to the operation in S407, the flutter framework can execute S413 after completing the production of the buffer corresponding to the N+2th frame image, and initiate the postTraversal callback and postAnimation callback to the main drawing thread. So that after the subsequent Vsync signal arrives, the electronic device can perform the corresponding display and / or rendering of the new image according to the similar process of S402-S407 or S408-S413.
[0158] In order to make it more clear Figure 4 The process shown is explained. Figure 5 The figure shows the timing logic of the image sending and rendering process.
[0159] like Figure 5 As shown, before the Vsync signal V0 arrives, the TextureBufferQueue may include buffer0 corresponding to the Nth frame image that has been produced. Before the Vsync signal V0 arrives, the main drawing thread can receive the postTraversal callback and postAnimation callback from the flutter framework, so after receiving the Vsync signal V0, DoFrame can be triggered to consume buffer0 and produce buffer1 corresponding to the new N+1th frame image.
[0160] Exemplarily, after receiving the Vsync signal V0, the native framework may instruct the corresponding rendering thread M to consume the current buffer0 through a rendering instruction. For example, at time T0, the native framework completes the display consumption of buffer0.
[0161] After receiving the Vsync signal V0, the native framework can also wake up the flutter framework through a wake-up instruction to produce the corresponding buffer of the new frame image. For example, before the next Vsync signal V1 arrives, the flutter framework can complete the waiting resources and rendering processing, thereby storing the buffer1 including the N+1th frame image obtained by rendering in the TextureBufferQueue. Among them, the waiting resources (also called Runnable) can correspond to the process of waiting for the electronic device to allocate corresponding processing resources (such as computing power resources, storage resources, etc.) after the flutter UI thread is awakened. After the waiting resources are allocated, the flutter UI thread can instruct the rendering process F to perform the rendering processing of the new frame image, which corresponds to the Running stage.
[0162] It is understandable that, combined with Figure 4 As described in S407, in this example, the Flutter framework can also send a callback indication to the native framework, which can include a postTraversal callback and a postAnimation callback. As a result, after the next Vsync signal arrives, the main drawing thread can consume buffer1 according to the received postTraversal callback, and wake up the Flutter framework according to the postAnimation callback to produce a new buffer.
[0163] In such Figure 5 In the example, after receiving the Vsync signal V1, the native framework can cooperate with the flutter framework to perform buffer consumption (such as the consumption of buffer1) and the production of new buffers (such as buffer2) similar to the above examples. The flutter framework can also send a postTraversal callback to the native framework after putting buffer2 into the TextureBufferQueue to facilitate the display consumption in the next cycle (that is, after the next Vsync signal arrives). In addition, when the flutter framework determines that there is a new image to be rendered, it can send a postAnimation callback to the native framework before the next cycle arrives (that is, before the next Vsync signal arrives), so that after the next cycle arrives, the native framework wakes up the flutter framework to render the new buffer.
[0164] In this example, it is taken that the native framework completes the display consumption of buffer1 corresponding to the N+1th frame image at time T1.
[0165] Similarly, at the subsequent time T2, the native framework can complete the display consumption of buffer2 corresponding to the N+2th frame image. The buffer2 can be produced by the flutter framework in the period between the Vsync signal V1 and the Vsync signal V2.
[0166] At the subsequent time T3, the native framework can complete the display consumption of buffer3 corresponding to the N+3th frame image. The buffer3 can be produced by the flutter framework in the period between the Vsync signal V2 and the Vsync signal V3.
[0167] Thus, between the frames T0 and T1, the electronic device can display the Nth frame image. Between the frames T1 and T2, the electronic device can display the N+1th frame image. Between the frames T2 and T3, the electronic device can display the N+2th frame image. In the display frames after T3, the electronic device can display the N+3th frame image. Figure 4 The solution shown is implemented so that the electronic device can continuously and timely display the page update corresponding to the user sliding operation (such as operation 16). Among them, the display frame can correspond to the refresh rate of the current display screen. The duration of a display frame is based on the display duration of 1 frame image corresponding to the refresh rate.
[0168] It can be seen that in order to enable each display frame to display each frame image continuously and accurately, it is necessary to ensure that the frame image currently to be displayed can be successfully produced and put into the TextureBufferQueue during the display process of the previous frame image.
[0169] However, even for continuous frame images, there may be situations where the frame images that need to be displayed cannot be produced and put into the TextureBufferQueue in time.
[0170] For example, a new frame image may have to wait for a long time before being rendered. This may be due to the high load of the current electronic device or the low priority of the command for configuring the current resource. In this way, the rendering process cannot be executed in time, and the buffer that needs to be consumed cannot be rendered and put into the TextureBufferQueue in time.
[0171] For example, during the rendering process of a new frame image, due to the heavy load of the current electronic device, insufficient computing power allocation, or the elements of the current frame image are more complex or more in number, the rendering time is long. In this way, the buffer of the frame image cannot be produced in time and put into the TextureBufferQueue.
[0172] This will affect the display of continuous frame images, causing problems such as freeze, frame loss, and frame skipping.
[0173] As an example, it is taken that the rendering time of the N+2th frame image is longer.
[0174] refer to Figure 6 , buffer0 can be consumed normally at time T0, and buffer1 can be consumed normally at time T1. In this way, between time T0 and time T1, the electronic device can display the Nth frame image normally. The electronic device can display the N+1th frame image starting from time T1.
[0175] In this example, the rendering time corresponding to buffer2 is longer, which results in buffer2 not being completed before the arrival of the Vsync signal V2. In this way, before the arrival of the Vsync signal V2, since the new buffer has not been completed, the native framework will not receive the corresponding postTraversal callback. Then the native framework will not trigger the consumption of the new buffer according to the arrival of the Vsync signal V2. As a result, the switching display of the N+2th frame image that should have been performed at time T2 will not be performed. Correspondingly, between the time T2 and the time T3, the electronic device can only continue to display the N+1th frame corresponding to the previous frame.
[0176] In this example, the production of buffer2 can be completed in the period between the Vsync signal V2 and the Vsync signal V3, so that when the Vsync signal V3 arrives, the native framework can have received the postTraversal callback. Therefore, at time T3 after the Vsync signal V3 arrives, the native framework can consume the buffer2 to display the N+2th frame image.
[0177] That is to say, in this Figure 6 In the example, from time T1 to time T2 to time T3, the electronic device will continue to display the N+1th frame, which causes a freeze.
[0178] In order to solve the above problems, the embodiment of the present application provides an image processing method, which can flexibly adjust the wake-up timing of the flutterUI thread, so that Figure 4The flutter framework in S406 shown can perform the buffer production operation in advance. Therefore, even if waiting for resources and / or rendering processing (i.e., drawing process) takes a long time, the new buffer production can be effectively completed before the next Vysnc signal arrives. Then, after the next Vysnc signal arrives, the native framework can smoothly obtain the new buffer from TextureBufferQueue for display consumption. This can also avoid the following Figure 6 As shown or due to other reasons, the buffer cannot be produced in time, resulting in problems such as jamming.
[0179] The technical solution provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0180] It should be noted that the technical solution provided in the embodiment of the present application can be applied to the user's electronic device. The electronic device may include a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, a cellular phone, a personal digital assistant (personal digital assistant, PDA), an augmented reality (augmentedreality, AR) device, a virtual reality (virtual reality, VR) device, an artificial intelligence (artificialintelligence, AI) device, a wearable device, a vehicle-mounted device, a smart home device, or at least one of a smart city device. The embodiment of the present application does not impose any special restrictions on the specific type of the electronic device.
[0181] In different embodiments of the present application, the electronic device may have different compositions.
[0182] Exemplarily, from the perspective of hardware composition, an embodiment of the present application provides an example of the composition of an electronic device.
[0183] In this example, the electronic device may include a processor, an external memory interface, an internal memory, a universal serial bus (USB) connector, a charging management module, a power management module, a battery, an antenna 1, an antenna 2, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, an earphone interface, a sensor module, a button, a motor, an indicator, a camera module, a display screen, and a subscriber identification module (SIM) card interface, etc. The sensor module may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.
[0184] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0185] In some implementations of this example, the processor may include one or more processing units, for example, the processor may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0186] The processor can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.
[0187] A memory may also be provided in the processor for storing instructions and data. In some embodiments, the memory in the processor may be a cache memory. The memory may store instructions or data that have been used by the processor or are frequently used. If the processor needs to use the instruction or data, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor, and thus improves the efficiency of the system.
[0188] Electronic devices can realize display functions through GPU, display screen, and application processor. It is a microprocessor for image processing, connecting the display screen and application processor. GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor may include one or more GPUs, which execute program instructions to generate or change display information.
[0189] The display screen is used to display images, videos, etc. The display screen includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device may include one or more display screens.
[0190] It is understandable that the above is only a schematic diagram of the composition of an electronic device provided in this application. In other embodiments, the electronic device may also have other components.
[0191] Exemplary, reference Figure 7 , is a schematic diagram of the software composition of an electronic device provided in an embodiment of the present application.
[0192] In this example, the software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the electronic device is illustrated.
[0193] In such Figure 7 In the example of , the layered architecture of the electronic device divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime, ART) and the native C / C++ library, the hardware abstraction layer (HAL) and the kernel layer.
[0194] The application layer, or application layer for short, can include a series of application packages.
[0195] like Figure 7 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications. In some embodiments, the application package may also include a shopping application. Specifically, the shopping application may be At least one of the common shopping applications such as .
[0196] Take the example of a user using a shopping application. The electronic device can receive a sliding operation from the user during the running of the shopping application. Correspondingly, the electronic device can Figure 4 The solution shown implements rendering of new images and display consumption of existing images, thereby presenting a continuous page sliding effect to the user.
[0197] The application framework layer, also known as the Application Framework layer, is referred to as the framework layer or Framework layer. This layer can provide application programming interface (API) and programming framework for the application programs in the application layer. The application framework layer includes some predefined functions.
[0198] like Figure 7As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, and the like. The window manager provides a window management service (WindowManager Service, WMS), which can be used for window management, window animation management, surface management, and as a transfer station for the input system. The content provider is used to store and obtain data and make the data accessible to the application. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, and the like. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and the like. The view system can be used to build applications. The display interface may be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures. The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, and the like. The notification manager can also be a notification that appears in the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is prompted in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc. The activity manager can provide an activity management service (Activity Manager Service, AMS), which can be used to start, switch, and schedule system components (such as activities, services, content providers, and broadcast receivers) and manage and schedule application processes. The input manager can provide an input management service (Input Manager Service, IMS), which can be used to manage system input, such as touch screen input, key input, sensor input, etc. IMS takes events from input device nodes and distributes events to appropriate windows through interaction with WMS.
[0199] In this example, the framework layer can be used to support Figure 2 The operation of the rendering processing framework shown. For example, the framework layer may include a main rendering module and a flutter rendering module. The main rendering module corresponds to the capabilities of the native framework and can be used to consume buffers through the main drawing unit and the main rendering unit. The flutter rendering unit corresponds to the capabilities of the flutter framework and can be used to produce buffers by running the flutter drawing unit and the flutter rendering unit.
[0200] Among them, the main drawing unit can correspond to Figure 4 The main rendering unit may correspond to the main drawing thread shown in FIG. Figure 4 The rendering thread M shown. The flutter drawing unit can correspond to Figure 4 The flutter UI thread shown in the figure. The flutter rendering unit can correspond to the following Figure 4 The rendering thread F is shown.
[0201] Based on the solution provided in the embodiment of the present application, the main rendering module can send a wake-up instruction to the flutter rendering module in advance according to a preset strategy, so that even if such a situation occurs, Figure 6 When the rendering time is long, or the buffer production (i.e., the drawing process) takes a long time due to other reasons (such as failure to obtain resources in time), the flutter rendering module can also have sufficient time to produce a new buffer.
[0202] The Android runtime includes the core library and the Android runtime. The Android runtime is responsible for converting source code into machine code. The Android runtime mainly uses the ahead of time (AOT) compilation technology and the just in time (JIT) compilation technology.
[0203] The core library is mainly used to provide basic Java class library functions, such as basic data structures, mathematics, IO, tools, databases, networks and other libraries. The core library provides an API for users to develop Android applications. The native C / C++ library can include multiple functional modules. For example: surface manager, media framework, libc, OpenGLES, SQLite, Webkit, etc. Among them, the surface manager is used to manage the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media framework supports a variety of commonly used audio and video format playback and recording, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. OpenGL ES provides drawing and operation of 2D graphics and 3D graphics in applications. SQLite provides a lightweight relational database for applications of electronic device 100.
[0204] The hardware abstraction layer runs in user space, encapsulates the kernel layer driver, and provides a calling interface to the upper layer. The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
[0205] Above Figure 7The software partitioning for one electronic device is not sufficient to constitute a practical restriction on the electronic device.
[0206] refer to Figure 8 , is a schematic diagram of the software composition of another electronic device provided in an embodiment of the present application.
[0207] like Figure 8 As shown, a shopping application may be configured in the application layer of the electronic device. The user may instruct the electronic device to run the shopping application, and correspondingly, the electronic device may display the interface of the shopping application to the user on the display screen.
[0208] It is understandable that when the user wants to view more shopping interfaces, the user can input instructions on the interface of the shopping application. For example, the user can input a sliding operation on the interface of the shopping application to instruct the electronic device to slide and update the display.
[0209] The process can be referred to as Figure 1 The instructions shown in the figure are not repeated here.
[0210] In such Figure 8 In the example, a plurality of modules may be configured in the application framework layer of the electronic device, and each module may further include a corresponding unit for supporting the electronic device to perform a sliding update display operation according to a user's sliding operation.
[0211] Exemplarily, the application framework layer may include a main rendering module, a flutter rendering module, and a display module.
[0212] The main rendering module is used to implement the capabilities of the native framework. For example, Figure 4 According to the description, the main drawing unit in the main rendering module can start the main drawing thread and, upon receiving the postTraversal callback, trigger the display consumption of the existing buffer in the TextureBufferQueue according to the arrival of the Vysnc signal.
[0213] For example, the main rendering unit can read the existing buffer from the TextureBufferQueue in the memory through the rendering instruction. The main rendering unit can also transmit the existing buffer to the display module so that the display module updates the display screen according to the buffer.
[0214] The display module may correspond to the surfaceflinger configured in the electronic device. After receiving the buffer for display consumption sent by the main rendering module, the surfaceflinger may transfer the data in the buffer to the buffer for synthetic display. The surfaceflinger may also instruct the display screen to update the display according to the data in the buffer for synthetic display.
[0215] The main drawing unit in the main rendering module can also trigger the wake-up of the flutter rendering module to produce a new buffer according to the arrival of the Vysnc signal when receiving the postAnimation callback.
[0216] The flutter drawing unit in the flutter rendering module can start the flutter UI thread and, upon receiving a wake-up instruction from the main drawing unit, instruct the flutter rendering unit to Figure 4 The rendering thread F shown obtains a new buffer from the TextureBufferQueue and puts the newly rendered frame image into the buffer. The Flutter rendering unit can also put the buffer including the newly rendered frame image into the TextureBufferQueue to facilitate the subsequent display consumption of the main rendering module. For example, after completing the production of the new buffer and putting it into the TextureBufferQueue, the Flutter rendering module can send a postTraversal callback to the main rendering module so that after the next Vsync signal arrives, the main rendering module continues to consume and realize display update.
[0217] In the embodiment of the present application, the memory of the electronic device may include a TextureBufferQueue. The TextureBufferQueue may include one or more buffers.
[0218] For example, a buffer is included in TextureBufferQueue. After the main rendering module completes buffer consumption, the buffer in TextureBufferQueue is empty. Correspondingly, after the flutter rendering module completes buffer production, the number of buffers in TextureBufferQueue is 1. If the buffer in TextureBufferQueue is not consumed, if the flutter rendering module completes new buffer production, the new buffer can overwrite the existing buffer stored in TextureBufferQueue because the number of buffers in TextureBufferQueue is limited to 1.
[0219] Take the example of two buffers included in TextureBufferQueue. When TextureBufferQueue is empty, the Flutter rendering module can put the buffer into the head of TextureBufferQueue each time it completes the production of a buffer. If the buffer has not been consumed, and the Flutter rendering module completes the production of a new buffer, the new buffer can be configured in the TextureBufferQueue in sequence. That is, in this example, the TextureBufferQueue can include an earlier produced buffer located at the head, and another buffer produced subsequently. In this way, when the main rendering module performs display consumption, the main rendering module can sequentially obtain the buffer in the TextureBufferQueue starting from the head. Thereby, the display consumption of the earlier produced buffer is realized. After the head buffer is consumed, the subsequent buffers can be progressively advanced to the head of the TextureBufferQueue, so that after the next Vysnc signal arrives, the main rendering module can continue to consume the buffer at the head of the TextureBufferQueue.
[0220] In the following example, we take the case where two buffers are configured in TextureBufferQueue as an example. It is understandable that when two buffers are configured in TextureBufferQueue, the situation where the new buffer overwrites the earlier buffer as described above will not exist. Correspondingly, based on the above solution of obtaining TextureBufferQueue from the header, there will be no display problems such as frame loss and frame skipping.
[0221] The solution for configuring multiple buffers in the TextureBufferQueue will be described in detail later.
[0222] In this example, to solve the problem Figure 6 In the scenario shown, the display freeze problem caused by the failure to complete any buffer production between two adjacent Vsync signals, the embodiment of the present application provides an image processing method based on the following example. Figure 8 The electronic device composition shown in the figure enables the main rendering module (or the main drawing unit therein) to wake up the flutter rendering module in advance to produce a new buffer when it is determined that the postAnimation callback has been received. In this way, the buffer with a long waiting time for resources and / or rendering processing time can be processed in advance, thereby significantly improving the Figure 6 Display stuttering issue in the scenario shown.
[0223] In the following description, each execution action of the native framework can be performed by Figure 8 The main rendering module shown is implemented. Among them, the operation of the main drawing thread in the native framework can be performed by the main drawing unit, and the operation of the rendering thread M in the native framework can be performed by the main rendering unit. Similarly, the operation of the flutter UI thread in the flutter framework can be performed by the flutter drawing unit, and the operation of the rendering thread F in the flutter framework can be performed by the Flutter rendering unit.
[0224] refer to Fig. 9 , is a flow chart of an image processing method provided in an embodiment of the present application. Fig. 9 The implementation of the scheme shown can be viewed as Figure 4 The improvement of the existing process shown in the figure. By waking up the Flutter framework in advance, the new buffer can be processed in advance, thus avoiding display jamming caused by long buffer production time.
[0225] like Fig. 9 As shown, the program may include:
[0226] S901, the main drawing thread receives the operation 16 input by the user and transparently transmits the operation information to the flutter UI thread.
[0227] The implementation of S901 can refer to Figure 4 S401 in.
[0228] In this example, operation 16 may correspond to a sliding operation input by the user on the shopping application interface. In response to operation 16, the electronic device may perform a sliding switching display of the current page.
[0229] It is understandable that during the sliding switching display process, the electronic device will quickly switch and display multiple consecutive frame images. For example, each display frame can display a new frame image. In this way, the electronic device will generate multiple consecutive Vsync instructions to trigger the consumption of buffers and the production of new buffers. In the scenario where the frame images need to be displayed continuously and quickly, it is easy to have the following situation: Figure 6 The stuttering situation shown.
[0230] In such Fig. 9 In the implementation of the solution shown, combined with Figure 4 According to the description in, the Flutter UI thread can determine the need for continuous buffer production based on the multiple operation information transmitted by the main drawing thread to support the subsequent buffer production. Then, the Flutter UI thread can send the above postAnimation callback to the main drawing thread before the next Vsync signal arrives. If the display interface needs to be switched later, the Flutter UI thread can also send the above postTraversal callback to the main drawing thread before the next Vsync signal arrives to support the switching display of the next frame image.
[0231] The Fig. 9 The scenario example shown will illustrate the execution of the postAnimation callback and the postTraversal callback.
[0232] S902 , the main drawing thread receives a Vsync signal V0 , triggering execution of a DoFrame operation D1 .
[0233] The implementation of S902 can refer to Figure 4 The Vsync signal V0 may also be referred to as the first Vsync signal. The subsequent Vsync signal V1 may also be referred to as the second Vsync signal, and the Vsync signal V2 may also be referred to as the third Vsync signal.
[0234] The Vsync signal V0 may trigger the switching display of the currently displayed N-1th frame image. For example, the Vsync signal V0 may trigger the switching display of the Nth frame image.
[0235] The Vsync signal V0 may also trigger the generation of a new buffer. For example, the Vsync signal V0 may trigger the generation of buffer 1 of the N+1th frame image.
[0236] Correspondingly, the execution of the DoFrame operation D1 may include executing the Traversal method to realize the display consumption of buffer0 corresponding to the Nth frame image. Before executing the Traversal method, the main drawing thread has received the postTraversal callback. For details, please refer to the following S903-S904.
[0237] The execution of the DoFrame operation D1 may include executing the Animation method to realize the generation of buffer1 corresponding to the N+1th frame image. Before executing the Animation method, the main drawing thread has received the postAnimation callback. For details, please refer to the following S905-S906.
[0238] Similar to the above Figure 4 As described in, in this example, the main drawing thread can send a wake-up instruction 92 (such as executing S905) before sending a rendering instruction 91 (such as executing S903). After the rendering thread M receives the rendering instruction 91 and the flutter UI thread receives the wake-up instruction 92, the two units can each asynchronously and independently perform corresponding operations. The following are explained separately.
[0239] S903: The main drawing thread sends a rendering instruction 91 to the rendering thread M.
[0240] The implementation of S903 can refer to Figure 4 The rendering instruction 91 may correspond to the rendering instruction 21 in S403.
[0241] For example, the rendering instruction 91 may include a Draw command, which is used to instruct the rendering thread M to consume the current buffer.
[0242] It is understandable that before the Vsync signal V0 arrives, the TextureBufferQueue may include buffer0 corresponding to the Nth frame image that has been produced. In this way, the rendering instruction 91 may be used to instruct the rendering thread M to send and consume buffer0.
[0243] S904, the rendering thread M obtains the buffer from the texture buffer queue (TextureBufferQueue) for consumption.
[0244] The implementation of S904 can refer to Figure 4 S404 in.
[0245] Exemplary, combined Figure 3In the example, the rendering thread M can obtain the currently stored buffer0 for consumption through acquireBuffer(). For example, the rendering thread M can send the storage address of the acquired buffer0 or the display content P0 stored in buffer0 to the display module (such as SurfaceFlinger). Correspondingly, SurfaceFlinger can store the display content P0 in the display buffer used for display on the display screen. As a result, SurfaceFlinger can perform relevant processing (such as synthesis processing) on the display content P0 entering the display buffer, and then update the display interface according to the processing results. This also realizes the switching of the current display interface from the N-1th frame image to the Nth frame image.
[0246] Combination Figure 3 as well as Figure 4 As described in , in some implementations, the rendering thread M can also release the consumed buffer0 through the releaseBuffer() function. In this way, the number of buffers in TextureBufferQueue can be 0 before a new buffer is produced.
[0247] S905. The main drawing thread sends a wake-up instruction 92 to the Flutter UI thread.
[0248] S906, Flutter UI thread instructs rendering thread F to draw a new frame image and store it in the texture buffer queue. For example, the flutter UI thread can instruct rendering thread F to produce a new buffer through drawing instruction 93. Correspondingly, rendering thread F can apply for a new buffer (such as buffer1) in TextureBufferQueue and render the N+1th frame image on buffer1. After the display content P1 corresponding to the N+1th frame image is placed in buffer1, buffer1 is placed in TextureBufferQueue.
[0249] The operations of S905-S906 can refer to the following. Figure 4 S405-S406 shown in the figure are not described in detail here.
[0250] It is understandable that, through the above S903-S906, the electronic device can complete the consumption of buffer0 and the production of buffer1 in TextureBufferQueue. The order of the two operations of consuming buffer0 and producing buffer1 is not limited.
[0251] It should be noted that in the existing solution, the number of buffers in TextureBufferQueue is limited to not more than 1. Therefore, if buffer1 is produced before buffer0 is consumed, buffer1 will overwrite buffer0 in TextureBufferQueue. This may cause buffer0 to fail to be displayed and consumed normally, resulting in freezes or frame skips when the Nth frame image cannot be displayed.
[0252] Correspondingly, in the present application, based on the aforementioned description of the number of buffers in the TextureBufferQueue in the memory, the number of buffers can be 1 or more. For example, the number of buffers can be 2. At the same time, the present application also provides a consumption mechanism when the number of buffers is greater than 1, that is, taking out the buffers from the head in sequence for display consumption. As a result, even if buffer1 is produced before buffer0 is consumed, buffer1 and buffer0 can be stored in the TextureBufferQueue at the same time. Among them, the earlier produced buffer0 can be located at the head of the TextureBufferQueue, and correspondingly, buffer1 can be located after buffer0. In this way, when consuming buffers, buffer0 can be taken out first for display consumption. That is, it can support the flexible sequential implementation of the above-mentioned two operations of buffer0 consumption and buffer1 production while avoiding the inability to display the Nth frame image.
[0253] In this way, due to the flexible configuration of the number of buffers in TextureBufferQueue, the buffer consumption logic of the native framework can be relatively isolated from the buffer production logic of the Flutter framework, and the problem of frame image loss due to different completion orders will not occur.
[0254] In different implementations of the present application, the number of buffers in TextureBufferQueue is configured to be 1 or 2 or other values, which can be flexibly determined by the electronic device according to the current rendering situation. For example, the electronic device can determine that the consumption of buffers is earlier than the production of new buffers in the first few Vysnc cycles of the current application. In this way, the electronic device can configure the number of buffers in TextureBufferQueue to 1, thereby saving the storage space overhead when multiple buffers are configured. For another example, the electronic device may have a situation where the buffer consumption is later than the buffer production in the first few Vysnc cycles of the current application. In this way, the electronic device can configure the number of buffers in TextureBufferQueue to 2 or a larger value, thereby ensuring that there will be no loss of frame images. For another example, the electronic device can configure at least 2 buffers for TextureBufferQueue by default. In this way, after the production of the new buffer is completed in the current Vysnc cycle, the existing unconsumed buffer can be stored in the TextureBufferQueue without overwriting it. In the present application, the Vysnc cycle can be the time period between two adjacent Vsync signals. For example, the Vsync signal V0 and the Vsync signal V1 may correspond to a Vysnc cycle (such as called Vsync cycle 0-1, or called the first Vsync cycle). For another example, the Vsync signal V1 and the Vsync signal V2 may correspond to a Vysnc cycle (such as called Vsync cycle 1-2), etc.
[0255] In the following example, it is taken that two buffers are configured in TextureBufferQueue. The two buffers may correspond to the first buffer and the second buffer respectively. The first buffer may be located before the second buffer in TextureBufferQueue. For example, the first buffer may be located at the head of TextureBufferQueue, and the second buffer may be located after the first buffer in TextureBufferQueue (such as at the tail).
[0256] Take Vsync cycle 0-1 as an example. Before the buffer is consumed, buffer0 can be stored in the first buffer, and the second buffer is empty. Then, in the Vsync cycle 0-1, the native framework can obtain buffer0 in the first buffer for display consumption. After display consumption, the first buffer and the second buffer in TextureBufferQueue are both empty. Correspondingly, in the Vsync cycle 0-1, after completing the production of the new buffer (buffer1), the flutter framework can put the buffer1 into the first buffer and send a postTraversal callback to the native framework to facilitate the consumption of buffer1 in the first buffer in the next Vysnc cycle.
[0257] It is understandable that in Fig. 9 In the example, the consumption of buffer0 is earlier than the production of buffer1. Then after the consumption of buffer0, the first buffer and the second buffer are both empty. After the production of buffer1, the first buffer corresponds to buffer1, in which the display content P1 can be stored. The second buffer is still empty.
[0258] In some other implementations, the production of buffer1 in the Vsync cycle 0-1 may be earlier than the consumption of buffer0. Then, before the production of buffer1, the first buffer may store buffer0, and the second buffer may be empty. After the production of buffer1, buffer0 may be stored in the first buffer, and buffer1 may be stored in the second buffer. In this way, in the Vsync cycle 0-1, when consuming buffers, the native framework may obtain buffer0 from the first buffer in the header for consumption. After completing the consumption of buffer0, buffer1 may enter the first buffer, and the corresponding second buffer is released as empty.
[0259] In such Fig. 9 In the example, after completing the production of buffer1, the flutter framework can make corresponding callbacks to the native framework to facilitate the subsequent smooth production and consumption.
[0260] For example, as shown in S907, the flutter UI thread may send a postTraversal callback CB1 and a postAnimation callback CB2 to the main drawing thread.
[0261] In some embodiments, the main drawing thread may store the received callback information in a callback queue. For example, the postTraversal callback CB1 and the postAnimation callback CB2 are stored in the callback queue. The postTraversal callback CB1 and the postAnimation callback CB2 may be respectively marked with flutter, thereby indicating that the two callbacks are from the flutter framework. As an example, the main drawing thread may identify the types of the postTraversal callback CB1 and the postAnimation callback CB2 as flutter types.
[0262] In this way, before executing DoFrame, the main drawing thread can check the callback queue and execute the Traversal method to consume the buffer when the callback queue includes the flutter type postTraversal callback. In addition, before executing DoFrame, the main drawing thread can check the callback queue and execute the Animation method to wake up the flutter framework for buffer production when the callback queue includes the flutter type postAnimation callback.
[0263] It is understandable that after executing the Traversal method, the main drawing thread can also delete the postTraversal callback from the callback queue, so that the same postTraversal callback will not trigger multiple Traversal methods for buffer consumption. Similarly, after executing the Animation method, the main drawing thread can also delete the postAnimation callback from the callback queue, so that the same postAnimation callback will not trigger multiple Animation methods to wake up the Flutter framework for buffer production.
[0264] Thus, through Fig. 9 As shown in S901-S907, the electronic device can complete the display consumption of the Nth frame image and the production of the N+1th frame image in the Vsync cycle 0-1.
[0265] like Fig. 9As shown, in the Vsync cycle 0-1, the native framework can also initiate the production of more new buffers. Combined with the description of S905-S906, the flutter framework can execute the production of buffer1 after being awakened by the native framework. Then, thereafter, in the Vsync cycle 0-1, the native framework can also wake up the flutter framework to execute the subsequent production of buffer2. The buffer2 can be a buffer for storing display content P2, and the display content P2 corresponds to the N+2th frame image.
[0266] In this way, by executing the production of buffer2 in the Vsync cycle 0-1, even if the production of buffer2 takes a long time (such as the runnable process and / or the running process of buffer2 takes a long time), buffer2 can also be produced and placed in the TextureBufferQueue before it needs to be sent to the display for consumption later.
[0267] As an example, Fig. 9 As shown, after S907, the method further includes:
[0268] S908: The main drawing thread determines that there is a need to wake up.
[0269] Exemplarily, the main drawing thread may check the callback queue, and when a flutter-type postAnimation callback exists in the callback queue, determine that there is a wake-up requirement.
[0270] It can be understood that, combined with the above description of the postAnimation callback, the flutter framework can register the postAnimation callback with the native framework when determining that subsequent image rendering is required.
[0271] For example, in S907, the flutter framework may register the postAnimation callback CB2 with the native framework when determining that a subsequent image (such as an N+2th frame image corresponding to a new buffer2 exists after the generated buffer1) needs to be rendered.
[0272] Correspondingly, the postAnimation callback CB2 may be placed in a callback queue, and its type may be a flutter type.
[0273] Then in S908, the main drawing thread can check the callback queue after completing the production and enqueueing of buffer1 (i.e. entering TextureBufferQueue). The main drawing thread can determine that the callback queue includes the flutter type postAnimation callback CB2, thereby determining that there is a subsequent new frame image drawing requirement. The drawing of the new image can be implemented based on waking up the flutter framework. Therefore, when it is determined that the callback queue includes the flutter type postAnimation callback CB2, the main drawing thread can determine that there is a wake-up requirement.
[0274] S909. The main drawing thread sends a wake-up instruction 95 to the flutter UI thread.
[0275] The wake-up indication 95 can be used to wake up the flutter framework again to produce a new buffer within the Vsync period 0-1.
[0276] It can be understood that, through the execution of S908-S909, the main drawing thread can send the wake-up instruction 95 to the flutter UI thread in advance, so as to advance the production of buffer2 corresponding to the wake-up instruction 95 to the Vsync cycle 0-1.
[0277] As a specific implementation, Figure 10-12 A specific execution logic of S908-S909 is provided. The solution can be applied to when the Vsync signal V0 arrives, by setting a delay task, so that S908-S909 is executed after S901-S907.
[0278] like Fig.10 As shown, after the Vsync signal V0 arrives, the main drawing thread can start to execute the corresponding operation. For example, trigger the execution of run. Then, the main drawing thread can execute the DoFrame operation D1. This process can correspond to the implementation in the aforementioned S901-S907. Fig.10 In the example, the main drawing thread can also determine whether the early FlutterAnim is effective.
[0279] The following example illustrates how to determine whether the early FlutterAnim is effective.
[0280] In this example, the condition for determining whether the early FlutterAnim is effective may also include at least one of the following: the current thread is the main thread; a hand-off action of a sliding operation is received; the current display is not in a multi-window state; the current application uses a single textureview.
[0281] Take the condition for judging whether the early FlutterAnim is effective as an example, where the current thread is the main thread.
[0282] When the current thread is the main thread, it is determined that the early FlutterAnim takes effect. On the contrary, when the current thread is not the main thread, it is determined that the early FlutterAnim does not take effect.
[0283] Take the condition for determining whether the early FlutterAnim is effective including receiving the hand-off of the sliding operation as an example.
[0284] If there is no hand-off action for the sliding operation and the sliding operation is currently in progress, it is determined that the early FlutterAnim takes effect. On the contrary, if the hand-off action has been received, it is determined that the early FlutterAnim does not take effect.
[0285] Take the condition for judging whether the early FlutterAnim is effective including the current non-multi-window display as an example.
[0286] Multi-window display corresponds to two or more Draw instructions appearing at the same time to draw the window, which may occur when the user inputs window switching. In this example, when the current display is not multi-window, it is determined that FlutterAnim takes effect in advance.
[0287] Take the condition for judging whether the early FlutterAnim is effective, including that the current application uses a single textureview, as an example.
[0288] A single textureview corresponds to the rendering process performed using the flutter framework. Then, if the class of the View currently used is fluttertextureView, it is determined that FlutterAnim takes effect in advance.
[0289] The above example provides multiple solutions to determine whether the advance FlutterAnim is effective. In other embodiments, the multiple solutions can use two or more of them at the same time. If each condition determines that the advance FlutterAnim is effective, it is considered that the current scene can produce buffers in advance, triggering the generation of a delay task.
[0290] In the present application, the delay task may correspond to a first duration. In some embodiments, the first duration may be 2 / 3 of the Vsync cycle 0-1. That is, after receiving the Vsync signal V0, the main drawing thread may configure a delay task. The duration of the delay task may be 2 / 3 of the Vsync cycle 0-1. In this way, after the delay task ends, the main drawing thread may start to execute related actions.
[0291] refer to Fig.11 When the delayed task corresponding to the first duration ends, the main drawing thread can detect whether the current callback queue includes the PostAnimation callback. Before the first duration ends, the electronic device can complete the production of buffer1 and execute S907, so that the postTraversal callback CB1 can be included in the callback queue. In this way, it can be determined that the animation method needs to be executed in advance.
[0292] like Fig.11 As shown, after the first time length after the Vsync signal V0, if it is determined that there is a PostAnimation callback, the main drawing thread can obtain the timestamp corresponding to the execution of the animation method in the next frame. For example, the timestamp can be as follows Figure 4 The timestamp carried in the wake-up indication 25 in the UE. Take the timestamp as the moment TB as an example.
[0293] In this way, according to the time TB corresponding to the determined timestamp, the main drawing thread can execute doCallbacks.
[0294] For example, the main drawing thread can execute doCallbacks at the first moment. The first moment can be moment TB-delta. Delta can be the preset second duration. Since delta is advanced based on moment TB, the animation method in doCallbacks is executed in advance by delta duration. This also enables the processing of buffer2 to be advanced by the second duration.
[0295] As an example, Fig.12 As shown, the main drawing thread can configure the timestamp of the animation method to be executed currently based on the time TB, so that the animation method can be executed before the preset second duration.
[0296] Specifically corresponding to Fig. 9 In the example, after the second time period is advanced, the flutter framework may trigger execution of the following S910.
[0297] S910. The Flutter UI thread instructs the rendering thread F to draw a new frame image and store it in a texture buffer queue.
[0298] Combined with the description in S909, the flutter UI thread may receive a wake-up instruction 95 carrying a timestamp of (TA-delta). Therefore, the flutter UI thread may send a drawing instruction 96 to the rendering thread F after the (TA-delta) duration.
[0299] The rendering thread F can obtain a new buffer 2 from the TextureBufferQueue according to the rendering information corresponding to the N+2 frame image carried in the drawing instruction 96, and draw the display content P2 corresponding to the N+2 frame image on buffer 2. After the drawing is completed, the flutter framework can put the buffer 2 including the display content P2 into the TextureBufferQueue.
[0300] Combined with the above description on setting the first buffer and the second buffer in TextureBufferQueue, in this example, the first buffer may store buffer1 that has been produced but not consumed, and the produced buffer2 may be stored in the second buffer.
[0301] In such Fig. 9 In the example, the production of buffer2 can be completed before the arrival of Vsync signal V1. In other embodiments, the production of buffer2 may also be completed after the arrival of Vsync signal V1. It can be understood that due to the early production of buffer2, as long as buffer2 enters TextureBufferQueue before VsyncV2 arrives, the normal display consumption of buffer2 can be achieved.
[0302] S911, the flutter UI thread sends a callback to the main thread.
[0303] Similar to the description in S907 above, after the production of buffer2 is completed, the flutter framework can register a callback with the main drawing thread so that buffer2 can be consumed later and a new buffer can be produced.
[0304] For example, the postTraversal callback CB3 can be used to register the consumption of buffer2. For another example, the postAnimation callback CB4 can be used to register the production of buffer3.
[0305] It should be noted that in this Fig. 9In the example, buffer2 and buffer1 can complete the production in the same Vsync cycle 0-1. Correspondingly, the present application provides a TextureBufferQueue configured with two buffers for storing buffer1 and buffer2 respectively, so as to avoid buffer2 covering buffer1 and causing frame loss.
[0306] In addition, the callback registration of S911 and S907 may also be performed in the same Vsync cycle 0-1.
[0307] In the present application, the flutter framework can encapsulate the callback before executing S911 to avoid two postTraversal callbacks or postAnimation callbacks in the same Vsync cycle.
[0308] For example, the flutter framework can encapsulate the postTraversal callback CB3 and the postAnimation callback CB4 into other callbacks different from the flutter type, so that the main drawing thread will not directly consume buffer2 according to the postTraversal callback CB3 or produce buffer3 according to the postAnimation callback CB4 after the Vsync signal V1 arrives.
[0309] In some implementations, the encapsulated postTraversal callback CB3 and postAnimation callback CB4 may have a delay attribute. The delay attribute corresponds to the duration of a Vsync cycle. After the delay attribute ends, the type of the postTraversal callback CB3 and postAnimation callback CB4 can be automatically changed to the flutter type.
[0310] In this way, in the Vsync cycle 0-1, the main drawing thread can receive the flutter type postTraversal callback CB1. As a result, in the next cycle (such as Vsync cycle 1-2), the main drawing thread can send and consume buffer1 according to the postTraversal callback CB1.
[0311] In addition, during the Vsync cycle 0-1, the main drawing thread can also receive the encapsulated postTraversal callback CB3 and postAnimation callback CB4. Since the types of the postTraversal callback CB3 and postAnimation callback CB4 are not flutter types, the main drawing thread will not trigger the corresponding Traversal method or Animation method after the Vsync signal V1 arrives.
[0312] After receiving the encapsulated postTraversal callback CB3 and postAnimation callback CB4 for a Vsync cycle, the types of postTraversal callback CB3 and postAnimation callback CB4 can be changed to flutter type. In this way, it is equivalent to the main drawing thread receiving flutter type postTraversal callback CB3 and postAnimation callback CB4 within Vsync cycle 1-2. Therefore, when the next Vsync signal V2 arrives, the main drawing thread can consume buffer2 based on the postTraversal callback CB3, and wake up the flutter framework based on the postAnimation callback CB4 to produce buffer3.
[0313] It is understandable that in Fig. 9 In the above example, the flutter framework encapsulates postTraversal callback CB3 and postAnimation callback CB4 and then sends them.
[0314] In other embodiments, the encapsulated operation may also be executed by the main drawing thread.
[0315] Exemplarily, the flutter framework can also directly register postTraversal callback CB3 and postAnimation callback CB4 to the main drawing thread according to the existing scheme. Correspondingly, the main drawing thread can check the current callback queue after receiving postTraversal callback CB3 and postAnimation callback CB4. When it is determined that there is an unresponsive postTraversal callback CB1, the main drawing thread can actively encapsulate the postTraversal callback CB3 and postAnimation callback CB4, so that after a Vsync cycle, the type of the postTraversal callback CB3 and postAnimation callback CB4 is automatically changed to the flutter type, so as to facilitate the corresponding response after the next Vsync signal V2 arrives.
[0316] In this way, through S908-S911 in the Vsync cycle 0-1, the production of buffer2 is advanced, thereby avoiding the problem of failure to send and display normally in Vsync cycle 2-3 caused by failure to complete the production of buffer2 in time in Vsync cycle 1-2.
[0317] It is understandable that after completing the above operations in the Vsync cycle 0-1, the electronic device can consume the N+1th frame image according to the arrival of the next Vsync signal V1. Exemplarily, the method further includes the following S912-S914.
[0318] S912: The main drawing thread executes the DoFrame operation D2.
[0319] Among them, the main drawing thread can trigger the execution of DoFrameD2 after the Vsync signal V1 arrives.
[0320] Similar to the description of the aforementioned DoFrame operation D1, the main drawing thread may determine to include the postTraversal callback CB1 in the callback queue. Correspondingly, the DoFrame operation D2 may include a response to the postTraversal callback CB1.
[0321] That is, the buffer in the first buffer in the current TextureBufferQueue is consumed. The first buffer may store display content P1 corresponding to buffer1, corresponding to the N+1th frame image.
[0322] S913: The main drawing thread sends a rendering instruction 94 to the rendering thread M. The rendering instruction 94 is used to instruct the rendering thread M to consume the buffer in the TextureBufferQueue.
[0323] S914. Rendering thread M obtains a buffer from the texture buffer queue for consumption.
[0324] Specifically, the rendering thread M may obtain the content in the buffer (such as the first buffer) stored in the head from the TextureBufferQueue for consumption. The first buffer may store display content P1.
[0325] After acquiring the display content P1, the rendering thread M may transmit the display content P1 to the surfaceflinger, so that the surfaceflinger displays the N+1th frame on the display screen according to the display content P1.
[0326] It is understandable that after the Vsync signal V1 arrives, according to the existing solution (such as Figure 4 As shown), the main drawing thread can wake up the flutter framework to produce buffer2.
[0327] In this example, the production of buffer2 and its entry into TextureBufferQueue have been completed in Vsync cycle 0-1 through S908-S910. Therefore, there is no need to perform the production process of buffer2 again in Vsync cycle 1-2.
[0328] Thus, based on the above Figure 9-12 In the solution provided in the embodiment of the present application, the main drawing thread can be Figure 10-12 The judgment mechanism is to advance the animation method triggered by the arrival of the next Vsync signal to the current Vsync cycle when the wake-up demand is confirmed, thereby realizing the rapid production of subsequent buffers. This avoids the failure of subsequent display consumption due to the long buffer production time, thereby solving the corresponding display freeze problem.
[0329] As an example, Fig.13 Provides a timing example of buffer consumption and production when executing the solution of this application. Fig.13 The example is Figure 6 By using the problem scenario background shown in Fig. 9 The solution shown solves Figure 6 Buffer2 cannot complete production in time, resulting in the problem that the N+1 frame image cannot be displayed normally between T2 and T3.
[0330] For example, Fig.13 As shown in the figure, after the Vsync signal V0 arrives, the flutter framework can produce buffer1 and put it into TextureBufferQueue after waiting for resources and rendering according to the wake-up instruction. Fig.13 In the example, the first buffer and the second buffer are still configured in TextureBufferQueue. The buffer1 can be put into the first buffer of TextureBufferQueue. Fig. 9 The solution shown in the figure is that after the Vsync signal V0 arrives, the electronic device can determine that the DoAnimation process needs to be triggered in advance. In this way, before the Vsync signal V1 arrives, the electronic device can wake up the flutter framework again to perform drawing processing on the new buffer (such as buffer2). At 2 / 3 of the Vsync cycle 0-1, the electronic device can start executing the DoAnimation processing corresponding to buffer2.
[0331] This allows the DoAnimation processing corresponding to buffer2 to be advanced. Fig.13 As shown, by drawing buffer2 in advance, buffer2 can be finished and enter TextureBufferQueue before Vsync signal V2 arrives. As a result, after Vsync signal V2 arrives, the electronic device can smoothly consume buffer and switch to display the N+2 frame image at time T2. This can avoid the following Figure 6 As shown in the figure, the buffer2 drawing process is too long to be produced in time, which affects the normal display. This also avoids the display freeze problem caused by this.
[0332] In addition, combined Fig. 9 As described in S911, in order to avoid two postAnimation callbacks in the same Vsync cycle, the main drawing thread can actively encapsulate the received multiple callback instructions.
[0333] In the embodiment of the present application, during the specific execution of S911, the main drawing thread can perform a corresponding callback encapsulation operation according to the preset first encapsulation interval and the second encapsulation interval, and temporarily encapsulate the flutter type into other types, thereby avoiding the display problem caused by two or more postAnimation callbacks in the same Vsync cycle.
[0334] As an example, refer to Fig.14 , combined with Fig.13 The encapsulation mechanism provided by this application is described with reference to the timing logic.
[0335] Combined with the aforementioned native framework (such as the main drawing thread therein), DoFrame can be executed after a new Vsync signal is received.
[0336] Exemplarily, the DoFrame operation may include traversing the callbacks received in the callback queue.
[0337] Take Vsync cycle 0-1 as an example. Before the Vsync signal V0 arrives, the postAnimation callback and the postTraversal callback have entered the callback queue.
[0338] So, if Fig.14 As shown, after the Vsync signal V0 arrives, the main drawing thread of the native frame can execute DoFrame. When traversing to the postAnimation callback, the main drawing thread can execute DoAnimation, corresponding to executing the animation method configured in the postAnimation callback, thereby generating and sending a wake-up instruction 92.
[0339] Similarly, when traversing to the postTraversal callback, the main drawing thread can execute DoTraversal, and correspondingly execute the Traversal method configured in the postTraversal callback, thereby generating and issuing a rendering instruction 91.
[0340] Based on the existing solution, the wake-up instruction 95 for waking up the flutter framework to draw the next frame (such as the N+2 frame) should be issued in the Vsync cycle 1-2. In this way, a freeze may occur due to the long rendering time of the N+2 frame image.
[0341] Based on Fig. 9 In the implementation of the solution shown, the wake-up instruction 95 can be sent in advance in the Vysnc cycle 0-1, so as to achieve early processing of the N+2th frame image.
[0342] In the specific implementation process, Fig.14 As shown, in the Vsync cycle 0-1, after the main drawing thread performs the DoFrame operation in the existing solution, DoAnimation can be executed again before the Vsync signal V1 arrives, and a corresponding wake-up instruction 95 is generated and issued, thereby achieving the following Fig. 9 Execution of S909.
[0343] In this example, in order to avoid the main drawing thread receiving multiple postAnimation callbacks in the same Vsync cycle, the main drawing thread can Fig.14 The callbacks received in the first encapsulation interval and the second encapsulation interval are encapsulated.
[0344] The encapsulation processing of the callback may include: packaging the encapsulated callback, and the class of the callback instruction after packaging may be different from the callback instruction before packaging. For example, taking the encapsulated callback as a flutter type as an example. Then, after the encapsulation processing, the type of the encapsulated callback may be different from the flutter type.
[0345] In some implementations, the encapsulation process may also include: configuring a delay mechanism for the callback instruction after the encapsulation process. The delay mechanism is used to unpack the callback instruction after the encapsulation process after a certain period of time. The type of the callback instruction after the unpacking process is the same as the callback instruction before the encapsulation process. For example, the callback instruction to be encapsulated is of the flutter type before the encapsulation process. Then, after the encapsulation process, the type of the callback instruction to be encapsulated is different from the flutter type. After the unpacking process, the type of the callback instruction to be encapsulated is of the flutter type. In some implementations, the duration corresponding to the delay mechanism may be the duration of one Sync cycle. As a result, after one Sync cycle, the main drawing thread can respond normally to the callback instruction.
[0346] As a specific implementation, Fig.14 In the example, the start time of the first encapsulation interval may correspond to the time when the main drawing thread starts to execute DoAnimation for the second time in any Vsync cycle (such as Vsync cycle 0-1). Correspondingly, the end time of the first encapsulation interval may correspond to the end time of the second execution of DoAnimation by the main drawing thread in any Vsync cycle (such as Vsync cycle 0-1), that is, the time when the main drawing thread generates and issues a wake-up indication (such as wake-up indication 95). Accordingly, the main drawing thread can encapsulate all types of postanimation callbacks in the first encapsulation interval.
[0347] The start time of the second encapsulation interval may be the same as the end time of the first encapsulation interval. For example, the start time of the second encapsulation interval may correspond to the end time of the second execution of DoAnimation by the main drawing thread in any Vsync cycle (such as Vsync cycle 0-1), that is, the time when the main drawing thread generates and issues a wake-up indication (such as wake-up indication 95). Accordingly, the main drawing thread can encapsulate the flutter type callback (that is, the postanimation callback sent by the flutter framework) in the second encapsulation interval.
[0348] Thus, through Fig.14 The solution shown can achieve the effect of accurately encapsulating the postAnimation callback. It is understandable that in some scenarios, the main rendering thread can also encapsulate the postTraversal callback based on a similar solution, and the specific implementation is not repeated here.
[0349] It should be noted that, combined with Fig. 9 as well as Fig.14 In some embodiments of the present application, TextureBufferQueue may include two or more buffers. For example, in the above example, TextureBufferQueue includes a first buffer and a second buffer.
[0350] like Fig.15 As shown, reference Figure 4 Description of existing schemes.
[0351] In the current implementation, the length of TextureBufferQueue can be 1 buffer. That is, the data of 1 buffer can be stored in TextureBufferQueue at the same time. For example, an electronic device is configured with 3 buffers for data caching. The three buffers can be: Buffer_B1, Buffer_B2, Buffer_B3. Then, the data of one of the three buffers can be stored in TextureBufferQueue at the same time. When the data of other buffers arrives, the buffer data in TextureBufferQueue is discarded, and the data of the new buffer is stored in TextureBufferQueue. For example, Fig.15As shown, the current TextureBufferQueue can store the data of Buffer_B3. Then, when the data of Buffer_B1 arrives, the TextureBufferQueue can discard the data of Buffer_B3 and store the data of Buffer_B1.
[0352] Correspondingly, in the TextureBufferQueue configuration provided in the embodiment of the present application, the length of the TextureBufferQueue can be 2 buffers or larger. For example, the length of the TextureBufferQueue can be 2 buffers. In this way, the data of two buffers can be stored in the TextureBufferQueue at the same time.
[0353] For example, Fig.15 As shown, the data of Buffer_B2 can be stored in the first buffer, and the data of Buffer_B3 can be stored in the second buffer. In this way, when the data of Buffer_B3 is not consumed, if new data of Buffer_B2 arrives, the data in Buffer_B3 will not be discarded, so that the data of Buffer_B3 will not be lost when the buffer is consumed next time, and normal display consumption is achieved.
[0354] In this application, combined with the description of S910, the above problem is solved by configuring the first buffer and the second buffer, so that the newly generated buffer will not overwrite the existing buffer. This avoids frame loss due to buffer overwriting. In this way, after the Vsync signal V1 arrives, the native framework can consume buffer1 in the first buffer and correctly display the N+1th frame image.
[0355] In the examples of this application, refer to Fig.16 Based on the setting method of multiple buffers in the TextureBufferQueue, a command control mechanism corresponding to multiple buffers is also provided, so that the consumption and production of each buffer can be successfully completed when there are multiple buffers in the TextureBufferQueue.
[0356] like Fig.16As shown, the buffer management process may involve multiple software modules. The multiple software modules or units (such as buffer queue producer (BufferQueueProducer.cpp), Android graphics surface texture (Android_graphics_SurfaceTexture.cpp), surface texture (surfaceTexture.java), structure view (TextureView.java), general view or view (View.java), etc.) can be implemented under the control of the native framework. Fig.16 Therefore, from another logical perspective, Fig.16 The buffer control scheme shown can be as follows: Figure 8 Specifically, in some implementations, the main rendering module shown in FIG. Fig.16 The solution shown can be implemented by a main drawing unit.
[0357] In this example, BufferQueueProducer.cpp is the server-side implementation of IGraphicBufferProducer, which is responsible for buffer application. Correspondingly, BufferQueueConsumer.cpp is the server-side implementation of IGraphicBufferProducer, which is a buffer consumer. In this application, when new buffer production is required (such as when the flutter framework receives a wake-up instruction), a new buffer can be applied for through the BufferQueueProducer.cpp so that new content can be drawn and rendered on the newly applied buffer.
[0358] When the data frame is valid, that is, when onFrameAvailable is called, you can call GLSurfaceView.requestRender() to display the request for rendering, which triggers the Renderer's onDrawFrame(). Fig.16 As shown, Android_graphics_SurfaceTexture.cpp can also send a native callback event (postEventFramNative) to surfaceTexture.java, so that surfaceTexture.java can send an empty message (such as sendEmptyMessage) to TextureView.java, triggering TextureView.java to send an invalid message (invalidate) to View.java to consume the current buffer.
[0359] It is understandable that after SurfaceTexture is initialized, a JNISurfaceTextureContext listener is set to GLConsumer, which calls back to the Java layer SurfaceTexture.postEventFromNative method, and further calls back to the OnFrameAvailableListener listener registered in SurfaceTexture to notify the business layer that a new GraphicBuffer has been queued.
[0360] In the present application, since there are multiple buffer configurations, surfaceTexture.java can reasonably configure the consumption of each buffer by triggering the operation of obtaining the current number of buffers in S1601 and confirming the current number of valid buffers.
[0361] like Fig.16 As shown in S1601 in the example, surfaceTexture.java executes getBufferCount, i.e., obtains the number of currently valid (consumable) buffers. For example, when there are buffers that have not been consumed and new buffers are produced, the number of valid buffers is 2. Correspondingly, after a buffer consumption is executed, the number of valid buffers is reduced by 1. And so on.
[0362] Next, surfaceTexture.java sends a nativeGetBufferCount indication to Android_graphics_SurfaceTexture.cpp. Thus, the number of valid buffers is shared with the surfacetexture in the native framework. In some implementations of the present application, when it is confirmed that the number of valid buffers is greater than or equal to 1 (that is, when there are available buffers), Android_graphics_SurfaceTexture.cpp can continue to perform subsequent operations.
[0363] In this example, Android_graphics_SurfaceTexture.cpp can instruct BufferQueueProducer.cpp to obtain the current buffer size, that is, execute getQueueBufferSize. The size of the buffer can correspond to the size of the texture stored in the buffer. It can be understood that when the texture size is consistent with the interface size of the currently running application, the corresponding buffer can be the buffer displayed by the current application (such as a shopping application).
[0364] In such Fig.16 In the example, the invalidate corresponding to 1603a can realize the consumption of the first buffer. Through S1601, the main drawing unit (corresponding to the main drawing thread) can determine that there are other buffers waiting to be consumed in the TextureBufferQueue.
[0365] BufferQueueProducer.cpp can send the mCore->mQueue.size() message to surfaceTexture.java through Android_graphics_SurfaceTexture.cpp to transfer the buffer size, and then instruct surfaceTexture.java to trigger the consumption of the unconsumed buffer content.
[0366] It should be noted that, in this example, combined with the description of S910 above, the display content P1 in the first buffer is as follows: Fig.16 After being consumed in 1603a, the display content P2 in the second buffer can be progressively transferred to the first buffer. Thus, when a new buffer (such as display content P3) is subsequently produced, if the display content P2 has not been consumed, the display content P3 can enter the second buffer and be consumed after the display content P2.
[0367] Continue to refer Fig.16 As shown in S1602, surfaceTexture.java can delay sending sendEmptyMessage to TextureView.java, that is, sendEmptyMessageDelayed. The delay time corresponding to the delayed sending can be the length of a Vysnc cycle. As a result, the sendEmptyMessage is sent in the next Vsync cycle, triggering 1603b to execute consumption of the display content P2.
[0368] It is understandable that through Fig.16 The internal implementation shown can be implemented by the solution of determining the number of valid buffers in S1601 and the solution of delaying the sending of the consumption buffer indication in S1602, so that TextureView.java will not receive two consumption buffer indications in one Vsync cycle. This ensures that when there are multiple buffers in TextureBufferQueue, the buffer consumption is carried out normally.
[0369] Thus, through the above Figure 9-Figure 16 In this example, the native framework wakes up the flutter framework in advance, so that the buffers produced in the next Vsync cycle can start production in the current Vsync cycle. In this way, for buffers with a long production time, production and queueing can be completed before the corresponding Vsync signal for display arrives. This avoids display failures caused by the inability to produce buffers in time, and can also solve the corresponding display freeze problem.
[0370] Combination Fig. 9 as well as Fig.16 As explained in the above example, a multi-buffer configuration scheme for TextureBufferQueue is still used. This ensures that the buffer produced in advance will not overwrite the previously produced but not consumed buffer. In the above example, in the multi-buffer configuration, the new buffer may be caused by the corresponding wake-up indication being initiated in advance. It is understandable that in other scenarios, even if the buffer is not produced in advance, two buffers may overlap.
[0371] Thus, if we continue to use Figure 4 or Fig.15 In the existing single buffer solution shown in , a new buffer may overwrite an existing unconsumed buffer, thereby causing frame loss and frame skipping problems.
[0372] The following examples illustrate the implementation of the multi-buffer configuration solution and the corresponding scenarios. Fig. 9 The technical solutions S908-S911 shown perform advance buffer production. The configuration of multiple buffers can solve the frame loss and frame skipping problems caused by buffer coverage loss in existing scenarios.
[0373] As an example, refer to Fig.17 , which is an example of a buffer production and consumption scenario.
[0374] The Fig.17 Figure 1 shows a specific implementation of normal buffer production and consumption. Fig.17 The execution of each step can correspond to the following Figure 4 In S401-S407 shown.
[0375] A brief description is given below. Fig.17 As shown, the program may include:
[0376] S1701. After receiving operation 16 input by the user, the main drawing thread transparently transmits the operation information to the flutter UI thread.
[0377] The execution of this step can refer to Figure 4 S401 shown. In this way, the flutter UI thread can determine whether the sliding operation corresponding to the operation 16 has ended according to the acquired operation information. If the sliding operation has not ended, the flutter UI thread can determine that subsequent operations such as waking up the corresponding buffer production need to be performed according to the operation information.
[0378] The Flutter UI thread can determine the subsequent production and consumption of at least the Nth frame image to the N+3th frame image according to the operation information.
[0379] In this example, similar to the above scenario, the Vsync signal V0 is used to trigger the consumption of the Nth frame image and the production of the N+1th frame image. Similarly, the Vsync signal V1 is used to trigger the consumption of the N+1th frame image and the production of the N+2th frame image. The Vsync signal V2 is used to trigger the consumption of the N+2th frame image and the production of the N+3th frame image.
[0380] The Nth frame image corresponds to the display content P0, the N+1th frame image corresponds to the display content P1, the N+2th frame image corresponds to the display content P2, and the N+3th frame image corresponds to the display content P3.
[0381] In this Fig.17 In the example of FIG. 1 , the electronic device can realize the production of a new buffer according to S1702a to S1702e. The electronic device can also realize the consumption of an existing buffer according to S1703a to S1703c.
[0382] S1702a. After the Vsync signal V0 arrives, the main drawing thread sends a wake-up instruction 22 to the flutter framework.
[0383] For the execution of this step, please refer to Figure 4 S405 shown.
[0384] The wake-up instruction 22 may be generated and sent when the main drawing thread receives the postAnimation callback before the Vsync signal V0 and executes the Animation method in the DoFrame operation after receiving the Vsync signal V0. For specific implementation, please refer to the above description.
[0385] In some implementations, the wake-up indication 22 may carry a timestamp and rendering information. The timestamp may be used to indicate that the flutter framework starts producing a new buffer at the time corresponding to the timestamp. The rendering information may be used by the flutter framework to draw a new buffer map accordingly.
[0386] S1702b, the Flutter UI thread sends a drawing instruction 23 to the rendering thread F.
[0387] For the execution of this step, please refer to Figure 4 The relevant description of S406 is shown. The drawing instruction 23 can be used to draw and queue a new buffer according to the rendering information.
[0388] In some embodiments, the flutter UI thread may send a drawing instruction 23 to the rendering thread F when the time corresponding to the timestamp corresponding to the wake-up instruction 22 arrives.
[0389] In other embodiments, the flutter UI thread may directly send a drawing instruction 23 to the rendering thread F after receiving the wake-up instruction 22. In this implementation, the rendering thread F may start to execute the production of a new buffer when the time stamp corresponding to the wake-up instruction 22 arrives according to the time information carried in the drawing instruction 23.
[0390] S1702c, rendering thread F draws and displays content P1.
[0391] The drawing process may include Figure 5 or Figure 6 The process of waiting for resources (runnable) and rendering (running) is shown in FIG. Fig.17 In the example of FIG. 1 , the rendering thread F may complete drawing of the display content P1 after a period of time.
[0392] S1702d, rendering thread F sends the display content P1 to the texture buffer queue (TextureBufferQueue).
[0393] S1702e. The display content P1 is stored in Buffer1 in the texture buffer queue.
[0394] In this way, the production corresponding to buffer 1 can be completed. The buffer 1 may include display content P1 corresponding to the N+1th frame image.
[0395] In addition, in parallel with the process from S1702a to S1702e, the electronic device may execute S1703a-S1703b to consume the existing buffer (eg, buffer0).
[0396] S1703a. The main drawing thread sends a rendering instruction 21 to the rendering thread M.
[0397] The execution of S1703a-S1703c can refer to Figure 4 The relevant instructions in S403 are shown.
[0398] The rendering instruction 21 may include a Draw instruction for consuming the existing buffer.
[0399] S1703b, rendering thread M obtains the buffer from the texture buffer queue for consumption.
[0400] Exemplarily, the rendering thread M may obtain an existing buffer from the current TextureBufferQueue. The existing buffer may be buffer0 storing the display content P0.
[0401] The rendering thread M may also transmit the display content P0 in the buffer0 to SurfaceFlinger, so that SurfaceFlinger can perform subsequent processing.
[0402] S1703c, SurfaceFlinger performs synthesis processing based on the display content P0 and sends it for display.
[0403] In this way, the Nth frame image can be sent and displayed for consumption through the implementation of S1703a-S1703c.
[0404] It should be noted that the execution process of S1702a-S1702e and the execution process of S1703a-S1703c are independent of each other, and there is no fixed order between the two.
[0405] S1704. The Flutter UI thread sends a postTraversal callback CB1 and a postAnimation callback CB2 to the main drawing thread.
[0406] For example, after completing the production of buffer1 in S1702a-S1702e, the Flutter UI thread can send a postTraversal callback CB1 to the main drawing thread, so that after the next Vsync signal V1 arrives, the main drawing thread can repeat a similar solution in S1703a-S1703c to produce buffer1.
[0407] The Flutter UI thread can also send a postAnimation callback CB2 to the main drawing thread when it determines that the Flutter framework needs to be awakened to produce a new buffer.
[0408] In this example, the postTraversal callback CB1 and the postAnimation callback CB2 are sent adjacently or simultaneously. In other implementations, the sending of the postTraversal callback CB1 and the postAnimation callback CB2 can be independent of each other, and the sending order and time between them are not specifically limited.
[0409] Thus, by Fig.17 S1701-S1704 shown completes the consumption of buffer0 in the Vsync cycle 0-1, clears the TextureBufferQueue, and the buffer1 to be produced thereafter can re-enter the TextureBufferQueue to wait for the next consumption.
[0410] After consuming buffer0, the Nth frame image can be displayed on the display screen of the electronic device.
[0411] Under normal circumstances, if in the next Vsync cycle 1-2, the electronic device can Fig.17 As shown, after completing the consumption of buffer 1 and the production of buffer 2, the N+1th frame image can be switched for display.
[0412] However, due to the obvious differences in the rendering time corresponding to different buffers, it is difficult to ensure that each Vsync cycle can be Fig.17 The production and consumption of one buffer are completed in the order shown. This will cause display freezes, frame drops, frame skips, and other problems.
[0413] For example, Fig.18 An example of buffer production and consumption in an abnormal scenario is shown.
[0414] The Fig.18 The scheme shown can be in Fig.17 After S1704 is shown, an example of a corresponding process of receiving a subsequent Vsync signal is shown. For example, after executing S1704, the following can be executed: Fig.18 S1705a shown. And / or, after executing S1704, the following may be executed: Fig.18 S1706a shown.
[0415] S1705a. After the Vsync signal V1 arrives, the main drawing thread sends a wake-up instruction 25 to the flutter framework.
[0416] S1705b, the Flutter UI thread sends a drawing instruction 26 to the rendering thread F.
[0417] S1705c. Rendering thread F draws and displays content P2.
[0418] S1705d. Rendering thread F sends display content P2 to TextureBufferQueue.
[0419] S1705e, TextureBufferQueue stores display content P2 in Buffer2.
[0420] S1705f, Flutter UI thread sends postTraversal callback CB4 to the main drawing thread.
[0421] In this example, the production time of the display content P2 is relatively long. For example, before starting to render, the waiting time for resources is relatively long. And / or after starting to render, the rendering process takes a relatively long time. In this way, the display content P2 can be rendered and queued only after the next Vsync signal V2 arrives.
[0422] Correspondingly, after the new buffer2 (corresponding to the display content P2) is queued, the Flutter UI thread executes S1705f, which means that the main drawing thread receives the postTraversal callback CB4 only after the Vsync signal V2 arrives.
[0423] S1706a. The main drawing thread sends a rendering instruction 24 to the rendering thread M.
[0424] S1706b, rendering thread M obtains the buffer from the texture buffer queue for consumption.
[0425] S1706c, SurfaceFlinger performs synthesis processing based on the display content P1 and sends it for display.
[0426] In the Vsync period 1-2, the electronic device can implement switching display of the N+1th frame image based on S1706a-S1706c.
[0427] In this example, since there is a rendering requirement for subsequent frame images (such as the N+3th frame image corresponding to buffer3), before the Vsync signal V2, the flutter UI thread can execute S1707 and send the postAnimation callback CB3 to the main drawing thread.
[0428] In this way, within the Vsync cycle 1-2, the electronic device can complete the consumption of the N+1 frame image (i.e., buffer1), and the corresponding TextureBufferQueue is cleared to 0. However, within the Vsync cycle 0-1, the production of the new buffer2 is not completed. Therefore, when the next Vsync signal V2 arrives, the main drawing thread does not receive the postTraversal callback CB4 for consuming the new buffer.
[0429] This will make the main drawing thread not trigger the consumption of the buffer in the corresponding Vsync cycle 2-3 after the Vsync signal V2 arrives, and will not update the currently displayed content.
[0430] Exemplarily, within the Vsync period 2-3, the electronic device may further execute S1708a-S1708f, and S1709.
[0431] S1708a. The main drawing thread sends a wake-up instruction 27 to the flutter framework.
[0432] S1708b, the Flutter UI thread sends a drawing instruction 28 to the rendering thread F.
[0433] S1708c. Rendering thread F draws and displays content P3.
[0434] S1708d. Rendering thread F sends display content P3 to TextureBufferQueue.
[0435] S1708e, TextureBufferQueue stores display content P3 in Buffer3.
[0436] S1708f, Flutter UI thread sends postTraversal callback CB5 to the main drawing thread.
[0437] S1708a-S1708f can be used to execute the production of new buffers in Vsync cycle 2-3. It can be understood that, combined with the description of S1707, in Vsync cycle 1-2, the flutter UI thread can determine that there is a need to produce new buffers in the future, and issue the postAnimation callback CB3 accordingly. Then, after the Vsync signal V2 arrives, the main drawing thread can execute the Animation method in the DoFrame operation to produce buffer3 corresponding to the N+3 frame image.
[0438] It should be noted that due to the delay in the production of the previous buffer2, no display consumption of the existing buffer2 is performed in the Vsync cycle 2-3. Therefore, when the production of buffer3 is completed, the buffer2 is still stored in the TextureBufferQueue.
[0439] Then, due to the current single buffer configuration, after buffer3 is produced and put into the queue, it will overwrite the original unconsumed buffer2. In other words, after buffer3 is completed and put into the queue, it will overwrite the unconsumed buffer2.
[0440] In the Vsync cycle 2-3, the flutter UI framework may determine that there is a subsequent demand for new buffer production, and execute S1709, that is, send a postAnimation callback CB6 to the main drawing thread.
[0441] It is understandable that, when the Vsync cycle 2-3 ends, the electronic device can still display the N+1th frame image corresponding to buffer1, and the TextureBufferQueue stores the display content P3 corresponding to the N+3th frame image.
[0442] Next, after the Vsync signal V3 arrives, the electronic device may execute the following S1710a-S1710c to consume the new buffer.
[0443] S1710a: The main drawing thread sends a rendering instruction 29 to the rendering thread M.
[0444] S1710b, rendering thread M obtains the buffer from the texture buffer queue for consumption.
[0445] S1710c, SurfaceFlinger performs synthesis processing based on the display content P3 and sends it for display.
[0446] In this way, after the Vsync signal V3 arrives, the electronic device can execute the Traversal method in DoFrame according to the postTraversal callback CB5 and the triggering of the Vsync signal V3, and read the buffer from the current TextureBufferQueue for consumption.
[0447] Combined with the above description of Vsync cycle 2-3, when the Vsync signal V3 arrives, buffer 3 may be stored in TextureBufferQueue. Thus, through S1710a-S1710c, the electronic device may switch to display the corresponding N+3 frame image on the display screen.
[0448] like Figure 17-Figure 18 In the scenario shown, the corresponding buffer production and consumption logic is as follows Fig.19 shown.
[0449] like Fig.19 As shown, after the Vsync signal V0 arrives, the electronic device can complete the consumption of the existing buffer0 at time T0 and start displaying the Nth frame image. Correspondingly, before the Vsync signal V1 arrives, the production of buffer1 is completed. Therefore, before the Vsync signal V1 arrives, the main drawing thread can receive the callback corresponding to the consumption of buffer1.
[0450] After the Vsync signal V1 arrives, the electronic device can consume buffer1 and switch to display the N+1th frame image at time T1. After the Vsync signal V1 arrives, the electronic device can also start to produce buffer2.
[0451] like Fig.18 As shown in the figure, buffer2 can complete production and be queued after the Vsync signal V3 arrives.
[0452] In this way, within the Vsync cycle 1-2, the main drawing thread will not receive a callback for buffer consumption after the Vsync signal V2 arrives.
[0453] Therefore, after the Vsync signal V2 arrives, the consumption of the buffer will not be triggered. Then in the Vsync cycle 2-3, the electronic device will not update the display image.
[0454] In Vsync cycle 2-3, the electronic device can complete the production and queueing of buffer 2 and buffer 3. However, due to the single buffer setting, after buffer 3 is produced, it will overwrite the produced but unconsumed buffer 2 when it is put into TextureBufferQueue.
[0455] In this way, when the Vsync signal 3 arrives, the electronic device can continue to display the N+1th frame image, and the TextureBufferQueue can include buffer3 but not buffer2.
[0456] Therefore, after the Vsync signal 3 arrives, the electronic device can obtain buffer 3 from TextureBufferQueue for consumption display, so that the display interface switches from the N+1th frame image to the N+3th frame image.
[0457] This results in the loss of the N+2th frame image, which corresponds to frame loss and frame skipping during the display process.
[0458] In this embodiment, by Fig.15 The multiple buffer configurations shown can effectively avoid the above problems.
[0459] It should be noted that, in some embodiments, the electronic device may configure multiple buffers for TextureBufferQueue in a flexible manner according to the current scenario.
[0460] For example, electronic devices can be combined with Fig. 9 Any one or more of the multiple methods provided in S908 can be used to determine the need to set multiple buffers in a similar manner. The specific implementation can refer to the above description, which will not be repeated here.
[0461] In the following example, a first buffer and a second buffer are configured in TextureBufferQueue as an example, wherein different buffers can be used to store different display contents.
[0462] In this application, when scheduling multiple buffers, the processing logic of different producers and consumers for different buffers can be controlled to achieve normal consumption and production logic.
[0463] Exemplary, combined Figure 3 ,refer to Fig. 20 .
[0464] When the first buffer and the second buffer are configured in TextureBufferQueue, the flutter framework, as a buffer producer, can be similar to Figure 3 The current solution provided requests a free buffer from TextureBufferQueue via dequeueBuffer(), fills the buffer via euqueueBuffer() and returns the queue.
[0465] It should be noted that, in this embodiment, euqueueBuffer() does not drop the existing buffer when putting the buffer into the queue. In this embodiment, if there is a buffer in the queue, the new buffer will continue to be stored in TextureBufferQueue.
[0466] For example, if there is already data in the first buffer, the Flutter framework can put the newly generated data into the second buffer while retaining the data stored in the first buffer, thereby avoiding the discarding of unconsumed buffers.
[0467] When the first and second buffers are configured in TextureBufferQueue, the native framework, as a buffer consumer, can be similar to Figure 3 The current solution provided is to obtain the buffer area from TextureBufferQueue through acquireBuffer(). The buffer is released after consumption through releaseBuffer() to avoid repeated consumption of the buffer.
[0468] It should be noted that, in this embodiment, since there are multiple buffers, acquireBuffer() can be used to acquire one of the multiple buffers without performing drop processing on other buffers.
[0469] Exemplarily, acquireBuffer() can be used to acquire the content in the first buffer for consumption. If the second buffer also includes other content, the content in the second buffer is retained.
[0470] In some embodiments of the present application, after the content in the first buffer is consumed and released, the content in the second buffer can be progressively entered into the first buffer. So that the next time it is consumed, the native framework can continue to obtain the unconsumed content through acquireBuffer() for display consumption. This implementation of first buffer priority consumption can also be described as: the priority of the first buffer is higher than that of the second buffer, so when consuming the buffer, the native framework can give priority to consuming the content in the first buffer with a higher priority.
[0471] In other embodiments of the present application, each buffer in TextureBufferQueue can be configured with a time stamp for each buffer in addition to storing the corresponding display content. For example, when the display content P2 is stored in the first buffer, the time stamp of the first buffer can be the time when the display content P2 enters the first buffer. For another example, when the display content P3 is stored in the second buffer, the time stamp of the second buffer can be the time when the display content P3 enters the second buffer.
[0472] In this way, when the native framework consumes buffers, it can consume one buffer at a time according to the order of the time stamps of each buffer, from earliest to latest. For example, when the TextureBufferQueue includes a first buffer storing display content P2 and a second buffer storing display content P3, the native framework can consume the display content P2 in the first buffer first because the time stamp of the first buffer is earlier than that of the second buffer.
[0473] It is understandable that in this implementation, after the display content P2 in the first buffer is consumed, the first buffer is released and becomes empty. The display content P3 in the second buffer can continue to be retained in the second buffer. The time stamp of the second buffer can be the moment when the display content P3 enters the second buffer. After the new buffer 4 is produced and queued, the buffer 4 can be stored in the idle first buffer, and the time stamp of the first buffer is changed to the moment when buffer 4 enters.
[0474] In this way, when the native framework consumes the buffer, it can obtain the display content P3 from the second buffer for consumption according to the time mark of the second buffer being earlier than that of the first buffer.
[0475] In the following example, the priority of the first buffer is higher than that of the second buffer. After the content of the first buffer is consumed, the content in the second buffer is progressively entered into the first buffer.
[0476] refer to Fig.21 , is a flow chart of an image processing method provided in an embodiment of the present application.
[0477] In this Fig.21 In the provided scenario, the electronic device can run the application under the user's instruction. Take the application as a shopping application as an example. Similar to the description of the above-mentioned scenarios, in this example, the user can input a sliding operation on the interface of the shopping application. For example, the sliding operation can correspond to operation 16 in the above-mentioned scenario.
[0478] In this example, a first buffer and a second buffer may be set in the TextureBufferQueue in the electronic device. The priority of the first buffer is higher than that of the second buffer.
[0479] Combination Figure 17-Figure 18 In the description of Fig.21 Before the Vsync signal V1 arrives, the electronic device can Fig.17 The solution shown is implemented to consume the buffer corresponding to the Nth frame image and to produce the buffer corresponding to the N+1th frame image. The display content P1 of the N+1th frame image can be stored in the first buffer. When the Vsync signal V1 arrives, the second buffer is empty.
[0480] like Fig.21 As shown, the program may include:
[0481] S2101a. After the Vsync signal V1 arrives, the main drawing thread sends a wake-up instruction 25 to the flutter framework.
[0482] S2101b, the Flutter UI thread sends a drawing instruction 26 to the rendering thread F.
[0483] S2101c, rendering thread F draws and displays content P2.
[0484] S2101d. Rendering thread F sends display content P2 to TextureBufferQueue.
[0485] S2101e, TextureBufferQueue stores display content P2 in Buffer2.
[0486] S2101f, Flutter UI thread sends postTraversal callback CB4 to the main drawing thread.
[0487] In this example, the execution of S2101a-S2101f can refer to Fig.18 S1705a-S1705f shown.
[0488] It should be noted that in this example, when buffer2 is finished producing, the previous buffer (such as buffer1) has finished sending to the display. Therefore, after the Vsync signal V2 arrives, when buffer2 is finished producing, the first buffer and the second buffer in TextureBufferQueue are both empty.
[0489] Therefore, in S2101d-S2101e, after completing the production of buffer2, the rendering thread F can put the display content P2 in buffer2 into the first buffer of TextureBufferQueue, so that when it is needed later, buffer2 can be consumed first.
[0490] In this way, after buffer2 is queued (ie, in Vsync cycle 2-3), the display content P2 corresponding to buffer2 can be stored in the first buffer of TextureBufferQueue, and the second buffer is empty.
[0491] Similar to Fig.18 As described in, in Vsync cycle 1-2, the electronic device can also perform consumption of the existing buffer.
[0492] For example, Fig.21 As shown, the existing buffer may be buffer1, and the consumption of buffer1 may be performed through S2102a-S2102c.
[0493] S2102a, the main drawing thread sends a rendering instruction 24 to the rendering thread M.
[0494] S2102b, rendering thread M obtains the buffer from the texture buffer queue for consumption.
[0495] S2102c, SurfaceFlinger performs synthesis processing based on the display content P1 and sends it for display.
[0496] In this example, the execution process of S2102a-S2102c can refer to the following Fig.18 S1706a-S1706c in.
[0497] In this way, the N+1th image can be displayed and consumed in the Vsync cycle 1-2.
[0498] It can be understood that in the Vsync cycle 1-2, the TextureBufferQueue can be described in terms of timing as follows: when the Vsync signal V1 arrives, the first buffer stores display content P1. After executing S2102c to complete consumption, both the first buffer and the second buffer are empty. In the Vysnc cycle 1-2, since the new buffer has not been produced, the first buffer and the second buffer are empty until the Vsync signal V2 arrives.
[0499] like Fig.21 As shown, in Vsync cycle 1-2, the flutter UI thread can also determine that it needs to wake up again in the next Vsync cycle to produce a new buffer. Correspondingly, the flutter UI thread can execute S2103 in Vsync cycle 1-2 and send a postAnimation callback CB3 to the main drawing thread.
[0500] In this way, the electronic device can start to execute the processing of the next Vsync cycle after the Vsync signal V2 arrives.
[0501] The processing in the Vsync cycle 2-3 may include the following S2104a-S2104f, and S2105, etc.
[0502] S2104a. The main drawing thread sends a wake-up instruction 27 to the flutter framework.
[0503] S2104b, the Flutter UI thread sends a drawing instruction 28 to the rendering thread F.
[0504] S2104c, rendering thread F draws and displays content P3.
[0505] S2104d. Rendering thread F sends display content P3 to TextureBufferQueue.
[0506] S2104e, TextureBufferQueue stores display content P3 in Buffer3.
[0507] S2104f, Flutter UI thread sends postTraversal callback CB5 to the main drawing thread.
[0508] In this example, through S2104a-S2104f, the electronic device can generate a new buffer based on the postAnimation callback CB3 received in S2103, after the Vsync signal V2 arrives. The new buffer may be buffer3.
[0509] Combined with the above description about the production of buffer2, in the Vsync cycle 2-3, the electronic device can complete the production of 2 buffers. The first to complete the production and be queued is buffer2. This buffer2 can be stored in the first buffer. Thereafter, the next to complete the production and be queued is buffer3. This buffer3 can be stored in the second buffer.
[0510] In addition, if there is a new buffer (such as buffer4 corresponding to the N+4th frame image) that needs to be produced, in the Vsync cycle 2-3, the corresponding S2105 is executed, and the flutter UI thread also sends a postAnimation callback CB6 to the main drawing thread.
[0511] In this way, at the end of the Vsync cycle 2-3, the display content P2 may be stored in the first buffer in the TextureBufferQueue, and the display content P3 may be stored in the second buffer.
[0512] The electronic device may execute the following S2106a-S2106c after the Vsync signal V3 arrives.
[0513] S2106a. The main drawing thread sends a rendering instruction 29 to the rendering thread M.
[0514] S2106b, rendering thread M obtains the buffer from the texture buffer queue for consumption.
[0515] S2106c, SurfaceFlinger performs synthesis processing based on the display content P2 and sends it for display.
[0516] Combination Fig. 20 In the example, the main drawing thread may give priority to consuming the first buffer when consuming the buffer. The first buffer may store display content P2 that has not been consumed.
[0517] Combination Fig. 22 The Fig.21 An example of the logic of buffer consumption and production during the execution of the solution.
[0518] and Fig.19By comparing with the existing solution shown in FIG. 1 , it can be seen that although in Vsync cycle 2-3, buffer 2 has not completed production, resulting in buffer 2 not being displayed in the Vsync cycle 2-3. Fig.21 The configuration of the first buffer and the second buffer shown enables the electronic device to display the N+2th frame image in the Vsync period 3-4, rather than directly skipping the N+2th frame image and displaying the N+3th frame image, thereby solving the problem of buffer loss in the case of a single buffer configuration.
[0519] It is understandable that Figure 15-Figure 22 In the example of not producing buffers in advance, the configuration and use of multiple (such as at least two) buffers in TextureBufferQueue are given. This solves the common problem of display freeze and frame skipping caused by buffer loss.
[0520] Thus, the embodiment of the present application is Figure 9-Figure 17 Provides a solution for early buffer production, and also Figure 15-Figure 22 A dual buffer configuration solution is provided. In different scenarios, one of the two solutions can be selected to optimize the problem of image display freeze after receiving the user's sliding operation. Alternatively, the two solutions can also be configured to optimize the image display freeze after the user's sliding operation.
[0521] It is understandable that the electronic device provided in the embodiment of the present application includes a hardware structure and / or software module corresponding to each function in order to realize the above functions. Those skilled in the art should easily realize that, in conjunction with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiment of the present application.
[0522] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0523] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of each functional module. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.
[0524] The above integrated modules can be implemented in the form of hardware or software functional modules. It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0525] For example, Fig.23 FIG. 2 shows a schematic diagram of the composition of an electronic device 2300. Fig.23 As shown, the electronic device 2300 may include: a processor 2301 and a memory 2302. The memory 2302 is used to store computer execution instructions. The electronic device may also be provided with a display screen 2303. The display screen 2303 may display the interface of the application under the instruction of the user. The display screen 2303 may also receive a sliding operation of the user to trigger the execution of any method provided in the embodiments of the present application. Exemplarily, in some embodiments, when the processor 2301 executes the instructions stored in the memory 2302, the electronic device 2300 may execute any of the methods shown in the above embodiments.
[0526] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.
[0527] Fig.24 A schematic diagram of the composition of a chip system 2400 is shown. The chip system 2400 may include: a processor 2401 and a communication interface 2402, which are used to support related devices to implement the functions involved in the above embodiments. In a possible design, the chip system also includes a memory for storing program instructions and data necessary for the electronic device. The chip system may be composed of chips, or may include chips and other discrete devices. It should be noted that in some implementations of the present application, the communication interface 2402 may also be referred to as an interface circuit.
[0528] It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.
[0529] The functions or actions or operations or steps in the above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or may include one or more servers, data centers and other data storage devices that can be integrated with the medium. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).
[0530] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. An image processing method, characterized in that: The method is applied to an electronic device, and the method comprises: Receiving a first operation from a user; In response to the first operation, running a first application and presenting an interface of the first application; receiving a second operation performed by a user on an interface of the first application, where the second operation is a sliding operation; In response to the second operation, performing a first drawing operation in a first synchronous Vsync period to obtain a first image; In response to the second operation, a second drawing operation is performed within the first synchronous Vsync cycle to obtain a second image, wherein the moment when the first Vsync cycle starts is the moment when the first Vsync signal is received, the moment when the first Vsync signal cycle ends is the moment when the second Vsync signal is received, and the first Vsync signal and the second Vsync signal are two adjacent Vsync signals, and the start time of the second drawing operation is later than the start time of the first drawing operation; the first image and the second image are images displayed sequentially by the electronic device in response to the second operation.
2. The method according to claim 1 or 2, characterized in that: The method further comprises: After the second Vsync signal is generated, the electronic device performs a first display sending operation, where the first display sending operation is used to control the display screen of the electronic device to display the first image; After the third Vsync signal is generated, the electronic device performs a second display sending operation, where the second display sending operation is used to control the display screen of the electronic device to display the second image; The third Vsync signal is later than the second Vsync signal, and the third Vsync signal is adjacent to the second Vsync signal.
3. The method according to claim 2, characterized in that The electronic device is provided with a texture buffer queue TextureBufferQueue, and the TextureBufferQueue includes at least one valid buffer, and each of the at least one valid buffer stores a drawn image.
4. The method according to claim 3, characterized in that The at least one valid buffer includes a first buffer; The performing of a first drawing operation within a first synchronous Vsync period to obtain a first image includes: The first drawing operation is performed within the first Vsync period to obtain the first image, and when the first buffer is empty, the first image is stored in the first buffer.
5. The method according to claim 4, characterized in that The at least one valid buffer includes a second buffer; The performing a second drawing operation within the first synchronous Vsync period to obtain a second image includes: The second drawing operation is performed within the first Vsync period to obtain the second image, and when the first buffer is not empty, the second image is stored in the second buffer.
6. The method according to claim 5, characterized in that The display priority of the first buffer is higher than that of the second buffer; The electronic device performs a first display operation, including: The first image is obtained from the first buffer having a higher display priority, so as to synthesize and display the first image.
7. The method according to claim 6, characterized in that After the electronic device performs the first display sending operation, the method further includes: The first image in the first buffer is released, and the first buffer is empty after being released.
8. The method according to claim 6 or 7, characterized in that: After performing the first display sending operation, the method further includes: The second image in the second buffer is stored in the first buffer; and the second image in the second buffer is released.
9. The method according to claim 8, characterized in that The electronic device performs a second display sending operation, including: The second image is obtained from the first buffer so as to synthesize and display the second image.
10. The method according to any one of claims 1 to 9, characterized in that The electronic device is provided with a callback queue, and the callback queue is used to store a display callback and / or a drawing callback; Before the electronic device performs the first drawing operation, the method further includes: After the first Vsync signal is generated, the electronic device determines that the callback queue includes a first drawing callback; the first drawing callback includes first image information; the first image information is used to draw the first image; The electronic device performs a first drawing operation, including: The electronic device generates a first wake-up instruction according to the first drawing callback; the first wake-up instruction carries the second image information; the first wake-up instruction is used to wake up the flutter rendering module configured in the electronic device to perform a drawing operation according to the second image information; According to the first wake-up indication, the flutter rendering module performs the first drawing operation to obtain the second image.
11. The method according to claim 10, characterized in that The electronic device performs a second drawing operation, including: The electronic device generates a second wake-up instruction; the second wake-up instruction carries second image information; the second wake-up instruction is used to wake up the flutter rendering module to perform a drawing operation according to the second image information; According to the second wake-up indication, the flutter rendering module performs the second drawing operation to obtain the second image.
12. The method according to claim 11, characterized in that The method further comprises: After the first Vsync signal is generated, the electronic device determines that there is a second image that needs to be continuously drawn; The electronic device generates a second wake-up instruction, including: When the electronic device determines that there is a second image that needs to be continuously drawn, the electronic device triggers the generation of the second wake-up indication after a preset time period.
13. The method according to claim 12, characterized in that The preset duration corresponds to 2 / 3 of the first Vsync period.
14. The method according to any one of claims 10 to 13, characterized in that: The method further comprises: The electronic device encapsulates the wake-up indication generated in the first encapsulation interval and the wake-up indication generated in the second encapsulation interval, so that the type of the wake-up indication related to the encapsulation process is adjusted, and the adjusted type of the wake-up indication is a non-flutter type; The starting point of the first encapsulation interval corresponds to the electronic device determining that the callback queue includes a drawing callback, and the end point of the first encapsulation interval corresponds to generating a wake-up indication corresponding to the drawing callback; The starting point of the second encapsulation interval corresponds to the electronic device generating the wake-up indication, and the end point of the second encapsulation interval corresponds to the electronic device determining whether the callback queue includes a drawing callback after the next Vsync signal is generated.
15. The method according to any one of claims 2 to 14, characterized in that Before the electronic device performs the second display sending operation, the method further includes: The electronic device obtains the number of valid buffers in TextureBufferQueue; The electronic device performs a second display operation, including: When the number of valid buffers is greater than or equal to 1, the electronic device performs the second display operation.
16. The method according to any one of claims 10 to 14, characterized in that The drawing callback is a postAnimation callback; The display callback is a postTraversal callback.
17. An electronic device, characterized in that: The electronic device comprises: a display screen, a memory and one or more processors; the display screen, the memory and the processor are coupled; The memory is used to store computer program codes, and the computer program codes include computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1 to 16.
18. A chip system, characterized in that: The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected through lines; the interface circuit is used to receive a signal from a memory of the electronic device and send the signal to the processor, the signal including a computer instruction stored in the memory; when the processor executes the computer instruction, the electronic device executes the method as described in any one of claims 1-16.
19. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 16.
Citation Information
Cited By
Image processing method and electronic device
EP4651500A1
