Image processing method, readable storage medium, electronic device and program product

Abstract

The present application relates to the technical field of image processing. Disclosed are an image processing method, a readable storage medium, an electronic device and a program product. In the method, a rendering thread of a first application, which is required to display first image data in a first image frame and currently runs on an electronic device, can send first information to a compositing process, which is used for compositing a display interface, in the electronic device before the rendering of the first image data is completed, for example, before the rendering thread finishes executing an instruction in a "beginDraw" stage. After receiving the first information sent by the first application, and in response to the first information, the compositing process can execute an instruction in a pre-compositing preparation stage, and composite and display the first image frame after the rendering of all the display content corresponding to the first image frame is completed. The compositing process starts to execute the instruction in the pre-compositing preparation stage even before the rendering of the first image data is completed, and therefore the time consumed by the electronic device to draw, render and composite an image frame can be reduced.

Description

Image processing method, readable storage medium, electronic device and program product TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image processing method, a readable storage medium, an electronic device and a program product. BACKGROUND

[0002] An electronic device can generally display a graphical user interface (GUI) through a display screen. A user can interact with the electronic device through operations on the display screen, such as a sliding operation, a clicking operation, a pinching operation, and the like, so that the electronic device can execute corresponding instructions and / or display corresponding images based on the user operations.

[0003] In some scenarios, the electronic device can fail to complete rendering and composition of a frame of image before a display time of the frame of image due to complex content of the frame of image, insufficient available resources of a processor, and the like. As a result, the electronic device can display a frame of image with a phenomenon of lag or delay, for example, the frame of image displayed by the electronic device is inconsistent with the operation of the user on the display screen, and a poor hand-following problem occurs.

[0004] SUMMARY

[0005] The present application provides an image processing method, a readable storage medium, an electronic device and a program product, which are beneficial to improve the hand-following of the electronic device.

[0006] In a first aspect, an embodiment of the present application provides an image processing method applied to an electronic device, the method comprising: running a first application; a rendering thread performing rendering processing on first image data of the first application to obtain second image data, and sending first information to a composition process before the rendering of the first image data is completed; in response to the first information, the composition process performing first processing, the first processing comprising processing before composition processing of the second image data; and the composition process performing the composition processing on the second image data.

[0007] In the method, since the processing before the composition processing of the second image data generally does not need to use the second image data itself, the composition process can start to perform the processing before the composition processing of the second image data (for example, execute instructions in a pre-composition preparation stage below) before the rendering of the first image data is completed, without waiting for the rendering of the first image data to the second image data and then performing the processing before the composition processing. As a result, the total time length from the start of drawing to the completion of composition of a frame of image can be shortened, which is beneficial to improve the hand-following of the electronic device.

[0008] In a possible implementation of the first aspect, the rendering thread corresponds to the first application. The rendering thread can be in a process of the first application, or in a process of another application or service.

[0009] In a possible implementation of the first aspect, the first information includes a first identifier of the first buffer for storing the second image data and a second identifier of the first fence, where the first fence is a software fence.

[0010] In this implementation, the first information can include the first identifier of the first buffer for storing the second image data and the first fence. Since the hardware fence of the first buffer is generated after the rendering of the first image data is completed, and the process before the composition process performed by the composition process usually requires an identifier of a buffer and an identifier of a fence, by sending the second identifier of the first fence and the first identifier of the first buffer to the composition process, the composition process can perform the process before the composition process before the rendering of the first image data is completed.

[0011] In this implementation, the state of the first fence can indicate whether the rendering of the first image data is completed. For example, when the first fence is in the released state, it can be indicated that the first image data has been rendered into the second image data, and the second image data has been stored in the first buffer.

[0012] In a possible implementation of the first aspect, the method further includes: setting, by the rendering thread, the first fence to the released state in a case where the hardware fence of the first buffer is released.

[0013] In this implementation, the state of the first fence is consistent with the state of the hardware fence of the first buffer, so that the rendering thread does not need to send the hardware fence of the first buffer to the composition process by cross-process communication after the hardware fence of the first buffer is released, which is beneficial to reduce the number of cross-process communications in the image processing process.

[0014] In a possible implementation of the first aspect, the composition process performs the composition process on the second image data, including:

[0015] The composition process performs the composition process on the second image data in a case where the first fence is in the released state.

[0016] In this implementation, the composition process can determine the timing of performing the composition process on the second image data based on the state of the first fence, that is, performing the composition process on the second image data in a case where the first fence is in the released state.

[0017] In a possible implementation of the first aspect, the processing before the synthesis of the second image data comprises at least one of the following processing tasks: obtaining the first buffer from a graphics buffer queue of the electronic device, determining whether the first buffer is used for the synthesis of the first image frame, and determining a synthesis manner of the first image frame; the synthesis processing comprises synthesizing the first image frame by the kernel thread; and the first image frame is an image frame on which the first image data is to be displayed.

[0018] In a possible implementation of the first aspect, the method further comprises displaying an image frame obtained after the synthesis of the second image data.

[0019] In a possible implementation of the first aspect, the rendering processing comprises executing a prepareTree stage, a dequeueBuffer stage, a queueBuffer stage and a beginDraw stage on the first image data; and the sending of the first information to the synthesis process before the rendering of the first image data is completed comprises: the rendering thread sends the first information to the synthesis process before the execution of the beginDraw stage is completed.

[0020] In this implementation, the rendering thread can send the first information to the synthesis process in any one of the prepareTree stage, the dequeueBuffer stage and the queueBuffer stage, or can send the first information to the synthesis process before the execution of the beginDraw stage is completed or in the beginDraw stage.

[0021] In a possible implementation of the first aspect, the execution order of the prepareTree stage, the dequeueBuffer stage, the queueBuffer stage and the beginDraw stage is any one of the following orders: the prepareTree stage, the dequeueBuffer stage, the queueBuffer stage and the beginDraw stage; the dequeueBuffer stage, the prepareTree stage, the queueBuffer stage and the beginDraw stage; the dequeueBuffer stage, the queueBuffer stage, the prepareTree stage and the beginDraw stage.

[0022] In a possible implementation of the first aspect, the sending of the first information to the synthesis process before the rendering of the first image data is completed comprises: the rendering thread sends the first information to the synthesis process in the queueBuffer stage or the dequeueBuffer stage.

[0023] In a possible implementation of the first aspect, the synthetic process is a process corresponding to surfaceflinger.

[0024] In some embodiments, the synthetic process can be a process corresponding to surfaceflinger, or a process including a thread of surfaceflinger.

[0025] In a second aspect, an embodiment of the present application provides an electronic device, including: a memory, configured to store one or more programs; and a processor, configured to execute the one or more programs to enable the electronic device to implement the image processing method in the first aspect and any possible implementation of the first aspect.

[0026] In a third aspect, an embodiment of the present application provides a readable storage medium, including one or more programs, which when executed on an electronic device, enable the electronic device to implement the image processing method in the first aspect and any possible implementation of the first aspect.

[0027] In a fourth aspect, an embodiment of the present application provides a program product, which when executed on an electronic device, enable the electronic device to implement the image processing method in the first aspect and any possible implementation of the first aspect.

[0028] The advantages of the second aspect to the fourth aspect can refer to the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0029] FIG. 1 shows a schematic diagram of a software architecture according to some embodiments of the present application.

[0030] FIG. 2 shows a schematic diagram of a process of display processing of a mobile phone 10 according to some embodiments of the present application.

[0031] FIG. 3A shows another schematic diagram of a process of display processing of a mobile phone 10 according to some embodiments of the present application.

[0032] FIG. 3B shows a schematic diagram of a change of a display interface of a mobile phone 10 according to some embodiments of the present application.

[0033] FIG. 4 shows a schematic diagram of a process of processing an image of a mobile phone 10 according to some embodiments of the present application.

[0034] FIG. 5A shows a schematic diagram of processing an image of a mobile phone 10 based on an order O1 according to some embodiments of the present application.

[0035] FIG. 5B shows a schematic diagram of processing an image of a mobile phone 10 based on an order O2 according to some embodiments of the present application.

[0036] FIG. 5C shows a process diagram of a mobile phone 10 sending first information to an SF to process an image in a dequeueBuffer stage, according to some embodiments of the present application.

[0037] FIG. 6 shows a diagram of a mobile phone 10 configuring a first fence, according to some embodiments of the present application.

[0038] FIG. 7 shows a diagram of an interaction flow of an image processing method, according to some embodiments of the present application.

[0039] FIG. 8 shows a diagram of an interaction flow of another image processing method, according to some embodiments of the present application.

[0040] FIG. 9 shows a diagram of a structure of a mobile phone 10, according to some embodiments of the present application. DETAILED DESCRIPTION

[0041] The illustrative embodiments of the present application include, but are not limited to, display methods, readable storage media, and electronic devices.

[0042] The technical solutions of the present application are described below with reference to the drawings.

[0043] The image processing method provided by the embodiments of the present application can be applied to any electronic device with display function, and the electronic device can also be referred to as a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present application do not limit the specific technology and specific device form of the electronic device. For ease of description, the electronic device is described below as a mobile phone 10.

[0044] For ease of understanding, first introduce the software architecture of the electronic device involved in the embodiments of the present application.

[0045] The software system of the mobile phone 10 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture, etc. The embodiments of the present application take the Android system with a layered architecture as an example to illustrate the software structure of the mobile phone 10. TM

[0046] Exemplarily, FIG. 1 shows a schematic diagram of a software architecture according to some embodiments of the present application.

[0047] The layered architecture divides the software into several layers, each of which has a clear role and division of labor. The layers communicate with each other through software interfaces. As shown in FIG. 1, in some embodiments, the Android system is divided into five layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, the hardware abstraction layer, and the kernel layer.

[0048] The application layer can include a series of application programs.

[0049] The application programs can include desktop, phone, email, calendar, camera, instant messaging application, etc.

[0050] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the application programs of the application layer. The application framework layer includes some pre-defined functions, such as choreographer, user interface (UI) thread, render thread, surface flinger (SF), etc.

[0051] The UI thread is used as the main thread responsible for handling the UI events and drawing operations of the application programs or system services. When a Vsync signal corresponding to a frame display interface arrives, the UI thread coordinates with the choreographer to determine the display content corresponding to the frame display interface, and triggers the drawing, rendering and composition of the frame display interface. For example, the UI thread can send a start drawing instruction or message to the render thread to instruct the render thread to draw the frame display interface.

[0052] The render thread is used to draw and render the display content of the application program. When an application program is running, a render thread corresponding to the application program can be pulled up to draw and render the display content corresponding to the application program. The render thread can be in the process of the application program, or in the process of other application programs or services.​

[0053] In some embodiments, the rendering can refer to image rendering of the display interface.

[0054] In some embodiments, the rendering can refer to image rendering of the display interface.

[0055] In some embodiments, the rendering thread can include a prepareTree stage, a dequeueBuffer stage, a beginDraw stage, a queueBuffer stage, and the like in the process of rendering and drawing the image of the corresponding application. In the prepareTree stage, the rendering thread can calculate the view hierarchy, determine the redrawn view, the traversal and update of the view tree, and the like. In the dequeueBuffer stage, the rendering thread can obtain a buffer from the graphic buffer queue, which is used to store the content of the image to be drawn next. In the beginDraw stage, the rendering thread can call the graphics processing unit (GPU) or other modules to draw and render the image. In the queueBuffer stage, the rendering thread can pass the identifier of the fence corresponding to the buffer storing the drawn image and the identifier of the buffer to the SF. In some embodiments, the process of rendering the first image data (such as the display elements in the application window, such as icons, text, images, tables, and the like) in a frame of image to be displayed by the application into the second image data (such as an image corresponding to the display elements in the window of the application) by the rendering thread based on the above process can be referred to as the rendering process.

[0056] It should be noted that the rendering thread can call one or more functions (or instructions, modules) corresponding to each stage to implement the corresponding operation. For example, the rendering thread can call the function or module with the same name as each stage to implement the corresponding operation.

[0057] It should be noted that the fence is a resource synchronization mechanism. In general, the fence can be understood as a resource lock used to coordinate access to shared resources by different threads or components, to ensure data consistency and accuracy. For example, a buffer in a graphics buffer queue can correspond to a fence, and the GPU can establish and occupy the fence of the buffer during the process of synthesizing the display content corresponding to the buffer, and release the fence (equivalent to setting the fence to a released state) after synthesizing the display content. SF can start synthesizing the frame display interface after all the fences of the buffers corresponding to the frame to be synthesized are released. The fence can be divided into software fence and hardware fence, wherein the software fence is a fence generated by software (such as threads and processes of application programs, system services, etc.) in the electronic device, and the hardware fence is a fence generated by hardware (such as central processing units, graphics processing units, neural network processing units, application specific integrated circuits, etc.) in the electronic device.

[0058] The SF is used to synthesize the images rendered by the rendering threads of each application program in the mobile phone 10 and the images rendered by other modules (such as video frames, etc.), to obtain a display interface. In some embodiments, the synthesis can refer to the process of synthesizing a plurality of rendered images into a display interface (such as a frame of image).

[0059] In some embodiments, after receiving the identification of the buffer delivered by one or more rendering threads and the hardware fence corresponding to the buffer, the SF can perform the preparation operation before synthesizing the display interface (hereinafter referred to as the pre-synthesis preparation stage, and the processing performed by the SF in the pre-synthesis preparation stage is referred to as the processing before the synthesis processing), such as taking out the received buffer from the graphics buffer queue, determining whether the taken out buffer is used for the synthesis of the current frame, determining the synthesis manner of the current frame (such as whether the synthesis manner is GPU synthesis), etc., and after the pre-synthesis preparation stage is executed and all the fences of the buffers corresponding to a frame of display interface are released, the SF synthesizes the display interface based on the images in each buffer (the processing of the SF to synthesize the image data in all the buffers corresponding to a frame of display interface into a frame of image can be referred to as the synthesis processing).

[0060] In some embodiments, the SF can be for an independent process (which can be referred to as a synthesis process), and the main thread or other threads of the process can be used to perform the processing before the synthesis processing and the synthesis processing. In some embodiments, the SF can also be a thread of another process (which can be referred to as a synthesis process), and the thread can be used to perform the processing before the synthesis processing and the synthesis processing.

[0061] In some embodiments, the process of SF synthesizing the display interface can include a commit stage and a composite stage. In the commit stage, SF can determine which images need to be processed in the current frame; in the composite stage, SF can identify, sort and synthesize the images that need to be drawn to the current frame, and then call a hardware compositor, GPU, data processing unit (DPU) or the like to synthesize the display interface corresponding to the current frame.

[0062] In some embodiments, the pre-synthesis preparation stage can also be referred to as the first processing stage; in the process of SF synthesizing the display interface, the process after the synthesis preparation stage can be referred to as the second processing stage.

[0063] In some embodiments, SF is also used to generate a corresponding software Vsync signal according to a hardware vertical synchronization (Vsync) signal generated by the hardware (such as the display screen, GPU, etc.) of the mobile phone 10, and to pass the software Vsync signal to other processes or threads (such as the main thread of the application program, the UI thread, the rendering thread, etc.), so that the other threads or processes can perform related operations based on the software Vsync signal.

[0064] The Android runtime includes a core library and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0065] The core library contains two parts: one is the function function library that the java language needs to call, and the other is the core library of Android.

[0066] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the java files of the application layer and the application framework layer into binary files. The virtual machine is used to perform functions such as management of object life cycle, stack management, thread management, security and exception management, and garbage collection.

[0067] The system library can include multiple functional modules, such as: image rendering library, image synthesis library, etc. Among them, the image rendering library can be used for rendering two-dimensional or three-dimensional images, and the image synthesis library can be used for synthesizing two-dimensional or three-dimensional images.

[0068] The hardware abstraction layer can include multiple library modules, such as a hardware compositor (HWC) and the like. The Android system can load the corresponding library module for the device hardware, thereby achieving the purpose of the application framework layer accessing the device hardware. The device hardware can include, for example, the GPU, DPU, etc. in the electronic device.

[0069] The kernel layer is a layer between hardware and software. The kernel layer at least includes a touch panel (TP) driver, a display driver, a Bluetooth driver, a wireless driver, a keyboard driver, a memory driver, and a camera driver, etc.

[0070] In some embodiments, the kernel thread in the kernel layer can include a crtc commit thread (not shown) for synthesizing and displaying a display interface. The SF can implement the synthesis of a display picture by calling the crtc commit thread.

[0071] It should be noted that the software architecture of the mobile phone 10 shown in FIG. 1 is only an example. In other embodiments, the software architecture of the mobile phone 10 can include more or fewer layers, each layer can include more or fewer modules, the layers to which different modules belong can be adjusted, or other structures can be used, which are not limited herein.

[0072] The technical solutions of the present application will be described below in conjunction with the software architecture shown in FIG. 1.

[0073] For example, FIG. 2 shows a process diagram of display processing of a mobile phone 10 according to some embodiments of the present application.

[0074] For example, assume that the images displayed by the mobile phone 10 in time sequence are frame F1, frame F2, and frame F3. Referring to FIG. 2, the application program in the mobile phone 10 can receive Vsync signals at t0 to t5, respectively, and process each frame image after receiving the Vsync signal. Taking frame F1 as an example, the mobile phone 10 can start drawing and rendering frame F1 after receiving the Vsync signal at t0; the SF synthesizes the well-drawn and rendered frame F1 after receiving the Vsync signal at t1. After the synthesis of frame F1 is completed, the electronic device can start the display driver by calling the kernel layer, and display the content corresponding to frame F1 on the display screen after the Vsync signal at t3 arrives. In FIG. 2, there is a lag of 2 Vsync signal periods (for different operating systems or electronic devices, there can be a lag of other number of Vsync signal periods, which are not limited herein) from the start of drawing of each frame image to the final display, and the electronic device has a lag in displaying the frame image.

[0075] In some cases, the mobile phone 10 can not be able to complete the rendering and synthesis of a frame image before the display time of the frame image due to complex content of the frame image, insufficient available resources of the processor, etc. As a result, the electronic device can have the phenomenon of frame loss, lag, and delay in displaying the frame image, for example, the frame image displayed by the electronic device does not match the operation of the user on the display screen, and the hand-following performance is poor.

[0076] For example, referring to FIG. 3A, if the content of frame F2 is complex or the load of mobile phone 10 is high, mobile phone 10 cannot complete the composition of frame F2 before t3 (the display time of frame F2), resulting in that mobile phone 10 can only continue to display frame F1 instead of frame F2 during t3 to t4, and display frame F2 during t4 to t5, resulting in that mobile phone 10 displays frame F2 with a delay, affecting the user experience. For example, referring to FIG. 3B, assuming that frame F1, frame F2 and frame F3 are interfaces of viewing pictures by the user through the up sliding operation on mobile phone 10, after the user slides on the display screen of mobile phone 10 from point A by a distance D to point A', if frame F2 is not rendered in time, the distance that the interface displayed by mobile phone 10 moves upward will be less than D, which will cause the interface displayed by mobile phone 10 to be inconsistent with the operation of the user.

[0077] FIG. 4 shows a process diagram of mobile phone 10 processing an image according to some embodiments of the present application, in combination with the software architecture shown in FIG. 1.

[0078] As shown in FIG. 4:

[0079] After the UI thread of the gallery application receives the Vsync signal of a frame of image (for example, frame F2 mentioned above) at t1, the UI thread can coordinate with the choreographer to determine the content that needs to be drawn for frame F2 and notify the gallery application rendering thread to start drawing the content of frame F2 at t2.

[0080] After receiving the notification of starting drawing sent by the UI thread, the rendering thread can in turn:

[0081] execute the instructions of the prepareTree stage to calculate the view hierarchy of frame F2, determine the views to be redrawn, update / traverse the view tree, etc.;

[0082] execute the instructions of the dequeueBuffer stage to obtain an available buffer for storing the content corresponding to frame F2 from the graphics buffer queue;

[0083] execute the instructions of the beginDraw stage to call the GPU or other modules to draw and render frame F2, wherein the GPU can release the hardware fence of the buffer corresponding to frame F2 after rendering frame F2;

[0084] execute the instructions of the queueBuffer stage to pass the identifier of the hardware fence of the buffer corresponding to frame F2 and the identifier (bufferID) of the buffer corresponding to frame F2 to SF to notify SF to start composing frame F2, for example, notify SF to start composing at t3.

[0085] SF receives the identification of the hardware fence of the buffer corresponding to frame F2 and the identification (bufferID) of the buffer corresponding to frame F2, can execute the instructions of the pre-composition preparation stage (such as instructions for taking out the received buffer from the graphics buffer queue, determining whether the taken out buffer is used for the composition of frame F2, determining the composition method of frame F2, etc.), and waits for the release of the hardware fence of the buffer corresponding to frame F2. SF executes the instructions of the pre-composition preparation stage at T4, and can determine that all the hardware fences of the buffer corresponding to frame F2 have been released at T5, call the crtc_commit thread to compose frame F2 and complete the composition of frame F2 at T6, so that the display screen of mobile phone 10 can display frame F2 after T6. If the display time of frame F2 is before T6, for example, T7 between T5 and T6, frame F2 will be delayed or not displayed.

[0086] Based on the process of FIG. 4, mobile phone 10 needs to go through multiple sequentially executed stages in the process of drawing, rendering and composing a frame of image. If a stage takes a long time, it may cause the frame of image to be unable to be displayed at the corresponding display time. For example, although the instructions of the pre-composition preparation stage executed by SF do not need the specific content of the buffer corresponding to a frame of image (equivalent to the pre-composition preparation stage can be executed after the completion of the beginDraw stage), SF still executes the instructions of the pre-composition preparation stage after the completion of the instructions of the beginDraw stage and the queueBuffer stage of the rendering process. In this way, even if the queueBuffer stage of a frame of image corresponding to the rendering thread of an application program ends, the entire content of the frame of image has been drawn and rendered, SF still needs to execute the instructions of the pre-composition preparation stage before starting the composition of the frame of image, which increases the total time length of drawing, rendering and composition of the frame of image.

[0087] Based on the above, in order to reduce the total time length of drawing, rendering and composition of an image by an electronic device, an image processing method is provided, which adjusts the execution order of part of the stages in the process of drawing, rendering and composing a frame of image by the electronic device, so that the pre-composition preparation stage can be executed before the completion of the rendering of the frame of image (for example, executed in parallel with part of the stages executed by the rendering thread), so as to reduce the total time length of drawing, rendering and composition of a frame of image by the electronic device, thereby alleviating the phenomenon of interface lag of the electronic device and improving the hand following performance of the electronic device.

[0088] More specifically, assume that the electronic device currently runs a first application which needs to display content in a first image frame, and the content that the first application needs to display in the first image frame is first image data. Before the rendering thread of the first application finishes rendering the first image data (e.g., before the rendering thread of the first application finishes executing the instruction of the beginDraw stage), the rendering thread of the first application can send first information to a composition process (or thread) in the electronic device for compositing the display interface (e.g., the process corresponding to the aforementioned SF), where the first information indicates a first buffer (e.g., includes an identifier of the first buffer) for storing second image data after the first image data is rendered and a first fence (e.g., includes an identifier of the first fence) corresponding to the first buffer (where the first fence is used to indicate whether the first image data has been rendered). After receiving the first information sent by the first application, the SF can execute the instructions of the aforementioned pre-composition preparation stage based on the first information (e.g., execute the instruction of taking each first buffer from the graphics buffer queue, the instruction of determining whether the first buffer is used for compositing the first image frame, the instruction of determining the compositing manner of the first image frame, etc.), and composite the first image frame after all display content corresponding to the first image frame is rendered (e.g., after the first fence indicates that the first image data has been rendered).

[0089] Since the first information is sent to the SF before the rendering thread finishes rendering the first image frame, the SF can execute the instructions of the pre-composition preparation stage in parallel with or before the process of the rendering thread rendering the first image frame, instead of executing the instructions of the pre-composition preparation stage after the rendering thread of the first application finishes rendering the first image frame. In this way, the time-consuming of the electronic device drawing, rendering and compositing an image frame can be reduced.

[0090] In some embodiments, the aforementioned first fence can be a software fence set by the rendering thread, and the first information can include an identifier of the first fence, where the state of the first fence is consistent with the state of the hardware fence corresponding to the first buffer. After the GPU finishes rendering the first image data, the GPU can release the hardware fence corresponding to the first buffer; accordingly, the rendering thread of the first application can also set the state of the first fence to the release state. In the case where the first fence is in the release state, indicating that the GPU has rendered the first image data into the second image data and stored in the first buffer, the SF can start compositing the first image frame.

[0091] Since the SF executes the instruction of the pre-composition preparation phase based on the received identifier of the buffer and the corresponding fence identifier, the first information including the identifier of the first fence and the identifier of the first buffer is sent to the SF, without the modification of the processing logic of the SF, which is beneficial to reduce the development cost of adapting the method provided in the embodiments of the present application.

[0092] In addition, since the hardware fence of the first buffer is generated after the first image data is rendered into the second image data, if the rendering process directly delivers the hardware fence of the first buffer to the SF, it will also increase a cross-process communication. Therefore, the first fence in the form of the software fence can also reduce the cross-process communication resources occupied by the electronic device in the image processing process.

[0093] In some embodiments, the rendering thread can also indicate the SF to execute the instruction of the pre-composition preparation phase through other forms of the first information, which is not limited herein.

[0094] In some embodiments, the first fence can also be a hardware fence following the state of the hardware fence of the first buffer, which is not limited herein.

[0095] It should be noted that there can be one or more first applications, and in the case that there are multiple first applications displaying content in the first image frame, the SF can synthesize the first image frame when the first image data corresponding to each first application has been completely rendered (for example, all rendered into the corresponding second image data) (for example, the first fences set by the rendering threads of the multiple first applications are all in the released state).

[0096] In some embodiments, the rendering thread of each application program can adjust the instruction of the aforementioned queueBuffer phase to be executed before the beginDraw phase, and send the aforementioned first information to the SF in the queueBuffer phase. For example, a fence monitoring module (fenceMonitor) for monitoring the hardware fence of the buffer can be added in the queueBuffer phase, which can establish a software fence (i.e., the aforementioned first fence) for monitoring the hardware fence of the first buffer, and set the state of the first fence to be released after monitoring that the hardware fence of the first buffer is released.

[0097] In some embodiments, the execution order of the aforementioned prepareTree phase, dequeueBuffer phase, beginDraw phase, and queueBuffer phase can be any one of the following orders:

[0098] O1: dequeueBuffer stage -> queueBuffer stage -> prepareTree stage -> beginDraw stage;

[0099] O2: prepareTree stage -> dequeueBuffer stage -> queueBuffer stage -> beginDraw stage;

[0100] O3: dequeueBuffer stage -> prepareTree stage -> queueBuffer stage -> beginDraw stage.

[0101] The following describes the technical solutions of the present application in the order of execution of the prepareTree stage, the dequeueBuffer stage, the beginDraw stage, and the queueBuffer stage as O1, and the electronic device as the mobile phone 10.

[0102] Exemplarily, FIG. 5A shows a process diagram of the mobile phone 10 processing an image based on the order O1 according to some embodiments of the present application.

[0103] As shown in FIG. 5A:

[0104] The UI thread of the first application can coordinate with the choreographer to determine the first image data that needs to be drawn for the first image frame (for example, the aforementioned frame F2) and notify the rendering thread of the first application to start drawing the first image data at T2 time after receiving the Vsync signal of the first image frame at T1 time.

[0105] After receiving the notification of starting drawing sent by the UI thread, the rendering thread of the first application can in turn:

[0106] execute the instruction of the dequeueBuffer stage to obtain a first buffer for storing second image data processed by rendering the first image data from the graphics buffer queue.

[0107] execute the instruction of the queueBuffer stage, and send the first information to the SF at T8 time in the process of executing the instruction of the queueBuffer stage to notify the SF to start compositing and pass the identifier of the first buffer and the first fence indicating the rendering state of the first image data to the SF. Exemplarily, referring to FIG. 6, the rendering thread can create the first fence (software fence) to monitor the state of the hardware fence of the first buffer in the process of executing the instruction of the queueBuffer stage.

[0108] The instructions of the prepareTree stage are executed to calculate the view hierarchy of the first image data, determine the redrawn view, update / traverse the view tree, etc.

[0109] The instructions of the beginDraw stage are executed to call the GPU or other module to draw and render the first image data, and complete the rendering of the first image data at T3 to obtain the second image data. In some embodiments, referring to FIG. 6, after the rendering thread calls the GPU to render the first image data, the GPU can generate a hardware fence corresponding to the first buffer, and release the hardware fence of the first buffer after the rendering of the first image data is completed. Accordingly, the rendering thread (e.g., the aforementioned fenceMonitor module) can set the state of the first fence to release in the case that the release of the hardware fence of the first buffer is detected.

[0110] After the SF receives the first information at T8, the SF can start to execute the instructions of the pre-composition preparation stage (e.g., instructions of taking out the buffer (e.g., the first buffer) indicated by the first information from the graphics buffer queue, determining whether the taken-out buffer (e.g., the first buffer) is used for the composition of the first image frame, determining the composition manner of the first image frame, etc.), and complete the execution of the instructions of the pre-composition preparation stage at T9. After the SF detects the release of the first fence (equivalent to the completion of the rendering of the first image data) at T3, the SF can call the crtc_commit thread to compose the first image frame and complete the composition at T 10 , so that the electronic device can start to display the first image frame after T 10 .

[0111] The following introduces the technical solutions of the present application with the execution order of the prepareTree stage, the dequeueBuffer stage, the beginDraw stage, and the queueBuffer stage as O2, and the electronic device as the mobile phone 10.

[0112] Exemplarily, FIG. 5B shows a process diagram of the mobile phone 10 processing images based on the order O2 according to some embodiments of the present application.

[0113] As shown in FIG. 5B:

[0114] The UI thread of the first application can coordinate with the choreographer to determine the first image data that needs to be drawn for the first image frame after receiving the Vsync signal of the first image frame (e.g., the aforementioned frame F2) at T1, and notify the rendering thread of the first application to start to draw the first image data at T2.

[0115] The rendering thread of the first application can, in sequence, after receiving the notification of starting drawing sent by the UI thread:

[0116] instructions of the prepareTree phase to calculate the view hierarchy of the first image data, determine the redrawn view, update / traverse the view tree, etc.

[0117] instructions of the dequeueBuffer phase to obtain a first buffer for storing second image data after rendering processing of the first image data from the graphics buffer queue.

[0118] instructions of the queueBuffer phase, and send the first information to the SF at T 11 moment during the execution of the instructions of the queueBuffer phase to inform the SF to start composition and pass the identity of the first buffer and the first fence indicating the rendering state of the first image data to the SF.

[0119] instructions of the beginDraw phase to call the GPU or other modules to draw and render the first image data, and complete the rendering of the first image data at T3 to obtain the second image data.

[0120] The SF can start to execute instructions of the pre-composition preparation phase (such as instructions of taking out the buffer (such as the first buffer) indicated by the first information from the graphics buffer queue, determining whether the taken out buffer (such as the first buffer) is used for the composition of the first image frame, determining the composition mode of the first image frame, etc.) after receiving the first information at T 11 moment, and execute the instructions of the pre-composition preparation phase at T 12 moment. The SF can call the crtc_commit thread to compose the first image frame after detecting the release of the first fence (equivalent to the completion of the rendering of the first image data) at T3, and complete the composition at T 10 moment, so that the electronic device can start to display the first image frame after T 10 moment (such as the aforementioned T7 moment).

[0121] In some embodiments, the rendering thread can also send the first information to the SF in other phases outside the queueBuffer phase, such as the dequeueBuffer phase, the prepareTree phase, etc.

[0122] Exemplarily, FIG. 5C shows a process diagram of a mobile phone 10 sending the first information to the SF to process the image in the dequeueBuffer phase according to some embodiments of the present application.

[0123] As shown in FIG. 5C:

[0124] The UI thread of the first application can coordinate with the choreographer to determine the first image data that needs to be drawn for the first image frame (e.g., the aforementioned frame F2) and notify the rendering thread of the first application to start drawing the first image data at T2, after receiving the Vsync signal of the first image frame at T1.

[0125] The rendering thread of the first application can in turn:

[0126] execute instructions of the prepareTree stage to calculate the view hierarchy of the first image data, determine the views to be redrawn, update / traverse the view tree, and the like.

[0127] execute instructions of the dequeueBuffer stage to obtain a first buffer for storing second image data processed by rendering the first image data from the graphics buffer queue, and send first information to the SF at T 13 to notify the SF to start composition and pass the identifier of the first buffer and a first fence indicating the rendering state of the first image data to the SF

[0128] execute instructions of the queueBuffer stage.

[0129] execute instructions of the beginDraw stage to call the GPU or other modules to draw and render the first image data, and complete the rendering of the first image data to obtain the second image data at T3.

[0130] The SF can start to execute instructions of the pre-composition preparation stage (e.g., instructions to obtain the buffer (e.g., the first buffer) indicated by the first information from the graphics buffer queue, determine whether the obtained buffer (e.g., the first buffer) is used for composition of the first image frame, determine the composition mode of the first image frame, and the like) after receiving the first information at T 13 and complete the execution of the instructions of the pre-composition preparation stage at T 14 The SF can call the crtc_commit thread to compose the first image frame and complete the composition at T 10 after detecting the release of the first fence (equivalent to the completion of the rendering of the first image data) at T3. 10 The electronic device can start to display the first image frame after T

[0131] In some embodiments, the process of the phone 10 processing the image based on the order O3 can refer to the process of the phone 10 processing the image based on the order O2 shown in FIG. 5B or FIG. 5C, with the difference being that the execution order of the prepareTree stage and the dequeueBuffer stage is different, which will not be described here.

[0132] Based on the processes shown in FIGS. 5A-5C, compared with the process of FIG. 4, the time for the phone 10 to draw, render and composite a frame of image is shortened by T9-T8 (or T 12 -T 11 , or T 14 -T 13 , which is conducive to alleviating the interface lag, delay and other phenomena displayed by the phone 10.

[0133] Based on the scenarios shown in FIGS. 5A and 6, the technical solutions of the present application will be described below.

[0134] Exemplarily, FIG. 7 shows an interaction flow diagram of an image processing method according to some embodiments of the present application. In the interaction flow, the first application in the electronic device that needs to display content in the first image frame can be one or more, and each first application can include a UI thread and a rendering thread, and the rendering thread can include a fenceMonitor. As shown in FIG. 7, the interaction flow includes the following steps:

[0135] 701, the UI thread sends a notification to the rendering thread to start drawing the first image data.

[0136] Exemplarily, after the UI thread of the first application receives the Vsync signal of the first image frame, it can initiate a notification to the rendering thread of the first application to start drawing the first image data.

[0137] For example, the UI thread can coordinate with the aforementioned choreographer to determine the first image data that the first application needs to display in the first image frame, etc. After determining the first image data that the first application needs to display in the first image frame, it can initiate a notification to the rendering thread of the first application to start drawing the first image data.

[0138] It should be noted that in the case of multiple first applications, the first image data displayed by each first application in the first image frame can be different.

[0139] 702, the rendering thread acquires the first buffer.

[0140] The rendering thread, after receiving the notification of starting to draw the first image data, can select an idle buffer from the graphics buffer queue as a first buffer for storing second image data after rendering processing of the first image data in response to the notification. For example, the rendering thread can obtain the first buffer based on the instruction of performing the dequeueBuffer stage.

[0141] It should be noted that in the case of multiple first applications, the first buffer taken by the rendering thread of different first applications is different.

[0142] 703. The rendering thread establishes a first fence.

[0143] After obtaining the first buffer, the rendering thread can establish a first fence (software fence) for monitoring the hardware fence of the first buffer, and send an identifier of the first fence to the FenceMonitor.

[0144] 704. The fenceMonitor in the rendering thread records the first fence.

[0145] After receiving the identifier of the first fence, the fenceMonitor can record the identifier of the first fence and monitor the state of the hardware fence of the first buffer.

[0146] 705. The rendering thread sends a start composition notification including first information to the SF.

[0147] After establishing the first fence, the rendering thread can send a start composition notification including first information to the SF.

[0148] In some embodiments, the first information can be used to indicate the first fence and the first buffer. For example, the identifier of the first fence and the identifier of the first buffer can be included in the first information.

[0149] 706. The SF executes the instruction of the pre-composition preparation stage.

[0150] After receiving the start composition notification, the SF can execute the instruction of the pre-composition preparation stage based on the first information in the start composition notification.

[0151] Exemplarily, the instruction of the pre-composition preparation stage can include but is not limited to the instruction of taking the first buffer from the graphics buffer queue, the instruction of determining whether the taken buffer (e.g. the first buffer) is used for composition of the first image frame, the instruction of determining the composition mode of the first image frame, etc.

[0152] In some embodiments, the instructions of the SF in the pre-composition preparation stage can also include more or less instructions based on different software architectures or operating systems, which are not limited herein.

[0153] 707. The SF invokes the crtc_commit thread to prepare the composition of the first image frame.

[0154] After the SF executes the instructions in the pre-composition preparation stage, the SF can invoke the crtc_commit thread to prepare the composition of the first image frame.

[0155] 708. The crtc_commit thread waits for the first fence corresponding to the first image frame to be released.

[0156] The crtc_commit thread can wait for the first fence corresponding to the first image frame to be released in response to the invocation of the SF.

[0157] 709. The rendering thread executes the instructions in the prepareTree stage and the beginDraw stage and invokes the GPU to render the first image data to obtain second image data.

[0158] After the rendering thread sends the start-composition notification to the SF, the rendering thread can execute the instructions in the prepareTree stage and the beginDraw stage and invoke the GPU (or other hardware for rendering images) to draw and render the first image data to obtain second image data and store the second image data in the first buffer.

[0159] It should be noted that the GPU establishes a hardware fence for the first buffer during the process of drawing and rendering the first image data, and releases the hardware fence for the first buffer after the drawing and rendering of the first image data is completed (for example, after the second image data that is drawn and rendered is stored in the first buffer).

[0160] 710. The fenceMonitor in the rendering thread releases the first fence when the hardware fence for the first buffer is released.

[0161] The fenceMonitor in the rendering thread can continuously monitor the state of the hardware fence for the first buffer after recording the first fence, and release the first fence when the hardware fence for the first buffer is released.

[0162] 711. The crtc_commit thread composes the first image frame when the first fence corresponding to the first image frame is released.

[0163] The crtc_commit thread synthesizes the first image frame when the first fence corresponding to the first image frame is released.

[0164] After the crtc_commit thread synthesizes the first image frame, the mobile phone 10 can display the first image frame on the display screen.

[0165] Based on the above method, the SF of the mobile phone 10 can execute the instructions in the pre-synthesis preparation stage before the first image frame is rendered, which can avoid the situation that the first image frame cannot be synthesized before the display time arrives due to the waiting of the SF to execute the instructions in the pre-synthesis preparation stage after the first image data corresponding to the first image frame is rendered. In this way, the situation that the mobile phone 10 appears to be stuck and the display screen is delayed can be alleviated, and the responsiveness of the mobile phone 10 can be improved.

[0166] The embodiment of the present application further provides an image processing method, which is applied to an electronic device, and the electronic device runs at least one first application and a synthesis process, wherein each first application includes a rendering thread.

[0167] FIG. 8 shows an interaction flow diagram of another image processing method according to some embodiments of the present application. As shown in FIG. 8, the flow includes the following steps:

[0168] 801. When the first image frame to be displayed includes first image data corresponding to the first application, the rendering thread of the first application renders the first image data to obtain second image data, and sends first information to the synthesis process before the first image data is rendered.

[0169] For example, when it is determined that the first image frame to be displayed includes first image data corresponding to the first application, the UI thread of the first application can send a notification of starting to draw the first image data to the rendering thread. In response to the notification of starting to draw the first image data, the rendering thread can execute the instructions in the prepareTree stage, the dequeueBuffer stage, the queueBuffer stage, and the beginDraw stage, render the first image data into second image data, and send first information to the synthesis process (for example, the aforementioned SF) before the first image data is rendered (for example, before the execution of the beginDraw stage is completed). The execution order of the prepareTree stage, the dequeueBuffer stage, the queueBuffer stage, and the beginDraw stage can be the aforementioned order O1, order O2, or order O3. For details, refer to the embodiments of FIGS. 5A, 5B, 5C, and 7, which will not be described here.

[0170] In some embodiments, the first information can indicate the first buffer storing the first image data, and a resource lock (e.g. fence) corresponding to the first buffer. For example, the first information can include an identifier of the first buffer, and the aforementioned first fence (software fence), the status of the first fence being consistent with the hardware fence of the first buffer.

[0171] In some embodiments, the first fence can be a software fence established by the rendering thread in the queueBuffer stage, for monitoring the hardware fence of the first buffer.

[0172] In some embodiments, the rendering thread can set the status of the first fence to the released status, in case that the GPU has completed the rendering of the first image data (e.g. in case that the hardware fence of the first buffer has been released).

[0173] In some embodiments, the rendering thread can also establish the first fence and / or send the first information to the composition process in other stages, for example, after obtaining the first buffer in the dequeueBuffer stage, in the beginDraw stage, etc., which are not limited herein.

[0174] In some embodiments, the establishment of the first fence and the sending of the first information to the composition process can also be in different stages, for example, the first fence is established in the dequeueBuffer stage, and the first information is sent to the composition process in the stage after the dequeueBuffer stage (e.g. the prepareTree stage, the queueBuffer stage, etc. in the aforementioned O3 order), which are not limited herein.

[0175] 802, the composition process executes the instruction of the pre-composition preparation stage corresponding to the first image frame in response to the first information.

[0176] After receiving the first information, the composition process can execute the instruction of the pre-composition preparation stage corresponding to the first image frame in response to the first information. For example, the composition process can take out the first buffer from the graphics buffer queue, determine whether the taken out buffer (e.g. the first buffer) is used for the composition of the first image frame, determine the composition mode of the first image frame, etc. For details, reference can be made to the content of the aforementioned steps 706 and 707, which are not repeated here.

[0177] In some embodiments, the composition process can also call the crtc_commit thread to prepare the composition of the first image frame in the pre-composition preparation stage or after the pre-composition preparation stage.

[0178] 803, the composition process composites the first image frame based on the second image data.

[0179] The synthesizing process can synthesize the first image frame if all the first fences corresponding to the first image frame have been released, e.g., if all the first image data has been rendered into the second image data.

[0180] Exemplarily, the synthesizing process and / or the crtc commit thread can continuously monitor the status of all the first fences corresponding to the first image frame, and the crtc commit thread can synthesize the first image frame if all the first fences corresponding to the first image frame have been released.

[0181] In some embodiments, after the crtc commit thread has synthesized the first image frame, the electronic device can display the content of the first image frame when the display time of the first image frame arrives.

[0182] In some embodiments, if the first image frame further includes other image data in addition to the second image data, the synthesizing process can synthesize the second image data and the other image data after the other image data has been rendered.

[0183] In some embodiments, the synthesizing process can invoke the aforementioned crtc commit thread to synthesize the first image frame.

[0184] Based on the above method, the synthesizing process of the electronic device can start executing the instructions of the pre-synthesis preparation phase of the first image frame before the first image frame has been rendered, which can shorten the total time length of drawing, rendering and synthesizing the first image frame.

[0185] Further, FIG. 9 shows a structural schematic diagram of a mobile phone 10 according to some embodiments of the present application.

[0186] As shown in FIG. 9, the mobile phone 10 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0187] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the mobile phone 10. In other embodiments of the present application, the mobile phone 10 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0188] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a GPU, a DPU, an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors.

[0189] The processor 110 can also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can hold instructions or data that the processor 110 has just used or is using repeatedly. If the processor 110 needs to use the instructions or data again, it can be called from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thereby improving the efficiency of the system.

[0190] In some embodiments, the processor 110 can be used to execute instructions of the image processing method of the present application, such as instructions of the aforementioned UI thread, rendering thread, SF, crtc commit thread, etc. For example, the processor 110 can execute instructions to cause the rendering thread to send first information to the SF to instruct the SF to perform the pre-synthesis preparation stage before the first image frame is rendered.

[0191] The USB interface 130 is an interface that conforms to the USB standard specification, and can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc.

[0192] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is illustrative and does not limit the structure of the mobile phone 10. In other embodiments of the present application, the mobile phone 10 can also use different interface connection methods or combinations of multiple interface connection methods in the above embodiments.

[0193] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger.

[0194] The power management module 141 is used to connect the battery 142 and the charging management module 140 and the processor 110.

[0195] The wireless communication function of the mobile phone 10 can be realized through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0196] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. The antennas in the mobile phone 10 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas.

[0197] The mobile communication module 150 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the mobile phone 10. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0198] The modem processor can include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal.

[0199] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the mobile phone 10. The wireless communication module 160 can be one or more devices integrated with at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signal, and sends the processed signal to the processor 110.

[0200] In some embodiments, the antennas 1 and the mobile communication module 150 of the mobile phone 10 are coupled, and the antennas 2 and the wireless communication module 160 are coupled, so that the mobile phone 10 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidu navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0201] The mobile phone 10 implements a display function through a GPU, a display screen 194, and an application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0202] The display screen 194 is used to display images, display videos, and receive sliding operations, etc. The display screen 194 includes a display panel. In some embodiments, the mobile phone 10 can include 1 or N display screens 194, N being a positive integer greater than 1.

[0203] The mobile phone 10 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, etc.

[0204] ISP is used to process the data feedback from the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and conversion into an image visible to the human eye.

[0205] The camera 193 is used to capture still images or videos.

[0206] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the phone 10. The external memory card communicates with the processor 110 through the external memory interface 120 to realize data storage functions. For example, music, video, and other files are saved in the external memory card.

[0207] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like. The data storage area can store data created during the use of the phone 10, and the like. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like. The processor 110 executes various function applications and data processing of the phone 10 by running instructions stored in the internal memory 121 and / or instructions stored in a memory disposed in the processor.

[0208] The phone 10 can realize audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, the application processor, and the like. For example, music playing, recording, and the like.

[0209] The audio module 170 is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals.

[0210] The speaker 170A, also known as a “loudspeaker”, is used to convert an audio electrical signal into a sound signal. The phone 10 can listen to music or listen to a hands-free call through the speaker 170A.

[0211] The receiver 170B, also known as a “earpiece”, is used to convert an audio electrical signal into a sound signal. When the phone 10 answers a call or a voice message, the receiver 170B can be held close to the ear to listen to the voice.

[0212] Microphone 170C, also called "microphone", "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by approaching the microphone 170C with the mouth, inputting the sound signal into the microphone 170C. The mobile phone 10 can be provided with at least one microphone 170C.

[0213] The earphone interface 170D is used to connect a wired earphone.

[0214] The key 190 includes a power-on key, a volume key, etc. The key 190 can be a mechanical key or a touch key. The mobile phone 10 can receive key input and generate key signal input related to user settings and function control of the mobile phone 10.

[0215] The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompt or touch vibration feedback.

[0216] The indicator 192 can be an indicator light, which can be used to indicate the charging state, the power change, or can be used to indicate messages, missed calls, notifications, etc.

[0217] The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195 to realize contact and separation with the mobile phone 10.

[0218] It should be noted that the structure of the mobile phone 10 shown in FIG. 9 is only an example. In other embodiments, the mobile phone 10 can include more or fewer modules, and some modules can be combined or split, or other structures can be used, which are not limited herein.

[0219] It should be noted that the electronic device to which the image processing method of the embodiments of the present application is applicable can adopt the same or similar structure as the mobile phone 10, or a different structure from the mobile phone 10, which is not limited herein.

[0220] In possible implementation manners, the computer execution instruction in the embodiments of the present application can also be referred to as application program code, which is not limited in the embodiments of the present application.

[0221] The embodiments of the present application provide an electronic device, the memory of the electronic device can be used to store at least one program instruction, and the processor is used to execute the at least one program instruction to realize the technical solutions of the above method embodiments. The implementation principle and technical effects are similar to the above method related embodiments, which will not be repeated here.

[0222] The embodiment of the present application provides a computer program product, when the computer program product is executed in an electronic device, causes the electronic device to execute the technical solutions in the above embodiment. The implementation principle and technical effects are similar to the above related embodiments, and will not be repeated here.

[0223] The embodiment of the present application provides a computer readable storage medium, which stores program instructions, when the program instructions are executed in an electronic device, causes the electronic device to execute the technical solutions in the above embodiment. The implementation principle and technical effects are similar to the above related embodiments, and will not be repeated here.

[0224] It should be noted that in the embodiments of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", etc., and the sequence is not limited. Those skilled in the art can understand that "first", "second", etc. do not limit the number and execution order, and "first", "second", etc. do not necessarily mean different.

[0225] It should be noted that in the embodiments of the present application, "exemplary" or "for example" is used to represent an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner.

[0226] It should be noted that in the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The association relationship of the associated objects is described, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following three cases: A exists alone, A and B exist together, and B exists alone, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c, can represent a, b, c, a-b, a-c, b-c, or a-b-c, wherein a, b, and c can be single or multiple.

[0227] It should be noted that the embodiments of this application are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0228] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method applied to electronic devices, characterized in that, include: Run the first application; The rendering thread renders the first image data of the first application to obtain the second image data, and sends the first information to the compositing process before the first image data is rendered. In response to the first information, the synthesis process performs a first process, which includes processing of the second image data before synthesis; The compositing process performs the compositing processing on the second image data.

2. The method according to claim 1, characterized in that, The first information includes a first identifier of the first buffer of the second image data and a second identifier of the first fence, wherein the first fence is a software fence.

3. The method according to claim 2, characterized in that, The method further includes: When the hardware fence of the first buffer is released, the rendering thread sets the first fence to a released state.

4. The method according to claim 3, characterized in that, The compositing process performs the compositing processing on the second image data, including: The compositing process performs the compositing processing on the second image data while the first fence is in a released state.

5. The method according to any one of claims 2 to 4, characterized in that, The processing prior to compositing the second image data includes at least one of the following processing tasks: obtaining the first buffer from the graphics buffer queue of the electronic device, determining whether the first buffer is used for compositing the first image frame, and determining the compositing method of the first image frame; the compositing process includes compositing the first image frame through a kernel thread; wherein, the first image frame is the image frame on which the first image data is to be displayed.

6. The method according to claim 1, characterized in that, The method further includes: Displays the image frame obtained after synthesizing the second image data.

7. The method according to any one of claims 1 to 6, characterized in that, The rendering process includes performing the prepareTree stage, dequeueBuffer stage, queueBuffer stage, and beginDraw stage on the first image data; Furthermore, sending the first information to the compositing process before the first image data rendering is completed includes: The rendering thread sends the first information to the compositing process before the beginDraw stage is completed.

8. The method according to claim 7, characterized in that, The execution order of the prepareTree phase, the dequeueBuffer phase, the queueBuffer phase, and the beginDraw phase can be any of the following orders: The prepareTree stage, the dequeueBuffer stage, the queueBuffer stage, and the beginDraw stage; The dequeueBuffer stage, the prepareTree stage, the queueBuffer stage, and the beginDraw stage; The dequeueBuffer stage, the queueBuffer stage, the prepareTree stage, and the beginDraw stage.

9. The method according to claim 8, characterized in that, Sending the first information to the compositing process before the rendering of the first image data is completed includes: The rendering thread sends the first information to the compositing process during the queueBuffer stage or the dequeueBuffer stage.

10. The method according to claim 1, characterized in that, The synthesis process is the process corresponding to SurfaceFlinger.

11. An electronic device, characterized in that, include: Memory, used to store one or more programs; A processor for executing the one or more programs to cause the electronic device to implement the image processing method according to any one of claims 1 to 10.

12. A readable storage medium, characterized in that, The readable storage medium includes one or more programs that, when executed on an electronic device, cause the electronic device to perform the image processing method according to any one of claims 1 to 10.

13. A program product, characterized in that, When the program product is run on an electronic device, it enables the electronic device to implement the image processing method according to any one of claims 1 to 10.

Images