Image processing method and device

Abstract

The invention provides an image processing method and device, which can reduce the power consumption of a GPU (Graphics Processing Unit). The method is applied to the electronic equipment, the electronic equipment comprises a graphics processing unit (GPU) and a hardware synthesizer (HWC), and the method comprises the following steps: responding to a first operation at a first moment, the GPU obtains a plurality of layers, the plurality of layers comprise a first fuzzy layer, and the plurality of layers belong to part or all of M layers; the GPU performs superposition processing and fuzzy processing on at least one of the multiple layers except the first fuzzy layer to obtain a first layer; and the HWC performs superposition processing on the M image layers and the first image layer to obtain a first frame image.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and device. Background Art

[0002] The images displayed by electronic devices are usually composed of multiple layers. At present, there are two main methods of layer synthesis: Graphics Processing Unit (GPU) synthesis and hardware compositor (HWC) synthesis. Among them, GPU is a general-purpose image processing device. In addition to being used for synthesizing layers, it is also used to complete other graphics processing tasks, such as image blur processing; HWC is a special-purpose image processing device, usually used for synthesizing layers. Compared with GPU synthesis, HWC synthesis has the advantages of low power consumption and high efficiency.

[0003] When the processing capability of the HWC is insufficient, for example, if there is a layer that needs to be blurred among the multiple layers to be synthesized, the multiple layers need to be synthesized by both GPU synthesis and HWC synthesis to obtain the target image. However, such an image processing method consumes a lot of power in the GPU. Summary of the invention

[0004] The present application provides an image processing method and device, which can reduce the power consumption of a GPU.

[0005] In a first aspect, an image processing method is provided, which is applied to an electronic device, wherein the electronic device includes a graphics processor (GPU) and a hardware synthesizer (HWC), and the method includes: at a first moment, in response to a first operation, the GPU obtains multiple layers, the multiple layers include a first blurred layer, and the multiple layers belong to part or all of M layers; the GPU performs superposition processing and blur processing on at least one layer among the multiple layers except the first blurred layer to obtain a first layer; the HWC performs superposition processing on the M layers and the first layer to obtain a first frame image.

[0006] In the image processing method of the present application, the GPU obtains a blurred layer and at least one layer below the blurred layer, performs superposition processing and blur processing on the at least one layer to obtain a first layer; the HWC performs superposition processing on the M layers used to synthesize the first frame image and the first layer, and makes the first layer adjacent to the blurred layer, and the first layer is located below the blurred layer, thereby obtaining the first frame image. It can be seen that in such an image processing method, the GPU does not need to superimpose the blurred layer on the first layer. Compared with the GPU obtaining the first layer and then superimposing the blurred layer on the first layer, the power consumption of the GPU is lower.

[0007] It should be understood that the first moment may refer to the moment when the GPU described in this application acquires M layers and the first layer. The first operation may be an operation that causes a blurred layer to exist in the M layers included in the display interface (first frame image) of the electronic device. Exemplarily, the first operation may refer to the user's input operation, and the electronic device, in response to the input operation, needs to blur some of the layers included in the display interface. For example, the input operation may be an operation of the user adjusting the volume. In response to the input operation, the electronic device needs to display a volume bar. The interface of the electronic device displaying the volume bar includes M layers, and the volume bar is a first blurred layer. The first operation may also refer to the electronic device receiving a first message. After the electronic device receives the first message, it needs to display a pop-up message corresponding to the first message. For example, the electronic device receives a text message and needs to display a pop-up message of the text message. The interface of the electronic device displaying the pop-up message of the text message includes M layers, and the pop-up message of the text message is a first blurred layer. The first operation may also have no specific action. For example, when the electronic device plays a video and the electronic device displays the first image in the video, the background of the first image needs to present a blurred effect. Then, the interface of the electronic device displaying the first image includes M layers, and the first image is a first blurred layer. This application does not specifically limit the first operation.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the GPU performs superposition processing and blur processing on at least one layer among the multiple layers except the first blurred layer, including: the GPU performs superposition processing and blur processing on the first target area of ​​each layer among the at least one layer to obtain the first layer, and the first target area is determined based on the first blurred layer.

[0009] It should be understood that the first target area is a partial or entire area of ​​each layer in at least one layer. In this way, when the first target area is a partial area of ​​each layer in at least one layer, the GPU does not need to perform overlay processing on the entire area of ​​each layer in at least one layer, which can further reduce the power consumption of the GPU.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: at a second moment, in response to a second operation, the HWC obtains N layers and the first layer, the N layers include a second blurred layer, and the layer of the N layers located below the second blurred layer is the same as the at least one layer, and the second moment is later than the first moment; the HWC performs superposition processing on the N layers and the first layer to obtain a second frame image, the first layer and the second blurred layer are adjacent, and the first layer is located below the second blurred layer.

[0011] It should be understood that the second moment may be the moment when the HWC described in this application acquires N layers and the first layer. The second operation may be an operation that makes the blur effect presented in the interface (second frame image) displayed by the electronic device the same as the blur effect presented in the first frame image. The second operation is similar to the first operation, and please refer to the description above. Through the above technical solution, the N layers and the first layer can all be superimposed through the HWC to obtain the second frame image, without the GPU performing superimposition and blur processing on the image below the second blurred layer, which can further reduce the power consumption of the GPU.

[0012] The fuzzy area corresponding to the first fuzzy layer is the same as the fuzzy area corresponding to the second fuzzy layer.

[0013] In combination with the first aspect, in some implementations of the first aspect, the electronic device also includes an image synthesis service; before the HWC obtains N layers and the first layer, the method also includes: the image synthesis service determines whether a preset condition is met, and the preset condition includes at least one of the following: the number of the second blurred layers is one, or, when the first layer is obtained, the timer turned on by the image synthesis service has not timed out, or, the counter for recording the number of frames of the image synthesized using the first layer is less than or equal to a preset threshold; the HWC obtains N layers and the first layer, including: when the preset condition is met, the HWC obtains the N layers and the first layer.

[0014] In combination with the first aspect, in some implementations of the first aspect, the method further includes: if the preset condition is not met, the GPU obtains the second blurred layer and the layer below the second blurred layer; the GPU performs superposition processing and blurring processing on the layer below the second blurred layer to obtain a second layer; the HWC performs superposition processing on the N layers and the second layer to obtain the second frame image.

[0015] It should be understood that by using the first layer to synthesize the second frame of the image when the preset conditions are met, even if the image synthesis service determines that the first layer can be used to synthesize the image, the electronic device will not use the first layer to synthesize the image for a long time. Because the electronic device may make a misjudgment when judging whether the layer below the second blurred layer is the same as at least one layer through the image synthesis service, such a solution can reduce the phenomenon of image synthesis errors caused by misjudgment of the electronic device.

[0016] In combination with the first aspect, in certain implementations of the first aspect, the GPU performs superposition processing and blurring processing on the layers located below the second blurred layer, including: the GPU performs superposition processing and blurring processing on the second target area of ​​each layer in the layers located below the second blurred layer to obtain the second layer, and the second target area is determined based on the second blurred layer.

[0017] It should be understood that the second target area can be understood as the blurred area corresponding to the second blurred layer. The second target area is a partial or entire area of ​​each layer in the layers below the second blurred layer, so that the GPU does not need to perform overlapping processing on the entire area of ​​each layer in the layers below the second blurred layer, which can further reduce the power consumption of the GPU.

[0018] In combination with the first aspect, in some implementations of the first aspect, the method also includes: the HWC determines a synthesis mark of each of the M layers and the first layer based on the M layers, the first layer and the hardware specifications of the HWC, and the synthesis mark is a HWC synthesis mark or a GPU synthesis mark; the HWC performs an overlay process on the M layers and the first layer, including: when the synthesis mark of each of the M layers and the first layer is a HWC synthesis mark, the HWC performs an overlay process on the M layers and the first layer to obtain the first frame image.

[0019] In combination with the first aspect, in some implementations of the first aspect, the method further includes: when there is a synthesis mark of a target layer in the M layers and the first layer as GPU synthesis, the GPU performs an overlay process on the target layer to obtain a third layer; the HWC performs an overlay process on the third layer and the remaining layers to obtain the first frame image, and the remaining layers are the layers among the M layers and the first layer except the target layer.

[0020] It should be noted that, when the target layer includes the first blurred layer, since the GPU has already synthesized the first layer, the GPU will neither perform the superposition processing and blur processing on the layers below the first blurred image, nor synthesize the first layer in a local synthesis manner, but will directly perform the superposition processing on the target image. In addition, when the target image includes the first layer, since the first layer is a layer obtained after the blurring process, the image synthesis service will no longer control the GPU to perform the blurring process on the first layer.

[0021] In combination with the first aspect, in some implementations of the first aspect, the GPU acquires multiple layers, including: when the number of layers of the first blurred layer is one, the GPU acquires the multiple layers. In this way, the phenomenon of synthesizing an erroneous first frame image can be reduced.

[0022] In combination with the first aspect, in certain implementations of the first aspect, among the M layers, the at least one layer is all the layers located below the first blurred layer; the method also includes: the GPU obtains layer parameters of the M layers, and the layer parameters of the M layers include parameters used to represent the upper and lower positional relationship of the M layers; the GPU obtains multiple layers, including: the GPU uses the layer parameters of the M layers to obtain the multiple layers.

[0023] It should be understood that the layer parameters of the M layers include parameters for indicating the upper and lower positional relationship of the M layers, so that the GPU can obtain the first blurred layer and the layer (at least one layer) located below the first blurred layer. Among the multiple layers, the first blurred layer is located at the topmost layer, and the at least one layer is all the layers located below the first blurred layer among the M layers.

[0024] In a second aspect, an image processing device is provided, which is used to execute the method in any possible implementation of the first aspect. Specifically, the device includes a module for executing the method in any possible implementation of the first aspect.

[0025] In a third aspect, the present application provides another image processing device, including a processor, the processor is coupled to a memory, and can be used to execute instructions in the memory to implement the method in any possible implementation of the first aspect. Optionally, the device also includes a memory. Optionally, the device also includes a communication interface, and the processor is coupled to the communication interface.

[0026] In another implementation, the device is a chip configured in an electronic device. When the device is a chip configured in an electronic device, the communication interface may be an input / output interface.

[0027] In a fourth aspect, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method in any possible implementation of the first aspect.

[0028] In the specific implementation process, the processor may be a chip, the input circuit may be an input pin, the output circuit may be an output pin, and the processing circuit may be a transistor, a gate circuit, a trigger, and various logic circuits. The input signal received by the input circuit may be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit may be, for example, but not limited to, output to a transmitter and transmitted by the transmitter, and the input circuit and the output circuit may be the same circuit, which is used as an input circuit and an output circuit at different times. The embodiments of the present application do not limit the specific implementation methods of the processor and various circuits.

[0029] In a fifth aspect, a processing device is provided, comprising a processor and a memory. The processor is used to read instructions stored in the memory, and can receive signals through a receiver and transmit signals through a transmitter to execute the method in any possible implementation of the first aspect.

[0030] Optionally, the number of the processors is one or more, and the number of the memories is one or more.

[0031] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.

[0032] In the specific implementation process, the memory can be a non-transitory memory, such as a read-only memory (ROM), which can be integrated with the processor on the same chip or can be set on different chips respectively. This application does not limit the type of memory and the setting method of the memory and the processor.

[0033] It should be understood that the related data interaction process, for example, sending indication information may be a process of outputting indication information from a processor, and receiving capability information may be a process of receiving input capability information from a processor.

[0034] The processing device in the fifth aspect mentioned above can be a chip. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading the software code stored in the memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0035] In a sixth aspect, a computer program product is provided, the computer program product comprising: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute a method in any possible implementation of the first aspect.

[0036] In a seventh aspect, a computer-readable storage medium is provided, which stores a computer program (also referred to as code, or instructions) which, when executed on a computer, enables the computer to execute a method in any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure of an electronic device;

[0038] Figure 2 A schematic diagram of a process for displaying a target image for an electronic device;

[0039] Figure 3 A schematic diagram of a process for synthesizing a target image for an electronic device;

[0040] Figure 4 A schematic diagram of a flow chart of an image processing method;

[0041] Figure 5 A flowchart of an image processing method provided in an embodiment of the present application;

[0042] Figure 6 A schematic diagram of a process for synthesizing a first frame of image provided in an embodiment of the present application;

[0043] Figure 7 A schematic diagram of the upper and lower positional relationship of multiple layers provided in an embodiment of the present application;

[0044] Figure 8 A schematic diagram of an interface of a display screen of a mobile phone provided in an embodiment of the present application;

[0045] Fig. 9 A schematic diagram of a process of determining a method for synthesizing an image provided in an embodiment of the present application;

[0046] Fig.10 A flowchart of another image processing method provided in an embodiment of the present application;

[0047] Fig.11 A schematic block diagram of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0049] In the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first value and the second value are only used to distinguish between different values, and do not limit their order. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0050] It should be noted that in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0051] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0052] The electronic device provided in the embodiment of the present application is an electronic device with a touch screen, for example, a terminal device, specifically a mobile phone, a tablet computer (pad), a desktop computer, a laptop computer, etc. The embodiment of the present application does not limit the specific technology and specific device form used by the electronic device.

[0053] In order to better understand the electronic device in the embodiment of the present application, Figure 1 The hardware structure of the electronic device according to the embodiment of the present application is described in detail.

[0054] Figure 1 Schematic diagram of the structure of the electronic device 100 provided in the embodiment of the present application. Figure 1As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, an indicator 192, a camera 193, and a display screen 194, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, etc.

[0055] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0056] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Optionally, a memory may be provided in the processor 110 for storing instructions and data. Different processing units may be independent devices or integrated in one or more processors.

[0057] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0058] Among them, the I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL); the I2S interface can be used for audio communication; the PCM interface can also be used for audio communication to sample, quantize and encode analog signals; the UART interface is a universal serial data bus for asynchronous communication; the MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193; the MIPI interface includes the camera serial interface (CSI), the display serial interface (DSI), etc.; the GPIO interface can be configured by software; the GPIO interface can be configured as a control signal or as a data signal.

[0059] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, can also be used to transmit data between the electronic device 100 and a peripheral device, and can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0060] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0061] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. While the charging management module 140 is charging the battery 142, it can also power the electronic device through the power management module 141. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.

[0062] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0063] Among them, antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0064] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G etc. applied on the electronic device 100 .

[0065] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0066] The wireless communication module 160 can provide wireless communication solutions for application in the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0067] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0068] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0069] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0070] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0071] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0072] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0073] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0074] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0075] The internal memory 121 may be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory provided in the processor.

[0076] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0077] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0078] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power changes, messages, missed calls, notifications, etc.

[0079] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. Figure 2 , taking the layered architecture Android system as an example, the process of the electronic device 100 displaying the target image is illustrated.

[0080] To facilitate understanding, we first explain the layered architecture of the Android system and the terms involved.

[0081] like Figure 2 As shown, the layered architecture divides the software system of the electronic device 100 into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into seven layers, from top to bottom, namely, the application layer, the application framework layer, the system library, the system service layer, the hardware abstraction layer, the kernel layer, and the hardware layer.

[0082] 1. The application layer may include a series of application packages. For example, the application package may include camera, gallery, video, short message, desktop manager, system UI and application renderer, etc.

[0083] The user interface (UI) can be used to measure the size of the layer, determine the layout of the layer elements included in the layer, and set layer parameters, etc. The layer elements included in the layer can be understood as the layer content included in the layer, such as the value indicating the volume size and the button for adjusting the volume size included in the layer corresponding to the volume bar. The layout of the layer elements can be understood as the position parameters of each layer element included in the layer.

[0084] The application renderer may be used to draw the layer elements included in the layer according to the layer parameters set by the system UI and the layout of the layer elements included in the layer determined by the system UI.

[0085] 2. The Application Framework Layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a choreographer and a ViewRootImpl, etc.

[0086] Choreographers can manage the execution of application input, animation, drawing and other tasks according to the vertical synchronization signal (Vsync). For example, they can control the system UI to draw the layer content.

[0087] ViewRootImpl can be used to dispatch input events to applications in the application layer; it can also be used to create ViewRootImpl objects, which can also be understood as choreographer objects. For example, the choreographer can instruct the system UI to measure the size of the layer, determine the layout of the layer elements included in the layer, and set the layer parameters through the ViewRootImpl object.

[0088] 3. The system library can include multiple functional modules, such as input reader, input dispatcher, and buffer queue.

[0089] The input reader is used to obtain input events, and perform translation, assembly, packaging and other processing on the input events, and send the processed input events to the input distributor for distribution.

[0090] The input dispatcher is used to dispatch input events from the input reader to the window corresponding to the input event.

[0091] The cache queue is used to cache the layer content of each layer. The cache queue may include at least one cache, and each cache in the at least one cache may be used to cache one layer.

[0092] 4. The system service layer includes the surface flinger (SF), which is used to uniformly manage the display system services of the surface. It is mainly responsible for the establishment, control, management and creation of display channels of the layers.

[0093] The image synthesis service receives all Window layers as input, and calculates the position of each layer in the final composite image according to the Z-Order (the order of each layer on the Z axis, that is, the upper and lower position relationship between each layer, transparency, size, position and other parameters), and then sends it to the hardware synthesizer (HWC) or OpenGL to generate the final frame buffer (FrameBuffer), which is then displayed on the display.

[0094] Image compositing services include SF compositor binder, SF compositor timer, SF compositor, and SF compositor render engine.

[0095] Among them, the SF synthesizer binder can be used to wake up the SF synthesizer timer.

[0096] SF synthesizer timer, which can be used to wake up the SF synthesizer.

[0097] The SF synthesizer can be used to obtain layer parameters of multiple layers, determine whether there is a blurred layer in the multiple layers according to the layer parameters of the multiple layers, and determine the synthesis mark of the layer; for example, when the SF synthesizer determines that the synthesis mark of the layer is GPU synthesis, the layer is synthesized through the GPU.

[0098] The SF compositor render engine can be understood as the logical control module of the GPU. It can be used to call the GPU to synthesize layers, such as calling the GPU to blur or overlay layers.

[0099] 5. The hardware abstraction layer (HAL Layer) mainly connects the application framework layer and the kernel layer. HAL can include HWC, which can provide hardware support for image synthesis services.

[0100] The hardware abstraction layer includes HWC, which can be used to determine the synthesis strategy of multiple layers and to perform overlay processing on layers that require HWC synthesis.

[0101] 6. The kernel layer is a layer between hardware and software. In an exemplary embodiment, the kernel layer includes a display driver, a touch driver, etc.

[0102] 7. The hardware layer can include a variety of hardware, such as display screens, GPUs, etc.

[0103] It should be noted that the embodiments of the present application are described using the Android system as an example, but its basic principles are also applicable to electronic devices using other operating systems, and the present application does not limit the operating system used by the electronic device.

[0104] Based on the above Android system, the process of the electronic device 100 displaying the target image includes the following steps:

[0105] S201, after receiving the touch operation, the display screen sends the information corresponding to the touch operation to the touch driver (the information may also be a command event), and the touch driver processes the touch operation into a raw input event through the kernel layer (the raw input event may include touch coordinates, touch force, timestamp of the touch operation, type of the touch subject, etc.), and the raw input event is stored in the kernel layer. The kernel layer reports the raw input event to the input reader in the system library through the input processing library.

[0106] S202: The input reader translates, assembles, and packages the original input events, and sends the processed input events to the input distributor for distribution.

[0107] S203: The input distributor sends the input event to the ViewRootImpl object.

[0108] S204. The ViewRootImpl object sends the input event to the system UI.

[0109] S205: The system UI instructs the SF synthesizer to create layer X.

[0110] S206: The SF synthesizer creates a layer control object and sends the layer control object to the choreographer. The layer control object can be used to set layer parameters of layer X.

[0111] S207 . The choreographer sends the layer control object to the ViewRootImpl object.

[0112] S208 . The ViewRootImpl object sends the layer control object to the system UI. The system UI uses the layer control object to measure the size of layer X, determine the layout of layer elements included in layer X, and set layer parameters of layer X.

[0113] S209: The system UI calls the application renderer.

[0114] S210: The application renderer obtains a first cache from the cache queue, and caches the layer content of the drawn layer X in the first cache.

[0115] S211. The application renderer sends the layer parameters of layer X and the identifier of the first cache to the SF synthesizer binder, and the SF synthesizer binder obtains the layer content of layer X from the first cache.

[0116] S212 , the SF synthesizer binder requests a Vsync-SF signal to wake up the SF synthesizer.

[0117] S213: When the timer times out, the SF synthesizer timer is awakened, and the SF synthesizer is awakened. The SF synthesizer obtains layer parameters of at least one layer, and the at least one layer includes layer X.

[0118] It should be understood that the at least one layer is a plurality of layers included in the target image currently required to be displayed by the electronic device. The creation method of the layers other than layer X in the plurality of layers is similar to the creation method of layer X, which will not be repeated here.

[0119] S214. The SF compositor sends the layer parameters of the at least one layer that needs GPU synthesis to the SF compositor render engine, and the SF compositor render engine sets an instruction, which is used to instruct the GPU to perform asynchronous synthesis. Exemplarily, the SF compositor render engine sets an instruction for instructing the GPU to overlay the layer below the blurred layer that needs to be blurred, and to blur the overlaid layer.

[0120] S215: Further, the SF synthesizer render engine can call the glFlush function to instruct the GPU to perform GPU synthesis, and call a function to obtain a fence identifier for tracking whether the GPU synthesis is completed. The process of GPU synthesis is the process of executing the instructions set by the render engine.

[0121] Optionally, the fence identifier used to track whether GPU synthesis is completed includes a first state and a second state, the first state is used to indicate that GPU synthesis is completed, and the second state is used to indicate that GPU synthesis is not completed.

[0122] It should be understood that the state of the fence identifier used to track whether GPU synthesis is completed can be set by the GPU. For example, when GPU synthesis starts, the state of the fence identifier can be set to the second state; when GPU synthesis ends, the state of the fence identifier can be reset to the first state.

[0123] S216. The SF compositor render engine sends a fence identifier used to track whether GPU synthesis is completed to the SF compositor.

[0124] S217. The SF synthesizer sends the layer parameters of the GPU synthesized layer and the fence identifier used to track whether the GPU synthesis is completed to the hardware synthesizer.

[0125] It should be understood that the hardware compositor can determine whether the GPU composition is finished by tracking the state of the fence flag whether the GPU composition is finished.

[0126] S218. The hardware synthesizer may further superimpose the layer synthesized by the GPU and at least one layer other than the layer synthesized by the GPU to obtain a target image, and send image data of the target image to the display driver.

[0127] S219: The display driver can control the display screen to map the image data of the synthesized target image onto the screen.

[0128] In the embodiments of the present application, various terms and English abbreviations, such as GPU, SF, HWC, etc., are illustrative examples given for the convenience of description and shall not constitute any limitation to the present application. The present application does not exclude the possibility of defining other terms that can achieve the same or similar functions in existing or future protocols.

[0129] from Figure 2 It can be seen from the process of displaying the target image by the electronic device shown that the layer synthesis methods mainly include GPU synthesis and HWC synthesis. Among them, GPU is a general image processing device, which is used not only for synthesizing layers, but also for completing other graphics processing tasks, such as image blur processing, etc.; HWC is a special image processing device, usually used for synthesizing layers. Compared with GPU synthesis, HWC synthesis has the advantages of low power consumption and high efficiency.

[0130] When the processing capability of the HWC is insufficient, for example, if there is a blurred layer that needs to be blurred among the multiple layers that need to be synthesized, the multiple layers need to be synthesized by both GPU synthesis and HWC synthesis to obtain a target image.

[0131] Next, combine Figure 3 and Figure 4, when the target image to be synthesized by the electronic device includes a blurred layer, a process of synthesizing the target image by the electronic device is described.

[0132] In one example, the process 300 of synthesizing a target image by an electronic device may be as follows: Figure 3 As shown. Assume that the target image includes layer A, layer B, layer C, layer D and layer E in order from bottom to top. Among them, layer D is a blurred layer, that is, the layer parameters of layer D include blur parameters. For the upper and lower positional relationship of the multiple layers included in the target image, layer B is above layer A, which means that when the electronic device synthesizes the target image, layer B is superimposed on layer A, that is, when the electronic device displays the target image, the layer content included in layer B is displayed on the layer content included in layer A. Exemplarily, layer A can be the layer corresponding to the desktop wallpaper, and layer B can be the display window of the music application, and the display window of the music application is displayed on the desktop wallpaper.

[0133] The electronic device first overlays layer B on layer A through the GPU to obtain layer A+B; then overlays layer C on layer A+B to obtain layer A+B+C; based on the blur parameter of layer D, the electronic device can determine the area that needs to be blurred. Assuming that the blur parameter of layer D includes parameter 1 for indicating that the area that needs to be blurred is the area covered by layer D, in one implementation, layer D can be a transparent or semi-transparent layer, and the electronic device blurs the area corresponding to parameter 1 in layer A+B+C through the GPU to obtain layer 1; then overlays layer D on layer 1 through the GPU to obtain layer 2. In this way, since layer D is transparent or semi-transparent, in layer 2, the area covered by layer D in layer A+B+C can also present a blurred effect, so that in layer 2, layer D has a three-dimensional effect, which helps to improve the user's viewing experience.

[0134] The electronic device sends the layer parameters of layer 2 and layer E to HWC through the image synthesis service. HWC uses the layer parameters of layer 2 and layer E to determine the position of layer E above layer 2, obtain layer 2 and layer E, and finally superimpose layer E on layer 2 to obtain the target image.

[0135] Figure 4 4 is a flow chart of an image processing method 400. The method 400 is applied to an electronic device, and the hardware structure of the electronic device can be as follows: Figure 1 As shown, the software structure of the electronic device can be as follows Figure 2 As shown. Figure 4 As shown, the method 400 includes the following steps:

[0136] S401, the system UI thread instructs the SF synthesizer main thread to create layer 3. Layer 3 may refer to one or more layers. For example, when the system UI thread needs to create a layer corresponding to a volume application, layer 3 may be a layer corresponding to a volume bar; when the system UI thread needs to create a layer corresponding to a desktop manager, layer 3 may be a layer corresponding to a desktop wallpaper.

[0137] S402, the SF synthesizer main thread creates a layer control object of layer 3, and sends the layer control object of layer 3 to the system UI thread. The created layer control object of layer 3 can be used to set layer parameters of layer 3.

[0138] S403, the system UI thread draws layer 3.

[0139] Optionally, S403 may be implemented in the following manner: the system UI thread obtains the size of the display screen, determines the size of layer 3 and the layer content included in layer 3, and sets the layer parameters of layer 3. The layer parameters of layer 3 may include parameters for indicating the upper and lower positional relationship of layer 3. The layer content may refer to the elements included in layer 3. For example, when layer 3 is the layer corresponding to the volume bar, the layer content of layer 3 may include elements such as a numerical value for indicating the volume size and a volume adjustment button.

[0140] S404: The system UI thread calls the application rendering thread.

[0141] S405 , the application rendering thread applies for cache 1 from the cache queue, and cache 1 is used to store the layer content included in layer 3.

[0142] S406 , applying the rendering thread to draw the layer content included in layer 3 , and cache the layer content of layer 3 in cache 1 .

[0143] S407 , the application rendering thread sends the layer parameters of layer 3 to the SF synthesizer binder thread. The layer parameters of layer 3 include the cache identifier of cache 1.

[0144] S408, the SF synthesizer binder thread stores the layer parameters of layer 3 in the process parameters, and requests a Vsync-SF signal. The process parameters may refer to the parameters of the image synthesis service process.

[0145] It should be understood that the Vsync signal is a periodic signal, and the Vsync signal can be divided into a software Vsync signal and a hardware Vsync signal. The software Vsync signal includes a Vsync-SF signal. The Vsync-SF signal can be used to trigger the layer synthesis process.

[0146] S409: When the timer times out, the SF synthesizer timer thread is awakened.

[0147] S410. The SF synthesizer timer thread wakes up the SF synthesizer main thread.

[0148] S411. The SF synthesizer main thread obtains layer parameters of L layers.

[0149] It should be understood that the L layers include layer 3. The L layers may also include other layers, for example, layer 3 is a layer corresponding to the volume bar. When the electronic device displays the volume bar, the volume bar may include other L layers below and / or above it, for example, a layer corresponding to the desktop wallpaper. Therefore, in addition to layer 3, the SF synthesizer main thread also obtains the layer parameters of the other layers. The creation process and the determination process of the layer parameters of the other layers are similar to those of layer 3. Please refer to the above description and will not be repeated here.

[0150] Optionally, the SF synthesizer main thread may also merge the layer parameters and display parameters of the L layers, update the layer parameters of the L layers, and update the visible layers. The display parameters may include parameters such as rotation processing parameters. After S411, the electronic device further processes the L layers using the updated layer parameters of the L layers.

[0151] S412. The SF synthesizer main thread determines whether there is a blurred layer in the L layers according to the layer parameters of the L layers, and sets the synthesis mark of the blurred layer and the layers below the blurred layer to GPU synthesis.

[0152] S413. The SF synthesizer main thread sends the layer parameters of the L layers to the HWC thread.

[0153] S414, the HWC thread determines the composition tags of the remaining layers except for the layers with the composition tag of GPU composition. The composition tag of GPU indicates that the layer needs to be composed by GPU. The composition tag is GPU composition or HWC composition.

[0154] S415. The HWC thread sends the composition marks of the L layers to the SF compositor main thread.

[0155] S416, the SF synthesizer main thread applies for cache 2 from the cache queue, and cache 2 is used to cache the layers synthesized by the GPU.

[0156] S417. The SF synthesizer main thread sends the layer parameters of the layer marked as GPU synthesis and the identifier of cache 2 to the SF synthesizer render engine rendering thread.

[0157] Furthermore, the SF compositor render engine rendering thread sets the instructions corresponding to S418 to S422 according to the identifier of the cache corresponding to the layer marked as GPU synthesis, the layer parameters of the layer marked as GPU synthesis, and the identifier of cache 2, and calls the glFlush function to notify the GPU to perform asynchronous synthesis. The process of the GPU performing asynchronous synthesis is the process of sequentially executing the instructions corresponding to S418 to S422.

[0158] S418: Determine whether there is a blurred layer.

[0159] S419: If a blurred layer exists, the layers below the blurred layer are overlapped and blurred to obtain layer 4.

[0160] S420, store layer 4 in cache 2.

[0161] S421, overlay the blurred layer on layer 4, and continue to draw the layer on the blurred layer to obtain layer 5, and layer 5 is cached in cache 2.

[0162] S422. The SF synthesizer render engine rendering thread notifies the GPU to perform asynchronous synthesis, and obtains a first fence identifier for tracking whether the GPU synthesis is completed.

[0163] Optionally, the SF compositor render engine rendering thread obtains the first fence identifier through eglDupNativeFenceFDANDROID.

[0164] S423. The SF synthesizer render engine rendering thread sends the first fence identifier to the SF synthesizer main thread.

[0165] S424. The SF synthesizer main thread sends the first fence identifier and the cache identifier of cache 2 to the HWC thread.

[0166] S425. The HWC thread can determine whether the GPU asynchronous synthesis is completed based on the first fence identifier. In the case of GPU asynchronous synthesis, the HWC thread further overlays the layer marked as HWC synthesis with layer 5 to obtain the target image, and sends the image data of the synthesized target image to the display driver.

[0167] S426. The HWC thread sends the second barrier identifier used for tracking whether screen display is completed and the third barrier identifier used for tracking whether synthesis of the L layers is completed to the SF synthesizer main thread.

[0168] S427, the SF synthesizer main thread monitors the callback of display completion and updates the layer timestamp parameter. In addition, if there is a buffer that has not been consumed, a Vsync signal is requested to further consume the buffer through the image synthesis service and HWC.

[0169] It can be seen from process 300 and method 400 that when the target image to be synthesized by the electronic device includes a blurred layer, the blurred layer and the layers below the blurred layer need to be synthesized by GPU synthesis, that is, the electronic device needs to superimpose the layers below the blurred layer by the GPU to obtain the superimposed layer; blur the superimposed layer to obtain the blurred layer; and superimpose the blurred layer on the blurred layer. However, such a method causes higher power consumption of the GPU.

[0170] In order to solve the above technical problems, the present application provides an image processing method and device, so that after the electronic device overlaps and blurs the layers below the blurred layer to obtain the first layer, the blurred layer will no longer be overlapped on the first layer, thereby reducing the power consumption of the GPU.

[0171] Combine the following Figures 5 to 10 The technical solution of the present application and how the technical solution of the present application solves the above technical problems are described in detail with specific embodiments. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0172] The embodiments shown in this application can be executed by an electronic device, which can be a terminal device, such as a mobile phone, a tablet computer, a smart bracelet, etc. The specific form and quantity of each device shown are only examples and should not constitute any limitation on the implementation of the method provided in this application. Below, the image processing method of the embodiment of the present application is described in detail from the perspective of the interaction of modules inside the electronic device.

[0173] It should be understood that the electronic device can be the electronic device itself, or a chip, a chip system or a processor that supports the electronic device to implement the image processing method, or a logic module or software that can implement all or part of the functions of the electronic device. The hardware structure of the electronic device can be as follows: Figure 1 As shown, the software structure can be as follows Figure 2 As shown, this application does not make any specific limitation to this.

[0174] Figure 5 The flowchart of an image processing method 500 provided in an embodiment of the present application is shown in FIG. The method 500 is applied to an electronic device including an image synthesis service, a GPU, and a HWC. Figure 5 As shown, the method 500 includes the following steps:

[0175] S501. The image synthesis service sends layer parameters of M layers to the GPU, where M is an integer greater than 1.

[0176] S502: Based on layer parameters of the M layers, the GPU obtains multiple layers, where the multiple layers include a first blurred layer, and the multiple layers belong to some or all of the M layers.

[0177] S503: The GPU performs superposition processing and blurring processing on at least one layer among the multiple layers except the first blurred layer to obtain a first layer.

[0178] S504: The image synthesis service sends the layer parameters of the M layers and the first layer to the HWC.

[0179] S505: Based on the layer parameters of the M layers and the first layer, the HWC obtains the M layers and the first layer, and performs superposition processing on the M layers and the first layer to obtain a first frame image.

[0180] Combination Figure 6 The method 500 is described in detail. Figure 6 A schematic diagram of a first frame image synthesis process provided in an embodiment of the present application. M layers are all layers used to synthesize the first frame image, for example Figure 6 Layer A, layer B, layer C, layer D and layer E in the first blur layer may refer to a layer that needs to be blurred, for example Figure 6 The multiple layers may refer to a first blurred layer and a layer below the first blurred layer, wherein the layer below the first blurred layer is, for example, Figure 6 The GPU obtains the layers A, B, C, and D, and performs superposition and blur processing on the layers A, B, and C to obtain the first layer; the HWC obtains the layers A, B, C, D, E, and the first layer, and superimposes the layers A, B, C, D, E, and the first layer to obtain the first frame image.

[0181] The layer parameters may include a cache identifier, which is used to indicate the cache storing the layer. For example, the cache identifier may be a buffer ID. Based on the layer parameters of the M layers, the GPU may determine the cache identifier corresponding to each of the multiple layers, and obtain each layer from the cache corresponding to each of the multiple layers according to the cache identifier corresponding to each of the multiple layers, thereby obtaining multiple layers. Similarly, the way in which the HWC obtains the M layers and the first layer is similar to the way in which the GPU obtains multiple layers, which will not be described in detail here.

[0182] It should be understood that based on the layer parameters of M layers, the GPU obtains multiple layers, which is only an example of the implementation method of the GPU obtaining multiple layers; based on the layer parameters of M layers and the first layer, the HWC obtains M layers and the first layer, which is only an example of the HWC obtaining M layers and the first layer. The embodiments of the present application do not limit the specific methods of the GPU and the HWC obtaining layers.

[0183] For example, the image synthesis service sends layer parameters of multiple layers to the GPU, and the GPU can obtain the multiple layers based on the layer parameters of the multiple layers.

[0184] Optionally, the layer parameters of each layer in the M layers also include the layer identifier of each layer and the corresponding relationship between the layer identifier of each layer and the cache identifier corresponding to each layer. In this way, the electronic device can determine the cache identifier corresponding to each layer according to the corresponding relationship.

[0185] In addition, the layer parameters may also include parameters for indicating the upper and lower positional relationship of the layer, such as the Z-order parameter of the layer, which may be understood as the coordinate of the layer on the Z axis; wherein the Z axis may refer to an axis perpendicular to the display screen of the electronic device and pointing from the back of the electronic device to the display screen of the electronic device. For example, Figure 7 As shown, it is assumed that the screen displayed by the electronic device 701 includes layer A, layer B, layer C, layer D and layer E. Among them, the Z-Order parameter included in the layer parameters of layer A is 1, the Z-Order parameter included in the layer parameters of layer B is 2, the Z-Order parameter included in the layer parameters of layer C is 3, the Z-Order parameter included in the layer parameters of layer D is 4, and the Z-Order parameter included in the layer parameters of layer E is 5. Then the order of the five layers from bottom to top is: layer A, layer B, layer C, layer D, layer E.

[0186] It should be understood that in the embodiment of the present application, the upper and lower positional relationship of the layers can be determined by the parameters included in the layer parameters for indicating the upper and lower positional relationship of the layers; in addition, the GPU and HWC performing the overlay processing on at least two layers can be understood as the GPU and HWC overlaying at least two layers in sequence from bottom to top based on the upper and lower positional relationship of the at least two layers. In order to make the description more concise, this will not be elaborated below.

[0187] It should also be noted that the blurred layer is a layer that needs to be blurred. It does not mean that the electronic device needs to perform blur processing such as Gaussian blur on the layer, but that the layer below the layer needs to be blurred. The blurred layer can usually be transparent or semi-transparent. In this way, the electronic device blurs the layer below the layer and then superimposes the blurred layer on the layer obtained after the blur processing. The resulting image can show a blurred effect, making the blurred layer seen by the user more three-dimensional.

[0188] Optionally, the electronic device may determine whether the layer is a blurred layer by using the layer parameters of the layer. For example, if the layer parameters of layer X include a blur parameter, the electronic device may determine that layer X is a blurred layer. The blur parameter may include, but is not limited to, one or more of the following parameters: a blur radius parameter, transparency, a parameter for setting the blurred area to rounded corners, a blur area parameter, etc., which is not specifically limited in this application.

[0189] Optionally, the fuzzy parameter may further include a fuzzy switch identifier, which may be a first identifier for identifying turning on fuzzy processing and a second identifier for turning off fuzzy processing. In this way, the electronic device may determine whether the layer is a fuzzy layer according to the fuzzy switch identifier included in the fuzzy parameter of the layer.

[0190] It should be understood that in S503, blurring is performed after superposition, that is, after superposition is performed on at least one layer, blurring is performed on the layer obtained after superposition, but not on each layer in the at least one layer.

[0191] The first layer may also be called a blur cache layer. Blurring may refer to Gaussian blurring or Kawase blurring, etc. This application does not specifically limit the blurring method.

[0192] Optionally, the image synthesis service sets the layer parameters of the first layer, and sets the parameters representing the upper and lower positional relationship of the first layer in the layer parameters of the first layer, so that the first layer is adjacent to the first blurred layer and is located below the first blurred layer. In this way, when the HWC performs the superposition processing on the first layer and the M layers, the first layer can be adjacent to the first blurred layer and is located below the first blurred layer.

[0193] Optionally, if the GPU acquires multiple layers at a first moment, and the HWC acquires N layers and the first layer at a second moment, the second moment is later than the first moment.

[0194] In the image processing method of the present application, the GPU obtains a blurred layer and at least one layer below the blurred layer, performs superposition processing and blur processing on the at least one layer to obtain a first layer; the HWC performs superposition processing on the M layers used to synthesize the first frame image and the first layer, and makes the first layer adjacent to the blurred layer, and the first layer is located below the blurred layer, thereby obtaining the first frame image. It can be seen that in such an image processing method, the GPU does not need to superimpose the blurred layer on the first layer. Compared with the GPU obtaining the first layer and then superimposing the blurred layer on the first layer, the power consumption of the GPU is lower.

[0195] As an optional embodiment, S503 may be implemented in the following manner: the GPU performs superposition processing and blurring processing on the first target area of ​​each layer in at least one layer to obtain a first layer, and the first target area is determined based on the first blurred layer.

[0196] It should be understood that the first target area of ​​each layer may be the entire area or a portion of the area of ​​each layer. In this way, compared with superimposing the entire area of ​​each layer in at least one layer, the power consumption of the GPU can be further reduced.

[0197] It should be noted that at least one layer may include a layer that does not include the first target area. For example, at least one layer includes layer Y, layer Y is an icon, and the first target area refers to an area outside the area where the icon is located. If layer Y does not include the first target area, the GPU will not perform the overlay process on layer Y.

[0198] Optionally, the layer parameters of the first blur layer include a first parameter, and the first parameter is used to indicate the blur region corresponding to the first blur layer, that is, the region to be blurred. The first parameter may also be referred to as a blur region parameter, and the electronic device may determine the region to be blurred by the first parameter. Correspondingly, the first target region is determined based on the first blur layer, and it can be understood that the first target region is the region indicated by the first parameter. For example, Figure 8 As shown, assuming that the first blurred layer is layer 801 corresponding to the volume bar, and the area indicated by the first parameter included in the layer parameters of layer 801 is the area where layer 801 is located, the area covered by layer 801 needs to be blurred. Figure 6 , the area represented by the first parameter included in the layer parameters of layer D is the area covered by layer D, so the GPU performs superposition processing and blur processing on the areas corresponding to the first parameter in layer A, layer B, and layer C to obtain the first layer. It can be seen that the GPU can further reduce the power consumption of the GPU by superimposing a part of the area in at least one layer.

[0199] It should be understood that the above-mentioned way in which the electronic device synthesizes the first frame image can be called a local synthesis way. For the convenience of description below, the way in which the electronic device synthesizes the first frame image is called a local synthesis way.

[0200] After the electronic device synthesizes the first frame image, the first layer may also be used to synthesize an image subsequent to the first frame image. The following is an example of the electronic device synthesizing the second frame image.

[0201] It should be noted that the second frame image and the first frame image do not necessarily refer to two frames of images synthesized continuously by the electronic device, that is, after the electronic device synthesizes the first frame image, it can also synthesize at least one other frame image, and after synthesizing at least one other frame image, it synthesizes the second frame image.

[0202] Furthermore, after the electronic device synthesizes the first frame of image, it may not clear the cache corresponding to the first layer, that is, the electronic device retains the first layer and sets the first layer to be invisible. For example, the electronic device may set the first layer to be invisible by setting a visible parameter (visible=0), so that when the electronic device does not need to use the first layer to synthesize subsequent images, the first layer will not affect the synthesis effect of subsequent images. In addition, even if the first layer is set to be invisible, the layer parameters of the first layer include parameters for indicating the upper and lower positional relationship of the first layer, which remain unchanged, so that when the electronic device needs to use the first layer to synthesize subsequent images, the first layer is located below the blurred layer, and the first layer is adjacent to the blurred layer.

[0203] If the electronic device sets the first layer to be invisible by setting visible=0 after synthesizing the first frame image, then when the image synthesis service determines that the second frame image can be synthesized using the first image, the first layer can be set to be visible by setting visible=1, so that the second frame image can be subsequently synthesized using the first layer.

[0204] On the basis of the above embodiments, after S505, method 500 also includes: S506, when the layer located below the second blurred layer in the N layers is the same as at least one layer, the image synthesis service sends the layer parameters of the N layers and the layer parameters of the first layer to the HWC; S507, the HWC obtains the N layers and the first layer according to the layer parameters of the N layers and the layer parameters of the first layer; S508, the HWC performs superposition processing on the N layers and the first layer to obtain a second frame image, the first layer and the second blurred layer are adjacent, and the first layer is located below the second blurred layer.

[0205] It should be understood that the implementation of S507 and S508 is similar to that of S504 and S505, and reference may be made to the above description, which will not be repeated here.

[0206] Optionally, in S506, the layer located below the second blurred layer in the N layers is the same as at least one layer, which can be determined in the following manner.

[0207] The layer parameters may also include translation, rotation, and scaling parameters and high dynamic range imaging (HDR) parameters; the layer below the second blurred layer among the N layers is the same as at least one layer, which includes: the layer below the second blurred layer among the N layers and at least one layer satisfy a first preset condition, and the first preset condition includes: the number of layers below the second blurred layer among the N layers is the same as that of at least one layer; and, in an arrangement order from bottom to top or from top to bottom, the kth layer among the layers below the second blurred layer is the same as the kth layer among at least one layer, and k takes all integers between 1 and the number of layers of at least one layer.

[0208] Optionally, the kth layer in the layers below the second blurred layer is the same as the kth layer in at least one layer, including: in the order of arrangement from bottom to top or from top to bottom, the cache identifier of the cache corresponding to the kth layer in the layers below the second blurred layer is the same as the cache identifier of the cache corresponding to the kth layer in at least one layer; and, the translation, rotation and scaling parameters of the kth layer in the layers below the second blurred layer are the same as the translation, rotation and scaling parameters of the kth layer in at least one layer; and, the HDR parameters of the kth layer in the layers below the second blurred layer are the same as the HDR parameters of the kth layer in at least one layer. Among them, the translation, rotation and scaling parameters of the kth layer in the layers below the second blurred layer are the same as the translation, rotation and scaling parameters of the kth layer in at least one layer can mean that the position parameters of the upper left corner vertex of the kth layer in the layers below the second blurred layer are the same as the position parameters of the upper left corner fixed point of the kth layer in at least one layer.

[0209] Through the above scheme, the electronic device directly uses the first layer to synthesize the second frame image, that is, in the process of synthesizing the second frame image by the electronic device, there is no need for the GPU to perform superposition and blurring processing on multiple layers. In this way, the N layers and the first layer used to synthesize the second frame image can be superimposed through the HWC, so that the GPU power consumption is low and the synthesis efficiency is high. For example, when the second blurred layer is the layer corresponding to the pop-up message, compared with synthesizing the second frame image in the manner of process 300, the power consumption of the GPU can be reduced by about 30mA, the power consumption of the electronic device can be reduced by about 70mA, and the time consumed by the electronic device to synthesize the second frame image can be shortened by about 6ms.

[0210] It should be understood that the above-mentioned method of synthesizing the second frame image can be called a frame skipping method. For the convenience of description below, the method of synthesizing the second frame image through the above-mentioned scheme is called a frame skipping method.

[0211] In a possible implementation, before S506, method 500 also includes: the image synthesis service determines whether a preset condition is met, the preset condition including at least one of the following: the number of second blurred layers is one, or the timer started by the image synthesis service has not timed out when the first layer or the first frame image is obtained, or the counter for recording the number of frames of the image synthesized using the first layer is less than or equal to a preset threshold; S506 can be implemented in the following manner: when the preset condition is met, the image synthesis service sends the layer parameters of N layers and the first layer to the HWC.

[0212] It should be understood that the timer not timing out can be understood as the frame skipping method can be used to synthesize the image within the preset time length. The counter being less than or equal to the preset threshold can be understood as after the electronic device synthesizes the first frame of the image, when the number of frames of the image synthesized using the first layer is less than or equal to the preset threshold, the frame skipping method can be used to synthesize the image. The implementation process of the frame skipping method can refer to the above description and will not be repeated here.

[0213] Optionally, the preset condition may further include that a counter for recording the number of images synthesized after synthesizing the first frame of image is less than or equal to a first threshold.

[0214] By synthesizing the second frame of the image using the first layer when the preset conditions are met, even if the image synthesis service determines that the image can be synthesized using the first layer, the electronic device will not synthesize the image by skipping frames for a long time. Since the electronic device may make a misjudgment when judging whether the layer below the second blurred layer is the same as at least one layer through the image synthesis service, such a solution can reduce the phenomenon of image synthesis errors caused by misjudgment of the electronic device.

[0215] In another possible implementation, method 500 also includes: if the preset condition is not met, the GPU obtains the second blurred layer and the layer below the second blurred layer; the GPU performs superposition processing and blur processing on the layer below the second blurred layer to obtain the second layer; the HWC obtains N layers and the second layer, and performs superposition processing on the N layers and the second layer to obtain a second frame image.

[0216] It should be understood that when the preset condition is not met, the electronic device synthesizes the second frame image in a partial synthesis manner. The above-mentioned method of synthesizing the second frame image when the preset condition is not met is similar to the implementation of S501 to S505, which can be referred to the above description and will not be repeated here.

[0217] Based on the above embodiment, the GPU performs superposition processing and blurring processing on the layers below the second blurred layer, which can be implemented in the following way: the GPU performs superposition processing and blurring processing on the second target area of ​​each layer in the layers below the second blurred layer to obtain the second layer, and the second target area is determined based on the second blurred layer.

[0218] Among them, the second target area is determined according to the second fuzzy layer. It can be understood that the second target area is determined according to the layer parameters of the second fuzzy layer. The layer parameters of the second fuzzy layer can include the second parameter, and the second parameter is used to represent the fuzzy area corresponding to the second fuzzy layer.

[0219] It should be understood that the second target area has a similar meaning to the first target area. For understanding the second target area and the second parameter, reference may be made to the above description of the first target area and the first parameter, which will not be repeated here.

[0220] In addition, during the synthesis process of the first frame image, S505 can also be implemented in the following manner.

[0221] As an optional embodiment, before S505, method 500 also includes: HWC determines a synthesis mark of each layer in the M layers and the first layer based on the M layers, the first layer, and the hardware specifications of the HWC and / or the double data rate (DDR) bandwidth of the HWC, where the synthesis mark is a HWC synthesis mark or a GPU synthesis mark; S505 can be implemented in the following way: when the synthesis mark of each layer in the M layers and the first layer is a HWC synthesis mark, the HWC performs superposition processing on the M layers and the first layer to obtain a first frame image.

[0222] The DDR bandwidth can also be understood as the memory read and write bandwidth. The HWC determines the synthesis mark of each layer in the M layers and the first layer based on the M layers, the first layer, the hardware specifications of the HWC, and / or the DDR bandwidth of the HWC. It can also be understood that the HWC determines whether the hardware specifications of the HWC and / or the DDR bandwidth of the HWC can support the HWC to implement the superposition processing of the M layers and the first layer.

[0223] Optionally, the HWC determines a synthesis mark of each of the M layers and the first layer based on the M layers, the first layer, the hardware specifications of the HWC, and / or the DDR bandwidth of the HWC, including: the HWC determines a synthesis mark of each of the M layers and the first layer based on layer parameters of the M layers, the layer parameters of the first layer, the hardware specifications of the HWC, and / or the DDR bandwidth of the HWC.

[0224] It should be understood that the HWC can determine the size of each layer and the number of layers of the M layers and the first layer based on the layer parameters of the M layers and the layer parameters of the first layer; then the HWC can determine the number of layers that the HWC can perform superimposition processing and the layers that the HWC cannot synthesize based on at least one of the hardware capabilities such as the hardware specifications of the HWC, the DDR bandwidth of the HWC, the memory frequency, and the read and write capabilities of the HWC, thereby determining the layers that need to be synthesized by the GPU and the layers that need to be synthesized by the HWC. For example, when there is a target layer with a larger size among the M layers and the first layer, so that the memory or calculation required to superimpose the target layer exceeds the hardware capabilities of the HWC, the HWC can set the synthesis mark of the target layer and the layer adjacent to the target layer to GPU synthesis.

[0225] In a possible implementation, method 500 further includes: when a synthesis mark of a target layer exists in the M layers and the first layer as GPU synthesis, the GPU performs an overlay process on the target layer to obtain a third layer, and the number of layers of the target layer is greater than 1; the HWC obtains the third layer and the remaining layers, and performs an overlay process on the third layer and the remaining layers to obtain a first frame image, and the remaining layers are the layers other than the target layer in the M layers and the first layer.

[0226] It should be noted that, when the target layer includes the first blurred layer, since the GPU has already synthesized the first layer, the GPU will neither perform superposition processing and blur processing on the layers below the first blurred image in the manner shown in process 300, nor synthesize the first layer in a local synthesis manner, but directly perform superposition processing on the target image. In addition, when the target image includes the first layer, since the first layer is a layer obtained after blurring, the image synthesis service no longer controls the GPU to perform blurring processing on the first layer.

[0227] As an optional embodiment, S502 may be implemented in the following manner: when the number of layers of the first blur layer is one, the GPU acquires multiple layers.

[0228] It should be understood that the number of the first blurred layer is one, which can also be understood as the number of blurred layers in the M layers is one. When the number of blurred layers in the M layers is multiple, synthesizing the first frame image in a local synthesis manner may cause errors in the synthesized first layer, thereby synthesizing an erroneous first frame image. Therefore, when the number of the first blurred layer is one, synthesizing the first frame image in a local synthesis manner can reduce the power consumption of the GPU while making the accuracy of the first frame image higher.

[0229] It should be noted that when the number of layers of the first blurred layer is one, the GPU obtains multiple layers, which does not mean that the GPU will not obtain multiple layers when the number of layers of the first blurred layer is multiple; rather, it means that when the number of layers of the first blurred layer is one, the electronic device synthesizes the first frame image in a local synthesis manner.

[0230] As an optional embodiment, S503 can be implemented in the following manner: the GPU selects an area corresponding to the first parameter from each layer of at least one layer, and performs superposition processing, downsampling processing and blurring processing in sequence to obtain a blurred layer; the GPU is controlled by the image synthesis service to enlarge the blurred layer to obtain the first layer.

[0231] It should be understood that, through downsampling processing, the size of the layer obtained after the superposition processing can be reduced, for example, the size of the layer obtained after the superposition processing can be reduced to 1 / 8 of the original size, so that the size of the layer after the downsampling processing is 1 / 8 of the size of the layer obtained after the superposition processing; the enlargement processing can be understood as enlarging the size of the blurred layer to the original size, and the original size can be understood as the size of the layer obtained after the superposition processing, that is, the size of the electronic device before the downsampling processing. Through the above technical solution, compared with directly blurring the layer obtained after the superposition processing, since the size of the layer after the downsampling processing is smaller, the power consumption of the GPU for blurring the layer after the downsampling processing is smaller.

[0232] As an optional embodiment, the first blurred layer and / or the second blurred layer is in a transparent or semi-transparent state.

[0233] It should be understood that when the first blurred layer and / or the second blurred layer are in a transparent or semi-transparent state, after superimposing the first blurred layer and / or the second blurred layer on the first layer, the background of the first blurred layer and / or the second blurred layer can be displayed as a blurred effect.

[0234] Optionally, the layer parameters of the first blurred layer include a first transparency, and the layer parameters of the second blurred layer include a second transparency; the electronic device can set the first blurred layer to a transparent or translucent state through the first transparency, and / or the electronic device can set the second blurred layer to a transparent or translucent state through the second transparency.

[0235] Fig. 9 The present invention provides a process diagram of a method 900 for determining a synthesis mode of an image. The method 900 is applied to an electronic device including an image synthesis service, a GPU, and a HWC. The hardware structure of the electronic device can be as follows: Figure 1 As shown, the software structure of the electronic device can be as follows Figure 2The method 900 comprises the following steps:

[0236] Before the electronic device executes S901, the electronic device synthesizes the image using an existing method, and the existing method can be understood as the electronic device not synthesizing the image using a local synthesis method or a frame skipping method. For example, the image is synthesized using the method shown in process 300 or method 400, that is, when there is a blurred layer in the i layers of the synthesized target image, the GPU obtains the blurred layer and the j layers below the blurred layer; the GPU performs superposition and blurring processing on the j layers to obtain layer M; the GPU superimposes the blurred layer on layer M to obtain layer N; the HWC performs superposition processing on layer N and the layer above the blurred layer to obtain the target image. Wherein, j is a positive integer, and i is an integer greater than j.

[0237] S901. The image synthesis service obtains layer parameters of P layers used to synthesize image 1, where P is an integer greater than 1.

[0238] S902: Using the layer parameters of the P layers, the image synthesis service determines whether there is a blurred layer in the P layers.

[0239] If yes, the electronic device executes S903.

[0240] If not, the electronic device executes S912 and synthesizes image 1 using the existing method.

[0241] S903: The blurred layer among the P layers is a third blurred layer, and the image synthesis service determines whether the number of the third blurred layer among the P layers is one.

[0242] If yes, the electronic device executes S904.

[0243] If not, the electronic device executes S912 and synthesizes image 1 using an existing method.

[0244] S904. The image synthesis service starts a counter, which is used to record the number of frames of the image synthesized by the electronic device after the synthesis of the image 1. That is, after the synthesis of the image 1, each time the electronic device synthesizes another frame of the image, the value recorded by the counter is increased by 1.

[0245] S905: The image synthesis service determines to synthesize the image 1 in a partial synthesis manner. Furthermore, the electronic device may synthesize the image 1 in a partial synthesis manner.

[0246] It should be understood that Image 1 may be the first frame image in method 500 or other images. The local synthesis method may refer to the synthesis method of the first frame image above, which will not be described in detail here.

[0247] S906. The image synthesis service obtains layer parameters of Q layers used to synthesize image 2, where Q is an integer greater than 1.

[0248] S907: The image synthesis service determines whether there is a blurred layer among the Q layers.

[0249] If yes, the electronic device executes S908.

[0250] If not, the electronic device executes S912 and synthesizes image 2 using an existing method.

[0251] S908: The blurred layer among the Q layers is a fourth blurred layer, and the image synthesis service determines whether the number of the fourth blurred layer among the Q layers is one.

[0252] If yes, the electronic device executes S909.

[0253] If not, the electronic device executes S912 and synthesizes image 2 using an existing method.

[0254] S909: The image synthesis service determines whether the layers below the fourth blurred layer are the same as the layers below the third blurred layer.

[0255] It should be understood that the implementation of S909 is similar to the way in which the image synthesis service determines whether the layer below the second blurred layer is the same as the layer below the first blurred layer. Please refer to the above description and will not be repeated here.

[0256] If yes, the electronic device executes S910.

[0257] If not, the electronic device executes S912 and synthesizes image 2 using an existing method.

[0258] S910: The image synthesis service determines whether the counter is less than or equal to a preset threshold.

[0259] If so, the electronic device executes S911.

[0260] If not, the electronic device executes S912 and synthesizes image 2 using an existing method.

[0261] S911: The image synthesis service determines to synthesize the image 2 by frame skipping. Furthermore, the electronic device may synthesize the image 2 by frame skipping.

[0262] It should be understood that Image 2 may be the second frame image in method 500 or other images. The local synthesis method may refer to the synthesis method of the second frame image above, which will not be described in detail here.

[0263] Fig.10The following is a process diagram of an image processing method 1000 provided in an embodiment of the present application. The method 1000 is applied to an electronic device including an image synthesis service, a GPU, and a HWC. The hardware structure of the electronic device can be as follows: Figure 1 As shown, the software structure of the electronic device can be as follows Figure 2 The method 1000 comprises the following steps:

[0264] S11. The system UI thread instructs the SF synthesizer main thread to create layer 8.

[0265] S12. The SF synthesizer main thread creates a layer control object for layer 8 and sends the layer control object for layer 8 to the system UI thread.

[0266] S13. The system UI thread draws layer 8.

[0267] S14. The system UI thread calls the application rendering thread.

[0268] S15 . The application rendering thread applies for cache 3 from the cache queue. Cache 3 is used to store layer content included in layer 8 .

[0269] S16 , applying a rendering thread to draw the layer content included in layer 8 , and caching the layer content of layer 8 in cache 3 .

[0270] S17. The application rendering thread sends the layer parameters of layer 8 to the SF synthesizer binder thread.

[0271] S18, the SF synthesizer binder thread stores the layer parameters of layer 8 in the process parameters, and requests a Vsync-SF signal. The process parameters may refer to the parameters of the SF process.

[0272] S19. When the timer times out, the SF synthesizer timer thread is awakened.

[0273] S20. The SF synthesizer timer thread wakes up the SF synthesizer main thread.

[0274] S21. The SF synthesizer main thread obtains the layer parameters of M layers.

[0275] S22. The SF synthesizer main thread determines the number of layers of blur parameters included in the M layers according to the layer parameters of the M layers.

[0276] S23. When the number of layers of blur parameters included in the M layers is one, the SF synthesizer main thread applies for cache 4 from the cache queue.

[0277] S24, the SF synthesizer main thread sends the visible layer list, the layer parameters of the M layers, and the cache identifier of cache 4 to the SF synthesizer render engine rendering thread. The visible layer list may refer to a list of layer identifiers of layers that can be displayed when the electronic device displays the synthesized target image, that is, a list of layer identifiers of layers that the user can see.

[0278] S25, the SF synthesizer render engine rendering thread sets a first instruction, the first instruction is: select an area corresponding to the first parameter from each layer of at least one layer, and perform superposition processing and blur processing to obtain a first layer. The first blurred layer is a blurred layer in the M layers, and the at least one layer is all layers in the M layers that are located below the first blurred layer.

[0279] It should be understood that the implementation of S25 is similar to the implementation of S503, and reference may be made to the above description, which will not be repeated here.

[0280] S26. The SF synthesizer render engine rendering thread sets a second instruction, and the second instruction is: cache the first layer in cache 4.

[0281] It should be understood that in S25, after the GPU selects the area corresponding to the first parameter from each layer in at least one layer, the area in each layer in at least one layer except the area corresponding to the first parameter will be cleared, that is, the area in each layer in at least one layer except the area corresponding to the first parameter will not be stored in cache 4.

[0282] S27, calling the glFlush function to notify the GPU to execute the first instruction and the second instruction.

[0283] It should be understood that the SF synthesizer render engine rendering thread can be understood as a logic control module of the GPU, and the GPU can be called to execute instructions set by the SF synthesizer render engine rendering thread.

[0284] S28. The SF synthesizer render engine rendering thread obtains the fourth fence identifier used to track whether GPU synthesis is completed.

[0285] S29. The SF compositor render engine rendering thread sends the fourth fence identifier to the SF compositor main thread.

[0286] S30. The SF synthesizer main thread sets layer parameters of the first layer so that the layer parameters of the first layer include the fourth fence identifier and the cache identifier of cache 4.

[0287] Optionally, the layer parameters of the first layer may further include a layer identifier of the first layer and an association relationship between the layer identifier of the first layer and a cache identifier of the cache 4 .

[0288] S31. The SF synthesizer main thread sets the parameters included in the layer parameters of the first layer for indicating the upper and lower positional relationship of the first layer according to the parameters included in the layer parameters of the first blurred layer for indicating the upper and lower positional relationship of the first blurred layer, so that the first layer is located below the first blurred layer.

[0289] S32. The SF synthesizer main thread traverses the parameters of each layer in the M layers and the first layer for indicating the upper and lower position relationship of the first layer, and determines that the first layer is located below the first blurred layer, and the first layer is adjacent to the first blurred layer.

[0290] S33. The SF synthesizer main thread sends the layer parameters of M layers and the first layer to the HWC thread.

[0291] Optionally, when the SF compositor main thread sets the composition tags of the M layers and part or all of the first layer to GPU composition, the SF compositor main thread sends the correspondence between the layer identifiers of the layers whose composition tags are set to GPU composition and the composition tags of GPU composition to the HWC thread.

[0292] S34, the HWC thread determines a synthesis strategy. The synthesis strategy can be understood as a synthesis mark of each layer in the M layers and the first layer.

[0293] Optionally, when the HWC thread receives the correspondence between the layer identifier of the layer marked as GPU synthesis and the synthesis tag of GPU synthesis from the SF compositor main thread, it further determines whether there are layers that need to be GPU synthesized in the M layers and the remaining layers in the first layer except the layer marked as GPU synthesis, and sets the layer tags of the layers that need to be GPU synthesized in the remaining layers to GPU synthesis; then the HWC sends the synthesis strategy to the SF compositor main thread; the SF compositor main thread can apply for cache 5 to instruct the SF compositor to render The engine rendering thread calls the GPU to perform overlay processing on the layers marked as GPU synthesis, and caches the new layers obtained after the overlay processing in cache 5; the SF synthesizer main thread sets the layer parameters of the new layer, the layer parameters of the new layer include parameters for indicating the upper and lower position relationship of the new layer and the cache identifier of cache 5, and sends the layer parameters of the new layer to the HWC thread. The HWC thread can determine the upper and lower position relationship between the M layers and the remaining layers in the first layer except the layers marked as GPU synthesis and the new layer according to the layer parameters of the new layer, and overlay the remaining layers except the layers marked as GPU synthesis with the new layer according to the upper and lower position relationship between the remaining layers and the new layer to obtain the target image. Among them, even if the layers marked as GPU synthesis include the first blurred layer, the SF synthesizer main thread will not instruct the SF synthesizer render engine rendering thread to blur the layers below the first blurred layer.

[0294] S35. In the absence of a layer marked as GPU synthesis, the HWC thread superimposes the first layer and the M layers to obtain a first frame image, and sends image data of the first frame image to the display driver.

[0295] S36. The HWC sends a second barrier identifier for tracking whether screen display is completed and a third barrier identifier for tracking whether synthesis of multiple layers is completed to the SF synthesizer main thread.

[0296] S37, the SF synthesizer main thread monitors the callback of display completion and updates the layer timestamp parameter. In addition, if there is a cache that has not been consumed, a Vsync signal is requested to further consume the cache through the image synthesis service, GPU and HWC.

[0297] It should be understood that in the embodiments of the present application, the types of parameters included in the layer parameters are only examples, and the layer parameters may also include more types of parameters, which are not specifically limited in the present application.

[0298] It should also be understood that the order of execution of the above methods does not necessarily mean the order in which they are executed; the order in which the methods are executed should be determined by their functions and internal logic.

[0299] Combination of the above Figures 3 to 10 , describes in detail the image processing method of the embodiment of the present application, and then combines Fig.11 , describes in detail the image processing device of an embodiment of the present application.

[0300] Fig.11 A schematic block diagram of an image processing device 1100 provided in an embodiment of the present application. The device 1100 includes a processor 1101, a communication interface 1102, and a memory 1103. Among them, the processor 1101, the communication interface 1102, and the memory 1103 communicate with each other through an internal connection path, the memory 1103 is used to store instructions, and the processor 1101 is used to execute the instructions stored in the memory 1103. The communication interface 1102 can be used to send signals to other devices (such as the processor 1101 or the touch screen of an electronic device), and can also be used to receive signals from other devices (such as the memory 1103). Exemplarily, the communication interface 1102 reads the instructions stored in the memory 1103 and sends the instructions to the processor 1101.

[0301] It should be understood that the device 1100 can be specifically an electronic device in the above-mentioned embodiment, and can be used to execute the various steps and / or processes corresponding to the electronic device in the above-mentioned method embodiment. Optionally, the memory 1103 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type. The processor 1101 may be used to execute instructions stored in the memory, and when the processor 1101 executes instructions stored in the memory, the processor 1101 is used to execute the various steps and / or processes of the above-mentioned method embodiment.

[0302] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0303] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0304] The present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to implement the method shown in the above method embodiment.

[0305] The present application also provides a computer program product, which includes a computer program (also referred to as code or instruction). When the computer program runs on a computer, the computer can execute the method shown in the above method embodiment.

[0306] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0307] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0308] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0309] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0310] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0311] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0312] The above is only a specific implementation of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method is applied to an electronic device, the electronic device comprising a graphics processor GPU and a hardware synthesizer HWC, and the method comprises: At a first moment, in response to a first operation, the GPU acquires a plurality of layers, the plurality of layers including a first blurred layer, and the plurality of layers belong to some or all of the M layers; The GPU performs a superposition process and a blur process on at least one layer among the multiple layers except the first blurred layer to obtain a first layer; The HWC performs superposition processing on the M layers and the first layer to obtain a first frame image.

2. The method according to claim 1, characterized in that The GPU performs superposition processing and blur processing on at least one layer among the multiple layers except the first blurred layer, including: The GPU performs superposition processing and blurring processing on a first target area of ​​each layer in the at least one layer to obtain the first layer, and the first target area is determined according to the first blurred layer.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: At a second moment, in response to a second operation, the HWC acquires N layers and the first layer, the N layers include a second blurred layer, a layer below the second blurred layer among the N layers is the same as the at least one layer, and the second moment is later than the first moment; The HWC performs superposition processing on the N layers and the first layer to obtain a second frame image, wherein the first layer and the second blurred layer are adjacent, and the first layer is located below the second blurred layer.

4. The method according to claim 3, characterized in that The electronic device also includes an image synthesis service; Before the HWC acquires N layers and the first layer, the method further includes: The image synthesis service determines whether a preset condition is satisfied, wherein the preset condition includes at least one of the following: the number of the second blurred layers is one, or the timer started by the image synthesis service has not timed out when the first layer is obtained, or the counter for recording the number of frames of the image synthesized using the first layer is less than or equal to a preset threshold; The HWC acquires N layers and the first layer, including: When the preset condition is met, the HWC obtains the N layers and the first layer.

5. The method according to claim 4, characterized in that The method further comprises: If the preset condition is not met, the GPU obtains the second blurred layer and a layer below the second blurred layer; The GPU performs superposition processing and blurring processing on the layer located below the second blurred layer to obtain a second layer; The HWC performs superposition processing on the N layers and the second layer to obtain the second frame image.

6. The method according to claim 5, characterized in that The GPU performs a superposition process and a blur process on the layer located below the second blurred layer, including: The GPU performs superposition processing and blurring processing on the second target area of ​​each layer in the layers below the second blurred layer to obtain the second layer, and the second target area is determined based on the second blurred layer.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The HWC determines, based on the M layers, the first layer, and hardware specifications of the HWC, a synthesis mark of each layer in the M layers and the first layer, where the synthesis mark is a HWC synthesis mark or a GPU synthesis mark; The HWC performs a superposition process on the M layers and the first layer, including: In the case that the synthesis mark of each layer in the M layers and the first layer is a HWC synthesis mark, the HWC performs superposition processing on the M layers and the first layer to obtain the first frame image.

8. The method according to claim 7, characterized in that The method further comprises: In a case where a synthesis mark of a target layer exists in the M layers and the first layer as GPU synthesis, the GPU performs a superposition process on the target layer to obtain a third layer; The HWC performs superposition processing on the third layer and the remaining layers to obtain the first frame image, where the remaining layers are the M layers and the layers in the first layer except the target layer.

9. The method according to any one of claims 1 to 8, characterized in that The GPU acquires multiple layers, including: When the number of layers of the first blur layer is one, the GPU obtains the multiple layers.

10. The method according to any one of claims 1 to 9, characterized in that Among the M layers, the at least one layer is all layers located below the first blurred layer; The method further comprises: The GPU acquires layer parameters of the M layers, where the layer parameters of the M layers include parameters for indicating a vertical position relationship of the M layers; The GPU acquires multiple layers, including: The GPU uses the layer parameters of the M layers to obtain the multiple layers.

11. An electronic device, characterized in that: include: A processor, wherein the processor is coupled to a memory, wherein the memory is used to store a computer program, and when the processor calls the computer program, the device is caused to execute the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1 to 10.

Images