Image processing method and electronic equipment

Abstract

The invention provides an image processing method and electronic equipment, and relates to the technical field of display. The method comprises the steps of obtaining a first image; the first image comprises a first image layer and a second image layer, and the first image layer carries fuzzy attribute information; performing fuzzy processing on the second image layer to obtain a fuzzy result image layer, and storing the fuzzy result image layer in a first cache; a second image is obtained, the second image comprises a third image layer, and the third image layer carries fuzzy attribute information; and performing synthesis processing on the fuzzy result layer and the third layer through a hardware synthesizer HWC. The second image multiplexes the blurring processing result of the first image, blurring processing does not need to be carried out, it is avoided that due to blurring processing in the layer synthesis process, layer synthesis and blurring processing are forcibly carried out through a GPU, and therefore power consumption of the electronic device caused by the fact that layer synthesis and blurring processing are carried out through the GPU is reduced, and the user experience is improved. And the performance of the electronic equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to an image processing method and electronic equipment. Background Art

[0002] People's daily lives are inseparable from a variety of electronic devices, and many electronic devices in daily life have display interfaces. With the development of science and technology, many electronic devices with display interfaces (such as mobile phones and tablets) have realized background blur functions. For example, background blur functions are applied in scenarios such as notification bar or status bar pull-down, pop-up window, desktop folder, lock screen, etc. This function blurs the background to highlight the beauty of the current interface and enhance the user experience.

[0003] The background blur function of electronic devices is generally divided into static blur and dynamic blur. In order to effectively improve the user's visual experience, the background blur function in current technology generally adopts dynamic blur, also known as real-time blur. Real-time blur will pass each frame of the background of the electronic device as a data source to the GPU for synthesis and blurring operations, and then crop the blurred background according to the scene requirements.

[0004] The current technology has the problem that the power consumption of electronic devices is high, the performance of electronic devices is reduced, and the user experience is seriously affected. Summary of the invention

[0005] The present application provides an image processing method and an electronic device, which reduce power consumption when implementing real-time blurring, improve the performance of the electronic device, and enhance the user experience.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, the present application provides an image processing method, the method comprising: obtaining a first image; the first image comprises a first layer and a second layer, wherein the first layer carries blur attribute information; blurring the second layer to obtain a blur result layer, and storing the blur result layer in a first cache; obtaining a second image, wherein the second image comprises: a third layer, and the third layer carries blur attribute information; synthesizing the blur result layer and the third layer through a hardware synthesizer HWC. The first image is blurred and the blurred layer is stored, and the second image reuses the blurred result layer of the first image, that is, reuses the blurred result of the first image, and does not need to be blurred, thereby avoiding the forced layer synthesis and blurring by the GPU during the layer synthesis process due to the blurring, and directly synthesizing the layer of the second image and the blur result layer through the HWC to realize the blurring of the first image, thereby reducing the power consumption of the electronic device caused by the layer synthesis and blurring by the GPU, improving the performance of the electronic device, and thus improving the user experience.

[0008] In a possible implementation, an adjustment operation for the volume of an electronic device is received, and in response to the adjustment operation for the volume of the electronic device, a first image is acquired, wherein the first layer in the first image is a volume bar layer. That is, when the volume of the electronic device is adjusted, a first image including the volume bar layer is acquired.

[0009] In a possible implementation, when a message window pops up on the electronic device, the first image is acquired, and the first layer of the first image is the pop-up window layer. That is, when the electronic device receives a notification message, the electronic device will notify through a message pop-up window, and the first image including the pop-up window layer will be acquired.

[0010] In a possible implementation, a pull-down operation on the status bar is received; in response to the pull-down operation on the status bar, a first image is acquired, and the first layer of the first image is the status bar display layer. That is, when the user pulls down the status bar of the electronic device, the electronic device acquires the first image including the status bar display layer in response to the pull-down operation on the status bar.

[0011] In a possible implementation, a shutdown operation of an electronic device is received; in response to the shutdown operation of the electronic device, a first image is acquired, and the first layer of the first image is a shutdown layer. That is, when the user shuts down the electronic device, the electronic device acquires the first image including the shutdown layer in response to the shutdown operation.

[0012] In a possible implementation, the second image is the next N frames of the first image, where N is a positive integer. After the first image is blurred to obtain a blurred result layer, the next N frames of the first image reuse the blurred result layer. The larger N is, the more images reuse the blurred result layer. For scenes with lower real-time requirements, N can be slightly larger, which can further reduce the power consumption problem caused by the GPU.

[0013] In a possible implementation, the number of the first layer and the third image is 1, and the shape, size and relative position of the first layer and the third image are the same. The third layer is a layer that does not change compared to the first layer, and the application scope of the image processing method provided by the present application is when the number of layers carrying fuzzy attribute information is only 1.

[0014] In a possible implementation, the second layer is located below the first layer, and in the process of blurring and compositing the first and second layers by the GPU, the GPU blurs the second layer to obtain a blurring result, and stores the blurring result in the blurring result layer. That is, in the original layer compositing process, only the blurring result is stored in the blurring result layer, which has little impact on the original layer compositing process and will not affect the display of the electronic device.

[0015] In a possible implementation, the third layer and the blur result layer are synthesized by a hardware synthesizer HWC in the order that the blur result layer is located in the next layer of the third layer. The blur result layer is ensured to be located close to the third layer, and the third layer and the blur processing layer are synthesized by HWC, so as to ensure the reuse of the blur processing result stored in the blur result layer, so as to achieve the blur processing effect of the second image reused by the first image.

[0016] In a possible implementation, after acquiring the first image, the method further includes: determining that the first image is in a first state according to a state of the first layer and a state of a previous frame of the first image. The first state represents that the first image is blurred to obtain a blurred result layer, so that the blurred result layer can be directly reused when subsequent images are layered.

[0017] In a possible implementation, the first image is determined to be in the first state according to the layer parameters of the first layer and the state of the previous frame of the first image. The layer parameters of the first layer can intuitively reflect the state of the first layer, simplifying the complexity of determining the state of the first image.

[0018] In a possible implementation, the parameters of the first layer include: the number of first layers and the first layer change parameter. When the number of the first layer is 1, the first layer change parameter indicates that the first layer is in a stable stage, and the previous frame of the first image is in the third state, the first image is determined to be in the first state; or when the number of the first layer is 1, the first layer change parameter indicates that the first layer is in a stable stage, the previous frame of the first image is in the second state, and the images in the second state accumulate to N frames, the first image is determined to be in the first state. After N consecutive frames of images in the second state, the first image is determined to be in the first state.

[0019] In a possible implementation, when the previous frame of the first image is in the third state, before determining that the first image is in the first state, the stable frame parameters carried by the previous frame of the first image are obtained, and it is determined that the number of third images indicated by the stable frame parameters reaches M. The third image includes a first layer, and the number of first layers included in the third image is 1, and the first layer is in a stable stage. That is, after continuously determining that the number of first layers of M frames is 1 and the third image is in a stable stage, the first image is determined to be in the first state, so as to avoid executing the synthesis process corresponding to the first state due to a false stability phenomenon, and there is no subsequent waste of resources caused by the corresponding second state, thereby avoiding increased power consumption losses.

[0020] In a possible implementation, when the number of the first layers is 1, the first layer change parameter indicates that the first layer is in a stable stage, and the previous frame image of the first image is in the third state, the stable frame parameter carried by the second image is obtained, and when the stable frame parameter indicates that the number of third images has not reached M, it is determined that the first image is in the third state, that is, the third image, and the first image executes the original synthesis process.

[0021] In a possible implementation, after acquiring the second image, the method further includes: determining that the second image is in a second state according to the state of the third layer and the state of the previous frame of the second image. The second state represents that the second image synthesizes the third layer and the blurred result layer through the HWC, and does not need to perform blur processing and layer synthesis through the GPU, thereby improving the performance of the electronic device and thus improving the user experience.

[0022] In a possible implementation, the second image is determined to be in the second state according to the layer parameters of the third layer and the state of the previous frame of the second image. The layer parameters of the third layer can intuitively reflect the state of the third layer, simplifying the complexity of determining the state of the first image.

[0023] In a possible implementation, the layer parameters of the third layer include: the number of third layers and a third layer change parameter; when the number of the third layer is 1, the third layer change parameter indicates that the third layer is in a stable stage, and the previous frame image of the second image is in the first state, the second image is determined to be in the second state; or when the number of the third layer is 1, the third layer change parameter indicates that the third layer is in a stable stage, the previous frame image of the second image is in the second state, and the image accumulation of the second state does not reach N frames, the second image is determined to be in the second state, where N is a positive integer.

[0024] In a possible implementation, a target image obtained by synthesizing the blurred result layer and the third layer is sent for display, so that the electronic device can normally display the target image with the blurred result.

[0025] In a second aspect, the present application provides an electronic device, which includes a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the method of the first aspect above.

[0026] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed, the method of the first aspect described above is implemented.

[0027] In a fourth aspect, the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the order of multiple surfaces on the Z axis provided in an embodiment of the present application;

[0029] Figure 2A A schematic diagram of a partially blurred interface of an electronic device provided in an embodiment of the present application;

[0030] Figure 2B A schematic diagram of a globally blurred interface of an electronic device provided in an embodiment of the present application;

[0031] Figure 2C A schematic diagram of another scenario of global blurring of an electronic device provided in an embodiment of the present application;

[0032] Figure 2D A schematic diagram of another scenario of global blurring of an electronic device provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of a layer of a local interface of an electronic device provided in an embodiment of the present application;

[0034] Figure 4 A simplified flowchart of a layer synthesis method for a frame of display image provided in an embodiment of the present application;

[0035] Figure 5A A schematic diagram of path tracking in a layer synthesis process without blurring provided in an embodiment of the present application;

[0036] Figure 5B A schematic diagram of path tracking in a layer synthesis process when blurring is present provided in an embodiment of the present application;

[0037] Fig. 6A A schematic diagram of the composition of an electronic device provided in an embodiment of the present application;

[0038] Figure 6B A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0039] Figure 7 A flowchart of an image processing method provided in an embodiment of the present application;

[0040] Fig. 8A A schematic diagram of an interface of an electronic device without a fuzzy layer provided in an embodiment of the present application;

[0041] Figure 8B A schematic diagram of an interface of an electronic device having a blurred layer provided in an embodiment of the present application;

[0042] Figure 8C A schematic diagram of an interface of an electronic device with two blurred layers provided in an embodiment of the present application;

[0043] Fig. 9 A schematic diagram of a volume bar layer change provided in an embodiment of the present application;

[0044] Fig. 10A A schematic diagram of a process flow of a layer synthesis method for copying a frame image provided in an embodiment of the present application;

[0045] Fig. 10B A schematic diagram of a process for synthesizing layers of a frame skipping image provided in an embodiment of the present application;

[0046] Fig.11 A schematic diagram of the arrangement of the copy frame state and the frame skip state provided in an embodiment of the present application;

[0047] Fig.12 A schematic diagram of an arrangement for determining the state of each frame of an image provided in an embodiment of the present application;

[0048] Fig.13A schematic diagram of an interaction for implementing object initialization and layer parameter collection provided in an embodiment of the present application;

[0049] Fig.14 An interactive schematic diagram of implementing the decision-making part of a state machine provided in an embodiment of the present application;

[0050] Fig.15 A schematic diagram of the interaction between a layer synthesis method and a position control part provided in an embodiment of the present application;

[0051] Fig.16 An interactive schematic diagram of implementing the blurred drawing part of the copy frame provided in an embodiment of the present application;

[0052] Fig.17 A flowchart of another image processing method provided in an embodiment of the present application;

[0053] Fig.18 A schematic diagram of path tracking for the layer synthesis process under different image processing schemes provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0055] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0056] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given:

[0057] Layer (surface): Each application may correspond to one or more graphical interfaces, and a graphical interface may correspond to at least one layer, called a surface. Each graphical interface has its position, size, and content to be displayed on the screen. A surface corresponding to a graphical interface can be understood as a container corresponding to the graphical interface, which is used to store the control information of the graphical interface (for example: the size, position, etc. of the graphical interface on the screen) and / or the data of the graphical interface (i.e., the content that the graphical interface needs to display). Each surface has corresponding layer properties, which may include the position, size, and other information of the corresponding layer.

[0058] Multiple layers (surfaces) may overlap, and the stacking relationship between layers (surfaces) can be described by Z-order. There is a Z-axis in the vertical outward direction of the screen plane. All layers (surfaces) are ordered according to the coordinates on the Z-axis, that is, describing the up-down covering relationship between each layer (surface). This order on the Z-axis can be called Z-order.

[0059] For ease of understanding, the following Figure 1 The order of multiple layers on the Z axis is shown as an example. In the order from top to bottom when displayed on the screen, the screen includes surface0, surface1 and surface2. Surface0 partially covers surface1, and surface1 partially covers surface2. The Z-order of surface0, surface1 and surface2 is represented as: 3, 2, 1 respectively.

[0060] Buffer queue: Each layer has a corresponding buffer queue, which is used to store the display data of the corresponding layer. The display data is used to indicate the display content of the corresponding layer, that is, the display data of the layer can be the image obtained after rendering the layer.

[0061] Typically, a buffer queue is an important data transfer component for a display system. The buffer queue connects a component that can generate a graphics data buffer (i.e., a producer, such as a surface) to a component that receives data for display or further processing (i.e., a consumer, such as a surfaceflinger).

[0062] Layer: A layer (surface) corresponds to a layer on the Z axis, called a layer. A layer (layer) provides operations on the control information of the layer (surface) corresponding to the layer (layer), as well as processing of the content of the layer (surface). The layer (layer) is the basic operation unit for the layer management module (surface flinger) to synthesize. The layer (layer) is created inside the layer management module (surface flinger) when the application requests to create a layer (surface). The layer (layer) corresponds to the surface (surface) one by one.

[0063] Furthermore, both the surface and the layer are components of the graphics buffer, which consists of the surface, buffer queue, layer, and graphic buffer. When an application needs to draw an image, it creates a surface and a layer. The producer (e.g., Skia, OpenGL, etc.) obtains a buffer (graphic buffer) through the surface and draws the layer on the graphic buffer. When the drawing is completed, the surface returns the graphic buffer to the buffer queue. After that, the consumer (e.g., surface flinger) calls the layer to obtain the drawn graphic buffer for synthesis and processing.

[0064] It can be understood that surface is for producers, while layer is for consumers (e.g. surfaceflinger). Since surfaces and layers correspond one to one, the order of layers on the Z axis (Z-order) is the same as the order of the corresponding surfaces on the Z axis.

[0065] Frame buffer is a continuous physical memory reserved for GPU (graphics processing unit), DPU (display processing unit), etc. in embedded systems. The display reads data from the frame buffer through a dedicated bus and displays it on the screen.

[0066] Layer management module (Surface flinger): also known as layer deliverer, Surface flinger is a system service. This system service is mainly used to receive graphic display data from multiple sources, and synthesize the graphic display data from multiple sources and send it to the display device. This service system can realize functions such as layer creation, control, and management. In the display system of an electronic device, Surface synthesis can be achieved through Surface flinger. For example, after the application is started, Surface flinger can create a layer for the application. During the operation of the terminal, Surface flinger can obtain the layers to be displayed of various applications running in the terminal, and synthesize the obtained layers through GPU and / or HWC.

[0067] Furthermore, the layer management module is also used to generate and distribute a vertical synchronization (Vertical Synchronization, Vsync) signal to ensure the synchronization of various processes of the display system.

[0068] The display pipeline of an electronic device mainly includes three processes: application rendering, layer synthesis, and driver display. In essence, the display pipeline of an electronic device is the flow of buffers.

[0069] GPU synthesis: GPU (Graphics Processing Unit) is a general-purpose image processing device. In addition to being used for layer synthesis, GPU is also used to complete other graphics processing tasks. GPU synthesis refers to the layer synthesis method that uses the GPU to synthesize layers.

[0070] HWC synthesis: HWC (Hardware Composer) is a dedicated image processing device used for layer synthesis and display. HWC can be an independent device or integrated into a system-level chip. HWC synthesis refers to the layer synthesis method of synthesizing layers through HWC. In the HWC synthesis process, since each layer needs to occupy a transmission channel of HWC, and the transmission channels of HWC are limited, the number of layers that HWC supports synthesis is limited.

[0071] It should be noted that the present application mainly relates to the layer synthesis process in the display pipeline process of an electronic device. The layer involved is a layer, that is, a layer for processing by consumers (eg, a surface flinger), so the layer below is a layer.

[0072] The following is a comparative explanation of the advantages of an image processing method provided in an embodiment of the present application, in combination with a real-time blurring method in the current technology.

[0073] With the development of science and technology, many electronic devices with display interfaces have realized background blurring functions, which can effectively improve the privacy and security of user data while effectively improving the aesthetics of the display interface of the electronic device. In some scenarios, the electronic device can realize local blurring functions or global blurring functions, that is, the display interface of the electronic device is partially blurred or globally blurred. For example: when a user is using a mobile phone, he receives text messages from other users. The mobile phone prompts the user through a pop-up window that he has received text messages from other users. The pop-up window can display the sender information of the text message (such as: the sender's phone number of the text message) and part of the content of the text message. In this case, in order to minimize the impact on the normal use of the mobile phone when the pop-up window prompts the user to receive the text message, and for the beauty of the overall display interface, the background of the pop-up window area is blurred. That is, when there is a pop-up window display on the mobile phone display interface, the pop-up window area may be partially blurred to achieve a local blur function; when the user requests to shut down or restart the phone, he can long press the power button. When the user long presses the power button until the shutdown interface appears, he can choose to shut down the phone or restart the phone. In order to make the shutdown interface more beautiful, the entire display interface of the phone is blurred. That is, when the phone displays the shutdown interface, a global blur function is implemented. For ease of understanding, the following combines Figure 2A-2D The following examples are provided to illustrate the situation where the display interface is partially blurred and the real interface is globally blurred, wherein the electronic device 10 is a mobile phone as an example.

[0074] like Figure 2A A partially blurred display interface of an electronic device is shown. After the user unlocks the electronic device 10, the electronic device 10 can present a desktop interface 200 to the user. The desktop interface 200 includes: wallpaper, application icons, time icons, etc., wherein the desktop interface 200 includes a variety of application icons, such as: phone icons, contact icons, Internet icons, SMS icons, etc.

[0075] In one case, after the user unlocks the electronic device 10, in order to prompt the user some important information or the current status of the electronic device (for example, abnormal status, task completion progress, task completion, etc.), a pop-up window will be used to prompt the user. Figure 2A The message pop-up window 201 shown is used as an example for explanation. The message pop-up window 201 is used to remind the user that the electronic device 10 has been connected to USB debugging, that is, the electronic device 10 is connected to other electronic devices (such as a computer) via a USB data cable, and when the electronic device 10 has turned on the USB debugging mode, the message pop-up window 201 will be used to remind the user that the electronic device 10 has been connected to USB debugging. In the current technology, in order to ensure the beauty of the display interface of the electronic device 10 while the message pop-up window can still play a prompting role, the background corresponding to the message pop-up window 201 will be blurred, that is, the area corresponding to the message pop-up window 201 is applied with a local blur function.

[0076] In some cases, the output volume of the electronic device 10 does not conform to the user's usage habits (the output volume is too loud or too low), and the user can adjust the output volume of the electronic device 10. For example, the user can adjust the volume through the volume adjustment key 11 of the electronic device 10, and the electronic device 10 adjusts the volume in response to the user's operation of the volume adjustment key 11, and the electronic device 10 displays the volume bar 202, which can display the output volume value of the electronic device 10 (the full value of the output volume is 100) or the output volume percentage, etc. In order to highlight the volume bar 202 without affecting the basic display of the electronic device 10, the background corresponding to the volume bar 202 area will be blurred, that is, the area corresponding to the volume bar 202 is applied with a local blur function.

[0077] In some cases, the user can wake up the voice assistant through voice or other operations. The user outputs a piece of voice data to the electronic device 10, and the corresponding function is realized by the voice assistant. For example, the default wake-up word of the voice assistant of the system supported by the electronic device 10 is "Hello YOYO". The user says "Hello YOYO" to the electronic device 10 to wake up the voice assistant of the electronic device 10. After the voice assistant wakes up, the electronic device 10 displays the voice assistant pop-up window 203 to prompt the user to wake up the voice assistant, and displays the user how to use the voice assistant, the functions that can be realized, etc. through the voice assistant pop-up window 203. In order to enable the voice assistant pop-up window 203 to highlight and clearly display relevant information without affecting the basic display of the electronic device 10, the background corresponding to the voice assistant pop-up window 203 area will be blurred, that is, the area corresponding to the voice assistant pop-up window 203 is locally blurred.

[0078] It should be noted that through Figure 2A Three cases where the local blur function is applied are described, and there are other cases where the local blur function is applied; and Figure 2A The three situations are shown to occur simultaneously, and the Z-order of the default volume bar 202 is greater than that of the message pop-up window 201 and the voice assistant pop-up window 203 .

[0079] like Figure 2BA globally blurred display interface of an electronic device is shown, the electronic device 10 is currently in the power-on state, the user long presses the power button 12 of the electronic device 10 for a preset time (for example: 3 seconds, that is, the user long presses the power button 12 of the electronic device 10 for 3 seconds), and the electronic device 10 presents a shutdown interface 210 to the user, and the shutdown interface 210 includes: a restart icon, a shutdown icon, etc. The user can trigger the shutdown icon or the restart icon, such as: click, long press, slide, etc. The electronic device 10 receives the user-triggered operation and starts the corresponding shutdown or restart process. The shutdown interface 210 completely covers the original interface of the electronic device 10. In order to improve the aesthetics of the display interface, the global area corresponding to the shutdown interface 210 is blurred, that is, the shutdown interface 210 applies a global blur function.

[0080] Combine the following Figure 2C (a) and Figure 2C (b) Another global blur scenario is introduced in detail. Figure 2C (a) shows a desktop interface of an electronic device. After the user unlocks the electronic device 10, the electronic device 10 can present a desktop interface 220 to the user. The desktop interface 220 includes icons of multiple applications, such as a phone icon, a contact icon, an Internet icon, a text message icon, etc., and there is a status bar 221 at the top of the desktop interface 220. The status bar 221 may include information such as message notifications 222, current time, current battery level, whether the current electronic device is connected to wifi, whether Bluetooth is turned on, etc. In response to the user's operation in the status bar 221 (e.g., sliding down), the electronic device can display the following information: Figure 2C (b) shows a globally blurred display interface of another electronic device.

[0081] like Figure 2C As shown in (b), the status bar interface 230 is an exemplary interface involved when the status bar is pulled down to display the notification page. The status bar interface 230 includes a detailed content part 231 involved in the message notification 222. For example, the "Mobile Manager" application of the electronic device 10 prompts the user that "Application A is consuming power in the background", and the detailed content part 231 can prompt the user that the current application A is consuming power in the background. At this time, in order to improve the beauty of the display interface, the global area of ​​the background part of the status bar interface 230 is blurred, and the detailed content part 231 involved in the message notification 222 and other information are displayed on the upper layer of the global blurred background.

[0082] Combine the following Figure 2D (a) and Figure 2D (b) Another global blur scenario is introduced in detail. Figure 2D(a) shows another desktop interface of an electronic device. After the user unlocks the electronic device 10, the electronic device 10 can present a desktop interface 240 to the user. The desktop interface 240 includes icons of multiple applications, such as a phone icon, a contact icon, an Internet icon, a text message icon, etc. Assuming that application A and application B in the electronic device 10 are running in the background at this time, in response to a specific user operation (such as a swipe up operation), the electronic device 10 can display the following: Figure 2D (b) shows a globally blurred display interface of another electronic device.

[0083] like Figure 2D (b) shows a multitasking interface 250, which displays the applications currently running in the background of the electronic device 10 (including: application A part 251 and application B part 252). The user can operate the application A part 251 and / or application B part 252 to close the application A and / or application B from the background of the electronic device 10. At this time, in order to improve the beauty of the display interface, the global area of ​​the background part of the multitasking interface 250 is blurred, and the application A part 251 and the application B part 252 are displayed on the upper layer of the global blurred background. While displaying the beautiful interface, the background running application content to be displayed by the multitasking interface 250 is highlighted.

[0084] It should be noted that in addition to the above, common global blur scenarios Figure 2B-Figure 2D (b) introduces three situations in which the global fuzzy function is applied. In addition, there are other situations in which the global fuzzy function is applied, and this application does not make specific limitations.

[0085] Furthermore, common background blur scenes of electronic devices are mainly divided into static blur scenes and dynamic blur scenes. Among them, the static blur scene is to use the background of a certain frame of the background as the blur data source, and only blur the background of the frame, that is, the blur area of ​​the electronic device only displays the blur background corresponding to the background of the frame, even if the background changes (for example: dynamic wallpaper, sliding background, etc.), the blur background corresponding to the background of the frame is still displayed, and it does not change with the background change, presenting a static mode; the dynamic blur scene (also known as: real-time blur scene) is to use each frame of the background as the blur data source, and blur each frame of the background, that is, the display of the blur area of ​​the electronic device changes with the background change. For example: when the background of the electronic device is the N-frame background, the display of the corresponding blur area of ​​the electronic device is the blur background corresponding to the N-frame background, and when the background of the electronic device is the N+1-frame background, the display of the corresponding blur area of ​​the electronic device is the blur background corresponding to the N+1-frame background, wherein the N-frame background and the N+1-frame background are different (the background changes). That is, when the background changes, the display of the blur area of ​​the electronic device changes dynamically, presenting a dynamic mode.

[0086] Combine the following Figure 3-5B , a background blurring scheme of electronic devices in the current technology is described in detail. Taking the case where the electronic device 10 adjusts the output volume of the electronic device 10 in response to the user's operation of the volume key 11 as an example, when the electronic device 10 displays a volume bar to the user, the background of the volume bar area in the display interface of the electronic device 10 is blurred, thereby achieving local blurring of the volume bar area of ​​the electronic device 10 (such as Figure 3 (a) is a display interface of an electronic device.

[0087] like Figure 3 (a) shows a display interface 300 of an electronic device, and the display interface 300 is divided into four layers as an example for explanation. Figure 3 As shown in (b), the four layers are status bar layer 301, volume bar layer 302, desktop layer 303 and wallpaper layer 304. The status bar layer 301 is located at the top layer, the volume bar layer 302 is located at the second layer, the desktop layer 303 is located at the third layer, and the wallpaper layer 304 is located at the bottom layer. The layer order Z is that the status bar layer 301 has a Z-order of 4, the volume bar layer 302 has a Z-order of 3, the desktop layer 303 has a Z-order of 2, and the wallpaper layer 304 has a Z-order of 1.

[0088] Electronic devices enable blur function in some scenarios, such as: Figure 3 In the scene shown in (a), the background of the volume bar area is blurred, and the electronic device 10 needs to perform layer synthesis and blur processing in the synthesis stage of the display process. However, since the display processing unit (DPU) synthesis and HWC synthesis do not currently support blur processing, the layers related to blur processing can only be blurred and synthesized by the GPU.

[0089] For ease of understanding, the following Figure 4 The following is an example of the blur processing and layer synthesis process for a frame of image in the current technology. The layers related to the blur processing include the volume bar layer 302, the desktop layer 303 and the wallpaper layer 304. The status bar layer 301 is irrelevant to the blur processing, so it does not need to be synthesized by the GPU.

[0090] When the layers of the display interface of the electronic device 10 are drawn (such as Figure 3After the status bar layer 301, volume bar layer 302, desktop layer 303 and wallpaper layer 304 are synthesized, the volume bar layer 302, desktop layer 303 and wallpaper layer 304 are synthesized by the graphics processor GPU (graphics Processing Unit) and blurred (generally, the volume bar layer 302, desktop layer 303 and wallpaper layer 304 are synthesized by the GPU and then blurred by the GPU), the GPU outputs the synthesis result (the synthesis result after layer synthesis and blurring), and then the hardware editor HWC (Hardware Composer) performs layer synthesis on the GPU synthesis result and the remaining layers to be synthesized (status bar layer 301), thereby completing the synthesis of the electronic device display interface.

[0091] The following is a detailed introduction to the layer synthesis and blurring of the volume bar layer 302, the desktop layer 303 and the wallpaper layer 304 through the GPU in combination with the buffer. For example: the wallpaper layer 304 is superimposed on buffer1, and then the desktop layer 303 is superimposed on the wallpaper layer 304, the result on buffer1 is copied to buffer2, and buffer2 is blurred and a blurred area with a position and size corresponding to the volume bar is cut out according to the volume bar layer 302, and the blurred area of ​​buffer2 is superimposed on the desktop layer 303 again (that is, the blurred area is superimposed on buffer1), and finally the volume bar layer 302 is superimposed on the blurred area (that is, the volume bar layer 302 is superimposed on buffer1), and the GPU outputs the synthesis result (that is, buffer1, buffer2 is just a transition for generating blur and is discarded after use).

[0092] The above is the blur processing and layer synthesis process for a frame of image. In the dynamic blur scene, each frame needs to be processed as follows: Figure 4 The blur processing and layer synthesis process shown is time-consuming, and GPU synthesis consumes more power than HWC synthesis, which will result in high power consumption, resulting in high power consumption of electronic devices, thereby reducing the performance of electronic devices, seriously affecting the user experience. For example: when the wallpaper of an electronic device is set to a dynamic wallpaper, and a video is played through a small window on the dynamic wallpaper, when a dynamic blur scene appears in the electronic device (for example: after the electronic device receives information sent by other electronic devices, it prompts the user through a pop-up window; the user adjusts the output volume of the electronic device through the volume button of the electronic device, etc.), in this case, it is necessary to perform relevant processing for real-time playback of the small window video and real-time blur processing, which results in excessive surface flinger load, resulting in high power consumption of the electronic device, thereby reducing the performance of the electronic device, seriously affecting the user experience.

[0093] Further, combined with Figure 5A and Figure 5B When the display image of the electronic device does not need to be blurred and only HWC synthesis is performed, the path tracking diagram of the layer synthesis process is as shown in FIG. Figure 5A As shown, when GPU synthesis is not required, it takes about 4 milliseconds (ms); when the display image of the electronic device needs to be blurred, GPU synthesis and blur processing are required. The path tracking diagram of the layer synthesis process is shown in Figure 5B As shown, it takes about 10 milliseconds (ms). Therefore, it takes a long time for the layers of the displayed image to be synthesized and blurred by the GPU. For real-time blurring, each frame of the image displayed by the electronic device needs to be synthesized and blurred by the GPU, resulting in high power consumption of the electronic device, thereby reducing the performance of the electronic device and seriously affecting the user experience. In addition, due to the long time consumption, performance issues such as stuttering and frame loss may occur.

[0094] The present application provides an image processing method, in the case of a stable blurred layer, blurring is performed on alternate frames, and blurring is not required for every frame. When the image state is a copy frame state, GPU synthesis and blurring are performed normally, and then the GPU synthesis result is synthesized with the remaining layers by HWC. During the GPU synthesis and blurring process, the blurring result is copied to a pre-created blurring result layer to obtain a blurring result layer with a blurring result, wherein the blurring result is the blurring result of the blurring layer generated by the GPU. When the image state is a skip frame state, the blurring result layer and all layers are synthesized by HWC, and GPU synthesis is not required. Since the skip frame image reuses the blurring result of the copy frame image, the forced GPU synthesis due to blurring is avoided, the power consumption of the electronic device is reduced, the performance of the electronic device is improved, and thus the user experience is improved.

[0095] In some embodiments, the electronic device may be a mobile phone, a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a smart watch, etc. The present application does not impose any special restrictions on the specific form of the above electronic devices. In this embodiment, the structure of the electronic device may be as follows: Fig. 6A As shown, Fig. 6A A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0096] like Fig. 6A As shown, the electronic device may include a processor 110 , a display screen 120 , a camera 130 , an internal memory 140 , an audio module 150 , a speaker 150A, a receiver 150B, a microphone 150C, a button 170 , and a touch sensor 160 .

[0097] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0098] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated into one or more processors. For example, according to the state of the first layer and the state of the second image, the state of the first image is judged; and each frame of the image is controlled to perform a corresponding layer synthesis process.

[0099] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0100] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0101] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0102] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 160, the charger, the flash, the camera 130, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 160 through the I2C interface, so that the processor 110 communicates with the touch sensor 160 through the I2C bus interface to realize the touch function of the electronic device.

[0103] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 120 and the camera 130. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 130 communicate via the CSI interface to implement the shooting function of the electronic device. The processor 110 and the display screen 120 communicate via the DSI interface to implement the display function of the electronic device.

[0104] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 130, the display 120, the audio module 150, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0105] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0106] The electronic device implements the display function through a GPU, a display screen 120, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 120 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0107] The display screen 120 is used to display images, videos, etc. The display screen 120 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 120, where N is a positive integer greater than 1.

[0108] A series of graphical user interfaces (GUIs) can be displayed on the display screen 120 of the electronic device, and these GUIs are the main screens of the electronic device. Generally speaking, the size of the display screen 120 of the electronic device is fixed, and only limited controls can be displayed on the display screen 120 of the electronic device. A control is a GUI element, which is a software component included in an application program, and controls all data processed by the application program and interactive operations on the data. Users can interact with the control through direct manipulation to read or edit relevant information of the application program. Generally speaking, a control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc.

[0109] The electronic device can realize the shooting function through the ISP, the camera 130, the video codec, the GPU, the display screen 120 and the application processor.

[0110] The ISP is used to process the data fed back by the camera 130. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 130.

[0111] The camera 130 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 130, where N is a positive integer greater than 1.

[0112] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy.

[0113] Video codecs are used to compress or decompress digital videos. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0114] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously self-learn. NPU can realize applications such as intelligent cognition of electronic devices, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0115] The internal memory 140 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 140. For example, in this embodiment, the processor 110 can recalculate the first display timestamp of the video stream and recalculate the second display timestamp of the audio stream by executing the instructions stored in the internal memory 140. The internal memory 140 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area may store data created during the use of the electronic device (such as audio data, a phone book, etc.). In addition, the internal memory 140 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 140 and / or the instructions stored in the memory provided in the processor.

[0116] The electronic device can implement audio functions such as music playing and recording through the audio module 150, the speaker 150A, the receiver 150B, the microphone 150C, and the application processor.

[0117] The audio module 150 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 150 can also be used to encode and decode audio signals. In some embodiments, the audio module 150 can be arranged in the processor 110, or some functional modules of the audio module 150 can be arranged in the processor 110.

[0118] The speaker 150A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device can listen to music or listen to a hands-free call through the speaker 150A.

[0119] The receiver 150B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device receives a call or voice message, the voice can be received by placing the receiver 150B close to the human ear.

[0120] Microphone 150C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting his mouth close to microphone 150C to input the sound signal into microphone 150C. The electronic device can be provided with at least one microphone 150C. In other embodiments, the electronic device can be provided with two microphones 150C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 150C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0121] The touch sensor 160 is also called a "touch control device". The touch sensor 160 can be arranged on the display screen 120. The touch sensor 160 and the display screen 120 form a touch screen, also called a "touch control screen". The touch sensor 160 is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 120. In other embodiments, the touch sensor 160 can also be arranged on the surface of the electronic device, which is different from the position of the display screen 120.

[0122] The key 170 includes a power key, a volume key, etc. The key 170 can be a mechanical key or a touch key. The electronic device can receive key input and generate key signal input related to user settings and function control of the electronic device.

[0123] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.

[0124] The operating system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device.

[0125] Figure 6B It is a software structure block diagram of the electronic device according to an embodiment of the present application.

[0126] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer (or framework layer), the Android runtime and system library, the hardware abstraction layer, and the kernel layer. For ease of explanation, Figure 6B The software structure diagram of the electronic device shown also includes Fig. 6A The hardware structure in, such as display processing unit, screen, etc.

[0127] The application layer can include a series of application packages. Figure 6B As shown, the application package may include applications such as beauty camera, gallery, video, short message, etc.

[0128] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 6B As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0129] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0130] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0131] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0132] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0133] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0134] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0135] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0136] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0137] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0138] The system library can include multiple functional modules, such as: layer management module (surface flinger), frame interval module (efficientblur.cpp), primitive operation encapsulation (Buffer State Layer), composition engine (CompositionEngine), output module (Output.cpp), rendering engine (Render Engine), 2D graphics engine (such as SGL), surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (such as OpenGLES), etc.

[0139] The layer management module is used to control image synthesis and generate vertical synchronization (Vsync) signals. Specifically, its main function is to receive graphics data buffers from multiple sources, synthesize them, and then send them to the display hardware (touch screen); at the same time, the image synthesis system is also responsible for error synchronization with the hardware HW-VSync signal and simulation of the software VSync signal, including SF-VSync and APP-VSync signals; the synthesizer flow control, signal simulator and connection manager collaborate to notify the VSync signal to the application and SurfaceFlinger itself.

[0140] The inter-frame module is used to collect the layer parameters of the blurred layer in each frame image, so as to determine the state of each frame image according to the parameters of the blurred layer in each frame image and the state of the previous frame image; control the creation and destruction of the blurred result layer, and control the position of the blurred result layer.

[0141] The primitive operation encapsulation is used to control the setting of the drawn buffer to the corresponding layer.

[0142] The compositing engine is used to iterate over all layers of each frame.

[0143] The output module is used to control the position of the visible layer and the layer synthesis method of the visible layer.

[0144] The drawing engine is used to draw the display content on the corresponding buffer.

[0145] A 2D graphics engine is a drawing engine for 2D drawings.

[0146] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0147] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0148] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0149] The hardware abstraction layer may include multiple library modules, such as hardware composer (HWC), camera library module, etc. The Android system may load the corresponding library module for the device hardware, thereby enabling the application framework layer to access the device hardware.

[0150] The hardware composer (HWC) mainly provides hardware support for the image synthesis system, supporting layer synthesis and display modules.

[0151] The kernel layer is the layer between hardware and software. The kernel layer includes at least the touch panel (TP) driver, display driver, Bluetooth driver, WIFI driver, keyboard driver, shared memory driver, ambient light sensor driver and camera driver.

[0152] The display driver can use the API to transmit instructions and data to the graphics processing unit (GPU) or dedicated graphics processing hardware and perform operations such as 3D rendering, video decoding, and GPU computing. The hardware may include display screens, touch hardware, processors, GPUs, etc.

[0153] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.

[0154] Embodiment 1:

[0155] Combine the following Figure 7-Figure 12 , an image processing method provided by an embodiment of the present application is introduced in detail. An image processing method provided by an embodiment of the present application is aimed at a real-time blur scene (dynamic blur scene) of an electronic device, that is, when there is a blur area (local or global) on the display interface of the electronic device, and the blur processing method of the blur area is a real-time blur processing method. And a specific introduction is given to how to determine whether the image processing scheme is turned on and off.

[0156] First, before executing an image processing method provided by an embodiment of the present application, the electronic device needs to perform object initialization in advance. For example: when the electronic device is converted from the off state to the on state, the object is initialized to initialize the layer parameters related to the blurred layer (also referred to as the layer parameters of the blurred layer) of the first frame image displayed after the electronic device is turned on. For example: taking the layer parameters related to the blurred layer as the number of blurred layers, the name of the blurred layer, and the blurred layer change parameters as an example, the number of blurred layers is initialized to 0, and there is no corresponding blurred layer name and blurred layer change parameter. At this time, the state of the first frame image displayed by the electronic device is the solution off state.

[0157] Among them, the solution off state is characterized by: one situation is that the frame image does not need to be blurred, that is, there is no blurred layer in the frame image, and the image processing solution provided by the present application will not be executed at this time; another situation is that there are two or more blurred layers in the frame image, and the image processing solution provided by the present application will not be executed at this time.

[0158] Among them, when there is a blurred layer in the display interface of an electronic device, the electronic device needs to perform relevant blur processing so that the blurred layer is displayed in a blurred state in the display interface of the electronic device, that is, all layers under the blurred layer need to be blurred to obtain the final blurred result. For example: when the user adjusts the output volume of the electronic device through the volume adjustment key, the electronic device adjusts the output volume in response to the operation of the user's volume adjustment key, and displays the volume bar to the user. At this time, the volume bar layer is the blurred layer. Furthermore, the blurred layer carries a fuzzy attribute. When the electronic device recognizes the fuzzy attribute carried by the blurred layer, it will blur all layers below the blurred layer.

[0159] S701, monitor and collect layer parameters related to the blurred layer in each frame of image displayed by the electronic device.

[0160] According to the display order of the electronic device, the layer parameters related to the blurred layer in each frame image are monitored and collected, which can also be called the layer parameters of the blurred layer.

[0161] Among them, the layer parameters related to the fuzzy layer include: the number of fuzzy layers, the name of the fuzzy layer, the fuzzy layer change parameters, etc.

[0162] The number of blurred layers represents the number of blurred layers in the frame image currently displayed by the electronic device. For example, when the electronic device currently displays only the desktop, Fig. 8A In the desktop interface of the electronic device shown in the figure, in the current case, the frame image displayed by the electronic device does not have a blurred layer, so the number of blurred layers is 0; when the electronic device currently displays only one blurred layer, such as Figure 8B A display interface of an electronic device shown in FIG. 1 , in the current case, the blurred layer in the frame image displayed by the electronic device includes the volume bar layer, so the number of blurred layers is 1; when the electronic device currently displays two blurred layers, such as Figure 8C Another display interface of the electronic device is shown. In the current case, the blurred layers in the frame image displayed by the electronic device include a volume bar layer and a pop-up window layer, so the number of blurred layers is 2.

[0163] The blur layer name corresponds to the blur layer one by one, and each blur layer has a unique corresponding blur layer name, and the blur layer name is an identifier of the blur layer. Furthermore, in some cases, dynamic blur scenes and static blur scenes can be distinguished based on the blur layer name, for example: a white list of dynamic blur scenes is pre-set, and the white list stores the names of blur layers that need to be dynamically blurred. When the blur layer name is in the white list of the dynamic blur scene, the electronic device performs real-time blur processing (that is, each frame of the image needs to be blurred).

[0164] The blur layer variation parameter indicates whether the blur layer is stable (i.e., whether the blur layer is in a stable stage), for example, whether the shape, position, and size of the blur layer are stable. The blur layer variation parameter indicates whether the blur layer in the current frame image has changed compared to the blur layer in the previous frame image. If there is a change, the blur layer is unstable; if there is no change, the blur layer is stable. Generally, when the blur layer variation parameter is 0, it indicates that the blur layer is stable (i.e., the blur layer is in a stable stage); when the blur layer variation parameter is 1, it indicates that the blur layer is unstable (i.e., the blur layer is not in a stable stage).

[0165] In order to facilitate the understanding of the fuzzy layer change parameters, the following Fig. 9 The volume bar layer change diagram shown in the figure introduces the blur layer change parameters as an example. Take the blur layer change parameter of 1 as an example, indicating that the blur layer changes / the blur layer is unstable, and the blur layer change parameter of 0 as an example, indicating that the blur layer does not change / the blur layer is stable.

[0166] During the normal display process of an electronic device, the user adjusts the output volume of the electronic device through the volume adjustment key. The electronic device responds to the user's adjustment operation of the volume adjustment key, adjusts the output volume of the electronic device, and displays a volume bar to the user, thereby prompting the user of the adjusted output volume of the electronic device. From the electronic device not displaying the volume bar to displaying the volume bar to the user, and then the volume bar disappears. The display and disappearance of the volume bar can be divided into three stages, namely, the entry stage, the stable stage, and the exit stage. Fig. 9 As shown, when the electronic device responds to the user's operation on the volume adjustment key, the first frame image is displayed in time sequence (composite display order), and the volume bar layer in the first frame image is a 1-frame volume bar layer, and the volume bar layer in the second frame image displayed in time sequence is a 2-frame volume bar layer, and so on. Fig. 9 The change from a 1-frame volume bar layer to an 8-frame volume bar layer is shown.

[0167] Among them, from the 1-frame volume bar layer to the 3-frame volume bar layer is the entry stage of the volume bar, and the size of the volume bar layer increases from small to large. If the size of the volume bar layer in the entry stage increases from small to large, the volume bar layer in the entry stage is unstable. By default, there is no volume bar layer in the previous frame image of the 1-frame volume bar layer, and the fuzzy layer change parameter of the 1-frame volume bar layer is 1. The 2-frame volume bar layer has changes compared to the 1-frame volume bar layer, and the 3-frame volume bar layer has changes compared to the 2-frame volume bar layer. Therefore, the fuzzy layer change parameter of the 1-frame volume bar layer is 1, the fuzzy layer change parameter of the 2-frame volume bar layer is 1, and the fuzzy layer change parameter of the 3-frame volume bar layer is 1. From the 4-frame volume bar layer to the 6-frame volume bar layer is the stable stage of the volume bar, and the size of the volume bar layer does not change. In the stable stage, the size of the volume bar layer remains unchanged (stable), and there is no change in the 4-frame volume bar layer compared to the 3-frame volume bar layer, so the fuzzy layer change parameter of the 4-frame volume bar layer is 0, and so on, the fuzzy layer change parameters of the 4-frame volume bar layer to the 6-frame volume bar layer are all 0. The volume bar layer from the 7-frame volume bar layer to the 8-frame volume bar layer is the exit stage of the volume bar, and the size of the volume bar layer decreases from large to small. The size of the volume bar layer in the exit stage decreases from large to small. The 7-frame volume bar layer changes compared to the 6-frame volume bar layer, and the 8-frame volume bar layer changes compared to the 7-frame volume bar layer. Therefore, the volume bar layer in the exit stage is unstable, and the fuzzy layer change parameter of the 7-frame volume bar layer is 1, and the fuzzy layer change parameter of the 8-frame volume bar layer is 1.

[0168] After monitoring and collecting the layer parameters related to the blurred layer in each frame of the image displayed by the electronic device, the electronic device determines the state of each frame of the image according to the collected layer parameters related to the blurred layer in each frame of the image in accordance with the synthesis order. The specific confirmation process is as follows: Fig.12 shown.

[0169] S702: Determine, based on layer parameters related to the blurred layer, whether there is and only one blurred layer in the current frame image.

[0170] When it is determined that there is not and only one blurred layer in the current frame image, for example, the current frame image does not have a blurred layer, or the current frame image has two or more blurred layers, at this time, the state of the current frame image is a scheme closed state.

[0171] When it is determined that the state of the current frame image is the scheme closed state, the next frame image of the image in the scheme closed state is updated as the current frame image according to the synthesis order, and S702 is repeated.

[0172] When it is determined that there is only one blurred layer in the current frame image, and the state of the previous frame image of the current frame image is the scheme closed state (after confirming the image of the scheme closed state, it is the first time to determine that there is only one blurred layer in the current frame image), the state of the current frame image is determined to be a normal frame state, that is, the current frame image can also be called a normal frame image. Then S703 is performed.

[0173] S703: Create a fuzzy result layer.

[0174] The current frame image is in the normal frame state, and the previous frame image is in the scheme-off state. At this time, the image processing scheme is turned on (that is, the image processing scheme is turned from the off state to the on state), and a blur result layer (blurlayer) will be created. Since the appearance of the blur layer is from nothing to something, and is a process from small to large until it is stable, the blur layer has appeared before the target interface to be blurred is determined. When the blur layer appears, the blur result layer (blurlayer) is created.

[0175] Furthermore, while creating the blurred result layer, a first cache (buffer) corresponding to the blurred result layer will be created to facilitate subsequent use in the layer synthesis process in the display pipeline of the electronic device, thereby realizing the flow of the buffer and thus realizing the sequential execution of the display pipeline.

[0176] The image processing method provided by the embodiment of the present application, at this time, only the layer parameter judgment of S702 is performed, indicating that the state of the current frame image has only the scheme closed state and the normal frame state. When the state of the current frame image is determined to be the normal frame state, the image processing scheme is turned on, and the image processing scheme is turned on from the closed state. When the image processing scheme is turned on from the closed state, a fuzzy result layer is created to facilitate the implementation of the subsequent image processing scheme.

[0177] At this time, only the blurred result layer is created and saved, and the blurred result layer does not participate in the layer synthesis of the normal frame image. That is, the layer synthesis process of the normal frame image is not affected. In a possible implementation method, the layer corresponding to the normal frame image still uses the GPU to perform layer synthesis and blur processing on the blurred layer and the layers related to the blurred layer (that is, all layers below the blurred layer), and synthesizes the synthesis result output by the GPU with the remaining layers of the normal frame image through the HWC to complete the layer synthesis of the normal frame image. In addition, the layer synthesis of the normal frame image can also be completed by other methods, and this application does not specifically limit the layer synthesis process of the normal frame image.

[0178] It should be noted that the normal frame state can be used as a buffer state between the scheme closed state and the copy frame state / frame skip state.

[0179] After the blur result layer is created, the next frame image of the image in the normal frame state is updated to the current frame image according to the synthesis order, and S704 is performed.

[0180] S704: Determine, based on layer parameters related to the blurred layer, whether there is and only one blurred layer in the current frame image.

[0181] According to the number of blurred layers, it is judged whether there is and only one blurred layer in the current frame image. When the number of blurred layers is 1, there is and only one blurred layer in the current frame image; when the number of blurred layers is not 1, there is not and only one blurred layer in the current frame image.

[0182] When it is determined that there is not and only one blurred layer in the current frame image (i.e., there is no blurred layer in the current frame image or there are two or more blurred layers in the current frame image), the state of the current frame image is the scheme closed state. Further, at this time, the current frame image is in the scheme closed state, and the previous frame image state is a normal frame state, which indicates that the image processing scheme is changed from the open state to the closed state, and S713 is performed.

[0183] When it is determined that there is only one blurred layer in the current frame image, proceed to S705.

[0184] S705. Determine, based on layer parameters related to the blurred layer, whether the only blurred layer in the current frame image is a stable blurred layer.

[0185] According to the blur layer variation parameter, it is determined whether the only blur layer in the current frame image is a stable blur layer (i.e. whether the only blur layer is in a stable stage). Generally, a blur layer variation parameter of 1 indicates that the blur layer is unstable (not in a stable stage), and a blur layer variation parameter of 0 indicates that the blur layer is stable (in a stable stage).

[0186] When it is determined that the only blurred layer in the current frame image is not a stable blurred layer, that is, the only blurred layer in the current frame image is a changing blurred layer, the state of the current frame image is a normal frame state.

[0187] When it is determined that the state of the current frame image is the normal frame state, the next frame image of the image in the normal frame state is updated as the current frame image according to the synthesis order, and S704 is repeated.

[0188] When it is determined that the only blurred layer in the current frame image is a stable blurred layer, the state of the current frame image is a copy frame state.

[0189] The copy frame state is used to indicate that the blurring result of the synthesis process of the image in the copy frame state is stored in the newly created blurring result layer. It should be noted that the image in the copy frame state can also be called a copy frame image.

[0190] S706. During the synthesis process of the copy frame image, the blur processing result in the copy frame image is stored in the blur result layer.

[0191] In a possible implementation, since the blur processing and layer synthesis of other layers related to the blurred layer in the electronic device are forced to be implemented through the GPU, that is, the electronic device synthesizes (superimposes) all layers under the blurred layer in the copy frame image through the GPU, blurs the superimposition processing result, and cuts it according to the shape and size of the blurred layer to obtain the corresponding blur processing result, and then synthesizes the blurred layer, the blur processing result and all layers under the blurred layer in the copy frame image through the GPU. If there are layers unrelated to the blurred layer (for example, layers located on the blurred layer), the synthesis result output by the GPU and the remaining layers are synthesized through the HWC to obtain the display result of the copy frame image. Specifically, in the process of layer synthesis and blur processing of the blurred layer in the copy frame image and all layers under the blurred layer through the GPU, the blur processing result of the copy frame image is stored in the blur result layer.

[0192] For ease of understanding, the following Fig. 10A Let's take step 706 as an example. Among them, the layers of the target interface after rendering include: video playback layer 1001, volume bar layer 1002, desktop layer 1003 and wallpaper layer 1004. The layers of each frame image in the target interface after rendering are the same, but the content rendered in each layer may be different (for example: the wallpaper displayed in the target interface is a dynamic wallpaper, then the wallpaper layer rendered for each frame image is different). Taking the direction of the Z axis perpendicular to the display screen outward as the positive direction, the video playback layer 1001 is located in the fourth layer (that is, the Z-order of the video playback layer 1001 is 4), the volume bar layer 1002 is located in the third layer (that is, the Z-order of the volume bar layer 1002 is 3), the desktop layer 1003 is located in the second layer (that is, the Z-order of the desktop layer 1003 is 2), and the wallpaper layer 1004 is located in the last layer (that is, the Z-order of the wallpaper layer 1004 is 1). It should be noted that Fig. 10A The layers shown in the figure are only simple schematic diagrams of the relative sizes and positions of the layers. They are only used as examples and do not show the content of the rendering of each layer.

[0193] like Fig. 10AAs shown, the blurred layer is the volume bar layer 1002, and the layers related to the blurred layer (the layers under the blurred layer) include: the desktop layer 1003 and the wallpaper layer 1004. The electronic device performs superposition (synthesis) processing on the desktop layer 1003 and the wallpaper layer 1004 through the GPU, and then blurs the superposition processing result of the desktop layer 1003 and the wallpaper layer 1004 through the GPU and cuts it according to the volume bar layer 1002 to obtain the blurred processing result, and then synthesizes the volume bar layer 1002, the blurred processing result, and the desktop layer 1003 and the wallpaper layer 1004 through the GPU, and outputs the GPU synthesis result. During the GPU synthesis process, the blurred processing result is stored in the blurred result layer 1005. Then, the GPU synthesis result and the video playback layer 1001 are synthesized through the HWC to obtain a copy frame image.

[0194] The blur result layer storing the blur processing result is set to a visible state during the synthesis process of the copy frame image, but does not participate in the layer synthesis process of the copy frame image. Therefore, the electronic device does not display the blur result layer when displaying the copy frame image. It should be noted that, since it does not participate in the layer synthesis, there is no blur cache layer in the final synthesized copy frame image, and there is no blur cache layer in the subsequently displayed copy frame image.

[0195] Furthermore, when the layer synthesis of the copy frame image is completed, the layer synthesis result of the copy frame image is sent to the display through the display driver according to the display order (synthesis order), so that the electronic device displays the copy frame image.

[0196] Further, the blur processing result is stored in the blur result layer to obtain the blur result layer storing the blur processing result, and the blur result layer is stored in the corresponding first cache so that the corresponding blur result layer can be directly called from the first cache later.

[0197] When it is determined that the state of the current frame image is the copy frame state, the next frame image of the image in the copy frame state is updated as the current frame image according to the display order (synthesis order), and S707 is performed.

[0198] S707: Determine whether there is and only has one blurred layer in the current frame image according to layer parameters related to the blurred layer.

[0199] When it is determined that there is not and only one blurred layer in the current frame image, the state of the current frame image is the scheme closed state. That is, when there is no blurred layer or the number of blurred layers is greater than 1, the image processing scheme is closed. Further, at this time, the state of the current frame image is the scheme closed state, and the state of the previous frame image is the copy frame state, which indicates that the image processing scheme is changed from the open state to the closed state, and S713 is performed.

[0200] When it is determined that there is only one blurred layer in the current frame image, proceed to S708.

[0201] S708. Determine whether the only blurred layer in the current frame image is a stable blurred layer according to layer parameters related to the blurred layer.

[0202] When it is determined that the only blurred layer in the current frame image is not a stable blurred layer, that is, the only blurred layer in the current frame image is a changing blurred layer (for example, the blurred layer change parameter is 1), the state of the current frame image is a normal frame state.

[0203] When it is determined that the state of the current frame image is the normal frame state, the next frame image of the image in the normal frame state is updated as the current frame image according to the display order (synthesis order), and S704 is repeated.

[0204] When it is determined that only one blurred layer in the current frame image is a stable blurred layer (for example, the blurred layer change parameter is 0), the state of the current frame image is a frame skipping state. When it is determined that the state of the current frame image is a frame skipping state, S709 is performed.

[0205] The frame skipping state is used to indicate that, in the synthesis process of the image in the frame skipping state, the blur result layer of the copy frame image is reused, that is, the blur result layer of the copy frame image and all layers of the frame image are synthesized through HWC to obtain the frame image. It should be noted that the image in the frame skipping state can also be called a frame skipping image.

[0206] S709, controlling all layers corresponding to the skipped frame image and the blurred result layer to be synthesized through HWC

[0207] During the synthesis of the skipped frame image, the HWC performs layer synthesis on the blur result layer storing the blur processing result and all the layers corresponding to the skipped frame image to obtain the skipped frame image. That is, the skipped frame image reuses the blur processing result of the copy frame image during the synthesis process and does not need to be blurred, that is, it is not necessary to perform blur processing on the layers related to the blur layer (all layers under the blur layer) through the GPU, and both the GPU and the HWC can perform layer synthesis. The skipped frame image does not need to be forced to perform layer synthesis and blur processing through the GPU, so the GPU blur processing process is skipped, and all the layers corresponding to the skipped frame image and the blur result layer are directly controlled to be synthesized through the HWC to obtain the skipped frame image.

[0208] Among them, during the process of synthesizing all the layers and blurred result layers corresponding to the skipped frame image through HWC, it is necessary to control the position of the blurred result layer to be located in the next layer of the blurred layer. Taking the vertical electronic device display screen outward as the positive direction, when the blurred layer is located at the xth layer, then the blurred result layer needs to be controlled to be located at the x-1th layer. For example: when the Z-order of the blurred layer is 4, the Z-order of the blurred result layer must be 3. It should be noted that the above example is only used to illustrate that the blurred result layer needs to be controlled to be located in the next layer of the blurred layer. The Z-order of the actual layer needs to be adaptively modified as a whole with the addition of the blurred result layer.

[0209] For ease of understanding, the following Fig. 10B Step 709 is described below with an example.

[0210] like Fig. 10B As shown, the electronic device controls all layers corresponding to the frame skipping image (video playback layer 1001, volume bar layer 1002, desktop layer 1003 and wallpaper layer 1004) and the blurred result layer 1005 to be synthesized through HWC according to the adjusted layer order to obtain a frame skipping image. When the blurred result layer is not added, the order of all layers corresponding to the frame skipping image is that the Z-order of the video playback layer 1001 is 4, the Z-order of the volume bar layer 1002 is 3, the Z-order of the desktop layer 1003 is 2, and the Z-order of the wallpaper layer 1004 is 1. After the blurred result layer is added to the frame skipping image, the adjusted layer order is that the Z-order of the video playback layer 1001 is 5, the Z-order of the volume bar layer 1002 is 4, the Z-order of the blurred result layer 1005 is 3, the Z-order of the desktop layer 1003 is 2, and the Z-order of the wallpaper layer 1004 is 1.

[0211] Furthermore, after the frame skipping image is obtained through HWC synthesis, the frame skipping image synthesized by HWC is sent to the display through the display driver according to the display order (synthesis order) of the electronic device to display the frame skipping image.

[0212] When the state of the current frame image is a skip frame state (i.e., a skip frame image), the electronic device directly sends the blur result layer and all layers corresponding to the skip frame image to the HWC for synthesis to obtain a skip frame image. That is, in the synthesis of the skip frame image, the blur processing result of the copy frame is reused, and there is no need to force the GPU to perform layer synthesis and blur processing on all layers under the blur layer, thus avoiding the need to force the use of GPU synthesis when performing blur processing in the skip frame image synthesis stage. All layers of the skip frame image and the blur result layer are directly synthesized through the HWC, thereby avoiding the high GPU load problem caused by forcing GPU synthesis, reducing the power consumption of the electronic device, improving the performance of the electronic device, and thus improving the user experience.

[0213] When it is determined that the state of the current frame image is the frame skipping state, the next frame image of the image in the frame skipping state is updated as the current frame image according to the display order (synthesis order), and S710 is performed.

[0214] S710. Determine, based on layer parameters related to the blurred layer, whether there is and only one blurred layer in the current frame image.

[0215] When it is determined that there is not and only one blurred layer in the current frame image, the state of the current frame image is the scheme closed state. Further, at this time, the state of the current frame image is the scheme closed state, and the state of the previous frame image is the adjustment state, which indicates that the image processing scheme is changed from the open state to the closed state, and then S713 is performed.

[0216] When it is determined that there is and only one blurred layer in the current frame image, proceed to S711.

[0217] S711. Determine whether the only blurred layer in the current frame image is a stable blurred layer according to layer parameters related to the blurred layer.

[0218] When it is determined that the only blurred layer in the current frame image is not a stable blurred layer, that is, the only blurred layer in the current frame image is a changing blurred layer (for example, the blurred layer change parameter is 1), the state of the current frame image is a normal frame state.

[0219] When it is determined that the state of the current frame image is the normal frame state, the next frame image of the image in the normal frame state is updated as the current frame image according to the display order (synthesis order), and S704 is repeated.

[0220] When it is determined that the only blurred layer in the current frame image is a stable blurred layer (for example, the blurred layer change parameter is 0), S712 is performed.

[0221] S712: Determine whether the accumulation of skipped frame images has reached N frames.

[0222] Among them, N is a positive integer, and N is pre-set according to actual conditions. For example, N may be related to the screen refresh rate of the electronic device. When the screen refresh rate of the electronic device is higher, the value of N may be greater than the value of N when the screen refresh rate of the electronic device is lower.

[0223] When the image accumulation in the frame skipping state does not reach N frames (ie, the frame skipping state of N frames of images is not continuously determined to be the frame skipping state), the state of the current frame of image is still the frame skipping state, and S709 is repeated.

[0224] When the image in the frame skipping state accumulates to N frames (ie, the frame skipping state of N frames of images is continuously determined to be the frame skipping state), the state of the current frame image is the copy frame state, and S706 is repeated.

[0225] For ease of understanding, the following Fig.11 The settings of copy frame status and skip frame status are introduced in detail. Take 8 frames of images (frame 1, frame 2, frame 3, frame 4, frame 5, frame 6, frame 7, frame 8) as an example, and each frame has and only has one stable (in a stable stage) blurred layer, that is, the number of blurred layers in the 8 frames is 1, and the blurred layer change parameter is 0 (indicating that the blurred layer is in a stable stage), and the default state of the first frame of the 8 frames is the copy frame state. It should be noted that Fig.11 What is shown in the figure is a simple schematic diagram of there being only one blurred layer in the 8-frame image.

[0226] When N=1 is preset, Fig.11 The status arrangement of the 8 frames of images shown is: the status of one frame of image is the copy frame status, and the status of one frame of image is the skip frame status. Fig.11 As shown in (a), the state of the first frame image is the copy frame state, the state of the second frame image is the skip frame state, the state of the third frame image is the copy frame state, the state of the fourth frame image is the skip frame state, the state of the fifth frame image is the copy frame state, the state of the sixth frame image is the skip frame state, the state of the seventh frame image is the copy frame state, and the state of the eighth frame image is the skip frame state. That is, the copy frame state and the skip frame state are alternately distributed.

[0227] When N=2 is preset, Fig.11 The status arrangement of the 8 frames of images shown is: the status of one frame of image is the copy frame status, and then there are two frames of image status in the skip frame status. Fig.11As shown in (b), the state of the 1st frame image is the copy frame state, the state of the 2nd frame image is the skip frame state, the state of the 3rd frame image is the skip frame state, the state of the 4th frame image is the copy state, the state of the 5th frame image is the skip frame state, the state of the 6th frame image is the skip frame state, the state of the 7th frame image is the copy state, and the state of the 8th frame image is the skip frame state.

[0228] When N=3 is preset, Fig.11 The status arrangement of the 8 frames of images shown is: the status of one frame of image is the copy frame status, and then there are three frames of image status in the skip frame status. Fig.11 As shown in (c), the state of the 1st frame image is the copy frame state, the state of the 2nd frame image is the skip frame state, the state of the 3rd frame image is the skip frame state, the state of the 4th frame image is the skip frame state, the state of the 5th frame image is the copy frame state, the state of the 6th frame image is the skip frame state, the state of the 7th frame image is the skip frame state, and the state of the 8th frame image is the skip frame state.

[0229] S713: When it is determined that the state of the current frame image is the scheme closed state, the created blur result layer is deleted.

[0230] When it is determined that the status of the current frame image is the scheme-off status, and the status of the previous frame image is the normal frame status, the copy frame status, or the skip frame status, it means that the image processing scheme is changed from the on state to the off state, and the blurred result layer is no longer involved in the layer synthesis process of the current frame image, so the created blurred result layer is deleted to avoid wasting resources.

[0231] Further, when it is determined that the status of the current frame image is the scheme closed state, and the status of the previous frame image is the scheme closed state, the image processing scheme is always in the closed state, and no blurred result layer is created, so there is no step of deleting the blurred result layer.

[0232] Furthermore, after the electronic device determines the first normal frame image, that is, according to the synthesis order, after determining for the first time that the current frame image has and only has one stable blur layer (that is, the blur layer stability parameter is 0), it is necessary to continuously determine that M frames have and only have one stable blur layer, that is, continuously determine that M frames have and only have one stable blur layer (it should be noted that at this time, the state of the M frames is a normal frame state), and then determine that the next frame image of the last frame image in the M frames has and only has one stable blur layer, the state of the frame image is a copy frame state (that is, before determining the state of the current frame image, it has been continuously determined that M frames have and only have one stable blur layer, then the state of the current frame image is a copy frame state). Wherein, M is a positive integer.

[0233] Specifically, each frame image carries a stable frame parameter, and the stable frame parameter is used to indicate the number of images that are continuously determined to have and have only one stable blurred layer. The current frame image has and has only one stable blurred layer, and the state of the previous frame image of the current frame image is a normal frame state, and the stable frame parameter carried by the previous frame image is judged: when the stable frame parameter indicates that the number of images that are continuously determined to have and have only one stable blurred layer reaches M, the state of the current frame image is a copy frame state; when the stable frame parameter indicates that the number of images that are continuously determined to have and have only one stable blurred layer does not reach M, the state of the current frame image is a normal frame state.

[0234] For ease of understanding, the following Fig.12 Here is an example. Here, 15 frames of images (namely, the first frame, the second frame, the third frame, the fourth frame, the fifth frame, the sixth frame, the seventh frame, the eighth frame, the ninth frame, the tenth frame, the eleventh frame, the twelfth frame, the thirteenth frame, the fourteenth frame, and the fifteenth frame) displayed by an electronic device in a display order (composition order) are used as an example for explanation. Fig.12 Only blurred layers in the image displayed by the electronic device are displayed. Among them, there is no blurred layer in the first frame image; there is and only one blurred layer in the second frame image, and the blurred layer is unstable; there is and only one blurred layer in the third frame image, and the blurred layer is unstable; there is and only one blurred layer in the fourth frame image, and the blurred layer is unstable; there is and only one blurred layer in the fifth to eleventh frames of images, and the blurred layer is stable; there is and only one blurred layer in the twelfth frame of image, and the blurred layer is unstable; there is and only one blurred layer in the thirteenth frame of image, and the blurred layer is unstable; there is no blurred layer in the fourteenth and fifteenth frames of image.

[0235] Take N=1 and M=3 as an example for illustration. Determine the states of the 1st to 15th frames according to the display order (synthesis order). There is no blurred layer in the 1st frame, so the state of the 1st frame is the scheme closed state; there is and only one blurred layer in the 2nd frame, and the blurred layer is unstable, so the state of the 2nd frame is the normal frame state; there is and only one blurred layer in the 3rd frame, and the blurred layer is unstable, so the state of the 3rd frame is the normal frame state; there is and only one blurred layer in the 4th frame, and the blurred layer is unstable, so the state of the 4th frame is the normal frame; there is and only one stable blurred layer in the 5th frame, and the 5th frame is the first image frame that has and only one stable blurred layer, the state of the 5th frame is a normal frame, and the stable frame parameter carried by the 5th frame is 1 (that is, it indicates that it has been continuously determined that 1 frame has and only one blurred layer in a stable stage); the 6th frame is the second image that has and only one stable blurred layer, and the state of the 6th frame is positive The state of the 7th frame is a normal frame, and the stable frame parameter carried by the 7th frame image is 3; the 8th frame image is the fourth image with only one stable blurred layer, and the stable frame parameter carried by the 7th frame image is 3. Since M=3, the state of the 8th frame image is a copy frame state; since N=1, the state of the 9th frame image is a skip frame state; the state of the 10th frame image is a copy frame state; the state of the 11th frame image is a skip frame state; there is only one blurred layer in the 12th frame image, but the blurred layer is unstable, and the state of the 12th frame image is a normal frame state; there is only one blurred layer in the 13th frame image, but the blurred layer is unstable, and the state of the 13th frame image is a normal frame state; there is no blurred layer in the 14th frame image and the 15th frame image, and the state of the 14th frame image is a scheme closed state; the state of the 15th frame image is a scheme closed state.

[0236] Continuously determining that M frames of images have and only have one stable blurred layer makes the determination of the image in the copied frame state more accurate, and avoids the situation where the electronic device has only one stable blurred layer in one frame of image, and the image after the frame of image does not satisfy the condition that there is and only one stable blurred layer, and the state of each frame of image is incorrectly determined, resulting in incorrect display of the electronic device, thereby affecting the user experience.

[0237] In an image processing method provided in an embodiment of the present application, the layer parameters related to the blurred layer and the state of the previous frame image in each frame image displayed by the electronic device are monitored and collected. The state of each frame image is determined, and the corresponding layer synthesis process is executed according to the state of each frame image. In the synthesis process of the copy frame image, the blur processing result performed by the GPU is stored in the blur cache layer, so that in the synthesis process of the skip frame image, all the layers and blur result layers corresponding to the skip frame image are controlled to be synthesized through the HWC to obtain the skip frame image. In the embodiment of the present application, the blur processing result of the copy frame image is reused in the synthesis process of the skip frame image, and the blur processing does not need to be performed by the GPU, which avoids the forced synthesis by the GPU due to the blur processing during the layer synthesis process. All layers and blur result layers of the skip frame image are synthesized directly through the HWC, thereby reducing the power consumption of the electronic device caused by the layer synthesis and blur processing by the GPU, improving the performance of the electronic device, and thus improving the user experience.

[0238] It should be noted that in the first case, when the state of a frame image is represented by the solution off state, there is no blurred layer in the frame image, and no blur processing is required. At this time, the image processing solution provided by the present application will not be executed; in the second case, when there are two or more blurred layers in the frame image, the reason why the image processing solution provided by the present application is not executed is that there are two or more blurred layers in the frame image, and it is necessary to create a layer covering the two blurred layers, and then process the two or more blurred layers separately, and then process them according to the relative relationship between the two or more blurred layers, which cannot effectively reduce the power consumption of the electronic device. Moreover, the time when two or more blurred layers are in a stable state (stable stage) may be very short, and the application time is very small, so it is difficult to effectively reduce the power consumption of the electronic device.

[0239] Embodiment 2:

[0240] Combine the following Figure 13-Figure 16 , which introduces in detail the interaction process between modules of an image processing method provided in an embodiment of the present application.

[0241] For the convenience of explanation, the interaction process between modules of the overall image processing method is divided into four parts for explanation, namely object initialization and layer parameter acquisition part, state machine decision part, layer synthesis method and position control part, and copy frame blur drawing part.

[0242] For ease of understanding, the following Fig.13 The object initialization and layer parameter collection parts provided in the embodiments of the present application are introduced in detail.

[0243] S1301. The layer management module (Surface flinger) calls the practical state setting function to initialize the object.

[0244] S1302. The surface management module (Surface flinger) sets the state of the current frame image displayed by the electronic device to a scheme closed state.

[0245] S1303, the layer management module (Surface flinger) calls the constructor, thereby calling the frame interval module to store the image processing function in the frame interval module into the 2D vector graphics processing function library (skia module).

[0246] Among them, the frame-interval module is a pre-developed programming language program (the frame-interval module is the main implementation module of an image processing method provided by this application), the purpose of which is to support the electronic device to determine the state of each frame of the image, switch on and off the image processing scheme, build and destroy the blur result layer, etc. The frame-interval module is stored in the system library of the electronic device operating system in the form of a source file of a programming language program, for example: efficientblur.cpp. The programming language program can be: C++ language, C language, etc., and this application does not make specific limitations.

[0247] S1304: When a blurred layer exists in the current frame image of the electronic device, the layer management module (Surfaceflinger) creates a blurred layer.

[0248] When there is a blurred layer in the current frame image displayed by the electronic device, the layer management module will call the layer creation function to create a corresponding blurred layer, so that the subsequent layer management module can control the blurred layer and other layers for HWC synthesis or GPU synthesis.

[0249] S1305. The layer management module (Surface flinger) sends the name of the created blur layer to the frame interval module.

[0250] The layer management module sends the name of the created fuzzy layer to the frame-interval module, so that the frame-interval module can collect the fuzzy layer name of the current frame image.

[0251] S1306. After the layer management module (Surface flinger) completes creating the blurred layer thread, it sends the number of blurred layers in the current frame image to the frame interval module.

[0252] After the thread of creating blurred layers in the current frame image of the electronic device is completed, that is, all blurred layers in the current frame image are created or there is no blurred layer in the current frame image, the blurred layer number parameter in the current frame image is sent to the frame interval module so that the frame interval module can collect the blurred layer number of the current frame image.

[0253] S1307, after the layer management module (Surface flinger) completes creating the blurred layer thread, it sends the blurred layer change parameters in the current frame image to the frame interval module.

[0254] After the blur layer creation thread in the current frame image of the electronic device is completed, that is, all blur layers in the current frame image are created or there is no blur layer in the current frame image, the blur layer change parameters in the current frame image are sent to the frame interval module so that the frame interval module can collect the blur layer change parameters of the current frame image.

[0255] S1308. The frame-by-frame module collects and stores layer parameters related to the blurred layer in each frame of the image.

[0256] Specifically, after receiving the fuzzy layer name sent by the layer management module, the fuzzy layer name is collected and stored; after receiving the fuzzy layer number sent by the layer management module, the fuzzy layer number ...

[0257] The above S1305, S1306, and S1307 implement the electronic device to collect the layer parameters related to the blurred layer in each frame image. Among them, S1305 is collected after the blurred layer is created when the blurred layer appears / exists in the current frame image. If the blurred layer is not created, the blurred layer does not exist in the current frame image, and it is impossible to collect the name of the blurred layer in the current frame image. Among them, S1306 and S1307 are required to be monitored and collected by the electronic device for each frame image.

[0258] The object initialization and layer parameter acquisition part provided in the embodiment of the present application realizes the initialization of object variables before the electronic device performs the image processing scheme, and initializes the state of the current frame image to the scheme closed state. At the same time, the layer parameters related to the blurred layer of each frame image are monitored and collected for subsequent use by the state machine decision part.

[0259] For ease of understanding, the following Fig.14 The state machine decision part provided in the embodiment of the present application is introduced in detail.

[0260] S1401. The layer management module (Surface flinger) sends a call request to the frame interval module.

[0261] The layer management module sends a call request to the frame interval module, thereby calling the frame interval module to determine the status of each frame image.

[0262] S1402, the frame-by-frame module determines the state of each frame of the image.

[0263] The frame-interval module responds to the call request and determines the state of each frame of the image.

[0264] Taking the current frame image as an example, the inter-frame module collects the layer parameters related to the blurred layer in the current frame image (i.e., the layer parameters of the blurred layer), and determines the state of the current frame image based on the layer parameters related to the blurred layer corresponding to the current frame image and the state of the previous frame image.

[0265] S1403. The frame-interval module determines whether to create a blurred result layer or destroy the blurred result layer according to the state of the current frame image and the state of the previous frame image.

[0266] When the frame interval module determines that the state of the current frame image is a normal frame state, and the state of the previous frame image is a scheme closed state, S1404 is performed to create a blur result layer.

[0267] When the frame interval module determines that the state of the current frame image is the scheme closed state, and the state of the previous frame image is the non-scheme closed state (normal frame state, copy frame state or skip frame state), S1405 is performed to destroy the blurred result layer.

[0268] S1404, the frame-by-frame module creates a blurred result layer.

[0269] The frame skipping module calls the 2D vector graphics processing function library (skia module) to create a blur result layer for storing the blur processing results corresponding to the copied frame image, and involves the blur result layer in the layer synthesis process of the skipped frame image.

[0270] Furthermore, when creating the fuzzy result layer, a buffer corresponding to the fuzzy result layer is created to store the data of the fuzzy processing result of the fuzzy result layer and participate in the rotation of the buffer of the fuzzy result layer. The combination of the buffer and the layer can display the corresponding fuzzy processing result to the user through the display screen of the electronic device. It can also be said that the layer is the carrier of the buffer, and the buffer is the carrier of the fuzzy processing result.

[0271] S1405, the frame-by-frame module destroys the blurred result layer.

[0272] When the frame-interval module destroys the blurred result layer, it also destroys the buffer corresponding to the blurred result layer to avoid occupying unnecessary storage space and to prevent too many layers from affecting the normal display of electronic devices.

[0273] S1406. The frame-interval module sets whether the blurred result layer is visible according to the state of the current frame image.

[0274] If the current frame image is in the copy frame state or the skip frame state, the blur result layer is set to be visible.

[0275] Specifically, the state of the current frame image is the copy frame state, the blurred result layer is set to visible, and GPU synthesis is performed together with the blurred layer of the current frame image and the layers related to the blurred layer (layers below the blurred layer). However, only the blurring result performed by the GPU is copied to the blurred result layer. Apart from this, the blurred result layer does not participate in layer synthesis.

[0276] The state of the current frame image is the frame skipping state. The blur result layer is set to be visible and HWC synthesis is performed together with all layers of the current frame image. The blur result layer participates in layer synthesis.

[0277] If the current frame image is in a normal frame state, the blurred result layer is set to be invisible.

[0278] Furthermore, when the state of the current frame image is the scheme closed state, there is no corresponding blur result layer.

[0279] Specifically, when the current frame image is in the normal frame state, the blur result layer is set to be invisible. Because when the current frame image is in the normal frame state, the layer synthesis process of the current frame image has not changed (GPU synthesis and blur processing are performed first, and then HWC synthesis is performed), so the blur result layer is set to be invisible, and the blur result layer is not allowed to participate in the layer synthesis process of the current frame image to avoid unnecessary resource consumption.

[0280] It should be noted that only when the blurred result layer is set to visible can the blurred result layer participate in the layer synthesis process of the current frame image; when the blurred result layer is set to invisible, the blurred result layer cannot participate in the layer synthesis process of the current frame image.

[0281] Furthermore, when the state of the current frame image is the copy frame state, during the synthesis process of the copy frame image, the blur processing result is stored in the blur result layer.

[0282] S1407: When the state of the current frame image is the copy frame state, in the synthesis process of the copy frame image, the alternate frame module sets the blur processing result to the buffer corresponding to the blur result layer.

[0283] In fact, it is to store the fuzzy result layer that stores the fuzzy processing results in the corresponding buffer.

[0284] S1408. The frame-interval module sends a call request to the primitive operation package.

[0285] The frame-by-frame module sends a call request to the primitive operation package, thereby calling the primitive operation package to set the buffer with the blurred processing result added to the blurred result layer.

[0286] S1409, the primitive operation encapsulation sets the buffer with the blurred processing result added to the blurred result layer.

[0287] The primitive operation encapsulation responds to the call request sent by the frame-by-frame module and sets the buffer with the blurred processing result added to the blurred result layer.

[0288] Draw the blur processing result on the buffer corresponding to the blur result layer, combine the buffer with the blur result layer, and store the blur processing result in the blur result layer.

[0289] It should be noted that S1407-S1409 realizes storing the blur processing results in the blur result layer during the synthesis process of the copy frame image.

[0290] The state machine decision part provided in the embodiment of the present application realizes determining the state of each frame image according to the layer parameters related to the blurred layer in each frame image, and based on the state of each frame image, decides on or off the scheme, creates or destroys the blurred result layer, sets whether the blurred result layer is visible, controls the copying process of the blurred processing result when copying the frame state, etc.

[0291] For ease of understanding, the following Fig.15 The layer synthesis method and position control part provided in the embodiment of the present application are introduced in detail.

[0292] S1501. The layer management module (Surface flinger) sends a call request to the synthesis engine.

[0293] The layer management module (Surface flinger) sends a call request to the synthesis engine, thereby calling the synthesis engine to traverse all layers of the current frame image.

[0294] S1502: The synthesis engine traverses all layers of the current frame image.

[0295] The synthesis engine traverses all layers of the current frame image in response to the call request sent by the layer management module.

[0296] The synthesis engine traverses all layers of the current frame image, and if there is a blurred result layer, S1503 is performed. If there is no blurred result layer, the image processing solution provided by the present application is not enabled, and the layer synthesis process of the current frame image is not affected.

[0297] S1503: The synthesis engine sends a call request to the output module.

[0298] The synthesis engine sends a call to the output module, thereby calling the synthesis engine to control the position of the visible layer.

[0299] S1504. The synthesis engine controls the position of the visible layer.

[0300] The synthesis engine controls the position of the visible layer in response to the call request sent by the output module.

[0301] Since the default position of the newly created blur result layer is the top of all layers, that is, the Z-order is the largest, you need to adjust the position of the blur result layer, that is, control the positions of all visible layers.

[0302] The output module is stored in the system library of the electronic device operating system in the form of a source file of a programming language, which may be called output.cpp (source file of C++ language).

[0303] The output module performs position control of the visible layer specifically as follows: S1505 and S1506.

[0304] S1505. The output module collects all visible layers in the current frame image.

[0305] Determine whether there is a blurred result layer among all the collected visible layers.

[0306] If there is no blurred result layer among all visible layers, there is no need to control / adjust the position of the visible layers.

[0307] If there is a blurred result layer among all visible layers, proceed to S1506.

[0308] S1506. The output module sends a call request to the inter-frame module.

[0309] The output module sends a call request to the frame-interval module, thereby calling the frame-interval module to control the blurred result layer to move to the next layer position of the blurred layer.

[0310] S1507, the frame interval module controls the blurred result layer to move to the next layer position of the blurred layer.

[0311] The frame interval module controls the blurred result layer to move to the next layer position of the blurred layer in response to the call request sent by the output module.

[0312] The blurred result layer is moved to the next layer of the blurred layer, that is, the blurred result layer needs to be located below the blurred layer and next to the blurred layer. For example, when the position control of the visible layer is not performed, assuming that the blurred result layer is located at the top layer, the Z-order of the blurred result layer is 5, and assuming that the Z-order of the blurred layer is 3, after the position control of the visible layer is performed, the Z-order of the blurred layer is 4, and the Z-order of the blurred result layer is 3.

[0313] S1508. The synthesis engine sends a call request to the output module.

[0314] The synthesis engine sends a call request to the output module, thereby calling the output module to control the layer synthesis method of the current frame image.

[0315] S1509: The output module controls the layer synthesis method of the current frame image.

[0316] The output module controls the layer synthesis method of the current frame image specifically as S1509-S1511.

[0317] S1510: The output module requests the frame-by-frame module to call back the status of the current frame image.

[0318] The output module responds to the call request sent by the synthesis engine and requests the frame interval module to call back the status of the current frame image.

[0319] Since the frame-interval module has determined the state of the current frame image according to the layer parameters related to the blurred layer in the current frame image and the state of the previous frame image, the output module requests the frame-interval module to call back the state of the current frame image.

[0320] S1511. The inter-frame module returns the status of the current frame image to the output module.

[0321] The frame interval module returns the status of the current frame image in response to the request for calling back the status of the current frame image sent by the output module.

[0322] S1512: The output module controls the layer synthesis method of the current frame image according to the state of the current frame image.

[0323] If the state of the current frame image is the frame skipping state, all layers of the current frame image and the blurred result layer are controlled to perform HWC synthesis.

[0324] If the state of the current frame image is the scheme closed state, the normal frame state or the copy frame state, the layer synthesis method of the current frame image will not be affected.

[0325] The layer synthesis method and position control part provided in the embodiment of the present application enable the electronic device to control the position of the newly created blur result layer, as well as control the layer synthesis method of each frame image.

[0326] For ease of understanding, the following Fig.16 The copy frame blur drawing part provided in the embodiment of the present application is introduced in detail.

[0327] When the output module requests the alternate frame module to call back the current frame state, the blurring result is copied to the blurring result layer during the layer synthesis process of the copy frame image (specifically, during the GPU synthesis process of the copy frame image), as follows:

[0328] S1601, the output module controls the current frame image to perform layer synthesis.

[0329] Specifically, the output module controls the current frame image to perform GPU synthesis and HWC synthesis through the compose surfaces() function. In the process of the output module controlling the current frame image to perform GPU synthesis, the blur processing result is copied to the blur result layer. Specifically, S1603-S1607.

[0330] S1602: The output module sends a call request to the rendering engine.

[0331] The output module sends a call request to the drawing engine, thereby calling the drawing engine to draw the layer.

[0332] S1603: The drawing engine performs layer drawing.

[0333] The rendering engine performs layer rendering in response to the call request sent by the output module.

[0334] The process of storing the fuzzy processing result in the fuzzy result layer is the process of drawing the fuzzy processing result on the fuzzy result layer. Therefore, the output module calls the drawing engine, and the drawing engine responds to the call request sent by the output module and controls the subsequent fuzzy processing result in the fuzzy result layer through the drawing engine.

[0335] S1604: The drawing engine sends a call request to the 2D graphics engine.

[0336] The drawing engine sends a call request to the 2D graphics engine, thereby calling the 2D graphics engine to draw the blur processing result.

[0337] S1605. The 2D graphics engine draws the fuzzy processing result.

[0338] The 2D graphics engine draws the blur processing result in response to the call request sent by the drawing engine.

[0339] Among them, the 2D graphics engine is a drawing engine for 2D drawing.

[0340] S1606. The 2D graphics engine requests the frame-by-frame module to obtain the blurred result layer.

[0341] The drawing engine of the 2D drawing is called to draw the blur processing result. During the drawing process of the blur processing result, the blur result layer is requested to be obtained from the frame-by-frame module.

[0342] S1607. The frame-by-frame module returns the blurred result layer to the 2D graphics engine.

[0343] The frame-interval module returns the blur result layer to the 2D graphic engine in response to the request of the 2D graphic engine.

[0344] S1608. The 2D graphics engine draws the blur processing result to the blur result layer.

[0345] The 2D graphics engine draws the blur processing result to the blur result layer, that is, completes the process of storing the blur processing result on the blur result layer.

[0346] It should be noted that S1606 to S1607 are actually because when the fuzzy result layer is created in advance, in order to show the normal operation of the pipeline, a buffer corresponding to the fuzzy result layer will be created at the same time, and the corresponding buffer and fuzzy result layer (layer) have been bound. Therefore, the 2D graphics engine requests the frame-by-frame module to obtain the fuzzy result layer, and at the same time, the 2D graphics engine requests the frame-by-frame module to obtain the buffer corresponding to the fuzzy result layer; the frame-by-frame module returns the fuzzy result layer and the corresponding buffer; the 2D graphics engine draws the fuzzy processing result to the fuzzy result layer, which is actually the 2D graphics engine copying the fuzzy processing result to the buffer corresponding to the fuzzy result layer, and the corresponding buffer is bound to the fuzzy result layer, so the fuzzy processing result is drawn to the fuzzy result layer.

[0347] The copy frame drawing part provided in the embodiment of the present application realizes that in the layer synthesis process of the copy frame image, the blur processing result is copied to the blur result layer, so that in the subsequent layer synthesis process of the skip frame image, the blur processing result of the copy frame image can be directly reused, thereby avoiding forced GPU synthesis due to blur processing during the image synthesis process, reusing the blur processing result of the copy frame image, and synthesizing the skip frame image through HWC, which reduces the power consumption of the electronic device, improves the performance of the electronic device, and thus improves the user experience.

[0348] Embodiment three:

[0349] Combine the following Fig.17 , another image processing method provided in an embodiment of the present application is introduced in detail.

[0350] S1701. Acquire a first image.

[0351] The first image includes a first layer and a second layer. The first layer carries blur attribute information, which is used to indicate that the electronic device needs to perform corresponding blur processing on the layer below the first layer, so the first layer can also be called a blur layer.

[0352] In the embodiment of the present application, the second layer is located below the first layer and is a layer related to the blur processing corresponding to the first layer.

[0353] Scenarios for acquiring the first image include: adjusting the volume of the electronic device, a message window popping up on the electronic device, pulling down the status bar, shutting down, a multi-tasking interface, etc.

[0354] In a possible implementation, an adjustment operation for the volume of an electronic device is received, and in response to the adjustment operation for the volume of the electronic device, a first image is acquired; at this time, the first layer in the first image is a layer of a volume bar.

[0355] In a possible implementation, when a message window pops up on an electronic device, a first image is acquired, and at this time, a first layer in the first image is a pop-up window layer.

[0356] In a possible implementation, a pull-down operation on the status bar is received, and in response to the pull-down operation on the status bar, a first image is acquired, and at this time, a first layer in the first image is a status bar display layer.

[0357] In a possible implementation, a shutdown operation of an electronic device is received, and in response to the shutdown operation of the electronic device, a first image is acquired, where a first layer in the first image is a shutdown interface layer.

[0358] S1702: Perform blur processing on the second layer to obtain a blur result layer, and store the blur result layer in the first cache.

[0359] The first buffer is a buffer pre-bound to the fuzzy result layer.

[0360] In a possible implementation, during the process of blurring and synthesizing the first layer and the second layer by the image processor GPU, the second layer is blurred by the GPU to obtain a blurred result, and the blurred result is stored in a blurred result layer to obtain a blurred result layer carrying the blurred result.

[0361] In a possible implementation, the second layer is located at a lower layer of the first layer, and the blurring of the second layer is related to the blurring of the first layer. The first layer and the second layer are synthesized and blurred by a graphics processor GPU, and the second layer is blurred by the GPU to obtain a blurring result; the blurring result is stored in a blurring result layer. Specifically, the second layer is synthesized (overlapped) by the GPU, and the second layer after the overlapping process is blurred to obtain an initial blurring result, and the initial blurring result is cropped according to the first layer to obtain a blurring result, and then the blurring result, the first layer, and the second layer are layer-synthesized to obtain a GPU synthesis result. At the same time, the blurring result is stored in the blurring result layer. If there are still layers that have not been layer-synthesized, the GPU synthesis result and the remaining layers (layers that are layer-synthesized) are synthesized by the hardware synthesizer HWC to complete the layer synthesis of the first image.

[0362] S1703: Acquire a second image.

[0363] The second image includes a third layer, and the third layer carries blur attribute information.

[0364] The number of the third layer and the first layer is 1, and the first layer and the third layer have the same shape, size and relative position. That is, the first layer and the third layer are layers with the same name, and the third layer is in a stable stage compared to the first layer. When the third layer is in a stable stage compared to the first layer, the third layer does not change compared to the first layer, so the shape, size and relative position of the first layer and the third layer are the same.

[0365] In a possible implementation manner, the second image is the next N frames of the first image, where N is a positive integer.

[0366] S1704. Perform synthesis processing on the blurred result layer and the third layer through the hardware synthesizer HWC.

[0367] The third layer and the blurred result layer in the second image are directly synthesized through HWC. The second image reuses the blurred processing result of the first image, so the second image does not need to be blurred, avoiding the need to force the GPU to perform layer synthesis and blur processing due to the need for blur processing. Skipping the blur processing process of the GPU, directly controlling the third layer and the blurred result layer of the second image to be synthesized through HWC, reducing the power consumption of electronic devices caused by forcing the GPU to perform layer synthesis and blur processing, improving the performance of electronic devices, and thus improving the user experience.

[0368] In a possible implementation, the blur result layer is stored in a position below the third layer, and the blur result layer and the third layer are synthesized by a hardware synthesizer HWC. The blur result layer is ensured to be close to the third layer, and the third layer and the blur processing layer are synthesized by HWC, so as to ensure the reuse of the blur processing result stored in the blur result layer, so as to achieve the blur processing effect of the second image reusing the first image.

[0369] An image processing method provided by an embodiment of the present application, in which the second image reuses the blur processing result of the first image, and no blur processing is required, thereby avoiding the need to force the GPU to perform layer synthesis and blur processing due to blur processing during layer synthesis, thereby reducing the power consumption of the electronic device caused by layer synthesis and blur processing through the GPU and improving the performance of the electronic device.

[0370] Furthermore, after S704, the method further includes: displaying a target image obtained by synthesizing the blur result layer and the third layer, so that the electronic device can normally display the second image with the blur processing effect.

[0371] The target image is an image to be displayed obtained by multiplexing the blur processing result of the first image with the second image.

[0372] Further, after acquiring the first image, it is determined that the first image is in the first state according to the state of the first layer and the state of the previous frame image of the first image.

[0373] In a possible implementation manner, it is determined that the first image is in the first state according to a layer parameter of the first layer and a state of a previous frame image of the first image.

[0374] The layer parameters of the first layer include the number of the first layer and the first layer change parameter.

[0375] Determine that the first image is in the first state, specifically as follows:

[0376] When the number of the first layer is 1, the first layer change parameter indicates that the first layer is in a stable stage, and the previous frame image of the first image is in the third state, it is determined that the state of the first image is the first state.

[0377] Further, when the number of first layers is 1, the first layer change parameter indicates that the first layer is in a stable stage, and the second image is in the third state, the stable frame parameter carried by the previous frame of the first image is obtained; when it is determined that the number of third images indicated by the stable frame parameter has accumulated to M, the first image is in the first state. The stable frame parameter can effectively avoid executing the synthesis process corresponding to the first state due to the false stability phenomenon, and there is no subsequent waste of resources caused by the corresponding second state, thereby avoiding increased power consumption loss.

[0378] The third image includes a first layer, and the number of the first layer is 1, which is in a stable stage.

[0379] When the number of first layers is 1, the first layer change parameter indicates that the first layer is in a stable stage, the previous frame image of the first image is in the second state, and the images in the second state accumulate to N frames, it is determined that the state of the first image is a copy frame state.

[0380] It should be noted that the first state can also be called "copy frame state", the second state can also be called "skip frame state", the third state can also be called "normal frame state", and the fourth state can also be called "scheme closed state", and this application does not make specific limitations.

[0381] Further, after acquiring the second image, it is determined that the second image is in the second state according to the state of the third layer and the state of the previous frame image of the second image.

[0382] In a possible implementation manner, it is determined that the second image is in the second state according to the layer parameters of the third layer and the state of a previous frame image of the second image.

[0383] The layer parameters of the third layer include the number of the third layer and the third layer change parameter.

[0384] Determine that the second image is in the second state, specifically as follows:

[0385] When the number of the third layer is 1, the third layer change parameter indicates that the third layer is in a stable stage, and the previous frame image of the second image is in the first state, it is determined that the second image is in the second state.

[0386] When the number of third layers is 1, the third layer change parameter indicates that the third layer is in a stable stage, the previous frame image of the second image is in the second state, and the image accumulation of the second state does not reach N frames, it is determined that the second image is in the second state.

[0387] Furthermore, an image in a third state is also included. Assuming that the image in the third state is a fourth image, the fourth image includes a fourth layer, and the fourth layer carries blur attribute information. Determining that the fourth image is in the third state is specifically as follows:

[0388] When the number of the fourth layer is 1, and the fourth layer change parameter indicates that the fourth layer is not in a stable stage, it is determined that the fourth image is in the third state.

[0389] When the number of the fourth layer is 1, and the previous frame image of the fourth image is in the fourth state, it is determined that the fourth image is in the third state.

[0390] Further, when the number of fourth layers is 1, the fourth layer change parameter indicates that the fourth layer is in a stable stage, and the previous frame image of the fourth image is in the third state, the stable frame parameter of the previous frame image of the fourth image is obtained; when it is determined that the stable frame parameter indicates that the cumulative number of third images has not reached M, then the fourth image is in the third state.

[0391] Furthermore, an image in a fourth state is also included, and it is assumed that the image in the fourth state is the fifth image.

[0392] The fifth image is determined to be in the fourth state, specifically as follows:

[0393] When the number of layers carrying blur attribute information is not 1 (ie, 0 or greater than 1), the fifth image is determined to be in the fourth state.

[0394] Furthermore, the images in the first state, the second state, and the third state carry indication bit information indicating that the scheme is turned on, and the image in the fourth state carries indication bit information indicating that the scheme is turned off.

[0395] In a possible implementation, the current frame image carries the indication bit information indicating that the scheme is turned on (the image in the first state, the second state or the third state), and the previous frame image of the current frame image carries the indication bit information indicating that the scheme is turned off (the image in the fourth state), and a blur result layer is created. This is convenient for use when the images in the first state and the second state are layered.

[0396] In a possible implementation, the current frame image carries the indication bit information indicating that the scheme is closed, and the previous frame image of the current frame image carries the indication bit information indicating that the scheme is opened, and the blurred result layer is deleted. At this time, the image processing scheme is turned from the open state to the closed state, and the blurred result layer is no longer involved in the layer synthesis process of the first image, so the blurred result layer is deleted to avoid wasting resources and further avoid affecting the layer synthesis of the image in the closed state of the scheme.

[0397] The embodiment of the present application provides an image processing method, including: obtaining a first image; the first image includes a first layer and a second layer, wherein the first layer carries blur attribute information; blurring the second layer to obtain a blur result layer, and storing the blur result layer in a first cache; obtaining a second image, wherein the second image includes a third layer, and the third layer carries blur attribute information; synthesizing the blur result layer and the third layer through a hardware synthesizer HWC. The first image is blurred and the blurred layer is stored, and the second image reuses the blur result layer of the first image, that is, reuses the blur processing result of the first image, and does not need to be blurred, thereby avoiding the forced layer synthesis and blur processing by the GPU during the layer synthesis process due to the blur processing, and directly synthesizing the layer of the second image and the blur result layer through the HWC to realize the blur processing of the first image, thereby reducing the power consumption of the electronic device caused by the layer synthesis and blur processing by the GPU, improving the performance of the electronic device, and thus improving the user experience.

[0398] Combine the following Fig.18 , an image processing method provided in this application is compared with the original real-time blur method.

[0399] like Fig.18 (a) shows the original real-time blurred path tracking diagram. The original real-time blurred image needs to be forcibly synthesized by the GPU for each frame, which will result in high power consumption, resulting in high power consumption of electronic devices, thereby reducing the performance of the electronic devices and seriously affecting the user experience.

[0400] like Fig.18 (b) shows an image processing method, and the path tracking diagram when N=1, that is, a copy frame image and a skip frame image are cycled. The copy frame image is synthesized and blurred by the GPU, and then synthesized by the HWC; the skip frame image reuses the blurred processing result of the copy frame image, avoiding the need to force the use of GPU synthesis when blurring is performed in the skip frame image synthesis stage, and directly synthesizes all layers and blurred result layers of the skip frame image through the HWC.

[0401] like Fig.18 (c) shows an image processing method, a path tracking diagram when N=2, that is, a one-frame copy frame image and two-frame skip frame image loop. The copy frame image is synthesized and blurred by the GPU, and then synthesized by the HWC; the skip frame image reuses the blurred processing result of the copy frame image, avoiding the need to force the use of GPU synthesis when blurring is performed in the skip frame image synthesis stage, and directly synthesizes all layers and blurred result layers of the skip frame image through the HWC.

[0402] Since the blur processing results of the copied frame are reused when skipping frames, there is no need to blur all layers under the blurred layer through the GPU, thus avoiding the need to force the use of GPU synthesis during the blur processing in the frame skipping image synthesis stage, avoiding the high GPU load problem caused by forced GPU synthesis, reducing the power consumption of electronic devices, improving the performance of electronic devices, and thus improving the user experience.

[0403] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed, the electronic device implements the above-mentioned functions or steps.

[0404] An embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, enables an electronic device to implement the above-mentioned functions or steps.

[0405] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Acquire a first image; the first image includes a first layer and a second layer; The first layer carries fuzzy attribute information; Performing blur processing on the second layer to obtain a blur result layer, and storing the blur result layer in the first cache; Acquire a second image; the second image includes a third layer; the third layer carries fuzzy attribute information; The blurred result layer and the third layer are synthesized by a hardware synthesizer HWC.

2. The method according to claim 1, characterized in that The acquiring of the first image comprises: Receiving an adjustment operation for the volume of an electronic device; In response to an adjustment operation on the volume of the electronic device, the first image is acquired, and the first layer in the first image is a volume bar layer.

3. The method according to claim 1, characterized in that The acquiring of the first image comprises: When a message window pops up on the electronic device, the first image is obtained, and the first layer of the first image is a pop-up window layer.

4. The method according to claim 1, characterized in that: The acquiring of the first image comprises: Receive the pull-down operation on the status bar; In response to the pull-down operation on the status bar, the first image is acquired, and the first layer of the first image is a status bar display layer.

5. The method according to claim 1, characterized in that The second image is the next N frames of image of the first image, wherein N is a positive integer.

6. The method according to claim 1, characterized in that The number of the first layer and the third layer is 1, and the first layer and the third layer have the same shape, size and relative position.

7. The method according to claim 1, characterized in that The second layer is located below the first layer; The blurring the second layer to obtain a blur result layer includes: In the process of blurring and compositing the first layer and the second layer by a graphics processor (GPU), blurring the second layer by the GPU to obtain a blurring result; The blur processing result is stored in the blur result layer.

8. The method according to claim 1, characterized in that The synthesizing process of the blur result layer and the third layer by a hardware synthesizer HWC includes: The blur result layer and the third layer are synthesized by a hardware synthesizer HWC in the order that the blur result layer is located in a layer below the third layer.

9. The method according to claim 1, characterized in that: After acquiring the first image, the method further includes: According to the state of the first layer and the state of a previous frame image of the first image, it is determined that the first image is in a first state.

10. The method according to claim 9, characterized in that The step of determining that the first image is in the first state according to the state of the first layer and the state of a previous frame of the first image includes: According to the layer parameters of the first layer and the state of a previous frame image of the first image, it is determined that the first image is in a first state.

11. The method according to claim 10, characterized in that The layer parameters of the first layer include: the number of the first layer and the first layer change parameter; The step of determining that the first image is in the first state according to the layer parameter of the first layer and the state of a previous frame of the first image includes: When the number of the first layers is 1, the first layer change parameter indicates that the first layer is in a stable stage, and the previous frame image of the first image is in the third state, determining that the first image is in the first state; When the number of the first layers is 1, the first layer change parameter indicates that the first layer is in a stable stage, the previous frame image of the first image is in the second state, and the image of the second state accumulates to N frames, it is determined that the first image is in the first state; wherein N is a positive integer.

12. The method according to claim 11, characterized in that When the number of the first layers is 1, the first layer change parameter indicates that the first layer is in a stable stage, and the previous frame image of the first image is in the third state, before determining that the first image is in the first state, the method further includes: Acquire stable frame parameters carried by a previous frame of the first image; Determining that the stable frame parameter indicates that the number of third images reaches M; the third image includes a first layer, and the number of first layers included in the third image is 1, and the first layer is in a stable stage; wherein M is a positive integer.

13. The method according to claim 1, characterized in that After acquiring the second image, the method further includes: According to the state of the third layer and the state of a previous frame image of the second image, it is determined that the second image is in the second state.

14. The method according to claim 13, characterized in that The step of determining that the second image is in the second state according to the state of the third layer and the state of a previous frame of the second image comprises: According to the layer parameters of the third layer and the state of the previous frame image of the second image, it is determined that the second image is in the second state.

15. The method according to claim 14, characterized in that The layer parameters of the third layer include: the number of the third layer and the third layer change parameter; The step of determining that the second image is in the second state according to the layer parameter of the third layer and the state of the previous frame image of the second image includes: When the number of the third layer is 1, the third layer change parameter indicates that the third layer is in a stable stage, and the previous frame image of the third image is in the first state, determining that the second image is in the second state; When the number of the third layers is 1, the third layer change parameter indicates that the third layer is in a stable stage, the previous frame image of the third image is in the second state, and the accumulation of images in the second state does not reach N frames, it is determined that the second image is in the second state; wherein N is a positive integer.

16. The method according to any one of claims 1 to 15, characterized in that: The method further comprises: The target image obtained by synthesizing the blurred result layer and the third layer is sent for display.

17. An electronic device, characterized in that: including a processor and a memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed, the method according to any one of claims 1 to 16 is implemented.

Images