Image processing method and electronic device

Abstract

The application provides an image processing method and an electronic device, and relates to the technical field of display. The method comprises the following steps: acquiring a first image; the first image comprises a first layer and a second layer, wherein the first layer carries blur attribute information; performing blur processing on the second layer to obtain a blur result layer, and storing the blur result layer in a first cache; acquiring a second image; the second image comprises a third layer, and the third layer carries blur attribute information; and performing synthesis processing on the blur result layer and the third layer by using a hardware synthesizer (HWC). The second image reuses the blur processing result of the first image, and does not need to perform blur processing, thereby avoiding forcibly performing layer synthesis and blur processing by using a GPU due to performing blur processing in the layer synthesis process, reducing the power consumption of the electronic device caused by performing layer synthesis and blur processing by using the GPU, and improving the performance of the electronic device.

Description

Technical Field

[0001] This application relates to the field of display technology, and more particularly to an image processing method and an electronic device. Background Technology

[0002] People's daily lives are inseparable from various electronic devices, many of which have display interfaces. With the development of science and technology, many electronic devices with display interfaces (such as mobile phones and tablets) now feature background blur functionality. This is used in various scenarios, including notification bars, status bar pull-downs, pop-ups, desktop folders, and lock screens. This function enhances the aesthetics of the interface by blurring the background, thereby improving the user experience.

[0003] Background blurring functions of electronic devices are generally divided into static blurring and dynamic blurring. In order to effectively improve the user's visual experience, the background blurring function in current technology generally adopts dynamic blurring, also known as real-time blurring. Real-time blurring transmits each frame of the background of the electronic device as a data source to the GPU for compositing and blurring operations, and then the blurred background is cropped according to the scene requirements.

[0004] Current technology suffers from high power consumption and reduced performance of electronic devices, which seriously affects the user experience. Summary of the Invention

[0005] This application provides an image processing method and electronic device that reduces power consumption when implementing real-time blurring and improves the performance of the electronic device, thereby enhancing the user experience.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, this application provides an image processing method, which includes: acquiring a first image; the first image comprising a first layer and a second layer, wherein the first layer carries blur attribute information; performing blur processing on the second layer to obtain a blurred result layer, and storing the blurred result layer in a first cache; acquiring a second image, the second image comprising a third layer, the third layer carrying blur attribute information; and performing composite processing on the blurred result layer and the third layer using a hardware compositer (HWC). By performing blur processing on the first image and storing the blurred layer, and reusing the blurred result layer of the first image in the second image (i.e., reusing the blur processing result of the first image without performing blur processing itself), this method avoids the forced layer compositing and blur processing via the GPU during layer compositing. Instead, it directly uses the HWC to composite the layers of the second image and the blurred result layer to achieve the blur processing of the first image. This reduces the power consumption of electronic devices caused by layer compositing and blur processing via the GPU, improves the performance of electronic devices, and thus enhances the user experience.

[0008] In one possible implementation, a volume adjustment operation on an electronic device is received, and in response to the volume adjustment operation, a first image is acquired, wherein the first layer of the first image is a volume bar layer. That is, when a volume adjustment operation is performed on the electronic device, a first image including the volume bar layer is acquired.

[0009] In one possible implementation, when a message window pops up on the electronic device, a first image is acquired, and the first layer of the first image is the pop-up window layer. That is, when the electronic device receives a notification message, it will notify the user via a message pop-up window, and thus acquire the first image including the pop-up window layer.

[0010] In one possible implementation, a pull-down operation on the status bar is received; in response to the pull-down operation on the status bar, a first image is acquired, wherein the first layer of the first image is the status bar display layer. That is, when the user pulls down the status bar of the electronic device, the electronic device, in response to the pull-down operation on the status bar, acquires a first image including the status bar display layer.

[0011] In one possible implementation, a power-off operation is received on the electronic device; in response to the power-off operation, a first image is acquired, wherein the first layer of the first image is a power-off layer. That is, when the user performs a power-off operation on the electronic device, the electronic device acquires a first image including the power-off layer in response to the power-off operation.

[0012] In one possible implementation, the second image is the N frames following the first image, where N is a positive integer. After blurring the first image to obtain a blurred result layer, the N frames following the first image reuse this blurred result layer. The larger N is, the more images reuse the blurred result layer. Therefore, for scenarios with lower real-time requirements, N can be slightly larger, which can further reduce power consumption issues caused by the GPU.

[0013] In one possible implementation, the number of both the first layer and the third image is one, and the first layer and the third image have the same shape, size, and relative position. The third layer is a layer that does not change compared to the first layer, and the image processing method provided in this application is applicable when the number of layers carrying blur attribute information is only one.

[0014] In one possible implementation, the second layer is located below the first layer. During the blurring and compositing process of the first and second layers using the GPU, the GPU blurs the second layer to obtain a blurred result; this result is then stored in a blurred result layer. In other words, during the original layer compositing process, only the blurred result is stored in the blurred result layer, minimizing the impact on the original layer compositing process and avoiding any impact on the display of the electronic device.

[0015] In one possible implementation, the third layer and the blurred result layer are composited using a hardware compositor (HWC) with the blurred result layer positioned below the third layer. This ensures the blurred result layer is placed immediately next to the third layer. The HWC composite of the third layer and the blurred layer guarantees the reuse of the blurred result layer's stored blurring effect, thus allowing the second image to reuse the blurring effect of the first image.

[0016] In one possible implementation, after acquiring the first image, the method further includes: determining the first image as a first state based on the state of the first layer and the state of the previous frame image of the first image. The first state indicates that the first image has undergone blurring processing to obtain a blurred result layer, so that the blurred result layer can be directly reused when subsequent images are composited.

[0017] In one possible implementation, the first image is determined to be in a first state based on the layer parameters of the first layer and the state of the previous frame of the first image. The layer parameters of the first layer can intuitively reflect the state of the first layer, simplifying the complexity of determining the state of the first image.

[0018] In one possible implementation, the parameters of the first layer include: the number of first layers and the first layer variation parameters. When the number of first layers is 1, the first layer variation parameters indicate that the first layer is in a stable phase, and the previous frame of the first image is in the third state, the first image is determined to be in the first state; or when the number of first layers is 1, the first layer variation parameters indicate that the first layer is in a stable phase, the previous frame of the first image is in the second state, and the second state images have accumulated to N frames, the first image is determined to be in the first state. After N consecutive frames of second state images, the first image is determined to be in the first state.

[0019] In one possible implementation, when the previous frame of the first image is in the third state, before determining the first image to be in the first state, the stable frame parameters carried by the previous frame of the first image are obtained, and the stable frame parameters indicate that the number of third images reaches M. Here, the third image includes a first layer, and the number of first layers included in the third image is 1, and the first layer is in a stable phase. That is, after continuously determining M frames of third images with a first layer count of 1 and in a stable phase, the first image is determined to be in the first state. This avoids the resource waste caused by performing the compositing process corresponding to the first state due to false stability, as there is no corresponding second state subsequently, and avoids increased power consumption.

[0020] In one possible implementation, when the number of the first layer is 1, the first layer change parameter indicates that the first layer is in a stable stage, and the previous frame of the first image is in the third state, the stable frame parameter carried by the second image is obtained. When the stable frame parameter indicates that the number of the third image has not reached M, the first image is determined to be in the third state, that is, the third image, and the first image performs the original compositing process.

[0021] In one possible implementation, after acquiring the second image, the method further includes: determining the second image as a second state based on the state of the third layer and the state of the previous frame of the second image. The second state indicates that the second image is composited using HWC (Hardware-Wide-Chip) with the third layer and the blurred result layer, eliminating the need for GPU-based blurring and layer compositing, thus improving the performance of the electronic device and enhancing the user experience.

[0022] In one possible implementation, the second image is determined to be in the second state based on the layer parameters of the third layer and the state of the previous frame of the second image. The layer parameters of the third layer can intuitively reflect the state of the third layer, simplifying the complexity of determining the state of the first image.

[0023] In one possible implementation, the layer parameters of the third layer include: the number of third layers and the third layer variation parameters; when the number of third layers is 1, the third layer variation parameters indicate that the third layer is in a stable phase, and the previous frame of the second image is in the first state, the second image is determined to be in the second state; or when the number of third layers is 1, the third layer variation parameters indicate that the third layer is in a stable phase, the previous frame of the second image is in the second state, and the accumulated images in the second state have not reached N frames, the second image is determined to be in the second state, where N is a positive integer.

[0024] In one possible implementation, the target image obtained by compositing the blurred result layer and the third layer is sent for display so that the electronic device can display the target image with the blurred result normally.

[0025] In a second aspect, this application provides an electronic device including a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the method described in the first aspect.

[0026] Thirdly, this application provides a computer-readable storage medium storing a computer program or instructions that, when executed, implement the method described in the first aspect.

[0027] Fourthly, this application provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the method described in the first aspect. Attached Figure Description

[0028] Figure 1 A schematic diagram illustrating the order of multiple surfaces along the Z-axis, provided for an embodiment of this application;

[0029] Figure 2A A schematic diagram of a partially blurred interface of an electronic device provided in an embodiment of this application;

[0030] Figure 2B A schematic diagram of a globally blurred interface for an electronic device provided in an embodiment of this application;

[0031] Figure 2C A schematic diagram illustrating a scenario of global blurring in an electronic device, as provided in an embodiment of this application;

[0032] Figure 2D A schematic diagram illustrating a scenario of global blurring in an electronic device, as provided in an embodiment of this application;

[0033] Figure 3 A layer diagram illustrating a partial interface of an electronic device provided in an embodiment of this application;

[0034] Figure 4 A simplified flowchart illustrating a layer composition method for displaying a single frame of an image, provided in an embodiment of this application;

[0035] Figure 5A A path tracing diagram of a layer compositing process without blurring provided in an embodiment of this application;

[0036] Figure 5B A path tracing diagram of a layer compositing process with blurring is provided in an embodiment of this application;

[0037] Figure 6A A schematic diagram illustrating the composition of an electronic device provided in an embodiment of this application;

[0038] Figure 6B A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0039] Figure 7 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0040] Figure 8A A schematic diagram of the interface of an electronic device without a blur layer provided in an embodiment of this application;

[0041] Figure 8B A schematic diagram of the interface of an electronic device with a blur layer provided in an embodiment of this application;

[0042] Figure 8C A schematic diagram of the interface of an electronic device with two blur layers provided in an embodiment of this application;

[0043] Figure 9 This is a schematic diagram illustrating the changes in a volume bar layer provided in an embodiment of this application;

[0044] Figure 10A A schematic flowchart illustrating a layer compositing method for a copied frame image provided in an embodiment of this application;

[0045] Figure 10B A schematic flowchart illustrating a layer composition method for frame skipping images provided in an embodiment of this application;

[0046] Figure 11 A schematic diagram illustrating the arrangement of copy frame states and skip frame states provided in an embodiment of this application;

[0047] Figure 12 A schematic diagram illustrating an arrangement for determining the state of each frame of an image, provided as an embodiment of this application;

[0048] Figure 13This application provides an interactive schematic diagram illustrating the implementation of object initialization and layer parameter acquisition.

[0049] Figure 14 An interactive schematic diagram illustrating the implementation of the state machine decision-making part in an embodiment of this application;

[0050] Figure 15 An interactive schematic diagram illustrating the implementation of layer compositing methods and position control in an embodiment of this application;

[0051] Figure 16 An interactive schematic diagram illustrating the implementation of a blurred drawing portion of a copy frame, provided as an embodiment of this application;

[0052] Figure 17 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0053] Figure 18 This is a schematic diagram illustrating the path tracing process of layer compositing under different image processing schemes provided in the embodiments of this application. Detailed Implementation

[0054] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0055] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0056] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0057] Surface: Each application may correspond to one or more graphical interfaces (GUIs). A GUI can correspond to at least one layer, called a surface. Each GUI has its position, size, and content to be displayed on the screen. A surface corresponding to a GUI can be understood as a container for that GUI, used to store the GUI's control information (e.g., its size and position on the screen) and / or its data (i.e., the content the GUI needs to display). Each surface has corresponding layer attributes, which can include information such as the layer's position and size.

[0058] Multiple surfaces may overlap, and the stacking relationship between these surfaces can be described using Z-order. There exists a Z-axis running vertically outwards on the screen plane. The order of all surfaces is determined by their coordinates on this Z-axis, which describes the vertical overlap between the various surfaces. This order on the Z-axis is called the Z-order.

[0059] To make it easier to understand, the following will be combined with... Figure 1 An illustrative example shows the order of multiple layers along the Z-axis. The screen includes surface0, surface1, and surface2, arranged from top to bottom when displayed. Surface0 partially covers surface1, and surface1 partially covers surface2. The Z-orders of Surface0, surface1, and surface2 are 3, 2, and 1, respectively.

[0060] Buffer queue: Each layer has a corresponding buffer queue, which stores the display data for that layer. The display data indicates the content to be displayed for that layer; that is, the layer's display data can be the image obtained after rendering that layer.

[0061] Typically, the buffer queue is an important component for data transfer in a display system. The buffer queue connects components that can generate graphics data buffers (i.e., producers, such as the surface) to components that receive data for display or further processing (i.e., consumers, such as the surfaceflinger).

[0062] Layer: A layer corresponds to a layer on the Z-axis. A layer provides control information for the layer it corresponds to, as well as the processing of its content. A layer is the basic unit of operation for compositing in the surface flinger module. Layers are created internally by the surface flinger module when an application requests the creation of a layer. There is a one-to-one correspondence between layers and surfaces.

[0063] Furthermore, both surfaces and layers are components of the graphics buffer, which consists of four parts: the surface, the buffer queue, the layer, and the graphic buffer. When an application needs to draw an image, it creates a surface and a layer. The producer (e.g., Skia, OpenGL, etc.) obtains a graphic buffer through the surface, draws the layer on the graphic buffer, and after drawing is complete, the surface returns the graphic buffer to the buffer queue. Then, the consumer (e.g., the surface flinger) calls the layer to obtain the drawn graphic buffer for compositing and processing.

[0064] This can be understood as the surface being for the producer, while the layer is for the consumer (e.g., surfaceflinger). Since there is a one-to-one correspondence between the surface and the layer, the order of the layer on the Z-axis is the same as the order of the corresponding surface on the Z-axis.

[0065] A frame buffer is a contiguous block of physical memory reserved in an embedded system for GPUs (graphics processing units) and DPUs (display processing units). The display reads data from the frame buffer through a dedicated bus and displays it on the screen.

[0066] The Surface Flinger module, also known as the layer delivery service, is a system service. This service primarily receives graphics display data from multiple sources, composites this data, and sends it to the display device. This service enables layer creation, control, and management. In electronic device display systems, Surface Flinger can be used to composite surfaces. For example, after an application starts, Surface Flinger can create a layer for that application. During terminal operation, Surface Flinger can acquire layers from various applications running on the terminal and composite these layers using the GPU and / or HWC.

[0067] Furthermore, the layer management module is also used to generate and distribute vertical synchronization (Vsync) signals to ensure the synchronization of various processes in the display system.

[0068] The display pipeline of electronic devices mainly includes three processes: application rendering, layer compositing, and driver delivery to the display. Essentially, the display pipeline of electronic devices is the flow of buffers.

[0069] GPU compositing: A GPU (Graphics Processing Unit) is a general-purpose image processing device. Besides layer compositing, GPUs are used to perform other graphics processing tasks. GPU compositing refers to a layer compositing method that uses the GPU to composite layers.

[0070] HWC Compositing: HWC (Hardware Composer) is a dedicated image processing device used for layer compositing and display. HWC can be a standalone device or integrated into a system-on-a-chip (SoC). HWC compositing refers to a layer compositing method that uses HWC to composite layers. During HWC compositing, each layer requires one transmission channel from the HWC, and since the number of transmission channels in an HWC is limited, the number of layers that can be composited is limited.

[0071] It should be noted that this application mainly relates to the layer composition process in the electronic device display production line, and the layers involved are layers, that is, layers that are processed by consumers (e.g., surface flingers). Therefore, the following layers are layers.

[0072] The advantages of an image processing method provided in this application embodiment are illustrated below by comparing it with current real-time blurring methods.

[0073] With the development of science and technology, many electronic devices with display interfaces have implemented background blurring functions, which effectively improve the security and privacy of user data while also enhancing the aesthetics of the electronic device's display interface. In some scenarios, electronic devices can achieve partial or global blurring functions, meaning the display interface of the electronic device is blurred in one area or all areas. For example, when a user receives a text message from another user while using their phone, the phone displays a pop-up notification. This pop-up can show the sender's information (e.g., the sender's phone number) and part of the message content. In this case, to minimize the impact of the pop-up notification on normal phone use and for aesthetic reasons, the pop-up area is blurred. That is, when a pop-up is displayed, a partial blurring effect may be applied to the pop-up area. Similarly, when a user requests to power off or restart their phone, they can press and hold the power button until the power off screen appears. To make the power off screen more visually appealing, the entire screen is blurred. For easier understanding, the following section will explain further. Figures 2A-2D Let's take an example to illustrate the situation of partial blurring of the display interface and global blurring of the actual interface. In this case, the electronic device 10 is a mobile phone.

[0074] like Figure 2A The diagram shows a partially blurred display interface of an electronic device. After the user unlocks the electronic device 10, the electronic device 10 can present a desktop interface 200 to the user. The desktop interface 200 includes: wallpaper, application icons, time icons, etc. Among them, the desktop interface 200 includes a variety of application icons, such as: phone icon, contact icon, internet icon, SMS icon, etc.

[0075] In one scenario, after a user unlocks the electronic device 10, a pop-up window will be displayed to provide important information or indicate the current status of the device (e.g., abnormal status, task completion progress, task completed, etc.). Figure 2A Taking the message pop-up window 201 as an example, message pop-up window 201 is used to notify the user that electronic device 10 is connected to USB debugging. That is, when electronic device 10 is connected to other electronic devices (e.g., a computer) via a USB data cable and USB debugging mode is enabled, message pop-up window 201 will notify the user that electronic device 10 is connected to USB debugging. In the current technology, in order to ensure the aesthetics of the display interface of electronic device 10 while still allowing the message pop-up window to serve its notification function, the background corresponding to message pop-up window 201 is blurred; that is, a partial blur function is applied to the area corresponding to message pop-up window 201.

[0076] In some cases, the output volume of electronic device 10 may not match the user's usage habits (output volume is too high or too low). The user can adjust the output volume of electronic device 10. For example, the user can adjust the volume using the volume adjustment button 11 of electronic device 10. Electronic device 10 responds to the user's operation of the volume adjustment button 11 to adjust the volume, and electronic device 10 displays a volume bar 202. The volume bar 202 can display the output volume value of electronic device 10 (output volume full value is 100) or the output volume percentage, etc. In order to highlight the volume bar 202 without affecting the basic display of electronic device 10, the background corresponding to the volume bar 202 area will be blurred, that is, a local blur function will be applied to the area corresponding to the volume bar 202.

[0077] In some cases, users can wake up the voice assistant through voice or other operations. The user speaks a voice message into the electronic device 10, and the voice assistant performs the corresponding function. For example, if the default wake-up word for the voice assistant supported by the electronic device 10 is "Hello YOYO," saying "Hello YOYO" will wake up the voice assistant. After the voice assistant is activated, the electronic device 10 displays a voice assistant pop-up window 203 to prompt the user that the voice assistant has been activated, and to display how to use the voice assistant, its functions, etc. To ensure that the voice assistant pop-up window 203 clearly displays relevant information without affecting the basic display of the electronic device 10, the background of the area corresponding to the voice assistant pop-up window 203 is blurred; that is, a partial blurring function is applied to the area corresponding to the voice assistant pop-up window 203.

[0078] It should be noted that, through Figure 2A This section introduces three scenarios where local blurring is applied; however, other scenarios also exist where local blurring is used. Figure 2A The example shows the simultaneous occurrence of three scenarios: the default volume bar 202 has a larger Z-order than the message pop-up 201 and the voice assistant pop-up 203.

[0079] like Figure 2BThe illustration shows a globally blurred display interface for an electronic device 10. The electronic device 10 is currently powered on. When a user presses and holds the power button 12 of the electronic device 10 for a preset time (e.g., 3 seconds), the electronic device 10 presents a shutdown interface 210 to the user. The shutdown interface 210 includes a restart icon and a shutdown icon. The user can perform actions on the shutdown or restart icons, such as clicking, long-pressing, or swiping. Upon receiving the user's action, the electronic device 10 initiates the corresponding shutdown or restart process. The shutdown interface 210 completely covers the original interface of the electronic device 10. To improve the aesthetics of the display interface, the corresponding global area of ​​the shutdown interface 210 is blurred; that is, the shutdown interface 210 applies a global blur function.

[0080] The following is combined Figure 2C (a) and Figure 2C (b) A detailed description of another scenario with global blur. For example... Figure 2C (a) shows a desktop interface of an electronic device. After the user unlocks the electronic device 10, the electronic device 10 can present a desktop interface 220 to the user. The desktop interface 220 includes icons for various applications, such as a phone icon, a contact icon, an internet icon, and a text message icon. A status bar 221 is located at the top of the desktop interface 220. The status bar 221 can include information such as message notifications 222, current time, current battery level, whether the electronic device is connected to Wi-Fi, and whether Bluetooth is enabled. In response to user actions in the status bar 221 (e.g., a swipe down), the electronic device can display... Figure 2C (b) shows another type of electronic device with a globally blurred display interface.

[0081] like Figure 2C As shown in (b), the status bar interface 230 is an exemplary interface involved when the status bar is pulled down to display the notification page. The status bar interface 230 includes a detailed content section 231 related to the message notification 222. For example, if the "Phone Manager" application of the electronic device 10 prompts the user that "Application A is consuming power in the background," the detailed content section 231 can indicate to the user that application A is currently consuming power in the background. At this time, in order to improve the aesthetics of the display interface, the global area of ​​the background of the status bar interface 230 is blurred, and the detailed content section 231 related to the message notification 222 and other information are displayed on top of the globally blurred background.

[0082] The following is combined Figure 2D (a) and Figure 2D (b) A detailed description of another scenario with global blur. For example... Figure 2D(a) shows another desktop interface of an electronic device. After the user unlocks the electronic device 10, the electronic device 10 can present a desktop interface 240 to the user. This desktop interface 240 includes icons for various applications, such as: phone icon, contact icon, internet icon, SMS icon, etc. Assuming that applications A and B in the electronic device 10 are running in the background at this time, in response to a specific user operation (e.g., a swipe up operation), the electronic device 10 can display as shown in Figure 10. Figure 2D (b) shows another type of electronic device with a globally blurred display interface.

[0083] like Figure 2D (b) shows the multitasking interface 250, which displays the applications currently running in the background of the electronic device 10 (including application A section 251 and application B section 252). Users can operate on application A section 251 and / or application B section 252 to close application A and / or application B from the background of the electronic device 10. To improve the aesthetics of the display, the background area of ​​the multitasking interface 250 is blurred globally, and application A section 251 and application B section 252 are displayed on top of this blurred background. This enhances the visual appeal of the interface while highlighting the background applications displayed by the multitasking interface 250.

[0084] It should be noted that, in addition to the above-mentioned common scenarios, global blurring is also found in other ways. Figures 2B-2D (b) Three cases of applying the global blur function are introduced. There are other cases of applying the global blur function, which are not specifically limited in this application.

[0085] Furthermore, common background blurring scenarios on electronic devices can be categorized into static blurring scenarios and dynamic blurring scenarios. Static blurring scenarios use a single frame of the background as the blurring data source, blurring only that frame. The blurred area of ​​the electronic device only displays the blurred background corresponding to that single frame, even if the background changes (e.g., live wallpapers, sliding backgrounds), maintaining the same blurred background and presenting a static mode. Dynamic blurring scenarios (also known as real-time blurring scenarios) use every frame of the background as the blurring data source, blurring each frame individually. The blurred area of ​​the electronic device changes its blurred background as the background changes. For example, if the background of the electronic device consists of N frames, the corresponding blurred area displays the blurred background corresponding to those N frames. If the background consists of N+1 frames, the corresponding blurred area displays the blurred background corresponding to those N+1 frames, where the N and N+1 frames are different (the background changes). In other words, as the background changes, the display of the blurred area of ​​the electronic device dynamically changes, presenting a dynamic mode.

[0086] The following is combined Figures 3-5B This section details current background blurring schemes for electronic devices. Taking the case where electronic device 10 adjusts its output volume in response to a user's operation of volume key 11 as an example, when electronic device 10 displays a volume bar to the user, the background of the volume bar area in the display interface of electronic device 10 is blurred, thereby achieving local blurring of the volume bar area of ​​electronic device 10 (e.g., Figure 3 (a) shows the display interface of an electronic device.

[0087] like Figure 3 (a) illustrates a display interface 300 of an electronic device, using the example of dividing the display interface 300 into four layers. For example... Figure 3 As shown in (b), the four layers are status bar layer 301, volume bar layer 302, desktop layer 303, and wallpaper layer 304. Status bar layer 301 is at the top, volume bar layer 302 is at the second layer, desktop layer 303 is at the third layer, and wallpaper layer 304 is at the bottom. The layer order Z is as follows: status bar layer 301 has a Z-order of 4, volume bar layer 302 has a Z-order of 3, desktop layer 303 has a Z-order of 2, and wallpaper layer 304 has a Z-order of 1.

[0088] Electronic devices enable blur functionality in some scenarios, such as: Figure 3 In the scenario shown in (a), the background of the volume bar area is blurred, and the electronic device 10 needs to perform layer compositing and blurring during the compositing stage of the display process. However, since display processing unit (DPU) compositing and HWC compositing do not currently support blurring, the layers related to blurring can only be blurred and composed by the GPU.

[0089] To make it easier to understand, the following will be combined with... Figure 4 Let's take an example to illustrate the current technology's blurring process and layer compositing workflow for a single frame of image. The layers related to blurring include volume bar layer 302, desktop layer 303, and wallpaper layer 304. Status bar layer 301 is unrelated to blurring and therefore does not require GPU compositing.

[0090] Once the various layers of the display interface of the electronic device 10 are drawn (such as...) Figure 3(b) After the status bar layer 301, volume bar layer 302, desktop layer 303, and wallpaper layer 304 shown, the graphics processing unit (GPU) performs layer compositing and blurring on the volume bar layer 302, desktop layer 303, and wallpaper layer 304 (generally, the GPU performs compositing on the volume bar layer 302, desktop layer 303, and wallpaper layer 304, and then performs blurring on the GPU). The GPU outputs the composite result (the composite result after layer compositing and blurring). Then, the hardware editor HWC (Hardware Composer) performs layer compositing on the GPU's composite result and the remaining layer to be composited (status bar layer 301) to complete the compositing of the electronic device display interface.

[0091] The following section details how the GPU performs layer compositing and blurring on volume bar layer 302, desktop layer 303, and wallpaper layer 304, using buffer as an example. For instance, wallpaper layer 304 is overlaid onto buffer1, then desktop layer 303 is overlaid onto wallpaper layer 304. The result from buffer1 is copied to buffer2, and buffer2 is blurred. Based on volume bar layer 302, a blurred area corresponding to the position and size of the volume bar is extracted. The blurred area of ​​buffer2 is then overlaid onto desktop layer 303 again (i.e., the blurred area is overlaid onto buffer1). Finally, volume bar layer 302 is overlaid onto the blurred area (i.e., volume bar layer 302 is overlaid onto buffer1). The GPU outputs the composite result (i.e., buffer1; buffer2 is only a transition for generating the blur and is discarded after use).

[0092] The above describes the blurring process and layer compositing workflow for a single frame of an image. In dynamic blur scenarios, each frame requires the following steps: Figure 4 The blurring and layer compositing process shown is time-consuming, and GPU compositing is more power-intensive than HWC compositing, leading to higher power consumption and reduced performance of the electronic device, severely impacting the user experience. For example, when the electronic device's wallpaper is set to a live wallpaper and a video is played in a small window over the live wallpaper, if a motion blur scene occurs (e.g., the electronic device receives a message from another electronic device and prompts the user via a pop-up; the user adjusts the output volume of the electronic device using the volume buttons), the device needs to handle both real-time video playback and real-time blurring. This results in excessive load on the Surface Flinger, leading to higher power consumption and reduced performance, severely impacting the user experience.

[0093] Furthermore, combined with Figure 5A and Figure 5B The path tracing diagram of the layer compositing process is shown below. When the displayed image of the electronic device does not require blurring and only HWC compositing is performed, the path tracing diagram of the layer compositing process is as follows. Figure 5A As shown, when GPU compositing is not required, the time is approximately 4 milliseconds (ms). When the displayed image of an electronic device needs to be blurred, both GPU compositing and blurring are required. The path tracing diagram of the layer compositing process is shown below. Figure 5B As shown, the processing time is approximately 10 milliseconds (ms). Therefore, the GPU compositing and blurring of the displayed image layers is time-consuming. For real-time blurring, each frame displayed by the electronic device requires GPU compositing and blurring, leading to higher power consumption and reduced performance, severely impacting the user experience. Furthermore, the extended processing time may cause performance issues such as stuttering and frame drops.

[0094] This application provides an image processing method that, when a stable blur layer exists, performs blurring by skipping frames, eliminating the need for blurring in every single frame. When the image is in a copy frame state, GPU compositing and blurring are performed normally. The GPU compositing result is then combined with the remaining layers using High-Wave Compatibility (HWC). During GPU compositing and blurring, the blurring result is copied to a pre-created blur result layer to obtain a blur result layer containing the blurring result, which is the blur result of the GPU-generated blur layer. When the image is in a skip frame state, the blur result layer and all layers are combined using HWC, eliminating the need for GPU compositing. Because skip frame images reuse the blurring result of copy frame images, the forced GPU compositing during blurring is avoided, reducing power consumption of electronic devices, improving performance, and thus enhancing the user experience.

[0095] In some embodiments, the electronic device may be a mobile phone, tablet computer, desktop computer, laptop computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, personal digital assistant (PDA), wearable electronic device, smartwatch, etc. This application does not impose any special limitations on the specific form of the aforementioned electronic device. In this embodiment, the structure of the electronic device may be as follows: Figure 6A As shown, Figure 6A This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0096] like Figure 6A As shown, the electronic device may include a processor 110, a display screen 120, a camera 130, internal memory 140, an audio module 150, a speaker 150A, a receiver 150B, a microphone 150C, buttons 170, and a touch sensor 160.

[0097] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0098] The processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors. For example, based on the state of the first layer and the state of the second image, the state of the first image is determined; the corresponding layer compositing process is controlled for each frame of the image.

[0099] The controller can serve as the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0100] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0101] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0102] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 160, charger, flash, camera 130, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 160 through the I2C interface, enabling the processor 110 and the touch sensor 160 to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device.

[0103] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 120 and the camera 130. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 130 communicate via the CSI interface to enable the electronic device to capture images. The processor 110 and the display screen 120 communicate via the DSI interface to enable the electronic device to display images.

[0104] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 130, a display screen 120, an audio module 150, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0105] It is understood that the interface connection relationships between the modules illustrated in this embodiment are merely illustrative and do not constitute a limitation on the structure of the electronic device. In other embodiments of this application, the electronic device may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0106] Electronic devices implement display functions through GPUs, display screens 120, and application processors. A GPU is a microprocessor for image processing, connecting the display screen 120 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0107] The display screen 120 is used to display images, videos, etc. The display screen 120 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 120, where N is a positive integer greater than 1.

[0108] The display screen 120 of an electronic device can display a series of graphical user interfaces (GUIs), which serve as the main screen of the electronic device. Generally, the size of the display screen 120 is fixed, and only a limited number of controls can be displayed on it. A control is a GUI element, a software component contained within an application, that controls all the data processed by the application and the interactive operations related to that data. Users can interact with controls through direct manipulation, thereby reading or editing information related to the application. Generally, controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0109] Electronic devices can achieve shooting functions through ISP, camera 130, video codec, GPU, display 120 and application processor.

[0110] The ISP (Image Signal Processor) is used to process data fed back from the camera 130. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 130.

[0111] Camera 130 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device may include one or N cameras 130, where N is a positive integer greater than 1.

[0112] Digital signal processors (DSPs) are used to process digital signals, including digital image signals and other digital signals. For example, when an electronic device is selecting a frequency, a DSP can perform Fourier transforms on the frequency energy.

[0113] Video codecs are used to compress or decompress digital video. Electronic devices can support one or more video codecs. This allows the electronic device to play or record video in various encoded formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0114] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0115] Internal memory 140 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 140. For example, in this embodiment, processor 110 can recalculate a first display timestamp of a video stream and a second display timestamp of an audio stream by executing the instructions stored in internal memory 140. Internal memory 140 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, phone book, etc.). In addition, internal memory 140 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 140 and / or instructions stored in memory disposed in the processor.

[0116] Electronic devices can implement audio functions such as music playback and recording through audio modules 150, speakers 150A, receivers 150B, microphones 150C, and application processors.

[0117] The audio module 150 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 150 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 150 may be located in the processor 110, or some functional modules of the audio module 150 may be located in the processor 110.

[0118] The speaker 150A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic devices can listen to music or make hands-free calls through the speaker 150A.

[0119] The receiver 150B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When an electronic device answers a phone call or voice message, the receiver 150B can be brought close to the ear to hear the voice.

[0120] Microphone 150C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 150C, inputting the sound signal into microphone 150C. Electronic devices can have at least one microphone 150C. In some embodiments, electronic devices can have two microphones 150C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic devices can have three, four, or more microphones 150C, enabling sound signal collection, noise reduction, sound source identification, and directional recording, among other functions.

[0121] Touch sensor 160, also known as a "touch device," can be disposed on display screen 120. The touch sensor 160 and display screen 120 together form a touchscreen, also known as a "touchscreen." Touch sensor 160 is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 120. In other embodiments, touch sensor 160 may also be disposed on the surface of the electronic device, in a different location than display screen 120.

[0122] Buttons 170 include a power button, volume buttons, etc. Buttons 170 can be mechanical buttons or touch-sensitive buttons. The electronic device can receive button input and generate key signal inputs related to user settings and function control of the electronic device.

[0123] In addition, an operating system runs on top of these components. Examples include Apple's iOS, Google's Android, and Microsoft's Windows. Applications can be installed and run on this operating system.

[0124] The operating system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of an electronic device.

[0125] Figure 6B This is a software structure block diagram of an electronic device according to an embodiment of this application.

[0126] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer (or framework layer), the Android runtime and system libraries, the hardware abstraction layer, and the kernel layer. For ease of explanation, in... Figure 6B The software architecture diagram of the electronic device shown also includes Figure 6A The hardware structure, such as the display processing unit and the screen.

[0127] The application layer can include a series of application packages. For example... Figure 6B As shown, the application package may include applications such as beauty camera, gallery, video, and SMS.

[0128] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 6B As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0129] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0130] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0131] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0132] A phone manager is used to provide communication functions for electronic devices. For example, it manages call status (including connection and disconnection).

[0133] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0134] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0135] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0136] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0137] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0138] The system library can include multiple functional modules. For example: layer management module (surface flinger), frame skipping module (efficientblur.cpp), primitive operation encapsulation (Buffer State Layer), compositing engine, output module (Output.cpp), rendering engine, 2D graphics engine (e.g., SGL), surface manager, media libraries, 3D graphics processing library (e.g., OpenGLES), etc.

[0139] The layer management module controls image compositing and generates vertical synchronization (Vsync) signals. Specifically, its main function is to receive graphics data buffers from multiple sources, composite them, and then send them to the display hardware (touchscreen). Simultaneously, the image compositing system is also responsible for error synchronization with the hardware HW-VSync signal and the simulation of software VSync signals, including SF-VSync and APP-VSync signals. The compositor flow control, signal simulator, and connection manager work together to notify the application and SurfaceFlinger itself of the VSync signal.

[0140] The frame-interval module is used to collect the layer parameters of the blur layer in each frame of the image, so as to determine the state of each frame of the image based on the parameters of the blur layer in each frame of the image and the state of the previous frame of the image; control the creation and destruction of the blur result layer, and control the position of the blur result layer.

[0141] Primitive operation encapsulation is used to control the setting of the drawn buffer to the corresponding layer.

[0142] The compositing engine is used to iterate through all layers of each frame of the image.

[0143] The output module is used to control the position of visible layers and the layer compositing method of visible layers.

[0144] The rendering engine is used to draw the content to be displayed on the corresponding buffer.

[0145] A 2D graphics engine is a graphics engine for 2D drawing.

[0146] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0147] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0148] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0149] The Hardware Abstraction Layer (HAL) can contain multiple library modules, such as a hardware compositor (HWC) and a camera library module. The Android system can load the corresponding library modules for the device hardware, thereby enabling the application framework layer to access the device hardware.

[0150] Hardware composers (HWC) primarily provide hardware support for image compositing systems, supporting layer compositing and display modules.

[0151] The kernel layer is the layer between hardware and software. The kernel layer includes at least the touch panel (TP) driver, display driver, Bluetooth driver, Wi-Fi driver, keyboard driver, shared memory driver, ambient light sensor driver, and camera driver.

[0152] The display driver can use APIs to send instructions and data to the graphics processing unit (GPU) or dedicated graphics processing hardware to perform operations such as 3D rendering, video decoding, and GPU computing. This hardware can include a display screen, touch hardware, a processor, and a GPU.

[0153] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.

[0154] Example 1:

[0155] The following is combined Figures 7-12 This application provides a detailed description of an image processing method based on its embodiments. The image processing method provided in this application addresses real-time blurred scenes (dynamic blurred scenes) on electronic devices, specifically scenarios where the display interface of an electronic device has blurred areas (local or global), and the blurring method for these blurred areas is real-time blurring. The application also details how the image processing scheme is determined to be enabled and disabled.

[0156] First, before executing the image processing method provided in this application embodiment, the electronic device needs to perform object initialization. For example, when the electronic device changes from a powered-off state to a powered-on state, object initialization is performed. This initialization sets the layer parameters (also called blur layer parameters) related to the blur layer for the first frame image displayed after the electronic device is powered on. For example, taking the layer parameters related to the blur layer as the number of blur layers, the name of the blur layer, and the blur layer variation parameters, if the number of blur layers is initialized to 0, then there is no blur layer name and no blur layer variation parameters. At this time, the state of the first frame image displayed by the electronic device is in the solution-off state.

[0157] The solution-off state is characterized by two cases: one is that the image frame does not need to be blurred, that is, there is no blur layer in the image frame, and the image processing solution provided in this application will not be executed; the other is that there are two or more blur layers in the image frame, and the image processing solution provided in this application will not be executed in this case either.

[0158] When a blurred layer exists on the display interface of an electronic device, the device needs to perform relevant blurring processing to ensure that the blurred layer is displayed in a blurred state. This means that all layers below the blurred layer need to be blurred to obtain the final blurred result. For example, when a user adjusts the output volume of an electronic device using volume control keys, the device responds by adjusting the output volume and displaying a volume bar. In this case, the volume bar layer is the blurred layer. Furthermore, the blurred layer carries blur attributes. When the electronic device recognizes the blur attributes of the blurred layer, it will blur all layers below it.

[0159] S701. Monitor and collect layer parameters related to the blurred layer in each frame of an image displayed by an electronic device.

[0160] Based on the display order of the electronic device, the layer parameters related to the blur layer in each frame of the image are monitored and collected; these can also be referred to as the layer parameters of the blur layer.

[0161] Among them, the layer parameters related to the blur layer include: the number of blur layers, the name of the blur layer, and the blur layer variation parameters.

[0162] The number of blurred layers indicates the number of blurred layers in the current frame of the image displayed by the electronic device. For example, when the electronic device is currently only displaying the desktop, such as... Figure 8A The desktop interface of the electronic device shown is currently displaying an image frame without a blur layer, so the number of blur layers is 0. If the electronic device currently displays only one blur layer, such as... Figure 8B The image shown is a display interface of an electronic device. Currently, the blurred layer in the frame image displayed by the electronic device includes a volume bar layer, therefore the number of blurred layers is 1. If the electronic device currently displays two blurred layers, such as... Figure 8C The electronic device shown is another display interface. In the current case, the blurred layers in the frame image displayed by the electronic device include the volume bar layer and the pop-up window layer, so the number of blurred layers is 2.

[0163] Each blurred layer has a unique name, which serves as an identifier for the layer. Furthermore, in some cases, dynamic and static blurred scenes can be distinguished by their blurred layer names. For example, a whitelist of dynamic blurred scenes can be pre-defined, storing the names of the blurred layers that require dynamic blurring. When a blurred layer name is in the whitelist for that dynamic blurred scene, the electronic device performs real-time blurring (i.e., blurring is applied to every frame of the image).

[0164] The blur layer variation parameter characterizes whether the blur layer is stable (i.e., whether the blur layer is in a stable phase). For example, it indicates whether the shape, position, and size of the blur layer are stable. The blur layer variation parameter determines whether the blur layer in the current frame has changed compared to the blur layer in the previous frame. If a change exists, the blur layer is unstable; if no change exists, the blur layer is stable. Generally, a blur layer variation parameter of 0 indicates a stable blur layer (i.e., the blur layer is in a stable phase); a blur layer variation parameter of 1 indicates an unstable blur layer (i.e., the blur layer is not in a stable phase).

[0165] To better understand the parameters of the blur layer transformation, the following is a combination of... Figure 9 The diagram showing the volume bar layer changes illustrates the blur layer change parameters. For example, a blur layer change parameter of 1 indicates a changing / unstable blur layer, while a blur layer change parameter of 0 indicates no change / stable blur layer.

[0166] During normal operation of an electronic device, the user adjusts the output volume using the volume control keys. The electronic device responds to this adjustment by adjusting the output volume and displaying a volume bar to indicate the adjusted volume. The process from the initial absence of a volume bar to its appearance and subsequent disappearance can be divided into three phases: the entry phase, the stabilization phase, and the exit phase. Figure 9 As shown, when an electronic device responds to a user's operation on the volume control keys, it displays the first frame image in chronological order (composite display order). The volume bar layer in this first frame image is frame 1 volume bar layer. The volume bar layer in the second frame image, displayed in chronological order, is frame 2 volume bar layer, and so on. Figure 9 The changes from the 1-frame volume bar layer to the 8-frame volume bar layer are shown.

[0167] The volume bar transition phase consists of three frames. From frame 1 to frame 3, the volume bar layer size increases progressively. During this transition, the volume bar layer size increases, indicating instability. By default, the volume bar layer did not exist in the previous frame of frame 1, so the blur layer change parameter for frame 1 is 1. Frames 2 and 3 show changes compared to frame 1, so their blur layer change parameters are also 1. From frame 4 to frame 6, the volume bar layer size remains constant. During this stable phase, the volume bar layer size remains unchanged (stable). Since frame 4 shows no change compared to frame 3, its blur layer change parameter is 0, and so on, with the blur layer change parameter for frames 4 through 6 always being 0. The volume bar transitions from frame 7 to frame 8, representing the exit phase of the volume bar, with the volume bar layer size decreasing. During this exit phase, the volume bar layer size changes from large to small. The volume bar layer at frame 7 shows a change compared to frame 6, and the volume bar layer at frame 8 shows a change compared to frame 7. Therefore, the volume bar layer is unstable during the exit phase. Consequently, the blur layer variation parameter for the frame 7 volume bar layer is 1, and the blur layer variation parameter for the frame 8 volume bar layer is also 1.

[0168] After monitoring and collecting the layer parameters related to the blur layer in each frame of the image displayed by the electronic device, the electronic device determines the state of each frame of the image according to the synthesis order based on the collected layer parameters related to the blur layer in each frame of the image. The specific confirmation process is as follows: Figure 12 As shown.

[0169] S702. Based on the layer parameters related to the blur layer, determine whether there is one and only one blur layer in the current frame image.

[0170] When it is determined that the current frame image does not contain one and only one blur layer (for example, the current frame image does not contain a blur layer, or the current frame image contains two or more blur layers), the current frame image is in the scheme off state.

[0171] Once it is determined that the current frame image is in the scheme closed state, the next frame image of the image in the scheme closed state is updated to the current frame image according to the synthesis order, and S702 is repeated.

[0172] When it is determined that there is one and only one blurred layer in the current frame image, and the state of the previous frame image is in the scheme off state (this is the first time it has been determined that there is one and only one blurred layer in the current frame image after confirming that the scheme is off), then the state of the current frame image is determined to be a normal frame state, that is, the current frame image can also be called a normal frame image. Then proceed to S703.

[0173] S703, Create a blurred result layer.

[0174] The current frame image is in a normal frame state, and the previous frame image was in a scheme-off state. At this time, indicating that the image processing scheme is turned on (i.e., the image processing scheme changes from the off state to the on state), a blur layer will be created. Since the appearance of the blur layer is a process of gradual increase until it stabilizes, the blur layer has already appeared before the target interface to be blurred is determined. When the blur layer appears, the blur layer is created.

[0175] Furthermore, while creating the blurred result layer, a first buffer corresponding to the blurred result layer is also created so that it can be used in the subsequent layer compositing process in the display pipeline of the electronic device, realizing the flow of the buffer and thus realizing the sequential execution of the display pipeline.

[0176] The image processing method provided in this application embodiment, in which only the layer parameter judgment in S702 indicates that the current frame image has only two states: a scheme off state and a normal frame state. When it is determined that the current frame image is in the normal frame state, it indicates that the image processing scheme is on, and that the image processing scheme changes from the off state to the on state. When the image processing scheme changes from the off state to the on state, a blur result layer is created to facilitate the implementation of subsequent image processing schemes.

[0177] At this point, only the blurred result layer is created and saved; the blurred result layer does not participate in the layer composition of the normal frame image. That is, the layer composition process of the normal frame image is unaffected. In one possible implementation, the layer corresponding to the normal frame image still undergoes layer composition and blurring processing of the blurred layer and related layers (i.e., all layers below the blurred layer) via the GPU. The GPU-output composition result is then combined with the remaining layers of the normal frame image using HWC to complete the layer composition of the normal frame image. Besides this, layer composition of the normal frame image can also be achieved through other methods; this application does not specifically limit the layer composition process of the normal frame image.

[0178] It should be noted that the normal frame state can serve as a buffer between the scheme-off state and the copy frame state / skip frame state.

[0179] After creating the blurred result layer, the next frame of the image in the normal frame state is updated to the current frame image according to the composition order, and S704 is performed.

[0180] S704. Based on the layer parameters related to the blur layer, determine whether there is one and only one blur layer in the current frame image.

[0181] The presence of a single blurred layer in the current frame is determined by the number of blurred layers. If the number of blurred layers is 1, then the current frame contains exactly one blurred layer; if the number of blurred layers is not 1, then the current frame does not contain exactly one blurred layer.

[0182] If it is determined that the current frame image does not contain one and only one blurred layer (i.e., the current frame image does not contain a blurred layer or the current frame image contains two or more blurred layers), then the current frame image is in the scheme off state. Furthermore, if the current frame image is in the scheme off state and the previous frame image is in the normal frame state, then this indicates that the image processing scheme has changed from the on state to the off state, and step S713 is performed.

[0183] If it is determined that there is one and only one blurred layer in the current frame image, then proceed to S705.

[0184] S705. Based on the layer parameters related to the blur layer, determine whether the only blur layer in the current frame image is a stable blur layer.

[0185] Based on the blur layer variation parameters, it is determined whether the only blur layer in the current frame image is a stable blur layer (i.e., whether the only blur layer is in a stable phase). Generally, a blur layer variation parameter of 1 indicates that the blur layer is unstable (not in a stable phase), and a blur layer variation parameter of 0 indicates that the blur layer is stable (in a stable phase).

[0186] When it is determined that the only blur layer in the current frame image is not a stable blur layer, that is, the only blur layer in the current frame image is a changing blur layer, then the state of the current frame image is a normal frame state.

[0187] When it is determined that the current frame image is in a normal frame state, the next frame image of the normal frame state image is updated to the current frame image according to the synthesis order, and S704 is repeated.

[0188] When it is determined that there is only one stable blur layer in the current frame image, the current frame image is in the copy frame state.

[0189] The copy frame state is used to indicate that the blurring result of the image in the copy frame state during the compositing process will be stored in a newly created blur result layer. It should be noted that the image in the copy frame state can also be called a copy frame image.

[0190] S706. During the synthesis of the copied frame image, the blurring result in the copied frame image is stored in the blurring result layer.

[0191] In one possible implementation, since the blurring and layer compositing of other layers related to the blurred layer in the electronic device are forcibly implemented through the GPU, the electronic device uses the GPU to composite (overlay) all layers below the blurred layer in the copied frame image. The overlay result is then blurred, and cropped according to the shape and size of the blurred layer to obtain the corresponding blurred result. The blurred layer, the blurred result, and all layers below the blurred layer in the copied frame image are then composited using the GPU. If there are layers unrelated to the blurred layer (e.g., layers above the blurred layer), the composite result output by the GPU and the remaining layers are composited using HWC to obtain the displayed copy frame image. Specifically, during the layer compositing and blurring process of the blurred layer and all layers below the blurred layer in the copied frame image using the GPU, the blurring result of the copied frame image is stored in a blurred result layer.

[0192] To make it easier to understand, the following will be combined with... Figure 10A Let's take step 706 as an example. The rendered layers of the target interface include: video playback layer 1001, volume bar layer 1002, desktop layer 1003, and wallpaper layer 1004. Each frame of the target interface has the same rendered layer, but the content rendered in each layer may differ (for example, if the wallpaper displayed in the target interface is a live wallpaper, then each frame will have a different rendered wallpaper layer). Taking the Z-axis perpendicular to the screen outwards as the positive direction, video playback layer 1001 is located in the fourth layer (i.e., Z-order = 4), volume bar layer 1002 is located in the third layer (i.e., Z-order = 3), desktop layer 1003 is located in the second layer (i.e., Z-order = 2), and wallpaper layer 1004 is located in the top layer (i.e., Z-order = 1). It should be noted that... Figure 10A The layers shown are only simplified diagrams illustrating the relative size and position of the layers, serving only as examples and do not represent the content rendered by each layer.

[0193] like Figure 10AAs shown, the blurred layer is the volume bar layer 1002. The layers related to the blurred layer (layers located below the blurred layer) include: desktop layer 1003 and wallpaper layer 1004. The electronic device uses the GPU to overlay (composite) the desktop layer 1003 and wallpaper layer 1004. Then, the GPU blurs the overlay result of the desktop layer 1003 and wallpaper layer 1004 and clips it according to the volume bar layer 1002 to obtain the blurred result. Then, the GPU composites the volume bar layer 1002, the blurred result, and the desktop layer 1003 and wallpaper layer 1004, outputting the GPU composite result. During the GPU composite process, the blurred result is stored in the blurred result layer 1005. Finally, the HWC composites the GPU composite result and the video playback layer 1001 to obtain the copy frame image.

[0194] In this process, the blurred result layer, which stores the blurring results, is made visible during the compositing of the copy frame image, but it does not participate in the layer compositing process. Therefore, the electronic device does not display the blurred result layer when displaying the copy frame image. It should be noted that since it does not participate in layer compositing, the final composite copy frame image does not contain a blurred cache layer, and consequently, the copied frame image displayed later also does not contain a blurred cache layer.

[0195] Furthermore, once the layers of the copied frame image are composited, the composited result of the copied frame image is sent to the display via the display driver according to the display order (composition order), thereby enabling the electronic device to display the copied frame image.

[0196] Furthermore, the blurring result is stored in a blurring result layer to obtain a blurring result layer containing the blurring result. The blurring result layer is then stored in the corresponding first cache so that the corresponding blurring result layer can be directly retrieved from the first cache later.

[0197] When it is determined that the current frame image is in the copy frame state, the next frame image of the copy frame state image is updated to the current frame image according to the display order (composition order), and S707 is performed.

[0198] S707. Based on the layer parameters related to the blur layer, determine whether there is one and only one blur layer in the current frame image.

[0199] When it is determined that there is one and only one blur layer in the current frame image, the current frame image is in the scheme off state. That is, when there is no blur layer or the number of blur layers is greater than 1, the image processing scheme is turned off. Furthermore, at this time, the current frame image is in the scheme off state, and the previous frame image is in the copy frame state, which indicates that the image processing scheme has changed from the on state to the off state, and S713 is performed.

[0200] If it is determined that there is one and only one blurred layer in the current frame image, then proceed to S708.

[0201] S708. Based on the layer parameters related to the blur layer, determine whether the only blur layer in the current frame image is a stable blur layer.

[0202] When it is determined that the only blur layer in the current frame image is not a stable blur layer, that is, the only blur layer in the current frame image is a changing blur layer (for example, the blur layer change parameter is 1), then the state of the current frame image is a normal frame state.

[0203] When it is determined that the current frame image is in a normal frame state, the next frame image of the normal frame state image is updated to the current frame image according to the display order (composition order), and S704 is repeated.

[0204] When it is determined that the only blur layer in the current frame image is a stable blur layer (e.g., the blur layer change parameter is 0), the current frame image is in a frame skipping state. When it is determined that the current frame image is in a frame skipping state, then proceed to step S709.

[0205] The frame skipping state indicates that during the image compositing process in the frame skipping state, the blurred result layer of the copy frame image is reused. That is, the blurred result layer of the copy frame image and all layers of the current frame image are composited using HWC to obtain the current frame image. It should be noted that an image in the frame skipping state can also be called a frame skipping image.

[0206] S709, Control all layers corresponding to the frame skipping image and the blurred result layer to be composited using HWC.

[0207] In the process of compositing skipped-frame images, the blur result layer storing the blurring results and all layers corresponding to the skipped-frame image are composited using HWC to obtain the skipped-frame image. That is, the skipped-frame image reuses the blurring results of the copy frame image during compositing, and does not require blurring processing itself. This means that the GPU does not need to blur the layers related to the blur layer (all layers under the blur layer), and both the GPU and HWC can perform layer compositing. Since skipped-frame images do not require forced layer compositing and blurring processing via the GPU, the GPU's blurring process is skipped, and all layers corresponding to the skipped-frame image and the blur result layer are directly composited using HWC to obtain the skipped-frame image.

[0208] In the process of compositing all layers corresponding to the skipped frame image and the blurred result layer using HWC, it is necessary to control the position of the blurred result layer to be located on the layer below the blurred layer. Taking the direction of the vertical electronic device display screen outward as positive, if the blurred layer is located on layer x, then the blurred result layer needs to be controlled to be located on layer x-1. For example, if the Z-order of the blurred layer is 4, the Z-order of the blurred result layer must be 3. It should be noted that the above example is only used to illustrate the need to control the blurred result layer to be located on the layer below the blurred layer. The actual Z-order of the layers needs to be adaptively modified as the blurred result layer is added.

[0209] To make it easier to understand, the following will be combined with... Figure 10B Let's take step 709 as an example.

[0210] like Figure 10B As shown, the electronic device controls all layers corresponding to the skipped frame image (video playback layer 1001, volume bar layer 1002, desktop layer 1003, and wallpaper layer 1004) and the blurred result layer 1005 to be composited using HWC according to the adjusted layer order to obtain the skipped frame image. Without the blurred result layer, the order of all layers corresponding to the skipped frame image is: video playback layer 1001 Z-order 4, volume bar layer 1002 Z-order 3, desktop layer 1003 Z-order 2, and wallpaper layer 1004 Z-order 1. After adding the blurred result layer, the adjusted layer order is: video playback layer 1001 Z-order 5, volume bar layer 1002 Z-order 4, blurred result layer 1005 Z-order 3, desktop layer 1003 Z-order 2, and wallpaper layer 1004 Z-order 1.

[0211] Furthermore, after obtaining the skip frame image through HWC synthesis, the skip frame image synthesized by HWC is sent to the display via the display driver according to the display order (synthesis order) of the electronic device to display the skip frame image.

[0212] When the current frame image is in a frame-skipping state (i.e., a frame-skipping image), the electronic device directly sends the blurred result layer and all layers corresponding to the frame-skipping image into HWC for compositing to obtain the frame-skipping image. In other words, the blurring result of the copied frame is reused in the frame-skipping image compositing process. It avoids the need to force GPU compositing during the blurring process, as was previously required during frame-skipping image compositing. Instead, HWC directly composites all layers of the frame-skipping image and the blurred result layer, thus avoiding the high GPU load caused by forced GPU compositing. This reduces the power consumption of the electronic device, improves its performance, and ultimately enhances the user experience.

[0213] When it is determined that the current frame image is in a frame skipping state, the next frame image of the frame skipping state image is updated to the current frame image according to the display order (composition order), and S710 is performed.

[0214] S710. Based on the layer parameters related to the blur layer, determine whether there is one and only one blur layer in the current frame image.

[0215] If it is determined that there is one and only one blurred layer in the current frame image, then the state of the current frame image is the scheme off state. Furthermore, since the current frame image is in the scheme off state and the previous frame image was in the adjustment state, this indicates that the image processing scheme has changed from the on state to the off state, and step S713 is executed.

[0216] If it is determined that there is one and only one blurred layer in the current frame image, then proceed to S711.

[0217] S711. Based on the layer parameters related to the blur layer, determine whether the only blur layer in the current frame image is a stable blur layer.

[0218] When it is determined that the only blur layer in the current frame image is not a stable blur layer, that is, the only blur layer in the current frame image is a changing blur layer (for example, the blur layer change parameter is 1), then the state of the current frame image is a normal frame state.

[0219] When it is determined that the current frame image is in a normal frame state, the next frame image of the normal frame state image is updated to the current frame image according to the display order (composition order), and S704 is repeated.

[0220] When it is determined that there is only one stable blur layer in the current frame image (e.g., the blur layer change parameter is 0), then proceed to S712.

[0221] S712, Determine whether the cumulative frame skipping image has reached N frames.

[0222] Where N is a positive integer, and N is preset according to the actual situation. For example, N can be related to the screen refresh rate of the electronic device. When the screen refresh rate of the electronic device is high, the value of N can be greater than the value of N when the screen refresh rate of the electronic device is low.

[0223] If the cumulative number of images in the frame skipping state has not reached N frames (i.e., there is no consecutive determination of the state of N frames in the frame skipping state), then the state of the current frame image is still in the frame skipping state, and S709 is repeated.

[0224] When the number of frames in the frame-skipping state accumulates to N (i.e., the state of N consecutive frames is determined to be in the frame-skipping state), the state of the current frame image is in the copy frame state, and S706 is repeated.

[0225] To make it easier to understand, the following will be combined with... Figure 11 This section details the settings for copy frame state and skip frame state. We'll use an example of 8 frames (frames 1-8), where each frame contains exactly one stable (in a stable phase) blur layer. That is, the number of blur layers in these 8 frames is 1, and the blur layer change parameter is 0 (indicating the blur layer is in a stable phase). By default, frame 1 is in copy frame state. It should be noted that... Figure 11 The image shown is a simplified diagram of a blur layer that exists in one and only one of eight frames.

[0226] When N=1 is preset, such as Figure 11 The state arrangement of the 8 frames shown is as follows: one frame is in copy frame state, and one frame is in skip frame state. For example... Figure 11 As shown in (a), the state of the first frame is a copy frame state, the state of the second frame is a skip frame state, the state of the third frame is a copy frame state, the state of the fourth frame is a skip frame state, the state of the fifth frame is a copy frame state, the state of the sixth frame is a skip frame state, the state of the seventh frame is a copy frame state, and the state of the eighth frame is a skip frame state. That is, the copy frame state and the skip frame state are distributed alternately.

[0227] When N=2 is preset, such as Figure 11 The state arrangement of the 8 frames shown is as follows: one frame is in copy frame state, followed by two frames in skip frame state. For example... Figure 11As shown in (b), the state of the first frame is a copy frame, the state of the second frame is a skip frame, the state of the third frame is a skip frame, the state of the fourth frame is a copy frame, the state of the fifth frame is a skip frame, the state of the sixth frame is a skip frame, the state of the seventh frame is a copy frame, and the state of the eighth frame is a skip frame.

[0228] When N=3 is preset, such as Figure 11 The state arrangement of the 8 frames shown is as follows: one frame is in copy frame state, followed by three frames in skip frame state. For example... Figure 11 As shown in (c), the state of the first frame is a copy frame state, the state of the second frame is a skip frame state, the state of the third frame is a skip frame state, the state of the fourth frame is a skip frame state, the state of the fifth frame is a copy frame state, the state of the sixth frame is a skip frame state, the state of the seventh frame is a skip frame state, and the state of the eighth frame is a skip frame state.

[0229] S713. Once it is determined that the current frame image is in the scheme off state, delete the created blurry result layer.

[0230] When the current frame image is determined to be in the scheme off state, and the previous frame image is in the normal frame state, copy frame state, or skip frame state, it means that the image processing scheme has changed from the on state to the off state. Furthermore, the layer compositing process of the current frame image no longer involves the blur result layer, so the created blur result layer is deleted to avoid wasting resources.

[0231] Furthermore, when the current frame image is determined to be in a scheme-off state, and the previous frame image is also in a scheme-off state, the image processing scheme is always in a off state, and no blur result layer is created. Therefore, there is no step to delete the blur result layer.

[0232] Furthermore, after the electronic device determines the first normal frame image, that is, according to the synthesis order, after determining for the first time that the current frame image has exactly one stable blur layer (i.e., the blur layer stability parameter is 0), it needs to continuously determine that M frames have exactly one stable blur layer. That is, continuously determining that M frames have exactly one stable blur layer (it should be noted that at this time, the state of the M frames is the normal frame state). When it is determined that the next frame image of the last frame image in the M frames has exactly one stable blur layer, the state of that frame image is the copy frame state (that is, before determining the state of the current frame image, if M frames have been continuously determined to have exactly one stable blur layer, then the state of the current frame image is the copy frame state). Here, M is a positive integer.

[0233] Specifically, each frame carries a stable frame parameter, which indicates the number of consecutively determined images with exactly one stable blur layer. The current frame has exactly one stable blur layer, and the previous frame was in a normal frame state. The stable frame parameter of the previous frame is determined as follows: if the stable frame parameter indicates that the number of consecutively determined images with exactly one stable blur layer is M, then the current frame is in a copy frame state; if the stable frame parameter indicates that the number of consecutively determined images with exactly one stable blur layer is less than M, then the current frame is in a normal frame state.

[0234] To make it easier to understand, the following will be combined with... Figure 12 Let's take an example. We'll use 15 frames of images displayed on an electronic device in the order they are displayed (the order in which they are composited) (frames 1-15). Figure 12 The image displayed on the electronic device only shows the blurred layer. Specifically, the first frame does not contain a blurred layer; the second frame contains one and only one blurred layer, which is unstable; the third frame contains one and only one blurred layer, which is unstable; the fourth frame contains one and only one blurred layer, which is unstable; frames 5 through 11 contain one and only one blurred layer, which is stable; the 12th frame contains one and only one blurred layer, which is unstable; the 13th frame contains one and only one blurred layer, which is unstable; and the 14th and 15th frames do not contain a blurred layer.

[0235] Let's take N=1 and M=3 as an example. The states of frames 1 through 15 are determined according to the display order (composition order). Frame 1 has no blur layer, so its state is "schema off". Frame 2 has one and only one blur layer, and this blur layer is unstable, so its state is "normal frame". Frame 3 has one and only one blur layer, and this blur layer is unstable, so its state is "normal frame". Frame 4 has one and only one blur layer, and this blur layer is unstable, so its state is "normal frame". Frame 5 has one and only one stable blur layer; it is the first frame with one stable blur layer, so its state is "normal frame", and the stable frame parameter is 1 (indicating that one frame has been continuously determined to have one and only one stable blur layer). Frame 6 is the second frame with one stable blur layer, so its state is "normal frame". The 6th frame is a normal frame, and the stable frame parameter carried by the 6th frame is 2; the 7th frame is the third image with only one stable blur layer, and the state of the 7th frame is a normal frame, and the stable frame parameter carried by the 7th frame is 3; the 8th frame is the fourth image with only one stable blur layer, and the stable frame parameter carried by the 7th frame is 3. Since M=3, the state of the 8th frame is a copy frame; since N=1, the state of the 9th frame is a skip frame; the state of the 10th frame is a copy frame; the state of the 11th frame is a skip frame; the 12th frame contains only one blur layer, but the blur layer is unstable, and the state of the 12th frame is a normal frame; the 13th frame contains only one blur layer, but the blur layer is unstable, and the state of the 13th frame is a normal frame; the 14th and 15th frames do not contain blur layers, the state of the 14th frame is scheme off; the state of the 15th frame is scheme off.

[0236] By continuously determining that M frames of an image have one and only one stable blur layer, the determination of the image state of the copied frame is more accurate. This avoids the situation where an electronic device has only one stable blur layer in one frame of an image, but the subsequent frames do not satisfy the condition of having one and only one stable blur layer. In such cases, the state of each frame of an image is incorrectly determined, leading to display errors on the electronic device and affecting the user experience.

[0237] In an image processing method provided in this application embodiment, layer parameters related to the blur layer and the state of the previous frame image are monitored and collected in each frame of an image displayed by an electronic device. The state of each frame image is determined, and a corresponding layer compositing process is executed based on the state of each frame image. During the compositing process of the copied frame image, the blurring result performed by the GPU is stored in a blur cache layer. This allows for the control of all layers corresponding to the skipped frame image and the blur result layer to be composited through HWC during the compositing process of the skipped frame image, resulting in the skipped frame image. In this application embodiment, the blurring result of the copied frame image is reused during the compositing process of the skipped frame image, eliminating the need for blurring processing by the GPU. This avoids the forced compositing process by the GPU during layer compositing due to blurring. Instead, all layers of the skipped frame image and the blur result layer are directly composited through HWC, thereby reducing the power consumption of the electronic device caused by layer compositing and blurring processing by the GPU, improving the performance of the electronic device, and thus enhancing the user experience.

[0238] It should be noted that in the first case, where the state of a frame image is "solution off," there is no blur layer in that frame image, and blurring is unnecessary; therefore, the image processing scheme provided in this application will not be executed. In the second case, where there are two or more blur layers in the frame image, the image processing scheme provided in this application is not executed because the presence of two or more blur layers necessitates creating layers that cover both blur layers, processing each blur layer individually, and then processing based on the relative relationship between the two or more blur layers. This approach cannot effectively reduce the power consumption of the electronic device. Furthermore, the time during which two or more blur layers are in a stable state (stable phase) may be very short, resulting in a small application time and making it difficult to effectively reduce the power consumption of the electronic device.

[0239] Example 2:

[0240] The following is combined Figures 13-16 This application provides a detailed description of the interaction process between modules in an image processing method provided in its embodiments.

[0241] For ease of explanation, the interaction process between the modules of the overall image processing method is divided into four parts: object initialization and layer parameter acquisition, state machine decision-making, layer compositing method and position control, and copy frame blur rendering.

[0242] To make it easier to understand, the following will be combined with... Figure 13 This application provides a detailed description of the object initialization and layer parameter acquisition sections provided in its embodiments.

[0243] S1301, The layer management module (Surface flinger) calls the practical status setting function to initialize the object.

[0244] S1302, The layer management module (Surface flinger) sets the status of the current frame image displayed on the electronic device to the scheme off state.

[0245] S1303, the layer management module (Surface flinger) calls the constructor, which in turn calls the frame skipping module to store the image processing functions in the frame skipping module into the 2D vector graphics processing function library (skia module).

[0246] The frame-interval module is a pre-developed programming language program (the frame-interval module is the main implementation module of the image processing method provided in this application). Its purpose is to support the electronic device in determining the state of each frame, switching the image processing scheme on and off, and constructing and destroying the blur result layer. The frame-interval module is stored in the system library of the electronic device's operating system as a source file of the programming language program, for example: efficientblur.cpp. The programming language can be C++, C, etc., and this application does not specifically limit its use.

[0247] S1304. When a blurred layer exists in the current frame image of the electronic device, the layer management module (Surfaceflinger) creates a blurred layer.

[0248] When a blurred layer exists in the current frame image displayed by the electronic device, the layer management module will call the layer creation function to create the corresponding blurred layer, so that the layer management module can control the blurred layer and other layers to perform HWC compositing or GPU compositing.

[0249] S1305, The layer management module (Surface flinger) sends the name of the created blurred layer to the frame-separating module.

[0250] The layer management module sends the name of the created blurred layer to the frame-interval module so that the frame-interval module can collect the names of the blurred layers in the current frame image.

[0251] S1306. After the layer management module (Surface flinger) completes the creation of the blur layer thread, it sends the number of blur layers in the current frame image to the inter-frame module.

[0252] Once the thread for creating blur layers in the current frame image of the electronic device has completed, that is, once all blur layers in the current frame image have been created or no blur layers exist in the current frame image, the number of blur layers in the current frame image is sent to the inter-frame module so that the inter-frame module can collect the number of blur layers in the current frame image.

[0253] S1307. After the layer management module (Surface flinger) completes the creation of the blur layer thread, it sends the blur layer change parameters in the current frame image to the inter-frame module.

[0254] Once the thread for creating blur layers in the current frame image of the electronic device has completed, that is, once all blur layers in the current frame image have been created or no blur layers exist in the current frame image, the blur layer change parameters in the current frame image are sent to the inter-frame module so that the inter-frame module can collect the blur layer change parameters of the current frame image.

[0255] S1308, the frame-interval module collects and stores the layer parameters related to the blurred layer in each frame of the image.

[0256] Specifically, upon receiving the blurred layer names from the layer management module, the system collects and stores these names; upon receiving the number of blurred layers from the layer management module, the system collects and stores these numbers; upon receiving data from the layer management module...

[0257] The above steps S1305, S1306, and S1307 enable the electronic device to collect layer parameters related to the blurred layer in each frame of the image. Specifically, S1305 collects parameters only after the blurred layer is created if it appears / exists in the current frame. If no blurred layer is created, it does not exist in the current frame, and therefore the name of the blurred layer in the current frame cannot be collected. S1306 and S1307 require the electronic device to monitor and collect parameters for each frame of the image.

[0258] The object initialization and layer parameter acquisition section provided in this embodiment initializes object variables before the electronic device performs image processing and initializes the state of the current frame image to the scheme off state. Simultaneously, it monitors and collects the layer parameters related to the blur layer for each frame image, so that they can be used by the state machine decision-making section later.

[0259] To make it easier to understand, the following will be combined with... Figure 14 This application provides a detailed description of the state machine decision-making part provided in its embodiments.

[0260] S1401, The layer management module (Surface flinger) sends a call request to the frame skipping module.

[0261] The layer management module sends a call request to the frame-interval module, which then calls the frame-interval module to determine the state of each frame of the image.

[0262] S1402, the frame-interval module determines the state of each frame of the image.

[0263] The frame-interval module responds to the call request and determines the status of each frame of the image.

[0264] Taking the current frame image as an example, the inter-frame module collects the layer parameters related to the blur layer in the current frame image (i.e., the layer parameters of the blur layer). Based on the layer parameters related to the blur layer in the current frame image and the state of the previous frame image, the state of the current frame image is determined.

[0265] S1403 The frame-interval module determines whether to create or destroy the blurred result layer based on the state of the current frame image and the state of the previous frame image.

[0266] When the inter-frame module determines that the current frame image is in a normal frame state and the previous frame image is in a scheme-off state, then S1404 is performed to create a blurred result layer.

[0267] When the frame skipping module determines that the current frame image is in the scheme off state and the previous frame image is in the non-scheme off state (normal frame state, copy frame state, or skip frame state), then S1405 is executed to destroy the blurred result layer.

[0268] S1404, the inter-frame module creates a blurred result layer.

[0269] The skip frame module calls the 2D vector graphics processing function library (skia module) to create a blurred result layer, which is used to store the blurred processing result corresponding to the copied frame image, and participates in the layer composition process of the skip frame image.

[0270] Furthermore, while creating the blurred result layer, a corresponding buffer is also created to store the blurring result data of the blurred result layer. This buffer participates in the rotation of the blurred result layer's buffer. Only when the buffer and the layer work together can the corresponding blurred result be displayed to the user through the electronic device's screen. In other words, the layer is the carrier of the buffer, and the buffer is the carrier of the blurred result.

[0271] S1405, the frame-interval module destroys the blurred result layer.

[0272] While the frame-interval module destroys the blurred result layer, it also destroys the buffer (cache) corresponding to the blurred result layer to avoid occupying unnecessary storage space and to prevent too many layers from affecting the normal display of electronic devices.

[0273] S1406, The frame-interval module sets whether the blurred result layer is visible based on the state of the current frame image.

[0274] If the current frame image is in copy frame state or frame skip state, then set the blurred result layer to visible.

[0275] Specifically, the current frame image is in copy frame state, the blurred result layer is set to visible, and is combined with the blurred layer of the current frame image and the layers related to the blurred layer (layers located below the blurred layer) for GPU compositing. However, only the blurring result processed by the GPU is copied to the blurred result layer, and the blurred result layer does not participate in layer compositing.

[0276] The current frame image is in a frame skipping state. The blurred result layer is set to visible and is combined with all layers of the current frame image using HWC compositing. The blurred result layer participates in the layer compositing.

[0277] If the current frame image is in a normal frame state, then the blurred result layer is set to invisible.

[0278] Furthermore, when the current frame image is in the scheme off state, there is no corresponding blurred result layer.

[0279] Specifically, when the current frame image is in a normal frame state, the blurred result layer is set to invisible. This is because the layer compositing process for the current frame image remains unchanged when it is in a normal frame state (GPU compositing and blurring are performed first, followed by HWC compositing). Therefore, setting the blurred result layer to invisible prevents it from participating in the layer compositing process of the current frame image, thus avoiding unnecessary resource consumption.

[0280] It should be noted that the blurred result layer can only participate in the layer composition process of the current frame image when it is set to visible; when it is set to invisible, the blurred result layer cannot participate in the layer composition process of the current frame image.

[0281] Furthermore, when the current frame image is in the copy frame state, the blurring result is stored in the blurring result layer during the synthesis process of the copy frame image.

[0282] S1407 When the current frame image is in copy frame state, during the synthesis process of the copy frame image, the inter-frame module sets the blur processing result into the buffer corresponding to the blur result layer.

[0283] Essentially, it involves storing the blurred result layer, which contains the results of the blurring process, into the corresponding buffer.

[0284] S1408, the frame-interval module sends a call request to the primitive operation encapsulation.

[0285] The frame-interval module sends a call request to the primitive operation encapsulation, thereby invoking the primitive operation encapsulation to set the buffer with the added blur processing result into the blur result layer.

[0286] S1409, The primitive operation encapsulation adds a buffer setting of the blur processing result to the blur result layer.

[0287] The primitive operation encapsulation responds to the call request sent by the frame-interval module by adding a buffer containing the blurred result to the blurred result layer.

[0288] The blurring result is drawn on the buffer corresponding to the blurring result layer. The buffer is combined with the blurring result layer to store the blurring result in the blurring result layer.

[0289] It should be noted that S1407-S1409 implements the storage of blurring results to a blurring result layer during the synthesis of copied frame images.

[0290] The state machine decision-making part provided in this application embodiment realizes the determination of the state of each frame image based on the layer parameters related to the blur layer in each frame image, and controls the opening or closing of the decision scheme, the creation or destruction of the blur result layer, the setting of whether the blur result layer is visible or not, and the copying process of the blur processing result when copying the frame state, etc., based on the state of each frame image.

[0291] To make it easier to understand, the following will be combined with... Figure 15 This application provides a detailed description of the layer compositing method and position control mechanism provided in its embodiments.

[0292] S1501, The Surface flinger module sends a call request to the compositing engine.

[0293] The Surface flinger module sends a request to the compositing engine, which then calls the compositing engine to traverse all layers of the current frame image.

[0294] S1502, The compositing engine traverses all layers of the current frame image.

[0295] The compositing engine responds to the call request sent by the layer management module by traversing all layers of the current frame image.

[0296] The compositing engine traverses all layers of the current frame image. If a blurred result layer exists, step S1503 is performed. If no blurred result layer exists, the image processing scheme provided in this application is not enabled, and the layer compositing process of the current frame image is unaffected.

[0297] S1503, The synthesis engine sends a call request to the output module.

[0298] The compositing engine sends a call to the output module, which in turn calls the compositing engine to control the position of visible layers.

[0299] S1504, the compositing engine controls the position of visible layers.

[0300] The compositing engine responds to the call requests sent by the output module and controls the position of visible layers.

[0301] Since the newly created blurred result layer is positioned at the very top of all layers by default, meaning it has the largest Z-order, it is necessary to adjust the position of the blurred result layer, which means controlling the position of all visible layers.

[0302] The output module is stored in the system library of the electronic device's operating system in the form of a source file in a programming language, which can be called output.cpp (a source file in C++).

[0303] The output module controls the position of visible layers specifically through S1505 and S1506.

[0304] S1505, The output module collects all visible layers in the current frame image.

[0305] Determine whether a blurred result layer exists among all the visible layers collected.

[0306] If there is no blurred result layer among all visible layers, then there is no need to control / adjust the position of the visible layers.

[0307] If a blurred result layer exists among all visible layers, then proceed to S1506.

[0308] S1506, The output module sends a call request to the frame-interval module.

[0309] The output module sends a call request to the frame-interval module, thereby calling the frame-interval module to control the blurred result layer to move to the next layer position of the blurred layer.

[0310] S1507, the frame-interval module controls the movement of the blurred result layer to the next layer below the blurred layer.

[0311] The frame-interval module responds to the call request sent by the output module and controls the blurred result layer to move to the next layer position of the blurred layer.

[0312] The blurred result layer is moved to the layer below the blurred layer; that is, the blurred result layer needs to be located below the blurred layer and adjacent to it. For example, without position control of the visible layers, assuming the blurred result layer is on top and its Z-order is 5, and the blurred layer's Z-order is 3, after position control of the visible layers, the blurred layer's Z-order becomes 4, and the blurred result layer's Z-order becomes 3.

[0313] S1508, The synthesis engine sends a call request to the output module.

[0314] The compositing engine sends a call request to the output module, which then controls the layer compositing method for the current frame image.

[0315] S1509, The output module controls the layer composition method of the current frame image.

[0316] The output module controls the layer composition method of the current frame image specifically as S1509-S1511.

[0317] S1510, The output module requests the status of the current frame image from the frame skipping module.

[0318] The output module responds to the call request sent by the compositing engine by requesting the callback status of the current frame image from the frame-interval module.

[0319] Since the frame skipping module has determined the state of the current frame image based on the layer parameters related to the blurred layer in the current frame image and the state of the previous frame image, the output module requests the frame skipping module to call back the state of the current frame image.

[0320] S1511, The frame skipping module returns the status of the current frame image to the output module.

[0321] The frame skipping module responds to the request from the output module to return the status of the current frame image.

[0322] S1512. The output module controls the layer composition method of the current frame image according to the state of the current frame image.

[0323] If the current frame image is in a frame skipping state, then control all layers of the current frame image and the blurred result layer to perform HWC compositing.

[0324] If the current frame image is in a scheme off state, a normal frame state, or a copy frame state, the layer composition method of the current frame image will not be affected.

[0325] The layer compositing method and position control provided in this application embodiment enable the electronic device to control the position of the newly created blurred result layer, as well as control the layer compositing method of each frame image.

[0326] To make it easier to understand, the following will be combined with... Figure 16 This application provides a detailed description of the copy frame blur rendering part provided in the embodiments.

[0327] When the output module requests a callback from the inter-frame module indicating that the current frame image is in the copy frame state, during the layer compositing process of the copy frame image (specifically, during the GPU compositing process of the copy frame image), the blurring result is copied to the blur result layer, as follows:

[0328] S1601, The output module controls the layer composition of the current frame image.

[0329] Specifically, the output module controls the GPU compositing and HWC compositing of the current frame image through the `compose surfaces()` function. During the GPU compositing process controlled by the output module, the blurring result is copied to the blur result layer. This is specifically described in steps S1603-S1607.

[0330] S1602, The output module sends a call request to the drawing engine.

[0331] The output module sends a request to the rendering engine, which then invokes the rendering engine to draw the layers.

[0332] S1603, The drawing engine performs layer drawing.

[0333] The rendering engine responds to the call request sent by the output module and performs layer rendering.

[0334] The process of storing the blurring result in the blurring result layer is the same as the process of drawing the blurring result onto the blurring result layer. Therefore, the output module calls the drawing engine, and the drawing engine responds to the call request sent by the output module, controlling the subsequent blurring results in the blurring result layer.

[0335] S1604, The rendering engine sends a call request to the 2D graphics engine.

[0336] The rendering engine sends a request to the 2D graphics engine, which then calls the 2D graphics engine to render the blurred result.

[0337] The S1605 2D graphics engine is used to render the results of the blurring process.

[0338] The 2D graphics engine responds to the call request sent by the rendering engine and renders the result of the blurring process.

[0339] Among them, the 2D graphics engine is the drawing engine for 2D drawing.

[0340] The S1606 2D graphics engine requests the blurred result layer from the frame-interval module.

[0341] The drawing engine of 2D drawing is called to draw the blurring result. During the drawing process of the blurring result, the blur result layer is requested from the frame-interval module.

[0342] S1607, The frame-interval module returns a blurred result layer to the 2D graphics engine.

[0343] The frame skipping module responds to the 2D graphics engine's request by returning a blurred result layer to the 2D graphics engine.

[0344] The S1608 2D graphics engine draws the blurring results onto the blurring result layer.

[0345] The 2D graphics engine draws the blurring results onto the blurring result layer, thus completing the process of storing the blurring results on the blurring result layer.

[0346] It should be noted that S1606-S1607 actually involves the following process: When the blur result layer is pre-created, a buffer corresponding to the blur result layer is simultaneously created to display the normal operation of the pipeline. This buffer and the blur result layer are already bound together. Therefore, the 2D graphics engine requests the blur result layer from the frame-interval module, and simultaneously requests the buffer corresponding to the blur result layer from the frame-interval module. The frame-interval module returns the blur result layer and its corresponding buffer. The 2D graphics engine then draws the blur processing result onto the blur result layer. In reality, the 2D graphics engine copies the blur processing result onto the buffer corresponding to the blur result layer. Since the buffer is bound to the blur result layer, this achieves the effect of drawing the blur processing result onto the blur result layer.

[0347] The copy frame drawing part provided in this application embodiment realizes that during the layer synthesis process of the copy frame image, the blur processing result is copied to the blur result layer, so that the blur processing result of the copy frame image can be directly reused in the subsequent layer synthesis process of the skip frame image. This avoids the forced GPU synthesis due to blur processing during the image synthesis process. By reusing the blur processing result of the copy frame image and compositing the skip frame image through HWC, the power consumption of the electronic device is reduced, the performance of the electronic device is improved, and thus the user experience is improved.

[0348] Example 3:

[0349] The following is combined Figure 17 This paper details another image processing method provided by the embodiments of this application.

[0350] S1701, Obtain the first image.

[0351] The first image includes a first layer and a second layer. The first layer carries blur attribute information, which is used to indicate to the electronic device that the layer located below the first layer needs to be blurred accordingly; therefore, the first layer can also be called the blur layer.

[0352] In this embodiment, the second layer is located below the first layer and is a layer related to the blurring process corresponding to the first layer.

[0353] Scenarios for acquiring the first image include: adjusting the volume of an electronic device, a message window popping up on the electronic device, pulling down the status bar, shutting down the device, multitasking interfaces, etc.

[0354] In one possible implementation, a volume adjustment operation on an electronic device is received, and in response to the volume adjustment operation, a first image is acquired; at this time, the first layer in the first image is the volume bar layer.

[0355] In one possible implementation, when a message window pops up on the electronic device, the first image is acquired, and the first layer in the first image is the pop-up window layer.

[0356] In one possible implementation, a pull-down operation on the status bar is received, and in response to the pull-down operation on the status bar, a first image is obtained, wherein the first layer in the first image is the status bar display layer.

[0357] In one possible implementation, a power-off operation is received for the electronic device, and in response to the power-off operation, a first image is acquired, wherein the first layer in the first image is the power-off interface layer.

[0358] S1702. Blur the second layer to obtain a blurred result layer, and store the blurred result layer in the first cache.

[0359] The first cache is a cache (buffer) pre-bound to the blurred result layer.

[0360] In one possible implementation, during the blurring and layer compositing of the first and second layers by the image processor GPU, the second layer is blurred by the GPU to obtain a blurring result, and the blurring result is stored in a blurring result layer to obtain a blurring result layer carrying the blurring result.

[0361] In one possible implementation, the second layer is located below the first layer, and its blurring is related to that of the first layer. The first and second layers are composited and blurred using a GPU. The GPU blurs the second layer to obtain a blurred result, which is then stored in a blurred result layer. Specifically, the GPU composites (overlays) the second layer, blurs the overlaid second layer to obtain an initial blurred result, crops the initial blurred result based on the first layer, and then composites the blurred result, the first layer, and the second layer to obtain the GPU composite result. Simultaneously, the blurred result is stored in the blurred result layer. If there are still layers that have not been composited, the GPU composite result and the remaining layers (those not yet composited) are composited using a hardware compositor (HWC) to complete the layer compositing of the first image.

[0362] S1703, Obtain the second image.

[0363] The second image includes a third layer, which carries blur attribute information.

[0364] The third layer and the first layer each have one instance of the same element. Furthermore, the first and third layers have the same shape, size, and relative position. This means the first and third layers have the same name, and the third layer is in a stable phase compared to the first layer. When the third layer is in a stable phase compared to the first layer, there is no change in the third layer compared to the first layer. Therefore, the shape, size, and relative position of the first and third layers are identical.

[0365] In one possible implementation, the second image is the N frames following the first image, where N is a positive integer.

[0366] S1704. Use the hardware compositor HWC to composite the blurred result layer and the third layer.

[0367] The third layer and the blurred result layer of the second image are directly composited using HWC. The second image reuses the blurring result of the first image, therefore, the second image does not require blurring itself, avoiding the forced layer compositing and blurring processing via the GPU. By skipping the GPU's blurring process and directly controlling the compositing of the third layer and the blurred result layer of the second image via HWC, the power consumption of electronic devices caused by forced layer compositing and blurring processing via the GPU is reduced, improving the performance of electronic devices and thus enhancing the user experience.

[0368] In one possible implementation, the blurred result layer is stored sequentially at the position below the third layer. The blurred result layer and the third layer are then composited using a hardware compositor (HWC). By ensuring the blurred result layer is adjacent to the third layer, the third layer and the blurred layer are composited using HWC, thus ensuring the reuse of the blurred result stored in the blurred result layer, achieving the effect of the second image reusing the blurred effect of the first image.

[0369] The image processing method provided in this application reuses the blurring result of the first image in the second image, without needing to perform blurring. This avoids the need for forced layer compositing and blurring through the GPU during the layer compositing process, thereby reducing the power consumption of electronic devices caused by layer compositing and blurring through the GPU and improving the performance of electronic devices.

[0370] Furthermore, after S704, the method further includes: sending the target image obtained by compositing the blurred result layer and the third layer to the display so that the electronic device can normally display the second image carrying the blurred processing effect.

[0371] The target image is the image to be displayed obtained by multiplexing the blurring result of the first image into the second image.

[0372] Furthermore, after acquiring the first image, the first image is determined to be in the first state based on the state of the first layer and the state of the previous frame of the first image.

[0373] In one possible implementation, the first image is determined to be in a first state based on the layer parameters of the first layer and the state of the previous frame of the first image.

[0374] The layer parameters for the first layer include the number of first layers and the parameters for changing the first layer.

[0375] The first image is determined to be the first state, as follows:

[0376] When the number of first layers is 1, the first layer change parameter indicates that the first layer is in a stable stage, and the previous frame of the first image is in the third state, the state of the first image is determined to be the first state.

[0377] Furthermore, when the number of first layers is 1, the first layer change parameters indicate that the first layer is in a stable stage, and the second image is in the third state, the stable frame parameters carried by the previous frame of the first image are obtained; when the stable frame parameters indicate that the number of third images has accumulated to M, the first image is in the first state. The stable frame parameters effectively avoid the waste of resources caused by performing the synthesis process corresponding to the first state due to false stability, as there is no corresponding second state subsequently, thus avoiding increased power consumption.

[0378] The third image includes the first layer, and the number of the first layer is 1, which is in a stable phase.

[0379] When the number of first layers is 1, the first layer change parameter indicates that the first layer is in a stable stage, the previous frame of the first image is in the second state, and when the images in the second state accumulate to N frames, the state of the first image is determined to be the copy frame state.

[0380] It should be noted that the first state can also be called the "copy frame state", the second state can also be called the "frame skipping state", the third state can also be called the "normal frame state" and the fourth state can also be called the "scheme closed state". This application does not make specific limitations.

[0381] Furthermore, after acquiring the second image, the second image is determined to be in the second state based on the state of the third layer and the state of the previous frame of the second image.

[0382] In one possible implementation, the second image is determined to be in the second state based on the layer parameters of the third layer and the state of the previous frame of the second image.

[0383] The layer parameters for the third layer include the number of third layers and the parameters for changing the third layer.

[0384] The second image is determined to be the second state, as follows:

[0385] When the number of third layers is 1, the third layer change parameter indicates that the third layer is in a stable phase, and the previous frame of the second image is in the first state, the second image is determined to be in the second state.

[0386] When the number of third layers is 1, the third layer change parameter indicates that the third layer is in a stable phase, the previous frame of the second image is in the second state, and the accumulated images in the second state have not reached N frames, the second image is determined to be in the second state.

[0387] Furthermore, it also includes the image of the third state. Assuming the image of the third state is the fourth image, the fourth image includes a fourth layer, which carries blur attribute information. The fourth image is determined to be the third state as follows:

[0388] When the number of fourth layers is 1, and the fourth layer variation parameters indicate that the fourth layer is not in a stable phase, the fourth image is determined to be in the third state.

[0389] When the number of fourth layers is 1, and the previous frame of the fourth image is in the fourth state, the fourth image is determined to be in the third state.

[0390] Furthermore, when the number of fourth layers is 1, the fourth layer change parameter indicates that the fourth layer is in a stable stage, and the previous frame of the fourth image is in the third state, the stable frame parameter of the previous frame of the fourth image is obtained; if the stable frame parameter indicates that the cumulative number of third images has not reached M, then the fourth image is in the third state.

[0391] Furthermore, it also includes the image of the fourth state, which is assumed to be the fifth image.

[0392] The fifth image is determined to be in the fourth state, as follows:

[0393] When the number of layers carrying fuzzy attribute information is not 1 (i.e., 0 or greater than 1), the fifth image is determined to be in the fourth state.

[0394] Furthermore, the images in the first, second, and third states carry indication bit information indicating that the scheme is enabled, while the image in the fourth state carries indication bit information indicating that the scheme is disabled.

[0395] In one possible implementation, the current frame image carries indicator bit information indicating whether the scheme is enabled (images in the first, second, or third state), and the previous frame image carries indicator bit information indicating whether the scheme is disabled (images in the fourth state), creating a blurred result layer. This layer is then used for layering subsequent images in the first and second states.

[0396] In one possible implementation, the current frame image carries an indicator bit indicating that the scheme is off, and the previous frame image carries an indicator bit indicating that the scheme is on. The blurred result layer is then deleted. At this point, the image processing scheme changes from an on state to a off state, so the blurred result layer is no longer involved in the layer composition process of the first image. Therefore, the blurred result layer is deleted to avoid wasting resources and further avoid affecting the layer composition of images with the scheme off.

[0397] This application provides an image processing method, including: acquiring a first image; the first image includes a first layer and a second layer, wherein the first layer carries blur attribute information; blurring the second layer to obtain a blurred result layer, and storing the blurred result layer in a first cache; acquiring a second image, the second image including a third layer, the third layer carrying blur attribute information; and compositing the blurred result layer and the third layer using a hardware compositer (HWC). The first image undergoes blurring processing, and the blurred processing layer is stored. The second image reuses the blurred result layer of the first image, i.e., it reuses the blurring result of the first image without needing to undergo blurring processing itself. This avoids the forced layer compositing and blurring processing via the GPU during layer compositing. Instead, the HWC directly composites the layers of the second image and the blurred result layer to achieve the blurring of the first image, thereby reducing the power consumption of the electronic device caused by layer compositing and blurring processing via the GPU, improving the performance of the electronic device, and thus enhancing the user experience.

[0398] The following is combined Figure 18 This paper compares and explains the image processing method provided in this application with the original real-time blurring method.

[0399] like Figure 18 (a) shows the original real-time blurred path tracking map. Each frame of the original real-time blurred map requires forced GPU synthesis, which results in high power consumption and high power consumption of electronic devices. This reduces the performance of electronic devices and seriously affects the user experience.

[0400] like Figure 18 (b) shows an image processing method and a path tracking graph with N=1, i.e., a loop of one copy frame image and one skip frame image. The copy frame image is combined and blurred by GPU, and then HWC is performed; the skip frame image reuses the blurring result of the copy frame image, avoiding the need to force the use of GPU compositing when blurring during the skip frame image compositing stage, and directly compositing all layers of the skip frame image and the blur result layer through HWC.

[0401] like Figure 18 (c) shows an image processing method with path tracing for N=2, which involves a loop of one copy frame image and two skip frame images. The copy frame image is combined and blurred by GPU, and then combined by HWC. The skip frame image reuses the blurring result of the copy frame image, avoiding the need to force the use of GPU compositing when blurring during the skip frame image compositing stage. All layers of the skip frame image and the blur result layer are directly combined by HWC.

[0402] Because the blurring result of the copy frame is reused when skipping frames, it is not necessary to blur all layers under the blur layer through the GPU. This avoids the need to force the use of the GPU for blurring during the frame skipping image synthesis stage, thus avoiding the high GPU load caused by forcing the use of the GPU for synthesis. This reduces the power consumption of electronic devices, improves the performance of electronic devices, and thus improves the user experience.

[0403] This application also provides a computer-readable storage medium storing a computer program or instructions that, when executed, cause an electronic device to perform the aforementioned functions or steps.

[0404] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, cause an electronic device to perform the aforementioned functions or steps.

[0405] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Acquire a first image; the first image includes a first layer and a second layer; the first layer carries blur attribute information; Based on the state of the first layer and the state of the previous frame of the first image, the first image is determined to be in the first state; The second layer is blurred to obtain a blurred result layer, and the blurred result layer is stored in the first cache; Acquire a second image; the second image includes a third layer; the third layer carries blur attribute information; Based on the state of the third layer and the state of the previous frame of the second image, the second image is determined to be in a second state; wherein, the second state is used to indicate skipping the blurring process of the second image; The blurred result layer and the third layer are composited using the hardware synthesizer HWC; Determining the first image as a first state based on the state of the first layer and the state of the previous frame of the first image includes: When the number of the first layer is 1, the first layer change parameter indicates that the first layer is in a stable stage, and the previous frame of the first image is in the third state, the first image is determined to be in the first state; wherein, the third state is used to indicate that the previous frame of the first image is an image with only one blurred layer that is determined for the first time. Obtain the stable frame parameters carried by the previous frame of the first image; The number of third images indicated by the stable frame parameter is determined to be M; the third image includes a first layer, and the number of first layers included in the third image is 1, and the first layer is in a stable phase; wherein, M is a positive integer; When the number of the first layer is 1, the first layer change parameter indicates that the first layer is in a stable stage, the previous frame of the first image is in the second state, and the images in the second state have accumulated to N frames, the first image is determined to be in the first state; where N is a positive integer; Determining the second image as the second state based on the state of the third layer and the state of the previous frame of the second image includes: When the number of the third layer is 1, the third layer change parameter indicates that the third layer is in a stable stage, and the previous frame of the second image is in the first state, the second image is determined to be in the second state; When the number of the third layer is 1, the third layer change parameter indicates that the third layer is in a stable stage, the previous frame of the second image is in the second state, and the cumulative number of images in the second state has not reached N frames, the second image is determined to be in the second state; where N is a positive integer.

2. The method according to claim 1, characterized in that, The acquisition of the first image includes: Receive volume adjustment commands from electronic devices; In response to an operation to adjust the volume of the electronic device, the first image is acquired, wherein the first layer of the first image is a volume bar layer.

3. The method according to claim 1, characterized in that, The acquisition of the first image includes: When a message window pops up on the electronic device, the first image is acquired, and the first layer of the first image is the pop-up window layer.

4. The method according to claim 1, characterized in that, The acquisition of the first image includes: Receive pull-down requests to the status bar; In response to the pull-down operation of the status bar, the first image is acquired, and the first layer of the first image is the status bar display layer.

5. The method according to claim 1, characterized in that, The second image is the N frames following the first image; where N is a positive integer.

6. The method according to claim 1, characterized in that, The number of the first layer and the third layer is 1, and the first layer and the third layer have the same shape, size and relative position.

7. The method according to claim 1, characterized in that, The second layer is located below the first layer; The process of blurring the second layer to obtain a blurred result layer includes: During the process of blurring and compositing the first layer and the second layer using the graphics processing unit (GPU), the second layer is blurred using the GPU to obtain the blurring result. The blurring result is stored in the blurring result layer.

8. The method according to claim 1, characterized in that, The process of compositing the blurred result layer and the third layer using a hardware compositor (HWC) includes: The blurred result layer and the third layer are composited using a hardware compositor (HWC) based on the position of the blurred result layer below the third layer.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: The target image obtained by combining the blurred result layer and the third layer is then displayed.

10. An electronic device, characterized in that, Including processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions that, when executed, implement the method as described in any one of claims 1-9.

Images