Image processing method and electronic equipment

By directly performing blur processing during layer synthesis or drawing hardware bitmaps with blur attributes, the problems of low screenshot blur efficiency and waste of storage resources in the existing technology are solved, and more efficient image processing is achieved and memory savings.

CN118982477APending Publication Date: 2024-11-19HONOR DEVICE CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202310550444.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-15
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

Existing electronic devices are inefficient and consume a lot of storage resources during screenshot blurring, requiring a lower cost and more efficient method of background blurring.

Method used

The blur process is directly performed during layer synthesis, or the hardware bitmap is drawn using a brush with blur attributes to avoid the conversion between the hardware bitmap and the software bitmap, and the blur process and drawing are directly completed in the GPU.

Benefits of technology

It reduces the generation of intermediate products during screenshot blurring, saves electronic device memory, improves the efficiency of screenshot blurring, and improves the efficiency of image transmission.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982477A_ABST
    Figure CN118982477A_ABST
Patent Text Reader

Abstract

The invention provides an image processing method and electronic equipment, and the method is implemented, the electronic equipment can carry out the fuzzy processing of an image in the process of synthesizing a layer after traversing the layer, or can carry out the fuzzy processing of the image after obtaining a hardware bitmap without a fuzzy effect. According to the method, a hardware bitmap with a fuzzy effect is drawn by using a paintbrush with a fuzzy attribute, so that the generation of intermediate products in the screenshot fuzzy process can be effectively reduced, and the efficiency of the screenshot fuzzy process of the electronic equipment is improved while the memory of the electronic equipment is saved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method and an electronic device. Background Art

[0002] Currently, many usage scenarios of electronic devices involve background blur. For example, when a user pulls down the notification bar or long presses a desktop folder, the electronic device will blur the image displayed on the current interface in response to the user operation and display the blurred image on the screen to enhance the visual effect and user experience.

[0003] However, the screenshot blurring method currently used by electronic devices has low processing efficiency and consumes a large amount of storage resources. Therefore, it is necessary to study a background blurring method with lower storage cost and higher efficiency. Summary of the invention

[0004] The purpose of the present application is to provide an image processing method and an electronic device. By implementing the method, the electronic device can blur the image in the process of synthesizing the layers after traversing the layers, or after obtaining a hardware bitmap without blur effect, use a brush with blur attribute to draw it to obtain a hardware bitmap with blur effect, which can effectively reduce the generation of intermediate products in the screenshot blur process, and improve the efficiency of the electronic device screenshot blur process while saving the electronic device memory.

[0005] The above-mentioned and other objects are achieved by the features of the independent claims. Further implementations are reflected in the dependent claims, the description and the drawings.

[0006] In a first aspect, the present application provides an image processing method, comprising: synthesizing some or all layers in a first image to obtain first image data; using a graphics processor (GPU) to perform blur processing based on the first image data to obtain second image data; converting the second image data into a second image, wherein the second image is a hardware bitmap stored in the GPU.

[0007] It should be noted that in the traditional screenshot blurring method, the electronic device does not blur the image when synthesizing the layers using the GPU, so the hardware bitmap obtained by the layer synthesis is also a hardware bitmap without blurring effect; in the subsequent process, the hardware bitmap without blurring effect needs to be converted into a software bitmap without blurring effect first, so that the CPU can access the software bitmap without blurring effect and use the corresponding blurring algorithm to blur it, and obtain a software bitmap with blurring effect. However, it takes a certain amount of time to convert the hardware bitmap into a software bitmap, and the software bitmap consumes more storage space than the hardware bitmap due to its specific storage method; especially in the subsequent display process, the electronic device may also need to obtain the HardwareBuffer corresponding to the software bitmap with blurring effect to perform corresponding operations; in this case, after converting the new software bitmap into a new hardware bitmap with blurring effect, the electronic device can obtain the HardwareBuffer corresponding to the software bitmap with blurring effect, and then perform the next step of displaying, which undoubtedly further consumes storage space and further slows down the execution efficiency of the screenshot blurring process.

[0008] However, in this method, the second image is already an image with a blurred effect, and in the subsequent process, the hardware bitmap can be drawn into the display screen. This method directly blurs the image in the synthesis stage of the image data to obtain a hardware bitmap with a blurred effect (i.e., the second image). In this way, the electronic device does not need to use the CPU to blur the hardware bitmap with the blurred effect again, and there is no need to convert the above-mentioned hardware bitmap into a software bitmap in order to facilitate the CPU to access the above-mentioned hardware bitmap. By implementing this method, in the process of blurring the screenshot, the electronic device can avoid the conversion process between the hardware bitmap and the software bitmap, while saving the memory of the electronic device, it can also improve the efficiency of the blurring process of the electronic device screenshot.

[0009] In combination with the first aspect, in a possible implementation, the using a graphics processor GPU to blur the first image data to obtain the second image data includes: when a first parameter indicates that the GPU is to be used to blur the image, using the GPU to blur the first image data to obtain the second image data, the first parameter is an input parameter of a screenshot function, and the screenshot function is used to obtain the first image.

[0010] It should be understood that in the method provided in the present application, the process of synthesizing and blurring the image is completed by the GPU, and different applications may have different performance requirements for the screenshot blurring process. Therefore, in this embodiment, this scheme encapsulates the native screenshot function and allows the application to set whether GPU blurring is required. After the electronic device responds to an operation with a screenshot blurring requirement (for example, the user pulls down the status bar or long presses a folder), when the electronic device calls the corresponding function that implements the screenshot function, the electronic device can obtain the user's parameter (i.e., the first parameter) for determining whether the GPU needs to be used for blurring, and send the first parameter to the GPU so that the GPU can determine whether the image obtained by layer synthesis needs to be blurred in the layer synthesis stage. When the first parameter indicates that the GPU is used to blur the image, the electronic device will use the GPU to blur the first image data in the layer synthesis stage to obtain the second image data.

[0011] In combination with the first aspect, in a possible implementation, after converting the second image data into a second image, the method further includes: using the GPU to draw the second image onto a display screen; or, copying the pixel data corresponding to the second image to the application memory, and after obtaining the third image, using the central processing unit CPU to draw the third image onto the display screen, wherein the third image is a software bitmap stored in the application memory.

[0012] It should be understood that the hardware bitmap with blur effect will be drawn on the screen of the electronic device later. However, for electronic devices, when sending images for display, there are two ways to draw images: one is that the electronic device uses the CPU to draw, which is also called "software drawing"; the other is that the electronic device uses the GPU to draw, which is also called "hardware drawing".

[0013] For hardware bitmaps, when drawing, they can be drawn directly using hardware drawing; however, since the hardware bitmap also exists in the logically independent storage space of the GPU, and the CPU cannot directly access the logically independent storage space of the GPU, it is also impossible to directly use software drawing to draw the hardware bitmap onto the display.

[0014] Therefore, in this embodiment, after acquiring the second image, the electronic device can directly use the hardware drawing method, that is, use the GPU to draw the second image to the display screen. Specifically, in order to avoid the situation where the picture is garbled due to the difference between the hardware drawing frequency and the display screen refresh frequency, in the process of sending the second image with a blurred effect to the display, the second image can be first drawn to an image buffer that can be accessed by the CPU, GPU, and display screen, and the content in the image buffer will be consumed by the display screen and displayed on the display screen.

[0015] Alternatively, if the electronic device needs to use software drawing, the electronic device can first convert the second image into a software bitmap (i.e., the third image) accessible to the CPU, that is, copy the pixel data corresponding to the second image to the application memory and then use the CPU to draw the third image onto the display screen.

[0016] In combination with the first aspect, in a possible implementation, after using the GPU to draw the second image into the display screen, the method further includes: in response to a user operation, drawing the second image into the display screen, the user operation being an operation that triggers a background blur requirement.

[0017] It should be noted that in the image processing method provided in the present application, the hardware bitmap obtained by the electronic device through GPU blur, that is, the second image, can be stored in a logically independent storage space of the GPU for a long time. Therefore, these hardware bitmaps can be exported and saved in other storage devices, or cached in the application process and used in subsequent drawing logic.

[0018] Specifically, the time for which the second image is stored in the logically independent storage space of the GPU may be 10 minutes, one hour, one day or permanently, which is not limited in the present application.

[0019] For example, assuming that the second image is a hardware bitmap obtained by blurring the wallpaper image in the main interface of the electronic device, and the electronic device obtains the second image through this method after the user long presses the folder on the display screen of the electronic device, the electronic device will draw the second image on the display screen. Afterwards, the hardware bitmap can still be saved in the logically independent storage space of the GPU. Afterwards, if the user long presses the folder on the display screen of the electronic device again, then in response to this operation, the electronic device needs to obtain the blurred image corresponding to the wallpaper image again. However, since the second image is still saved in the logically independent storage area of ​​the GPU, the electronic device does not need to trigger the screenshot blur process provided by this method again to obtain the hardware bitmap with a blurred effect, but can directly call the second image saved in the logically independent storage area of ​​the GPU and draw it on the display screen of the electronic device, further improving the efficiency of blurring and displaying the image.

[0020] In combination with the first aspect, in a possible implementation, after drawing the third image into the display screen using a central processing unit CPU, the method further includes: drawing the third image into the display screen in response to a user operation, wherein the user operation is an operation that triggers a background blur requirement.

[0021] Similarly, the third image can also be saved in the application process for a long time, so these software bitmaps can be exported and saved in other storage devices, or cached in the application process and used in subsequent drawing logic to further improve the efficiency of blurred image display.

[0022] In combination with the first aspect, in a possible implementation, the use of the GPU to blur the first image data to obtain the second image data includes: obtaining the pixel value of each pixel in the first image data, and adjusting the pixel value of each pixel to obtain third image data with a blurred effect; drawing the image content corresponding to the third image data with a blurred effect on the image content corresponding to the first image data to obtain the second image data.

[0023] It should be understood that before the GPU synthesizes the layers, the screenshot function will call the graphics system service provided by the electronic device to generate corresponding layer drawing instructions and blur drawing instructions, and send these instructions to the GPU. In this embodiment, when the GPU synthesizes the layers, it actually draws each layer and the corresponding elements in each layer on a canvas according to the size and position recorded in the drawing instructions. At this time, the image content recorded on the canvas does not have a blur effect, and the image corresponds to the first image data. When blurring the clear image, the GPU will then first obtain the existing image content on the canvas according to the blur instruction, obtain the pixel value of each pixel in the existing image content, and adjust the pixel value of each pixel to obtain the image content with a blur effect (i.e. corresponding to the third image data), and draw the image content with a blur effect on the above canvas, and draw it in the form of an overlay to the position where the original existing image content is located. At this time, all the image content recorded on the canvas corresponds to the second image data.

[0024] In combination with the first aspect, in a possible implementation, before the first image data is blurred by a graphics processor GPU to obtain second image data, the method further includes: determining one or more of the image saturation and the transparency of the blurred area used when blurring the first image data based on a second parameter, wherein the second parameter is an input parameter of a screenshot function, and the screenshot function is used to obtain the first image.

[0025] In this embodiment, the input parameter of the screenshot function may also include the second parameter, which may include multiple sub-parameters. These sub-parameters may be sent to the GPU along with the calling process of the screenshot function. When the GPU performs blur processing, the second parameter may be used to personalize the blur effect achieved by the GPU blur processing. For example, the second parameter may include one or more of the sub-parameters for determining the saturation of the image obtained by the blur processing, the sub-parameter for the user to adjust the transparency of the blurred area, and the sub-parameter for adding a color layer to the blur effect (i.e., making the blurred image appear as a color similar to a filter effect), so as to further optimize the visual effect provided to the user. Specifically, the second parameters corresponding to different applications may be different.

[0026] In second aspect, the present application provides an image processing method, characterized in that the method includes: synthesizing part or all of the layers in a fourth image to obtain fourth image data; converting the fourth image data into a fifth image, wherein the fifth image is a hardware bitmap stored in a graphics processor GPU; controlling the GPU to use a brush with a blur attribute to draw the fifth image to obtain a sixth image; and drawing the sixth image on a display screen.

[0027] In this method, after the electronic device obtains the hardware bitmap without blur effect, i.e., the fifth image, the electronic device still uses the GPU to perform blur processing on it, i.e., uses a brush with blur attribute to draw it through the GPU to obtain a hardware bitmap with blur effect, i.e., the sixth image, which can also be stored in a logically independent storage space of the GPU. In this way, the electronic device can also avoid the conversion process between the hardware bitmap and the software bitmap, save memory and improve the efficiency of the screenshot blur process.

[0028] Specifically, the electronic device can create a RenderNode class, and based on the methods contained in the RenderNode class, create and bind a hardware canvas, which can be used to record the drawing operation of the current RenderNode. Afterwards, the electronic device can set the brush properties used for the drawing of the RenderNode, such as setting a blur style and radius of the BlurMaskFilter property of the brush through corresponding parameters. When drawing, the GPU can implement a blur filter effect for the drawn image based on the BlurMaskFilter property of the brush. The drawing operation here is equivalent to the operation of blurring the fifth image that originally did not have a blur effect, and the new hardware bitmap obtained by drawing is the sixth image with a blur effect.

[0029] In a third aspect, the present application provides an electronic device, comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method in the first aspect or any possible implementation of the first aspect, or the method in the second aspect.

[0030] In a fourth aspect, the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method in the first aspect or any first possible implementation, or the method in the second aspect.

[0031] In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, causes the electronic device to execute a method as in the first aspect or any first possible implementation, or a method as in the second aspect.

[0032] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method in the first aspect or any first possible implementation, or the method in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a blurring process of a screenshot of an electronic device provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of an image data conversion process in a screenshot blurring process provided by an embodiment of the present application;

[0035] Figure 3 A schematic diagram of another electronic device screenshot blurring process provided by an embodiment of the present application;

[0036] Figure 4 A schematic diagram of another image data conversion process in the screenshot blurring process provided by an embodiment of the present application;

[0037] Figure 5 A schematic diagram of another electronic device screenshot blurring process provided by an embodiment of the present application;

[0038] Figure 6 A flowchart of using a brush with fuzzy attributes to draw a hardware bitmap provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of an image data conversion process in a screenshot blurring process provided by an embodiment of the present application;

[0040] Figure 8 A schematic diagram of a process of displaying a historically saved hardware bitmap by an electronic device provided in an embodiment of the present application;

[0041] Fig. 9 An architecture diagram of an electronic device provided in an embodiment of the present application;

[0042] Fig.10 A flowchart of an image processing method provided in an embodiment of the present application;

[0043] Fig.11 A flowchart of an image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0045] To facilitate understanding, the relevant terms involved in the embodiments of the present application are first introduced below.

[0046] (1) Application layer, application framework layer, local framework layer, hardware layer

[0047] The application layer (Application), application framework layer (Android Framework), local framework layer (Android Native), and hardware layer (Hardware) are the four layers in the Android system architecture. Among them:

[0048] The application layer includes system built-in applications and non-system level applications, which are usually developed based on Java language and are mainly responsible for direct interaction with users. For example, when a user operates an electronic device running on the Android system and pulls down the notification bar on the screen, the application layer can respond to the user operation and call the screenshot function in the application framework layer to complete the screenshot.

[0049] The application framework layer is the foundation of Android application development. This layer is written in Java code and provides developers with the APIs needed to develop applications. Many core applications also implement their core functions through this layer. For example, the screenshot function in the above description can be encapsulated in the SurfaceControl class in the application framework layer. This layer simplifies the reuse of components. Developers can directly use the components it provides to develop applications quickly, or they can achieve personalized expansion through inheritance. Specifically, the application framework layer can include multiple parts such as activity manager, window manager, content provider, view system, package manager, phone manager, resource manager, location manager, notification manager, etc.

[0050] The local framework layer provides the main feature support for the Android system through some C / C++ libraries, such as some local services and some link libraries. This layer is implemented based on C and C++ languages. When a device needs to perform a complex operation, if it is implemented through Java code, the efficiency will be very low. At this time, you can choose to implement it through C or C++ code. At this time, the electronic device needs to communicate with the upper-level Java code through the C or C++ code provided by the local framework layer (this communication method is called jni mechanism in the Android system). For example, the SurfaceFlinger module of the local framework layer can traverse some or all layers of the interface during the screenshot process of the electronic device, and arrange the position and draw the content of each layer (the drawing described here actually only generates drawing instructions and sends them to the GPU. The actual execution of the drawing instructions is in the GPU synthesis stage).

[0051] The hardware layer is the hardware abstraction layer, which encapsulates the underlying hardware driver and provides a general interface for calling the driver to the application framework layer. As long as the manufacturer implements the corresponding interface according to the specification and stores it in a specific directory in the form of a shared library, the upper layer only needs to load the shared library and find the pointer of the device corresponding to the corresponding module. After obtaining the pointer of the entire device, the underlying hardware can be operated. For example, after receiving the drawing instruction issued by the SurfaceFlinger module of the aforementioned local framework layer, the GPU in the hardware layer can draw (merge) all layers traversed by the SurfaceFlinger module according to the drawing instruction.

[0052] (2) Bitmap

[0053] A bitmap, also known as a bitmap or raster image, is made up of individual dots called pixels (picture elements). These dots can be arranged and colored in different ways to form a pattern. When you enlarge a bitmap, you can see the countless individual squares that make up the entire image. The effect of increasing the size of a bitmap is to increase the size of individual pixels, making lines and shapes appear jagged. However, if you look at it from a slightly farther distance, the colors and shapes of a bitmap image appear continuous. Photos taken with a digital camera, pictures scanned by a scanner, and computer screenshots are all bitmaps. Understandably, since bitmaps need to record the position and color value of each pixel when they are saved, bitmaps require a large amount of storage space in electronic devices.

[0054] In addition, depending on the storage location, the bit Figure 1 Generally, it can be divided into software bitmap and hardware bitmap. It should be understood that in electronic devices, the memory used by the CPU and GPU is generally on the same physical memory particle, but the GPU itself has a logically independent memory interval.

[0055] A software bitmap is a bitmap that stores pixel data (i.e., pixel byte array) in application memory. Figure 1 Generally, there is also a copy of the image stored in a logically independent memory interval of the GPU; a hardware bitmap refers to an image that only stores pixel data in a logically independent memory interval of the GPU. Therefore, compared to a software bitmap, a hardware bitmap only takes up half the memory. Generally speaking, the CPU cannot directly read the data in the logically independent memory interval of the GPU, and it is also impossible to use the blur algorithm to directly process the hardware bitmap stored in the GPU. It is necessary to use an electronic device to convert the hardware bitmap into a software bitmap (that is, copy the pixel content of the hardware bitmap to the application memory) before the CPU can blur the newly acquired software bitmap.

[0056] (3) Software drawing and hardware drawing

[0057] The drawing of user interfaces generally relies on two core hardware in electronic devices: the central processing unit (CPU) and the graphics processing unit (GPU). The CPU is the computing and control core of the computer system and the final execution unit for information processing and program running; the GPU is a processor specifically used for image computing. In computer systems, the core component usually referred to as the "graphics card" is the GPU.

[0058] Correspondingly, in the Android system, there are two ways to draw images: one is that the electronic device uses the CPU to draw, which is also called "software drawing"; the other is that the electronic device uses the GPU to draw, which is also called "hardware drawing". Among them:

[0059] The engine framework used for software drawing is Skia, which is a high-quality 2D cross-platform graphics framework for low-end devices. For example, Google's Chrome and Flutter both use this graphics rendering framework.

[0060] The engine framework used for hardware rendering is usually OpenGL, which converts graphics calculations that the CPU is not good at into GPU-specific instructions through the underlying software code, and the GPU completes the rendering task. In other words, the essence of hardware rendering is to use the GPU instead of the CPU to complete the rendering work in order to achieve better performance.

[0061] Currently, many usage scenarios of electronic devices involve background blur. For example, when a user pulls down the notification bar or long presses a desktop folder, the electronic device will blur the image displayed on the current interface in response to the user operation and display the blurred image on the screen to enhance the visual effect and user experience.

[0062] Figure 1 The entire screenshot blurring process of today’s electronic devices is shown from the perspective of the Android system architecture. Figure 1 As shown, the architecture involved in the screenshot blurring process of the electronic device may include the following four layers: application layer 101, application framework layer 102, local framework layer 103, and hardware layer 104. The specific functions of each layer of architecture in the Android system can refer to the aforementioned related instructions, which will not be repeated here. In the process of the electronic device blurring the currently displayed image and displaying the blurred image on the screen, the application layer 101, application framework layer 102, local framework layer 103, and hardware layer 104 can work together to complete the entire screenshot blurring process, specifically:

[0063] The application layer 101 can respond to user operations, such as the user's pull-down operation on the status bar or the user's long press operation on the folder, and call the screenshot function in the auxiliary management class SurfaceControl in the application framework layer, such as captureDisplay or captureLayer (it should be understood that the parameters required by the two functions captureDisplay and captureLayer are different. When taking screenshots, whether the application layer specifically calls captureDisplay or captureLayer depends mainly on whether the parameters that can be transmitted to the application framework layer 102 in the current scenario are applicable to captureDisplay or captureLayer).

[0064] Afterwards, the application framework layer 102 will further call the graphics system service SurfaceFlinger in the local framework layer 103 through the screenshot function, and transmit the parameters obtained based on the above screenshot function to the graphics system service SurfaceFlinger in the local framework layer 103.

[0065] After the graphics system service SurfaceFlinger in the local framework layer 103 obtains the parameters transmitted from the application framework layer, it will create a data buffer based on the current screenshot requirements of the electronic device (the data buffer exists in a logically independent memory interval of the GPU and belongs to a hardware buffer, so it is referred to as a hardware buffer below). It should be understood that the only user of this hardware buffer at this time is the GPU in the hardware layer 104, that is, other objects except the GPU cannot access or use the hardware buffer at this time.

[0066] In addition, as the manager of the layer, the graphics system service SurfaceFlinger will traverse some or all of the layers contained in the screen currently displayed by the electronic device after receiving the parameters transmitted by the application framework layer 102, and arrange the position and draw the content of each layer. It should be noted that the "position arrangement and drawing" mentioned here is actually just the graphics system service SurfaceFlinger generating the corresponding drawing instructions for this part of the action, and these drawing instructions will eventually be sent to the GPU in the hardware layer 104, and executed by the GPU to complete the actual layer synthesis work.

[0067] After the GPU in the hardware layer 104 receives the drawing instruction generated by the above-mentioned graphics system service SurfaceFlinger, the GPU will draw each layer according to the drawing instruction, and the cache data obtained by drawing will be stored in the above-mentioned hardware buffer.

[0068] During the process of GPU writing image cache data into the above hardware buffer, the hardware buffer only supports GPU use, and SurfaceFlinger cannot transmit the data in the hardware buffer at this time. After the GPU executes all drawing instructions, that is, writes all layers into the above hardware buffer in the form of cache data, the GPU can return a parameter to the graphics system service SurfaceFlinger, which is used to notify the local framework layer 102 that the above hardware buffer has been used up by the GPU. At this time, the hardware buffer no longer uses the GPU as the only user, but uses the graphics system service SurfaceFlinger as the user, so that the graphics system service SurfaceFlinger can return the hardware buffer to the application framework layer 102. It should be noted that "returning the hardware buffer to the application framework layer 102" here refers to returning the address or handle of the hardware buffer as a parameter to the application framework layer 102, so that the application framework layer can read the cached data in the hardware buffer.

[0069] After the application framework layer 102 obtains the cached data in the hardware buffer, the application framework layer 102 can convert the cached data in the hardware buffer into a hardware bitmap based on the corresponding function (such as the createBitmap function), and the hardware bitmap is also stored in a logically independent memory interval of the GPU. It should be understood that although the cached data stored in the hardware buffer is obtained by the GPU synthesizing the layers of the screenshot image, it can represent the screenshot image to a certain extent, but these image data are not images that the application layer can directly call the drawing function to draw; only after the application framework layer 102 can convert these data into a hardware bitmap, the application layer can directly call the drawing function to draw.

[0070] However, since the CPU cannot directly access the logically independent memory interval of the GPU, the CPU cannot directly blur the hardware bitmap stored in the logically independent memory interval of the GPU. Therefore, in order to obtain the blurred screenshot image, the electronic device needs to copy the hardware bitmap to the application memory to obtain a software bitmap corresponding to the hardware bitmap. After obtaining the software bitmap, the electronic device can use the CPU and the fuzzy algorithm stored therein (such as a Gaussian fuzzy algorithm or a Kawase fuzzy algorithm) to blur the software bitmap to obtain a software bitmap with a blurred effect.

[0071] Finally, the software bitmap with blur effect will be sent back to the application layer 101 and drawn on the screen of the electronic device. In addition, in some scenarios, during the display process, if the electronic device uses hardware drawing, the electronic device may also need to obtain the HardwareBuffer corresponding to the software bitmap with blur effect (that is, a copy of the software bitmap with blur effect stored in a logically independent memory interval of the GPU) to perform corresponding operations; in this case, after converting the new software bitmap into a new hardware bitmap with blur effect, the electronic device can obtain the HardwareBuffer corresponding to the software bitmap with blur effect and then perform the next step of display.

[0072] Specifically, after the application obtains the above-mentioned software bitmap with blur effect or hardware bitmap with blur effect, the relevant application that calls the screenshot function will trigger the drawing logic to draw the software bitmap with blur effect or hardware bitmap with blur effect to the display screen. In fact, in the process of displaying the software bitmap with blur effect or hardware bitmap with blur effect on the screen, in order to avoid the situation where the picture is garbled due to the difference between the hardware drawing frequency and the display screen refresh frequency, the above-mentioned software bitmap with blur effect or hardware bitmap with blur effect will be drawn to an image buffer first, and the content in the image buffer will be consumed by the display screen and displayed on the display screen. The image buffer is the same as the above-mentioned hardware buffer. It is also a display data buffer that can write data, and it is also located in a logically independent memory interval of the GPU, but after the image buffer is mapped, the CPU, GPU, and display screen all have access to it.

[0073] Combined with the above description, it can be seen that the electronic device needs to draw, synthesize and transform the image multiple times during the entire screenshot blurring process, and the whole process involves a lot of intermediate products. Figure 2 The specific transformation process of image data in the entire screenshot blurring process is explained.

[0074] like Figure 2 As shown, the image 201 may be an interface image currently displayed on the display screen of the electronic device. It should be understood that the image 201 is composed of multiple layers (i.e. Figure 2 All layers in the layer set 202 shown in FIG. 2 are synthesized in a certain order before being presented on the display screen as seen by the user.

[0075] When the user performs an operation on the display screen, such as pulling down the notification bar or long pressing the desktop folder, the electronic device responds to the operation and controls its internal CPU, GPU and other hardware to perform a series of operations such as drawing, synthesizing, blurring, and sending to display on the image 201, so as to display the blurred image 201 on the display screen. The final image presented can be referred to as Figure 2 The image 208 shown in the figure. It should be understood that the image 208 is only for the convenience of users to understand the specific meaning of "image blur", and it does not represent the actual style of the image obtained after the electronic device blurs the interface image in the actual scene; for example, in the actual application scene, other layers can also be displayed on the blurred image displayed in the image 208, such as the notification bar layer displayed by the electronic device in response to the user's pull-down notification bar operation, and the folder function list layer displayed by the electronic device in response to the user's long press of the desktop folder operation. The specific style and content of the image 208 are not limited in this application.

[0076] Specifically, in response to the user's operation, the GPU in the electronic device will synthesize the layers (all layers or part of the layers) in the image 201. In addition, before synthesizing these layers, the electronic device will call the corresponding API interface and use the graphics system service to traverse all layers in the image 201 to obtain layers 2021, 2022, and 2023 as shown in the layer set 202. Among them, layer 201 is the background layer, layer 2022 is the status bar layer (floating layer), and layer 2023 is the content layer. It should be understood that layers 2021-2023 are only for ease of understanding. In the actual process of traversing the layers, the image 201 can be divided into more layers, and this application does not limit this.

[0077] It should be noted that the drawing content of layers 2021-2023 is only sent to the GPU in the form of drawing instructions. These drawing instructions are generated according to the size and position of each element in each layer in the user interface (or display screen), as well as the display order between the layers. After the graphics service system generates these drawing instructions, these drawing instructions will be sent to the GPU so that the GPU draws and synthesizes the layers according to the drawing instructions to obtain image cache data. Combined with the above description, it can be seen that the image cache data obtained here is not an image that can be directly drawn by calling the drawing function. The GPU only writes these image cache data into the hardware buffer 203 based on the drawing instructions. This hardware buffer 203 can be a storage area initialized and created in a logically independent memory interval of the GPU when the electronic device calls the screenshot function.

[0078] When the GPU writes data into the hardware buffer 203, the GPU is the only user of the hardware buffer 203. After all drawing instructions are executed, that is, all image cache data corresponding to the image 203 is written into the hardware buffer 203, the use right of the hardware buffer 203 will be opened to other objects, and the electronic device will call the corresponding conversion function (such as the createBitmap function) to convert the image cache data stored in the hardware buffer 203 into the hardware bitmap 204. At this time, the hardware bitmap is also stored in the logically independent memory interval of the GPU, and the hardware bitmap 204 is already an image that can be used for drawing.

[0079] However, since the CPU cannot read the data in the video memory (logically independent memory interval of the GPU), the CPU cannot use the blur algorithm to directly blur the hardware bitmap 204. Therefore, in order to obtain an image with a blurred effect, the electronic device needs to convert the hardware bitmap 204 into a software bitmap 205 first. The "conversion" mentioned here actually means that the electronic device copies a copy of the pixel data corresponding to the conversion of the hardware bitmap 204 stored in the logically independent memory interval of the GPU to the application memory, and the pixel data copied and stored in the application memory is the software bitmap 205. It should be noted that the appearance of the software bitmap 205 does not mean the disappearance of the hardware bitmap 204, that is, after the software bitmap 205 is stored in the application memory, the hardware bitmap 204 is still stored in the logically independent memory interval of the GPU.

[0080] Since the CPU can read the application memory, after acquiring the software bitmap 205 , the CPU can directly use a corresponding fuzzy algorithm to fuzzy the software bitmap to obtain a software bitmap 206 with a fuzzy effect.

[0081] After obtaining the software bitmap 206 with blur effect, the electronic device will respond to the drawing logic triggered by the relevant application that calls the screenshot function, and draw the software bitmap 206 with blur effect to the display screen. In fact, in the process of displaying the software bitmap with blur effect or the hardware bitmap with blur effect on the screen, in order to avoid the situation where the picture is garbled due to the difference between the hardware drawing frequency and the display refresh frequency, the software bitmap 206 with blur effect will first be drawn to an image buffer that can be accessed by the CPU, GPU, and display screen, and the content in the image buffer will be consumed by the display screen and displayed on the display screen.

[0082] Optionally, in some scenarios, the application may also need to obtain the HardwareBuffer corresponding to the software bitmap 206 to perform certain operations. In this case, the software bitmap 206 needs to be converted into a hardware bitmap 207 in order to obtain the corresponding HardwareBuffer.

[0083] Therefore, looking at the entire screenshot blurring process, after obtaining the hardware bitmap 204, the electronic device needs to first convert the hardware bitmap into a software bitmap 205, and after blurring the software bitmap 205, a new software bitmap 206 with a blurring effect will be generated; if the application needs to get the HardwareBuffer corresponding to the software bitmap 206 during the display process, the electronic device also needs to convert the new blurred software bitmap into a hardware bitmap with a blurring effect. Combined with the above description, it can be seen that the conversion between the hardware bitmap and the software bitmap not only takes time to copy pixels, but also takes up more memory of the electronic device, which affects performance and occupies more memory space.

[0084] Based on the defects existing in the aforementioned screenshot blurring method, the present application provides an image processing method and an electronic device. When implementing this method, the electronic device can blur the image during the process of synthesizing the layers after traversing the layers, or after obtaining a hardware bitmap without a blurring effect, it can use a brush with a blurring attribute to draw it to obtain a hardware bitmap with a blurring effect. This can effectively reduce the generation of intermediate products in the screenshot blurring process, save the memory of the electronic device, and improve the efficiency of the screenshot blurring process of the electronic device.

[0085] First combine Figure 3 and Figure 4 The specific process of blurring the image during the process of synthesizing the layers after traversing the layers is described.

[0086] Figure 3 The entire screenshot blurring process of an electronic device is shown from the perspective of the Android system architecture. Figure 3 As shown, the architecture involved in the screenshot blurring process of the electronic device may include the following four layers: application layer 301, application framework layer 302, local framework layer 303, and hardware layer 304. The specific functions of each layer of architecture in the Android system can refer to the aforementioned related instructions, which will not be repeated here. In the process of the electronic device blurring the currently displayed image and displaying the blurred image on the screen, the application layer 301, application framework layer 302, local framework layer 303, and hardware layer 304 can work together to complete the entire screenshot blurring process, specifically:

[0087] The application layer 301 can respond to user operations, such as the user's pull-down operation on the status bar or the user's long press operation on the folder, and call the screenshot function in the auxiliary management class SurfaceControl in the application framework layer, such as captureDisplay or captureLayer (it should be understood that the parameters required by the two functions captureDisplay and captureLayer are different. When taking a screenshot, whether the application layer specifically calls captureDisplay or captureLayer depends mainly on whether the parameters that can be transmitted to the application framework layer 302 in the current scenario are applicable to captureDisplay or captureLayer).

[0088] It should be understood that, in combination with the above Figure 1 and Figure 2As can be seen from the relevant description, the synthesis of layers is generally completed by the GPU. Therefore, in an embodiment of the present application, the electronic device can encapsulate the native screenshot function, allowing the application to set whether GPU blur is required and specify a personalized style. That is to say, in the process of the application layer 301 calling the screenshot function in the auxiliary management class SurfaceControl, the parameters passed by the application layer to the application framework layer 302 may also include some related parameters, such as whether to use GPU blur parameters, image saturation and color value parameters used during blurring, and other parameters. These parameters will be passed to the local framework layer 303 along with the screenshot function in the subsequent process, so that the graphics system service SurfaceFlinger in the local framework layer 303 determines whether to generate corresponding blur drawing instructions for the GPU according to the parameters, as well as the blur radius, color value and other parameters during blur drawing.

[0089] After the application layer 301 passes the parameters into the screenshot function, the application framework layer 302 further calls the graphics system service SurfaceFlinger in the local framework layer 303 through the screenshot function, and transmits the parameters obtained based on the above screenshot function to the graphics system service SurfaceFlinger in the local framework layer 303.

[0090] After the graphics system service SurfaceFlinger in the local framework layer 303 obtains the parameters transmitted from the application framework layer, it will create a data buffer (hereinafter referred to as a hardware buffer) in the logically independent memory interval of the GPU based on the current screenshot requirements of the electronic device. It should be understood that at this time, the only user of this hardware buffer is the GPU in the hardware layer 304, that is, other objects except the GPU cannot access or use the hardware buffer.

[0091] In addition, as the manager of the layer, the graphics system service SurfaceFlinger will traverse some or all of the layers contained in the screen currently displayed by the electronic device after receiving the parameters transmitted by the application framework layer 302, and arrange the positions and draw the contents of each layer. It should be noted that the "position arrangement and drawing" mentioned here is actually just the graphics system service SurfaceFlinger generating the corresponding drawing instructions for this part of the action, and these drawing instructions will eventually be sent to the GPU in the hardware layer 304, and executed by the GPU to complete the actual layer synthesis work.

[0092] It should be noted that when the application layer calls the screenshot function, if the parameters passed in include parameters for determining the use of GPU for blurring, the graphics system service SurfaceFlinger will traverse some or all of the layers contained in the screen currently displayed by the electronic device and generate corresponding layer drawing instructions for the GPU. It will also generate corresponding blur drawing instructions for the GPU. The blur drawing instructions will be sent to the GPU together with the aforementioned layer drawing instructions, so that the GPU can blur the image content according to the above-mentioned blur drawing instructions after synthesizing all the layers according to the layer drawing instructions to obtain the entire image content displayed on the display screen.

[0093] Specifically, after the GPU in the hardware layer 304 receives the drawing instructions generated by the above-mentioned graphics system service SurfaceFlinger, the GPU will draw each layer according to the drawing instructions; after all the layer drawing instructions are executed, it can be considered that the GPU has completed the layer merging work, and at this time it can be considered that the GPU has drawn all the image content currently displayed on the display screen on a canvas. If the GPU also receives a blurred drawing instruction issued by the graphics system service SurfaceFlinger, the GPU will capture the image content that has been drawn on the canvas based on the blurred drawing instruction (that is, the image content drawn based on the layer drawing instruction), and convert the pixel value of each pixel in the image content based on the blur algorithm and the blur radius to obtain the blurred image content, and draw the blurred image content on the above-mentioned canvas. It should be understood that when drawing the blurred image content onto the canvas, the GPU can set the alpha value (transparency), color saturation, and add color value layer parameters according to the values ​​configured in the personalized style (i.e., the parameters passed by the screenshot function). When the alpha value (transparency) is set to 1 (different blur algorithms may have different alpha values ​​corresponding to opaque image content), the blurred image content will completely cover the originally displayed image content (i.e., the image content drawn based on the layer drawing instructions).

[0094] In addition, the cache data obtained by the GPU according to the layer drawing instructions and the fuzzy drawing instructions will be stored in the above-mentioned hardware buffer. In the process of the GPU writing the image cache data into the above-mentioned hardware buffer, the hardware buffer only supports the use of the GPU, and at this time SurfaceFlinger cannot transmit the data in the hardware buffer. After the GPU executes all the drawing instructions (including the layer drawing instructions and the fuzzy drawing instructions), the image data stored in the hardware buffer is the blurred screenshot image content. At this time, the hardware GPU can return a parameter to the graphics system service SurfaceFlinger, which is used to notify the local framework layer 302 that the above-mentioned hardware buffer has been used by the GPU. The hardware buffer no longer uses the GPU as the only user, but uses the graphics system service SurfaceFlinger as the user, so that the graphics system service SurfaceFlinger returns the hardware buffer to the application framework layer 302. Similarly, "returning the hardware buffer to the application framework layer 302" here refers to returning the address or handle of the hardware buffer as a parameter to the application framework layer 302, so that the application framework layer can read the cache data in the hardware buffer.

[0095] After the application framework layer 302 obtains the cached data in the hardware buffer, the application framework layer 302 can convert the cached data in the hardware buffer into a hardware bitmap based on a corresponding function (e.g., createBitmap function), and the hardware bitmap is also stored in a logically independent memory interval of the GPU. Similarly, the hardware bitmap obtained at this time is also a hardware bitmap with a blur effect. Therefore, the electronic device no longer needs to convert the hardware bitmap into a software bitmap for the CPU to perform blur processing.

[0096] Finally, the hardware bitmap with blur effect will be sent back to the application layer 301 and drawn on the screen of the electronic device. Similarly, in order to avoid the situation where the picture is garbled due to the difference between the hardware drawing frequency and the display screen refresh frequency, in the process of sending the hardware bitmap with blur effect to display, the hardware bitmap with blur effect can be drawn to an image buffer accessible by the CPU, GPU, and display screen, and the content in the image buffer will be consumed by the display screen and displayed on the display screen.

[0097] Optionally, if the rendering method adopted by the electronic device is software drawing, the electronic device can convert the above-mentioned hardware bitmap with blur effect into a software bitmap with blur effect again, and use the CPU to draw the software bitmap with blur effect on the display screen.

[0098] Combined with the above description, it can be seen that after the electronic device traverses the layers, it blurs the image in the process of synthesizing the layers, which can reduce the number of image data conversions. Figure 4 right Figure 3 The specific transformation process of image data during the entire screenshot blurring process is illustrated.

[0099] like Figure 4 As shown, image 401 is an interface diagram currently displayed on the display screen of the electronic device. It should be understood that image 401 is composed of multiple layers (i.e. Figure 4 All layers in the layer set 402 shown in FIG. 4 are synthesized in a certain order before being presented on the display screen as seen by the user.

[0100] When the user performs an operation on the display screen, such as pulling down the notification bar or long pressing the desktop folder, the electronic device responds to the operation and controls the CPU inside it (it should be noted that the CPU, as the computing and control core of the electronic device, still undertakes most of the computing and control work in the image processing method provided in the embodiment of the present application, but since the CPU is not involved in the specific conversion process of the subsequent image data, Figure 4 The CPU is not shown in the figure), and the GPU and other hardware perform a series of operations such as drawing, synthesizing, blurring, and sending to display on the image 401 to display the blurred image 401 on the display screen. The final image presented can be referred to as Figure 4 It should be understood that image 405 is only for the convenience of users to understand the specific meaning of "image blur", and it does not represent the actual style of the image obtained after the electronic device blurs the interface image in the actual scene. The specific style and content of image 405 are not limited in this application.

[0101] Specifically, in response to the user's operation, the GPU in the electronic device will synthesize the various layers in image 401. In addition, before synthesizing these layers, the electronic device will call the corresponding API interface and use the graphics system service to traverse all layers in image 401 to obtain layers 4021, 4022, and 4023 as shown in layer set 402. Among them, image 4021 is the background layer, layer 4022 is the status bar layer (floating layer), and layer 4023 is the content layer. It should be understood that layers 4021-4023 are only for ease of understanding. In the actual process of traversing the layers, image 401 can be divided into more layers, and this application does not limit this.

[0102] It should be noted that layers 4021-4023 are only sent to the GPU in the form of layer drawing instructions. These layer drawing instructions are generated according to the size and position of each element in each layer in the user interface (or display screen), as well as the display order between the layers. After the graphics service system generates these layer drawing instructions, these layer drawing instructions will be sent to the GPU so that the GPU draws and synthesizes the layers according to the drawing instructions to obtain image cache data. Combined with the above description, it can be seen that the image cache data obtained here is not an image that can be directly drawn by calling the drawing function. The GPU only writes these image cache data into the hardware buffer 403 based on the drawing instructions. This hardware buffer 403 can be a storage area initialized and created on the logically independent memory interval of the GPU when the electronic device calls the screenshot function.

[0103] In addition, when the electronic device calls the screenshot function, if the parameters passed in include parameters for determining the use of GPU for blurring, the graphics system service will traverse some or all of the layers contained in the screen currently displayed by the electronic device and generate corresponding layer drawing instructions for the GPU. It will also generate corresponding blur drawing instructions for the GPU. The blur drawing instructions will be sent to the GPU together with the aforementioned layer drawing instructions, so that after the GPU synthesizes all the layers according to the layer drawing instructions to obtain the entire image content displayed on the display screen, it can blur the image content according to the above-mentioned blur drawing instructions and write the blurred image content into the hardware buffer 403.

[0104] Figure 4Canvas 4031, canvas 4032 and canvas 4033 in the figure respectively show the process of GPU synthesizing layers based on layer drawing instructions, and blurring the image content after layer synthesis and drawing it on the canvas. It should be understood that canvas 4031-canvas 4033 are all the same canvas, and their corresponding different numbers are only for the convenience of readers to understand the changes in canvas content. Specifically, after the GPU receives the drawing instructions generated by the above-mentioned graphics system service, the GPU will draw each layer on the blank canvas 4031 according to the drawing instructions; after all the layer drawing instructions are executed, it can be considered that the GPU has completed the layer synthesis work. At this time, it can be considered that the GPU has drawn all the image content currently displayed on the display screen on canvas 4031. For details, please refer to the image content shown in canvas 4032. If the GPU also receives a blur drawing instruction issued by the graphics system service, the GPU will capture the image content that has been drawn on the canvas 4032 based on the blur drawing instruction, and convert the pixel value of each pixel in the image content shown on the canvas 4032 based on the blur algorithm and the blur radius to obtain the blurred image content, and draw the blurred image content on the canvas 4032. For details, please refer to the image content shown in the canvas 4033. It should be understood that when the blurred image content is drawn on the canvas, the GPU can set the alpha value (transparency), color saturation, and color value layer parameters according to the values ​​configured in the personalized style (i.e., the parameters passed by the screenshot function). When the alpha value (transparency) is set to 1 (the alpha value corresponding to the opaque image content of different blur algorithms may be different), the blurred image content will completely cover the image content originally displayed (i.e., the image content drawn based on the layer drawing instruction); that is, the image content originally in the canvas 4032 has been covered by the redrawn image content in the canvas 4033 and cannot be seen.

[0105] When the GPU writes data into the hardware buffer 403, the GPU is the only user of the hardware buffer 403. After all drawing instructions are executed, that is, all image cache data corresponding to the image 403 is written into the hardware buffer 403, the use right of the hardware buffer 403 will be opened to other objects. At this time, the electronic device will call the corresponding conversion function (such as the createBitmap function) to convert the image cache data stored in the hardware buffer 403 into the hardware bitmap 404. At this time, the hardware bitmap is also stored in the logically independent memory interval of the GPU, and the hardware bitmap 404 is already an image that the user can use for drawing, and the hardware bitmap 404 is a bitmap that already has a blurred effect. It should be understood that Figure 4The hardware bitmap 404 shown in FIG. 4 is only for the convenience of the reader's understanding and does not represent its appearance in actual application scenarios; for example, the blurriness of the software bitmap 404 may be greater than Figure 4 , or the blurred area of ​​hardware bitmap 404 may be greater than Figure 4 The specific blur effect can be determined based on the specific needs of the current application scenario and the parameters passed in when calling the screenshot function, and this application does not limit this.

[0106] After obtaining the hardware bitmap 404 with blur effect, the electronic device will respond to the drawing logic triggered by the relevant application that calls the screenshot function, and use the GPU (hardware drawing) to draw the hardware bitmap 404 with blur effect to the display screen. In fact, in the process of displaying the software bitmap with blur effect or the hardware bitmap with blur effect on the screen, in order to avoid the situation where the picture is garbled due to the difference between the hardware drawing frequency and the display refresh frequency, the above-mentioned hardware bitmap 404 with blur effect will first be drawn to an image buffer that can be accessed by the CPU, GPU, and display screen. The content in the image buffer will be consumed by the display screen and displayed on the display screen. The image displayed on the display screen can be referred to. Figure 4 The image 405 shown in .

[0107] Optionally, if the rendering method adopted by the electronic device is software drawing, the electronic device can convert the above-mentioned hardware bitmap with blur effect into a software bitmap with blur effect again, and use the CPU to draw the software bitmap with blur effect on the display screen.

[0108] Combined with the above description, it can be seen that in the embodiment of the present application, after the electronic device traverses the layers, it directly blurs the image in the process of synthesizing the layers, which can greatly reduce the number of conversions of the image data, thereby reducing the time required to copy pixels, and avoiding the occupation of the electronic device memory by the software bitmap, thereby improving the efficiency of the electronic device screenshot blur operation and saving the memory space of the electronic device. In addition, since the blurring of the image is completed by the GPU, and the GPU's image processing efficiency is much higher than that of the CPU, the efficiency of the electronic device screenshot blur operation is further improved.

[0109] Next, combine Figure 5-Figure 7 The specific process of obtaining a hardware bitmap with a blur effect by using a brush with a blur attribute to draw the hardware bitmap without a blur effect is described.

[0110] Figure 5 The entire screenshot blurring process of an electronic device is shown from the perspective of the Android system architecture. Figure 5As shown, the architecture involved in the screenshot blurring process of the electronic device may include the following four layers: application layer 501, application framework layer 502, local framework layer 503, and hardware layer 504. The specific functions of each layer of architecture in the Android system can refer to the aforementioned related instructions, which will not be repeated here. In the process of the electronic device blurring the currently displayed image and displaying the blurred image on the screen, the application layer 501, application framework layer 502, local framework layer 503, and hardware layer 504 can work together to complete the entire screenshot blurring process, specifically:

[0111] The application layer 501 can respond to user operations, such as the user's pull-down operation on the status bar or the user's long press operation on the folder, and call the screenshot function in the auxiliary management class SurfaceControl in the application framework layer. The application framework layer 502 will further call the graphics system service SurfaceFlinger in the local framework layer 503 through the screenshot function, and transmit the parameters obtained based on the above screenshot function to the graphics system service SurfaceFlinger in the local framework layer 503. After the graphics system service SurfaceFlinger in the local framework layer 503 obtains the parameters transmitted from the application framework layer, it will create a data buffer (hereinafter referred to as a hardware buffer) based on the current screenshot requirements of the electronic device. It will also traverse some or all layers contained in the screen currently displayed by the electronic device, and generate corresponding drawing instructions based on each layer. After the GPU in the hardware layer 504 receives the drawing instructions generated by the above graphics system service SurfaceFlinger, the GPU will draw each layer according to the drawing instructions, and the cached data obtained by drawing will be stored in the above hardware buffer. Afterwards, the application framework layer 502 can convert the cached data in the hardware buffer into a hardware bitmap based on the corresponding function (such as createBitmap function), and the hardware bitmap is also stored in a logically independent memory interval of the GPU. It should be noted that the hardware bitmap obtained here is a hardware bitmap that has not been blurred and has no blur effect.

[0112] The specific process of obtaining a hardware bitmap without blurring can refer to the above description. Figure 1 The relevant instructions will not be repeated here.

[0113] and Figure 1The process shown is different in that, in the embodiment of the present application, after the application framework layer 502 converts the cached data in the above-mentioned hardware buffer into a hardware bitmap, the application framework layer can return the above-mentioned hardware bitmap to the hardware layer 504 (the above-mentioned hardware bitmap is stored in a logically independent memory interval of the GPU, and the "returning the above-mentioned hardware bitmap to the hardware layer 504" here also refers to returning the physical address of the hardware bitmap to the GPU), and after setting the attributes for the brush, the GPU can use the brush with blur attributes to draw the above-mentioned hardware bitmap to obtain a new hardware bitmap with a blur effect. The hardware bitmap with a blur effect will be saved in a logically independent storage space of the GPU, and can be drawn by the GPU on the display screen in the subsequent process.

[0114] Figure 6 The specific operation flow of redrawing the above hardware bitmap using a brush with fuzzy attributes is shown.

[0115] like Figure 6 As shown, after obtaining the above hardware bitmap, the electronic device can create a RenderNode class based on the drawing program file RenderNode.cpp in the application framework layer. The RenderNode class can be used as a drawing node to construct a hardware accelerated drawing structure tree of the View.

[0116] After creating the RenderNode class, the electronic device can create and bind a hardware canvas based on the methods included in the RenderNode class. The hardware canvas can be used to record the drawing operation of the current RenderNode.

[0117] Afterwards, the electronic device can set the brush properties used by the RenderNode for drawing, such as setting the brush color, transparency, and whether to use anti-aliasing. Specifically, in order to achieve blur processing, the electronic device can set a blur style and radius through the corresponding parameters of the BlurMaskFilter property of the brush. When drawing, the GPU can achieve the effect of a blur filter for the drawn image based on the BlurMaskFilter property of the brush. The drawing operation here is equivalent to the operation of blurring the hardware bitmap that originally does not have a blur effect. The new hardware bitmap drawn is a hardware bitmap with a blur effect.

[0118] Finally, the hardware bitmap (address) with blur effect will be sent back to the application layer 501 and drawn on the screen of the electronic device.

[0119] Specifically, after the application obtains the above-mentioned hardware bitmap without blur effect, when the GPU uses the above-mentioned brush with blur attribute to draw, the GPU will first create a buffer for the drawing operation to write the blurred image data. The buffer is actually a display data buffer that can write data, and the buffer is also located in a logically independent memory interval of the GPU. Afterwards, in order to avoid the situation where the picture is garbled due to the difference between the hardware drawing frequency and the display refresh frequency, the hardware bitmap with blur effect stored in the buffer (hereinafter referred to as the B1 buffer) will first be drawn to another image buffer (hereinafter referred to as the B2 buffer); the content in the B2 buffer will be consumed by the display and displayed on the display. The B2 image buffer is the same as the B1 buffer used by the GPU to write blurred image data. It is also a display data buffer that can write data, and it is also located in a logically independent memory interval of the GPU. However, after the B2 image buffer is mapped, the CPU, GPU, and display all have access to it.

[0120] Combined with the above description, it can be seen that after the electronic device obtains the hardware bitmap without blur effect, it uses the brush with blur attribute to draw it through GPU during the process of sending the hardware bitmap to display, and obtains the hardware bitmap with blur effect, which can also avoid the conversion process between hardware bitmap and software bitmap. Figure 7 right Figure 5 The specific transformation process of image data during the entire screenshot blurring process is illustrated.

[0121] like Figure 7 As shown, image 701 is an interface diagram currently displayed on the display screen of the electronic device. It should be understood that image 701 is composed of multiple layers (i.e. Figure 7 All layers in the layer set 702 shown in FIG. 7 are synthesized in a certain order before being presented on the display screen as seen by the user.

[0122] When the user performs an operation on the display screen, such as pulling down the notification bar or long pressing the desktop folder, the electronic device responds to the operation and controls its internal CPU, GPU and other hardware to perform a series of operations such as drawing, compositing, blurring, and sending to display on the image 701, so as to display the blurred image 701 on the display screen. The final image presented can be referred to as Figure 7It should be understood that image 706 is only for the convenience of users to understand the specific meaning of "image blur", and it does not represent the actual style of the image obtained after the electronic device blurs the interface image in the actual scene; for example, in the actual application scene, other layers can also be displayed on the blurred image displayed in image 706, such as the notification bar layer displayed by the electronic device in response to the user's pull-down notification bar operation, and the folder function list layer displayed by the electronic device in response to the user's long press of the desktop folder operation. The specific style and content of image 706 are not limited in this application.

[0123] Specifically, in response to the user's operation, the GPU in the electronic device will synthesize the various layers in image 701. In addition, before synthesizing these layers, the electronic device will call the corresponding API interface and use the graphics system service to traverse all layers in image 701 to obtain layer 7021, layer 7022, and layer 7023 in layer set 702. Among them, layer 701 is the background layer, layer 7022 is the status bar layer (floating layer), and layer 7023 is the content layer. It should be understood that layers 7021-7023 are only for ease of understanding. In the actual process of traversing the layers, image 701 can be divided into more layers, and this application does not limit this.

[0124] It should be noted that layers 7021-7023 are only sent to the GPU in the form of drawing instructions. These drawing instructions are generated according to the size and position of each element in each layer in the user interface (or display screen), as well as the display order between the layers. After the graphics service system generates these drawing instructions, these drawing instructions will be sent to the GPU so that the GPU draws and synthesizes the layers according to the drawing instructions to obtain image cache data. Combined with the above description, it can be seen that the image cache data obtained here is not an image that can be directly drawn by calling the drawing function. The GPU only writes these image cache data into the hardware buffer 703 based on the drawing instructions. This hardware buffer 703 can be a storage area initialized and created on the logically independent memory interval of the GPU when the electronic device calls the screenshot function.

[0125] In the process of GPU writing data into hardware buffer 703, GPU is the only user of hardware buffer 703. After all drawing instructions are executed, that is, all image cache data corresponding to image 703 is written into hardware buffer 703, the use right of hardware buffer 703 will be opened to other objects, and then the electronic device will call the corresponding conversion function (such as createBitmap function) to convert the image cache data stored in hardware buffer 703 into hardware bitmap 704. At this time, the hardware bitmap is also stored in the logically independent memory interval of GPU, and the hardware bitmap 704 is already an image that can be used for drawing. However, hardware bitmap 704 has not been processed, and its image content does not have a blur effect.

[0126] Afterwards, in order to perform blur processing on the hardware bitmap 704, the GPU can redraw the hardware bitmap 704 using a brush with blur attributes to obtain a hardware bitmap 705 with a blur effect. The GPU can create a buffer for drawing images in its logically independent memory interval, and the buffer is used to store the data obtained by the GPU from drawing the hardware bitmap 704; after the GPU completes redrawing the hardware bitmap 704 using the brush data with blur attributes, the data stored in the buffer is the hardware bitmap 705.

[0127] Specifically, after obtaining the above hardware bitmap 704, the electronic device can create a RenderNode class, and based on the methods contained in the RenderNode class, create and bind a hardware canvas for recording the drawing operation of the current RenderNode. Afterwards, the electronic device can set the brush properties for drawing of the RenderNode, such as setting the color and transparency of the drawing, setting whether to use anti-aliasing, etc.; in order to achieve blur processing, the electronic device sets a blur style and radius of the BlurMaskFilter property of the brush through corresponding parameters, and the GPU can achieve the effect of a blur filter for the drawn image based on the BlurMaskFilter property of the brush when drawing, and the new hardware bitmap obtained by drawing is the hardware bitmap 705 with a blur effect.

[0128] Finally, the hardware bitmap 705 (address) with blur effect will be drawn on the form screen of the electronic device. Figure 7Image 706 in. Specifically, in order to avoid garbled images caused by the difference between the hardware drawing frequency and the display screen refresh frequency, the hardware bitmap 705 will first be drawn to another image buffer; the content in the image buffer will be consumed by the display screen and displayed on the display screen. This image buffer is the same as the buffer used for the GPU to write blurred image data. It is also a display data buffer to which data can be written, and it is also located in a logically independent memory interval of the GPU. However, after the image buffer is mapped, the CPU, GPU, and display screen all have access to it.

[0129] It should be particularly noted that in the image processing method provided by the present application, the hardware bitmap obtained by the electronic device through GPU blurring (such as the hardware bitmap 404 or the hardware bitmap 705 in the aforementioned description) can be stored in a logically independent storage space of the GPU for a long time, and the software bitmap with a blurring effect obtained by converting these hardware bitmaps can also be cached in the application process for a long time (the time can be 10 minutes, one hour or permanently, and the present application does not limit this), so these hardware bitmaps or software bitmaps can be exported and saved in other storage devices, and can also be used in subsequent drawing logic.

[0130] Figure 8 The specific process is shown in which an electronic device saves a hardware bitmap with a blur effect, and when the user subsequently triggers the electronic device to perform a background blur process again through other operations, the electronic device directly uses the previously saved hardware bitmap with a blur effect for display.

[0131] like Figure 8 As shown, the user interface 80 may be a main screen interface of an electronic device, and the user interface 80 may display an icon 801 of an application browser and an application It can be understood that the user interface 80 may also display other elements (layers) such as a status bar and a navigation bar, and this application does not limit this.

[0132] In response to the user's upward sliding operation at the lower boundary of the user interface 80, the electronic device can control the GPU inside it in response to the operation and execute the following operations: Figure 3 or Figure 5 In the screenshot blurring process shown, the background of the user interface 80 (i.e., the current wallpaper of the electronic device) is blurred (it is assumed here that this screenshot blurring is the first time that the electronic device blurs the background image, and the browser icon 801 and the application are not traversed during the layer traversal). The layer where the icon 802 is located), get Figure 8In the image 800. In combination with the above description, it can be known that the image 800 is a hardware bitmap wallpaper image with a blur effect, which is stored in a logically independent storage area of ​​the GPU by the electronic device.

[0133] After obtaining the image 800, the electronic device can use the GPU to draw the hardware bitmap into the display screen of the electronic device and display it as shown in FIG. Figure 8 User interface 81 shown in .

[0134] The user interface 81 is an application navigation interface. In order to present a better visual effect to the user, the image 800 with a blurred effect is displayed as a background image in the user interface 81. In addition, the user interface can display the application interface thumbnails of all applications currently opened by the electronic device. Among them, the application interface thumbnails 811 are application The user can click on the application interface thumbnail 811 to enable the electronic device to open the application Switch to the foreground application, in response to the user operation, the electronic device can display Figure 8 The user interface 82 shown in FIG. It should be understood that at this time, the image 800 is still stored in a logically independent storage area of ​​the GPU.

[0135] If the user swipes up again at the lower boundary of the user interface 82, the electronic device needs to display the application navigation interface again in response to the operation, and the background of the application navigation interface still needs to be set to the wallpaper image with a blurred effect. However, since the image 800 is still stored in the independent storage area of ​​the GPU logic, the electronic device does not need to trigger the activation of the image again. Figure 3 or Figure 5 Instead of obtaining a hardware bitmap with a blur effect by using the screenshot blur process shown, the hardware bitmap 800 stored in the GPU logic independent storage area can be directly called and drawn on the display screen of the electronic device. Finally, the interface displayed by the electronic device can refer to Figure 8 The user interface 83 shown in .

[0136] Similarly, during subsequent use, the hardware bitmap obtained by the electronic device when blurring screenshots of other interfaces can still be stored in the GPU's logically independent storage space for a long time or permanently. If subsequent electronic devices have the same interface blurring requirements, the electronic devices can directly use the hardware bitmap stored in the GPU's logically independent storage space, further improving the efficiency of blurring and displaying images.

[0137] Next, the electronic device provided by the embodiments of the present application is introduced.

[0138] The electronic device may be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) or a dedicated camera (such as a SLR camera, a card camera), etc. The present application does not impose any restrictions on the specific type of the electronic device.

[0139] Fig. 9 The structure of the electronic device is exemplarily shown.

[0140] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0141] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0142] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0143] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0144] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0145] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0146] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0147] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0148] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0149] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0150] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0151] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0152] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0153] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0154] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0155] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0156] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0157] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0158] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0159] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0160] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0161] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0162] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0163] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0164] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0165] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0166] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0167] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0168] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0169] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0170] The external memory interface 120 can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0171] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0172] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0173] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0174] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0175] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0176] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0177] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0178] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0179] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0180] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0181] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0182] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0183] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0184] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0185] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0186] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0187] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0188] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0189] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0190] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0191] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0192] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0193] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support NanoSIM cards, MicroSIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0194] In the embodiment of the present application, the electronic device 100 can respond to the user's operation, such as the user's pull-down operation on the status bar or the user's long press operation on the folder, to start the process of blurring the screenshot of the content of the screen currently displayed on the display screen 194. At this time, the electronic device can call the screenshot function stored in its processor 110 or internal memory 121, and traverse the layers of the screen currently displayed on the display screen 194 in combination with the image system service of the electronic device. Specifically, when calling the screenshot function, if there are relevant parameters in the input parameters of the screenshot function indicating that the GPU is used to blur the image obtained by layer synthesis, then after the GPU completes the synthesis of the layers, the GPU will further obtain the image content obtained by the layer synthesis (the image content is drawn on the canvas), obtain and convert the pixel value of each pixel in the image content to obtain the blurred image content, and draw the blurred image content accordingly on the above-mentioned canvas, and the drawing position is the same as the position of the original image content. The image data corresponding to the blurred image content can be saved in a logically independent storage space in the GPU. In the subsequent process, the electronic device can convert these image data into hardware bitmaps with a blurred effect, and send the hardware bitmaps with a blurred effect to the display screen for display.

[0195] In some embodiments, the electronic device 100 can respond to the user's operation, such as the user's pull-down operation on the status bar or the user's long press operation on the folder, to start the process of blurring the screenshot of the content of the screen currently displayed on the display screen 194. At this time, the electronic device can call the screenshot function stored in its processor 110 or internal memory 121, and traverse the layers of the screen currently displayed on the display screen 194 in combination with the image system service of the electronic device to obtain a hardware bitmap without a blur effect. Afterwards, the GPU can redraw the hardware bitmap without a blur effect by using a brush with a blur attribute to obtain a hardware bitmap with a blur effect, and send the hardware bitmap with a blur effect to the display screen for display.

[0196] Fig.10 An image processing method is provided in an embodiment of the present application. By implementing this method, the electronic device can blur the image during the process of synthesizing the layers after traversing the layers, which can effectively reduce the generation of intermediate products in the screenshot blurring process, and improve the efficiency of the screenshot blurring process of the electronic device while saving the memory of the electronic device. Fig.10 As shown, the method provided in the embodiment of the present application may include:

[0197] S101: The electronic device synthesizes some or all layers in a first image to obtain first image data.

[0198] The electronic device may be a mobile phone, a vehicle-mounted device (such as an on-board unit (OnBoardUnit, OBU)), a tablet computer (pad), a computer with a display function (such as a laptop computer, a PDA, etc.), etc. Specifically, the electronic device may be the electronic device 100 provided in an embodiment of the present application. It is understood that the present application does not limit the specific form of the electronic device.

[0199] The first image may be an image of the user interface currently displayed by the electronic device, or an image of the main interface of the electronic device. Specifically, some layers in the first image may be layers corresponding to the wallpaper image in the main interface of the electronic device. Whether the electronic device traverses all layers or some layers of the first image may be determined in combination with the specific needs of the current application scenario, and the embodiments of the present application do not limit this.

[0200] Optionally, before the electronic device synthesizes some or all layers in the first image to obtain the first image data, the electronic device can call a corresponding screenshot function to obtain the above-mentioned first image. Specifically, in response to a user operation, such as a user pulling down the notification bar or long pressing a folder and other operations with a fuzzy interface requirement, the electronic device can call a corresponding screenshot function, and use the screenshot function to further call the graphics system service in the electronic device to traverse all or part of the layers in the first image. In fact, the data generated by the traversal of the layers by the graphics system service are only some drawing instructions. These drawing instructions are generated according to the size and position of each element in each layer in the user interface (or display screen), and the display order between each layer. These drawing instructions will be sent to the GPU, and the GPU will perform the corresponding synthesis work. When the GPU synthesizes the layers, it actually writes the image data obtained by the layer synthesis into a buffer (hardware buffer), which exists in the logically independent storage space of the GPU.

[0201] S102: The electronic device uses a graphics processor (GPU) to perform blur processing based on the first image data to obtain second image data.

[0202] The second image data already corresponds to an image with a blur effect, but the image data is also stored in the hardware buffer, so at this time the second image data is not an image that can be drawn by a drawing method.

[0203] In this method, the process of synthesizing and blurring the image is completed by the GPU, and different applications may have different performance requirements for the screenshot blurring process. Therefore, in an optional embodiment of the present invention, the native screenshot function in the electronic device can be encapsulated to allow the application to set whether GPU blurring is required. After the electronic device responds to an operation with a screenshot blurring requirement (for example, the user pulls down the status bar or long presses a folder), when the electronic device calls the corresponding function that implements the screenshot function, the electronic device can obtain the user's parameter (i.e., the first parameter) for determining whether the GPU needs to be used for blurring, and send the above first parameter to the GPU, so that the GPU determines whether it is necessary to blur the image obtained by layer synthesis in the layer synthesis stage. When the first parameter indicates that the GPU is used to blur the image, the electronic device will use the GPU to blur the first image data to obtain the above second image data.

[0204] In addition, the input parameter of the above-mentioned screenshot function may also include a second parameter, which may include multiple sub-parameters. These sub-parameters may be sent to the GPU along with the calling process of the screenshot function. When the GPU performs blur processing, the second parameter may be used to personalize the blur effect achieved by the GPU blur processing. For example, the second parameter may include a sub-parameter for determining the saturation of the image obtained by the blur processing, a sub-parameter for the user to adjust the transparency of the blurred area, and a sub-parameter for adding a coloring layer to the blur effect (i.e., making the blurred image appear as a color similar to a filter effect), so as to further optimize the visual effect provided to the user. Specifically, the second parameter corresponding to different applications may be different.

[0205] Combined with the above description, it can be known that before the GPU synthesizes the layers, the screenshot function will call the graphics system service provided by the electronic device to generate corresponding layer drawing instructions and blur drawing instructions, and send these instructions to the GPU. In one possible implementation, when the GPU synthesizes the layers, each layer and the corresponding elements in each layer can be drawn on a canvas according to the size and position recorded in the drawing instructions. At this time, the image content recorded on the canvas does not have a blur effect, and the image corresponds to the first image data. When blurring the clear image, the GPU will then first obtain the existing image content on the canvas according to the blur instruction, obtain the pixel value of each pixel in the existing image content, and adjust the pixel value of each pixel to obtain the image content with a blur effect (i.e. corresponding to the third image data), and draw the image content with a blur effect on the above canvas, and draw it in the form of an overlay to the position where the original existing image content is located. At this time, all the image content recorded on the canvas corresponds to the second image data.

[0206] S103: The electronic device converts the second image data into a second image.

[0207] In this method, the second image is already an image with a blur effect; the second image is a hardware bitmap stored in a logically independent storage space of the GPU. In the subsequent process, the hardware bitmap can be drawn to the display screen by a drawing method. Specifically, the electronic device can convert the second image data into the second image by a corresponding data encapsulation method.

[0208] It should be understood that the second image with blur effect will be drawn to the screen of the electronic device later. However, for the electronic device, when sending the image for display, there are two ways to draw the image: one is that the electronic device uses the CPU to draw, which is also called "software drawing"; the other is that the electronic device uses the GPU to draw, also called "hardware drawing". For the hardware bitmap, when drawing, it can be directly drawn by hardware drawing; but because the hardware bitmap also exists in the logically independent storage space of the GPU, and the CPU cannot directly access the logically independent storage space of the GPU, it is impossible to directly use the software drawing method to draw the hardware bitmap to the display screen. Therefore, in a possible implementation, after obtaining the second image, the electronic device can directly use the hardware drawing method, that is, use the GPU to draw the second image to the display screen. Specifically, in order to avoid the situation where the picture is garbled due to the different hardware drawing frequency and the display screen refresh frequency, in the process of sending the second image with blur effect, the second image can be first drawn to an image buffer that can be accessed by the CPU, GPU, and display screen, and the content in the image buffer will be consumed by the display screen and displayed on the display screen. Alternatively, if the electronic device needs to use software drawing, the electronic device can first convert the second image into a software bitmap (hereinafter referred to as the third image) accessible to the CPU, that is, copy the pixel data corresponding to the second image to the application memory and then use the CPU to draw the third image to the display screen.

[0209] It is understandable that in this method, the hardware bitmap obtained by the electronic device through GPU blurring, that is, the above-mentioned second image, can be stored in the GPU logically independent storage space for a long time. Specifically, the time for the above-mentioned second image to be stored in the GPU logically independent storage space can be 10 minutes, one hour, one day or permanently, and this application does not limit this. Therefore, these hardware bitmaps can be exported and saved in other storage devices, or cached in the application process and used in subsequent drawing logic.

[0210] For example, assuming that the second image is a hardware bitmap obtained by blurring the wallpaper image in the main interface of the electronic device, and the electronic device obtains the second image through this method after the user long presses the folder on the display screen of the electronic device, the electronic device will draw the second image on the display screen. Afterwards, the hardware bitmap can still be saved in the logically independent storage space of the GPU. Afterwards, if the user long presses the folder on the display screen of the electronic device again, in response to the operation, the electronic device needs to obtain the blurred image corresponding to the wallpaper image again. However, since the second image is still saved in the logically independent storage area of ​​the GPU, the electronic device does not need to trigger the screenshot blur process provided by this method again to obtain the hardware bitmap with a blurred effect, but can directly call the second image saved in the logically independent storage area of ​​the GPU and draw it on the display screen of the electronic device, further improving the efficiency of blurring and displaying the image.

[0211] Similarly, the above-mentioned third image can also be saved in the application process for a long time, so these software bitmaps can be exported and saved in other storage devices, or cached in the application process and used in subsequent drawing logic to further improve the efficiency of blurring and displaying images.

[0212] This method obtains a hardware bitmap with a blur effect (i.e., the second image) by directly blurring the image in the synthesis stage of the image data. In this way, the electronic device does not need to use the CPU to blur the hardware bitmap with a blur effect again, and there is no need to convert the hardware bitmap into a software bitmap to facilitate the CPU to access the hardware bitmap. By implementing this method, during the blurring of the screenshot, the electronic device can avoid the conversion process between the hardware bitmap and the software bitmap, which can save the memory of the electronic device and improve the efficiency of the blurring process of the electronic device screenshot.

[0213] Fig.11 An image processing method provided in an embodiment of the present application, by implementing the method, an electronic device can obtain a hardware bitmap without blur effect, and then use a brush with blur attribute to draw it to obtain a hardware bitmap with blur effect, thereby saving the memory of the electronic device and improving the efficiency of the blur process of the electronic device screenshot. Fig.11 As shown, the method provided in the embodiment of the present application may include:

[0214] S201: The electronic device synthesizes part or all of the layers in the fourth image to obtain fourth image data.

[0215] The electronic device may be a mobile phone, a vehicle-mounted device (such as an on-board unit (OnBoardUnit, OBU)), a tablet computer (pad), a computer with a display function (such as a laptop computer, a PDA, etc.), etc. Specifically, the electronic device may be the electronic device 100 provided in an embodiment of the present application. It is understood that the present application does not limit the specific form of the electronic device.

[0216] The fourth image may be an image of the user interface currently displayed by the electronic device, or an image of the main interface of the electronic device. Specifically, some layers in the fourth image may be layers corresponding to the wallpaper image in the main interface of the electronic device. Whether the electronic device traverses all layers or some layers of the first image may be determined in combination with the specific requirements of the current application scenario, and the embodiments of the present application do not limit this.

[0217] Optionally, before the electronic device synthesizes some or all layers in the fourth image to obtain the fourth image data, the electronic device can call a corresponding screenshot function to obtain the above-mentioned fourth image. Specifically, in response to a certain user operation, such as the user pulling down the notification bar or long pressing the folder and other operations with interface fuzziness requirements, the electronic device can call the corresponding screenshot function, and use the screenshot function to further call the graphics system service in the electronic device to traverse all or part of the layers in the fourth image. In fact, the data generated by the traversal of the layers by the graphics system service are just some drawing instructions. These drawing instructions are generated according to the size and position of each element in each layer in the user interface (or display screen), and the display order between each layer. These drawing instructions will be sent to the GPU, and the GPU will perform the corresponding synthesis work. When the GPU synthesizes the layers, it actually writes the image data obtained by the layer synthesis into a buffer (hardware buffer), which exists in the logically independent storage space of the GPU.

[0218] Before the GPU synthesizes the layers, the screenshot function will call the graphics system service provided by the electronic device to generate corresponding layer drawing instructions, and send these instructions to the GPU. When the GPU synthesizes the layers, each layer and the corresponding elements in each layer can be drawn on a canvas according to the size and position recorded in the drawing instructions. At this time, the image content recorded on the canvas does not have a blur effect, and the image corresponds to the fourth image data.

[0219] S202: The electronic device converts the fourth image data into a fifth image.

[0220] In this method, the fifth image is an image without blur effect; the fifth image is a hardware bitmap stored in a logically independent storage space of the GPU. Specifically, the electronic device can convert the fourth image data into the fifth image through a corresponding data encapsulation method.

[0221] S203: The electronic device controls the GPU to draw the fifth image using a brush with a fuzzy attribute to obtain a sixth image.

[0222] After obtaining the above hardware bitmap, the electronic device can create a RenderNode class by drawing the program file RenderNode.cpp. The RenderNode class can be used as a drawing node to build a hardware accelerated drawing structure tree of the View. After creating the RenderNode class, the electronic device can create and bind a hardware canvas based on the methods contained in the RenderNode class. The hardware canvas can be used to record the drawing operation of the current RenderNode.

[0223] Afterwards, the electronic device can set the brush properties used by the RenderNode for drawing, such as setting the brush color, transparency, setting whether to use anti-aliasing, etc. Specifically, in order to achieve blur processing, the electronic device can set a blur style and radius of the BlurMaskFilter property of the brush through corresponding parameters, and the GPU can achieve the effect of a blur filter for the drawn image based on the BlurMaskFilter property of the brush when drawing. The drawing operation here is equivalent to the operation of blurring the hardware bitmap that originally does not have a blur effect. The new hardware bitmap drawn is the hardware bitmap with a blur effect, and the sixth image mentioned above.

[0224] Specifically, after the application obtains the above-mentioned fifth image without the blur effect, when the GPU uses the above-mentioned brush with blur attributes for drawing, the GPU will first create a buffer for the drawing operation to write the blurred image data and the above-mentioned sixth image. The buffer is actually a display data buffer to which data can be written, and the buffer is also located in a logically independent memory interval of the GPU.

[0225] S204: The electronic device draws the sixth image on a display screen.

[0226] Afterwards, in order to avoid the situation where the picture is garbled due to the difference between the hardware drawing frequency and the display screen refresh frequency, the sixth image with a blurred effect stored in the buffer (hereinafter referred to as the B1 buffer) will first be drawn to another image buffer (hereinafter referred to as the B2 buffer); the content in the B2 buffer will be consumed by the display screen and displayed on the display screen. The B2 image buffer is the same as the storage area used for the GPU to write the fifth image. It is also a display data buffer that can write data, and it is also located in a logically independent memory interval of the GPU. However, after the B2 image buffer is mapped, the CPU, GPU, and display screen all have access to it.

[0227] An embodiment of the present application also provides an electronic device, which includes: one or more processors and a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method shown in the aforementioned embodiment.

[0228] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0229] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0230] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. An image processing method, characterized in that: The method comprises: Synthesizing some or all of the layers in the first image to obtain first image data; Using a graphics processor (GPU) to perform blur processing based on the first image data to obtain second image data; The second image data is converted into a second image, where the second image is a hardware bitmap stored in the GPU.

2. The method according to claim 1, characterized in that The using a graphics processor (GPU) to perform blur processing on the first image data to obtain the second image data comprises: In the case where the first parameter indicates that the GPU is used to blur the image, the GPU is used to blur the first image data to obtain the second image data, and the first parameter is an input parameter of a screenshot function, and the screenshot function is used to obtain the first image.

3. The method according to claim 1 or 2, characterized in that: After converting the second image data into a second image, the method further includes: drawing the second image onto a display screen using the GPU; Or, copy the pixel data corresponding to the second image to the application memory, and after obtaining the third image, use the central processing unit CPU to draw the third image onto the display screen, and the third image is a software bitmap stored in the application memory.

4. The method according to claim 3, characterized in that After drawing the second image onto a display screen using the GPU, the method further includes: In response to a user operation, the second image is drawn into a display screen, where the user operation is an operation that triggers a background blur requirement.

5. The method according to claim 3, characterized in that: After drawing the third image into the display screen using a central processing unit (CPU), the method further includes: In response to a user operation, the third image is drawn on the display screen, where the user operation is an operation that triggers a background blur requirement.

6. The method according to any one of claims 1 to 5, characterized in that: The using the GPU to perform blur processing on the first image data to obtain second image data comprises: Obtaining a pixel value of each pixel in the first image data, and adjusting the pixel value of each pixel to obtain third image data with a blur effect; The image content corresponding to the third image data with the blur effect is drawn on the image content corresponding to the first image data to obtain the second image data.

7. The method according to any one of claims 1 to 6, characterized in that: Before the blurring of the first image data by using a graphics processor (GPU) to obtain the second image data, the method further includes: Based on a second parameter, one or more of the image saturation and the transparency of the blurred area used when blurring the first image data is determined, where the second parameter is an input parameter of a screenshot function, and the screenshot function is used to obtain the first image.

8. An image processing method, characterized in that: The method comprises: synthesizing part or all of the layers in the fourth image to obtain fourth image data; Convert the fourth image data into a fifth image, wherein the fifth image is a hardware bitmap stored in a graphics processor (GPU); Controlling the GPU to draw the fifth image using a brush with a fuzzy attribute to obtain a sixth image; The sixth image is drawn on a display screen.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors, a memory and a display screen; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 8.

10. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1-8.

11. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 8.

Citation Information

Patent Citations

  • Image data ground glass effect processing method and device

    CN105139356A

  • Method for customizing window popping-up control under Mac

    CN107450925A

  • Control fuzzy method and device, terminal equipment and readable storage medium

    CN114529485A

  • Real-time background fuzzy processing method and device and computer readable storage medium

    CN115469955A

  • Dynamic image blurring method, terminal equipment and computer readable storage medium

    CN116051387A