Image blurring method and device
Direct synchronous image blur processing on the graphics processor (GPU) solves the lag and delay problems caused by image blur processing in the prior art, and improves user experience and performance.
Patent Information
- Application Number
- PCT/CN2024/091458
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-18
- Filing Date
- 2024-05-07
- Publication Date
- 2024-11-21
AI Technical Summary
In the prior art, when image blurring is processed, the cutting-field motion effect can easily cause lag, affecting the user's visual experience, and the delay is long due to insufficient CPU performance, resulting in a sense of hysteresis.
Image blur processing is achieved through direct synchronization on the graphics processor (GPU), avoiding the CPU copying memory, saving memory consumption and processing time, and optimizing performance.
Significantly save memory and processing time, avoid lag, improve user's visual experience, and achieve fast response and smooth image blur processing.
Smart Images

Figure CN2024091458_21112024_PF_FP_ABST
Abstract
Description
Image blurring method and device Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image blurring method and device. Background Art
[0002] Image blurring is often referred to as smoothing filtering. It is one of the more common operations in the field of image processing and is used to reduce the level of detail in an image and remove trivial details. In terminal devices such as mobile phones and tablets, an image is usually displayed as a background image, and various application (Application, APP for short) icons or a collection of application icons are displayed on the background image. At this time, the background image is clear; and when the user performs certain operations, such as clicking to open a desktop folder, clicking to open an application folder, or pulling down to display a notification message, in order to highlight the message or application icon in the foreground, the background image needs to be blurred. In the prior art, image blurring algorithms usually rely on CPU (Central Processing Unit) implementation. Under the condition of strict performance requirements and low latency requirements, it is easy to cause the transition effect to freeze when responding to the above-mentioned user operations, affecting the user's visual experience, and making the user feel a sense of lag, which in turn affects the user's subsequent operations.
[0003] Summary of the Invention
[0004] The embodiments of the present application provide an image blurring method and device to solve the problem of stuttering and lag in transition effects when image blurring is required in the prior art.
[0005] In a first aspect, an embodiment of the present application provides an image blurring method, comprising:
[0006] In response to triggering a first operation to blur a first image, marking the first image, wherein the mark is used to indicate that the first image is to be blurred; transmitting the first image to a graphics processor according to the mark; setting a blurred drawing by the graphics processor; drawing a blurred first image by the graphics processor based on the blurred drawing and the first image; and displaying the blurred first image.
[0007] The embodiment of the present application is different from the prior art in which a memory of the same size as the first image is copied to carry the blur result, and the CPU performs blur processing on the first image and then sends it to the graphics processing unit (GPU) for drawing. Instead, the embodiment of the present application only requires marking the first image and then the GPU directly performs blur processing on the first image synchronously during the drawing process, which greatly saves memory consumption, saves processing time, and optimizes performance.
[0008] In a possible implementation of the first aspect, when marking the first image, the method further includes: selecting a first image blurring algorithm. In other words, different image blurring algorithms can also be selected in the embodiment of the present application.
[0009] In a possible implementation of the first aspect, when selecting the first image blurring algorithm, the method further includes setting parameters of the first image blurring algorithm. In other words, the embodiment of the present application can also set parameters of the selected first image blurring algorithm, which is highly flexible.
[0010] In a possible implementation manner of the first aspect, after selecting the first image blur algorithm, the method further includes: transmitting the name of the first image blur algorithm to the graphics processor.
[0011] In a possible implementation manner of the first aspect, after transmitting the name of the first image blur algorithm to the graphics processor, the method further includes: the graphics processor receiving the name of the first image blur algorithm.
[0012] That is, the graphics processor may receive the transmitted name of the first image blurring algorithm, so as to call the first image blurring algorithm according to the name of the first image blurring algorithm in subsequent steps.
[0013] In a possible implementation manner of the first aspect, after setting the parameters of the first image blurring algorithm, the method further includes: transmitting the set parameters of the first image blurring algorithm to the graphics processor.
[0014] In a possible implementation manner of the first aspect, after transmitting the set parameters of the first image blurring algorithm to the graphics processor, the method further includes: the graphics processor receiving the set parameters of the first image blurring algorithm.
[0015] That is, the graphics processor may receive the transmitted set parameters of the first image blurring algorithm so as to use the set parameters of the first image blurring algorithm in subsequent steps.
[0016] In a possible implementation manner of the first aspect, the graphics processor setting the blurred drawing includes at least one of creating a drawing brush, determining an image blur algorithm, or determining parameters of the image blur algorithm.
[0017] In a possible implementation manner of the first aspect, determining the image blur algorithm includes: the graphics processor adopting a default image blur algorithm.
[0018] In a possible implementation manner of the first aspect, determining parameters of the image blur algorithm includes: the graphics processor adopting default parameters of the image blur algorithm.
[0019] In a possible implementation manner of the first aspect, determining the image blur algorithm includes: the graphics processor adopting the first image blur algorithm.
[0020] In a possible implementation manner of the first aspect, determining parameters of the image blur algorithm includes: the graphics processor adopting the set parameters of the first image blur algorithm.
[0021] That is, the fuzzy drawing can be flexibly set, and the default image blur algorithm can be used; the parameters of the default image blur algorithm can be used; the selected first image blur algorithm can be used; and the set parameters of the first image blur algorithm can be used.
[0022] In the second aspect, an embodiment of the present application provides a terminal device, comprising: a memory for storing instructions executed by one or more graphics processors of the terminal device; a display for displaying images; and a graphics processor, which, when the graphics processor executes the instructions in the memory, enables the terminal device to implement the steps of the image blurring method proposed in the first aspect of the embodiment of the present application.
[0023] In a third aspect, an embodiment of the present application provides a chip, wherein the chip system includes a processing circuit and a storage medium, wherein the storage medium stores computer program code; when the computer program code is executed by the processing circuit, the steps of the image blurring method proposed in the first aspect of the embodiment of the present application are implemented.
[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program. When the computer program is run on a computer, the computer executes the steps of the image blurring method proposed in the first aspect of the embodiment of the present application.
[0025] Compared with the prior art, the embodiments of the present application have the following advantages: significantly saving memory and processing time, optimizing performance, avoiding lag, and improving the user's visual experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] FIG1( a ) is a schematic diagram showing a blurred background image when a user opens a folder 1 in the prior art;
[0027] FIG1( b ) is a schematic diagram showing a blurred background image when a user pulls down to display a notification message in the prior art;
[0028] FIG2 is a flowchart of an image blurring method according to an embodiment of the present application;
[0029] FIG3 is a schematic diagram of the structure of a mobile phone provided in an embodiment of the present application;
[0030] FIG4 is a software structure block diagram of a terminal device provided in an embodiment of the present application;
[0031] FIG5 is a second flowchart of an image blurring method provided in an embodiment of the present application;
[0032] FIG6 is a flowchart of steps for setting blur drawing in an image blurring method provided in an embodiment of the present application;
[0033] FIG7 is a first structural diagram of a terminal device provided in an embodiment of the present application;
[0034] FIG8 is a second structural diagram of a terminal device provided in an embodiment of the present application; DETAILED DESCRIPTION
[0035] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0036] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.
[0037] To enrich the user's visual experience, mobile devices and other terminal displays typically display a clear image as a background image (also known as wallpaper), with various application icons and / or a collection of application icons displayed above the background image. However, when the user performs certain actions, the background image needs to be blurred to highlight the message or application icon in the foreground. This processing is generally required to be real-time.
[0038] As shown in Figure 1(a), the display screen of mobile phone 100 displays a clear background image 101, with icons of various applications displayed on background image 101. A collection of application icons, such as folder 1 labeled 102, is also displayed on background image 101. When a user opens folder 1, the background image is blurred to highlight the content in the foreground. 103 is the blurred background image. It should be noted that Figure 1(a) only uses folder 1 labeled 102 as an example. The collection of application icons can also be a collection of user-defined icons such as "Utilities" or "Featured Apps."
[0039] As shown in FIG1(b), when the user pulls down on the display screen of mobile phone 100 to view the notification message, the originally clear background image 104 is blurred, and the blurred background image is shown as 106. The notification message is displayed on the background image 106, such as the notification message marked with "Text 1" at 105.
[0040] To avoid lag and ensure timely responses to the user's current and subsequent actions, the transition from a clear background image to a blurred one must be as fast as possible, completing within milliseconds. However, existing image blurring processes are typically performed on the CPU. In demanding scenarios or when the CPU is struggling due to numerous tasks, the blurring process often takes a long time, resulting in latency, choppy display, and lag, impacting user operations and creating a poor user experience.
[0041] Figure 2 shows a flowchart of an image blurring method in one embodiment of the present application. In this embodiment, in response to a user's operation to blur the image, the image is marked. Then, based on the image blur mark, memory space is requested and a surface blur layer of the image is created. The CPU then draws the blurred content corresponding to the surface blur layer of the image. Finally, the graphics processor draws the surface blur layer and blurred content of the image to achieve image blurring. In other words, in this embodiment, the CPU blurs the image before sending it to the graphics processor for drawing. During this process, a copy of memory of the same size as the image to be blurred is required to store the blurred result.
[0042] More preferably, another embodiment of the present application provides a method for image blurring by optimizing the image blurring processing flow on a GPU.
[0043] The image blurring method provided in the embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, laptop computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), wearable electronic devices, vehicle-mounted devices, and virtual reality devices, and the embodiment of the present application does not impose any restrictions on this.
[0044] For example, the terminal device referred to in the embodiments of the present application is a mobile phone. As shown in FIG3 , the mobile phone 100 includes:
[0045] Processor 310, memory (including external memory interface 320 and internal memory 321), universal serial bus (USB) interface 330, charging management module 340, power management module 341, battery 342, antenna 1, antenna 2, mobile communication module 350, wireless communication module 360, audio module 370, speaker 370A, receiver 370B, microphone 370C, headphone jack 370D, sensor module 380, button 390, motor 391, indicator 392, camera 393, display screen 394, and subscriber identification module (SIM) card interface 395, etc.
[0046] It should be understood that the illustrated structures of the embodiments of the present invention do not constitute a specific limitation on the mobile phone. In other embodiments of the present application, the mobile phone may include more or fewer components than shown, or may combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0047] The processor 310 may include one or more processing units. For example, the processor 310 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a flight controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. The GPU 311 shown in Figure 3 is integrated into the processor 310. In an embodiment of the present application, the processor 310 can quickly implement blur processing of a clear image through the GPU 311.
[0048] In some embodiments, the mobile phone 100 may also include one or more processors 310. The controller may generate an operation control signal based on the instruction opcode and timing signal to control instruction fetching and execution. In other embodiments, the processor 310 may also be provided with a memory for storing instructions and data. For example, the memory in the processor 310 may be a cache memory. This memory may store instructions or data that have just been used or are being recycled by the processor 310. If the processor 310 needs to use the instruction or data again, it may directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 310, and thereby improves the efficiency of the mobile phone 100 in processing data or executing instructions.
[0049] In some embodiments, the processor 310 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.
[0050] The external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external memory card communicates with the processor 310 through the external memory interface 320 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.
[0051] The internal memory 321 can be used to store computer executable program codes, which include instructions. The internal memory 321 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the mobile phone 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 310 executes various functional applications and data processing of the mobile phone 100 by running instructions stored in the internal memory 321 and / or instructions stored in a memory provided in the processor.
[0052] The charging management module 340 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 can receive charging input from the wired charger via the USB interface 330. In some wireless charging embodiments, the charging management module 340 can receive wireless charging input via the wireless charging coil of the mobile phone 100. While charging the battery 342, the charging management module 340 can also provide power to the mobile phone 100 via the power management module 341.
[0053] The power management module 341 is used to connect the battery 342, the charging management module 340, and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340 and provides power to the processor 310, the internal memory 321, the display 394, the camera assembly 393, and the wireless communication module 360. The power management module 341 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 341 can also be provided in the processor 310. In other embodiments, the power management module 341 and the charging management module 340 can also be provided in the same device.
[0054] The wireless communication function of the mobile phone 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 350, the wireless communication module 360, the modem processor and the baseband processor.
[0055] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.
[0056] The mobile communication module 350 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the mobile phone 100. The mobile communication module 350 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 350 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 350 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 350 can be set in the processor 310. In some embodiments, at least some of the functional modules of the mobile communication module 350 can be set in the same device as at least some of the modules of the processor 310.
[0057] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 370A, the receiver 370B, etc.) or displays an image or video through the display screen 394. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 310 and be provided in the same device as the mobile communication module 350 or other functional modules.
[0058] The wireless communication module 360 can provide wireless communication solutions including near field communication (NFC), wireless local area networks (WLAN) (such as WiFi networks), Bluetooth BT, global navigation satellite system (GNSS), frequency modulation (FM), infrared technology (IR), etc. applied on the mobile phone 100. The wireless communication module 360 can be one or more devices integrating at least one communication processing module. The wireless communication module 360 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 310. The wireless communication module 360 can also receive the signal to be sent from the processor 310, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0059] In some embodiments, the antenna 1 of the mobile phone 100 is coupled to the mobile communication module 350, and the antenna 2 is coupled to the wireless communication module 360, so that the mobile phone 100 can communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS) and / or satellite based augmentation system (SBAS). The mobile phone 100 can establish a communication connection with other devices through the mobile communication module 350 and perform data transmission with the other devices through the established communication connection.
[0060] Mobile phone 100 implements image blurring and display functions through GPU 311, display screen 394, and an application processor. GPU 311 is a microprocessor for image processing, connected to display screen 394 and application processor. GPU 311 is used to perform mathematical and geometric calculations for graphics rendering. Processor 310 may include one or more GPUs, which execute program instructions to blur images and generate or change display information. In this embodiment of the present application, GPU 311 can be used to blur a background image, and the blurred image is displayed on display screen 394.
[0061] The display screen 394 is used to display images, videos, etc. The display screen 394 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone 100 may include one or N display screens 394, where N is a positive integer greater than 1. In the embodiment of the present application, the display screen 394 is mainly used to display various images, such as application icons and background images.
[0062] The mobile phone 100 can realize the shooting function through the ISP, the camera component 393, the video codec, the GPU, the display screen 394 and the application processor.
[0063] The mobile phone 100 can implement audio functions such as music playback and recording through the audio module 370, speaker 370A, receiver 370B, microphone 370C, and application processor. For the specific operating principles and functions of the audio module 370, speaker 370A, receiver 370B, and microphone 370C, please refer to the description in conventional technology.
[0064] The sensor module 380 may include a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.
[0065] Keys 390 include a power button, a volume button, etc. Keys 390 may be mechanical keys or touch keys. Mobile phone 100 may receive key inputs and generate key signal inputs related to user settings and function control of mobile phone 100.
[0066] Motor 391 can generate vibration prompts. Motor 391 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 394, motor 391 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
[0067] Indicator 392 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.
[0068] The camera 393 is used to shoot pictures. The camera can be a wide-angle camera, a main camera, a macro camera, a telephoto camera, a time of flight (TOF) camera, etc.
[0069] The SIM card interface 395 is used to connect a SIM card. The SIM card can be connected to and disconnected from the mobile phone 100 by inserting it into or removing it from the SIM card interface 395. The mobile phone 100 can support one or N SIM card interfaces, where N is a positive integer greater than one. The SIM card interface 395 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 395 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 395 can also be compatible with different types of SIM cards. The SIM card interface 395 can also be compatible with external memory cards. The mobile phone 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the mobile phone 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the mobile phone 100 and cannot be separated from the mobile phone 100.
[0070] It should be noted that the hardware modules included in the mobile phone 100 shown in FIG3 are merely exemplary descriptions and do not limit the specific structure of the mobile phone 100. For example, the mobile phone 100 may also include other functional modules.
[0071] The following introduces the software system architecture of the device. For example, the software system of the terminal device provided in the embodiment of the present application can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. For example, the software system may include but is not limited to (Symbian), (Android), (iOS), (Blackberry), Hongmeng (Harmony) and other operating systems, this application is not limited.
[0072] Please refer to Figure 4, Figure 4 shows a layered architecture. Taking the operating system as an example, the software structure block diagram of the terminal device during image blur processing in the embodiment of the present application is specifically introduced.
[0073] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, The operating system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.
[0074] The application layer may include a series of application packages, including system applications and third-party applications, which may include video, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, short message and other applications. The applications on the terminal device may be native applications (such as applications installed in the terminal device when the operating system is installed before the terminal device leaves the factory) or third-party applications (such as applications downloaded and installed by the user through the application store), which is not limited in the embodiments of the present application.
[0075] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 4, the application framework layer may include a window manager, content provider, view system, telephony manager, resource manager, notification manager, and so on.
[0076] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.
[0077] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0078] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and the like. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures. For example, in an embodiment of the present application, the view system in the application framework layer is used to manage various drawn images, such as the background image 101 and the blurred background image 103 in FIG1(a), and the background image 104 and the blurred background image 106 in FIG1(b).
[0079] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.), management of voice call functions and video call functions, etc.
[0080] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0081] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.
[0082] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for System scheduling and management.
[0083] The core library consists of two parts: one is the function that the Java language needs to call, and the other is The core library.
[0084] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.
[0085] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), image processing library, etc.
[0086] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.
[0087] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0088] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0089] A 2D graphics engine is a drawing engine for 2D drawings.
[0090] The image processing library contains some commonly used image processing algorithms, such as image blurring, image denoising, image transformation, image enhancement, etc., for system calls.
[0091] In an embodiment of the present application, while calling an image blurring algorithm in an image processing library on a graphics processor, a two-dimensional graphics engine or a three-dimensional graphics processing library can be called to synchronously draw the blurred image.
[0092] The kernel layer is the layer between hardware and software. It includes at least the display driver, camera driver, audio driver, and sensor driver. For example, the display driver drives the display screen to display content. The camera driver drives the camera to capture images. The audio driver drives the audio player to play audio.
[0093] It should be noted that Figure 4 only shows the hierarchical architecture. This application does not limit the specific architecture of the software system of the terminal device. For the specific introduction of software systems of other architectures, reference can be made to conventional technologies.
[0094] For ease of understanding, some technical terms involved in the embodiments of this application are explained and illustrated below:
[0095] RenderThread: It is a processing thread controlled by the system. It can keep the animation smooth when the UI (User Interface) thread is blocked. That is, it takes on the rendering work of the UI thread and reduces the burden on the UI thread, such as calling skia for drawing.
[0096] skia: is a cross-platform 2D graphics processing library that includes fonts, coordinate conversion, and bitmaps. As shown in Figure 4 It also uses skia for graphics processing, combined with OpenGL / ES and specific hardware features to enhance the display effect. Its underlying principles include:
[0097] Canvas: All drawing operations in Skia are done on the canvas, which is implemented through Canvas. Canvas provides many methods to draw basic geometric shapes, text, bitmaps, etc.
[0098] Paint Engine: Skia's paint engine is its core component, which parses drawing instructions and converts them into final pixel data. The paint engine uses several optimization technologies, such as GPU acceleration, vector graphics rendering, and color space conversion.
[0099] Render Pipeline: Skia uses an advanced rendering pipeline to process drawing commands. The pipeline consists of multiple stages, including geometry clipping, shaders, texture mapping, blending, and masking.
[0100] Skia Pictures: Skia uses shadow pictures to store drawing command sequences for fast repetitive rendering. Shadow pictures are lightweight structures that only record drawing commands and do not contain any pixel data.
[0101] Path: A path in Skia is a series of line segments and curves that can be used to draw various shapes. Paths can also be used for operations such as clipping, transformation, and masking.
[0102] The technical solution in the present application is described in detail below with reference to specific embodiments. In the embodiment of the present application, in response to the user's operation of blurring the first image, a mark is made on the first image that needs to be blurred; the first image is transmitted to the graphics processor according to the mark; after the graphics processor sets the blurred drawing, the graphics processor draws the blurred first image according to the set blurred drawing and the first image; and finally, the blurred first image is displayed. The embodiment of the present application is different from the prior art in which the CPU blurs the first image and then sends it to the graphics processor (GPU) for drawing. Instead, the GPU directly blurs the first image synchronously during the drawing process, which greatly saves processing time and optimizes performance; there is no need to copy a memory of the same size as the first image to carry the blurred result, saving memory consumption. In addition, the blurring of the first image in the embodiment of the present application is uniformly implemented by the GPU, which avoids the disordered and competitive calls of different blur algorithms scattered in various application layers or framework layers, thereby improving efficiency.
[0103] Please refer to Figure 5, which shows a flow chart of the image blurring method provided by an embodiment of the present application. The image blurring method comprises the following steps:
[0104] S501: In response to triggering a first operation of blurring a first image, mark the first image.
[0105] The mark is used to indicate that blur processing is performed on the first image.
[0106] It should be noted that the first operation that triggers the blurring of the first image can be any form of operation. For example, as shown in Figure 1(a), the user clicks to open the folder 1 marked as 102, which triggers the first operation of blurring the background image 101; and as shown in Figure 1(b), the user pulls down the notification bar to display the notification message, which triggers the first operation of blurring the background image 104. In addition, the user's opening of the control center and blurring of the screenshot will also trigger the user's first operation of blurring the first image. It should be understood that the first operation can be any direct or indirect operation that triggers the blurring of the first image, and the embodiments of the present application are not limited to this. For example, the user directly triggers the need to blur the first image with a specific gesture.
[0107] It should be noted that the mark can be a very brief record information, which can be stored in a single byte or even a bit, and the embodiments of the present application are not limited to this.
[0108] It should be understood that the embodiments of the present application can perform blurring processing on images, videos, or other graphic, text, and image data that need to be blurred.
[0109] Please refer to step S202 in Figure 2. This step will apply for memory space based on the image blur mark and create a surface blur layer of the image, that is, a memory of the same size as the first image will be copied to carry the blur result; and unlike the image blur processing in step S202, it is necessary to copy a memory of the same size as the first image to carry the blur result, in the preferred embodiment of the present application, only the first image needs to be marked, and the storage capacity of the mark is much lower than the storage capacity of the memory of the same size as the first image, saving memory consumption.
[0110] In a possible implementation, while marking the first image, an image blurring algorithm may also be selected.
[0111] Different application scenarios may use different blur algorithms on the first image to achieve different blurring effects. Common image blurring algorithms include mean filtering, Gaussian filtering, median filtering, bilateral filtering, and filtering algorithms based on convolutional neural networks (CNN) in recent years. Due to different implementation principles, the visual effects after blurring vary greatly. Generally, a unified default blurring algorithm is used on devices to blur the first image. However, for a refined visual experience, different image blurring algorithms may be selected based on different application scenarios.
[0112] For example, the algorithm for blurring the background image 101 selected by the user when clicking to open the folder 1 marked as 102 in FIG1(a) can be different from the algorithm for blurring the background image 104 selected by the user when pulling down the notification message in FIG1(b), so as to achieve different blurred display effects for the background image.
[0113] In a possible implementation, when an algorithm for image blur processing is selected, parameters of the selected image blur algorithm are set.
[0114] It should be understood that in general, the blur algorithm uses the default parameter settings, and no additional settings are required when selecting the image blur processing algorithm. However, in order to meet the needs of refined application scenario experience, the settings can be opened.
[0115] For example, assuming that the background image 103 in FIG1(a) is blurred using a Gaussian filter algorithm, the Gaussian filter algorithm can set a filter radius (also known as a blur radius, generally the larger the value, the blurrier the image) parameter; the required filter radius parameter can be set when the background image 101 is Gaussian filtered in the scene in FIG1(a) where the user clicks to open the folder 1 marked as 102. More specifically, when the filter shown in FIG4 is used, Taking the system as an example, the sample code for setting the filter radius parameter is as follows:
[0116] Void Bitmap::blur(float sigma){
[0117] mBlurSigma=sigma;
[0118] }
[0119] Where mBlurSigma is the filter radius parameter of the Gaussian filter algorithm, and sigma is the value of the set filter radius.
[0120] S502: Transmit the first image to a graphics processor according to the mark.
[0121] When the first image is marked as requiring blur processing, there is no need to copy a memory of the same size as the first image to carry the blur result, and the first image is directly transmitted to the graphics processor through a specific channel.
[0122] In a possible implementation, if an image blurring algorithm is selected in step S501 , then in step S502 , relevant parameters of the selected image blurring algorithm may be transmitted to the graphics processor.
[0123] In a possible implementation, if the parameters of the selected image blur algorithm are set in step S501 , the parameters of the selected image blur algorithm may be transmitted to the graphics processor in step S502 .
[0124] S503: The graphics processor sets fuzzy drawing.
[0125] Fuzzy drawing is used to prepare for drawing the blurred first image. Setting up fuzzy drawing includes at least one of creating a drawing brush, determining an image blur algorithm, and determining parameters for the image blur algorithm. It should be understood that the image blur algorithm and its parameters may be determined using the system default settings.
[0126] It should be noted that, when blurring an image, an embodiment shown in FIG2 will copy a memory of the same size as the first image to carry the blur result (called a blurred temporary image), and then blur the first image using the selected image blur algorithm and the parameters of the image blur algorithm on the CPU (see step S203 in FIG2 ). After that, the blur result is filled into the blurred temporary image, and finally the filled blurred temporary image is sent to the graphics processor for drawing. Since the image blurring process is executed on the CPU, the graphics processor does not need to set up blurred drawing. However, the preferred embodiment of the present application needs to set up blurred drawing in the graphics processor so that the blurring of the first image can be synchronized during drawing in the subsequent step S504.
[0127] It should be noted that since setting the blurred drawing includes determining the image blur algorithm, in one possible implementation, if the name of the selected image blur algorithm is transmitted to the graphics processor in step S502, the selected image blur algorithm can be accepted as the image blur algorithm to be determined in step S503.
[0128] In a possible implementation, if the parameters of the selected image blur algorithm are transmitted to the graphics processor in step S502 , the parameters of the selected image blur algorithm may be accepted as the parameters of the image blur algorithm to be determined in step S503 .
[0129] Specifically, the setting of fuzzy drawing can be implemented by calling the skia function in RenderThread, which may include the following steps as shown in Figure 6:
[0130] S601: Set the canvas.
[0131] S602: Create a drawing brush.
[0132] S603: Determine whether the name of the image blur algorithm has been transmitted to the graphics processor. If yes, proceed to step S605; if no, proceed to step S604.
[0133] S604: adopting a default image blurring algorithm and default image blurring algorithm parameters.
[0134] S605: Accept the selected image blur algorithm as the image blur algorithm to be used.
[0135] S606: Determine whether the parameters of the selected image blurring algorithm have been transmitted to the graphics processor. If yes, proceed to step S607; if no, proceed to step S608.
[0136] S607 . While accepting the selected image blur algorithm as the blur algorithm to be used, accept the parameters of the selected image blur algorithm as blur parameters to be used.
[0137] S608 : While accepting the selected image blur algorithm as the blur algorithm to be used, adopt the default image blur algorithm parameters.
[0138] More specifically, assuming that the background image 103 in FIG1(a) is blurred by using a Gaussian filter algorithm, the Gaussian filter algorithm can set a filter radius parameter; and using the filter shown in FIG4 Taking the system as an example, when calling the skia function in RenderThread to implement steps S601 to S608, the sample code is as follows:
[0139] Among them, the first line of code, the get_canvas function, corresponds to step S601 to set the canvas; the second line of code defines a drawing pointer and assigns a value, which completes step S602 to create a drawing brush; the third line of code creates an empty blurred drawing, and the code in lines 4-5 determines if the value pointed to by the drawing pointer is not empty and assigns it to the blurred drawing; the seventh line of code has multiple functions: if the code in lines 4-6 is not executed, that is, the blurred drawing blurredPaint is not assigned a value, then imageFilter is assigned the default image blur algorithm and the default image blur algorithm parameters, which completes steps S603 and S604; if the code in lines 4-6 has been executed, it indicates that the blurred drawing has been assigned a value, and imageFilter accepts the selected image blur algorithm as the confirmation. The image blurring algorithm to be used is determined, and steps S603 and S605 are completed; the 8th line of code creates the filter radius parameter of the Gaussian filter algorithm and assigns it: if the filter radius parameter of the Gaussian filter algorithm is set in step S501, the value of the set filter radius is used when assigning the value, and then the final blurred drawing blurredPaint is set through the setImageFilter function in line 9, and steps S606 and S607 are completed; if the filter radius parameter of the Gaussian filter algorithm is not set in step S501, the default filter radius value is used when assigning the value, and then the final blurred drawing blurredPaint is set through the setImageFilter function in line 9, and steps S606 and S608 are completed.
[0140] S504: The graphics processor draws a blurred first image according to the blurred drawing and the first image.
[0141] It should be noted that the embodiment of the present application implements blur processing on the first image simultaneously with the graphics processor's drawing process. Specifically, an area is first selected as the display area; the blurred drawing and the first image are then input into the drawing function; and finally, blur processing is implemented simultaneously with the drawing process on the display area.
[0142] For example, when the Taking the system as an example, the sample code for calling the skia function in RenderThread to implement step S504 is as follows:
[0143] canvas->clipRect(dstLeft,dstTop,dstRight,dstBottom,SkClipOp::kIntersect);
[0144] canvas->drawBitmap(bitmap,srcLeft,srcTop,srcRight,srcBottom,dstLeft,dstTop,dstRight,dstBottom,&blurredPaint);
[0145] The clipRect function selects an area as the display area. The blurred drawing and the first image are then input into the drawBitmap function, and the blurred processing of the first image is synchronously implemented during the drawing process on the display area.
[0146] S505: Display the blurred first image.
[0147] As shown in FIG1(a), the blurred background image 103 is displayed on the mobile phone 100. As shown in FIG1(b), the blurred background image 106 is displayed below the notification message.
[0148] It should be understood that the various schemes of the embodiments of the present application can be reasonably combined and used, and the explanations or descriptions of the various terms appearing in the embodiments can be referenced or explained with each other in the various embodiments, without limitation to this.
[0149] It should also be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0150] It is understandable that, in order to implement the functions of any of the above-mentioned embodiments, the terminal device includes hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0151] The embodiments of the present application can divide the functional modules of the terminal device, etc. according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.
[0152] In the case of adopting an integrated unit, Figure 7 shows a possible structural diagram of the terminal device involved in the above embodiment. The terminal device 700 includes: an image marking module 701, a transmission module 702, an image blurring module 703 and an image display module 704. The image display module 704 is used to support the terminal device 700 to display specific information. For example, the display module 704 is used to support the terminal device 700 to execute the process S505 in Figure 5. The image marking module 701 is used to mark the image, for example, to execute the process S501 in Figure 5. The transmission module 702 is used to support data transmission in the terminal device 700. For example, the transmission module 702 is used to support the terminal device to execute the process S502 in Figure 5. The image blurring module 703 is used to blur the image. For example, the image blurring module 703 executes the processes S503 and S504 in Figure 5.
[0153] Figure 8 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. As shown in Figure 8, the terminal device 800 of this embodiment includes: at least one graphics processor 801 (only one is shown in Figure 8), memory 802, a display 803, and a computer program 804 stored in the memory 802 and executable on the at least one graphics processor 801. When the graphics processor 801 executes the computer program 804, the steps of the above-mentioned method embodiment for arbitrary image blurring are implemented.
[0154] The steps of the method or algorithm described in conjunction with the disclosure of the present application can be implemented in a hardware manner or can be implemented by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC.
[0155] Those skilled in the art will appreciate that, in one or more of the examples above, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.
[0156] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of this application. It should be understood that the above description is only the specific implementation methods of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of this application should be included in the scope of protection of this application.
Claims
1. An image blurring method, characterized in that: The method comprises: In response to a first operation of triggering blurring of a first image, marking the first image, wherein the mark is used to indicate that blurring is performed on the first image; transmitting the first image to a graphics processor according to the mark; The graphics processor sets blur drawing; The graphics processor draws a blurred first image according to the blurred drawing and the first image; The blurred first image is displayed.
2. The method according to claim 1, characterized in that: When marking the first image, the method further includes: A first image blur algorithm is selected.
3. The method according to claim 2, characterized in that When selecting the algorithm for blurring the first image, it also includes: Set the parameters of the first image blur algorithm.
4. The method according to claim 2, characterized in that: The method further comprises: The name of the first image blur algorithm is transmitted to the graphics processor.
5. The method according to claim 4, characterized in that The method further comprises: The graphics processor receives a name of the first image blur algorithm.
6. The method according to claim 3, characterized in that The method further comprises: The set parameters of the first image blur algorithm are transmitted to the graphics processor.
7. The method according to claim 6, characterized in that The method further comprises: The graphics processor receives the set parameters of the first image blur algorithm.
8. The method according to any one of claims 1 to 7, characterized in that The graphics processor sets blur drawing, including: At least one of creating a drawing brush, determining an image blur algorithm, or determining parameters of the image blur algorithm.
9. The method according to claim 8, characterized in that The method of determining the image blur algorithm comprises: The graphics processor uses a default image blur algorithm.
10. The method according to claim 8, characterized in that Determining the parameters of the image blur algorithm includes: The graphics processor uses default image blur algorithm parameters.
11. The method according to claim 8, characterized in that The method of determining the image blur algorithm comprises: The graphics processor uses the first image blur algorithm.
12. The method according to claim 8, characterized in that Determining the parameters of the image blur algorithm includes: The graphics processor adopts the set parameters of the first image blur algorithm.
13. A terminal device, characterized in that: include: a memory for storing instructions executed by one or more graphics processors of the terminal device; A display for displaying images; A graphics processor, when the graphics processor executes the instructions in the memory, can enable the terminal device to execute the image blurring method described in any one of claims 1 to 12.
14. A chip, characterized in that: The chip system includes a processing circuit and a storage medium, wherein the storage medium stores a computer program code; when the computer program code is executed by the processing circuit, the image blurring method according to any one of claims 1 to 12 is implemented.
15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed on a computer, the computer is enabled to execute the image blurring method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Image processing method and image processing device
CN106570847A
Control fuzzy method and device, terminal equipment and readable storage medium
CN114529485A
Fuzzy control creation method and device and computing equipment
CN114756323A
Method and system for providing blur filtering to emphasize focal regions or depths in ultrasound image data
US20210174476A1
