Image Processing Method, Apparatus, Electronic Device, and Storage Medium

Abstract

The present application discloses an image processing method, apparatus, electronic device, and storage medium, belonging to the field of image processing technology. The image processing method is applied to electronic devices and includes: performing bilateral filtering on an image to be processed to obtain a spatial domain kernel and a pixel domain kernel of the image to be processed, and a filtered image after the filtering process; performing edge detection on the image content of the filtered image based on the spatial domain kernel and the pixel domain kernel to detect a region to be processed in the filtered image; and performing image processing on the image content in the region to be processed.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image processing method, device, electronic device and storage medium. Background Art

[0002] In related technologies, after taking photos with electronic devices such as mobile phones, in order to make the photos more beautiful, the camera program generally performs certain post-processing on the original photos, and the imaging effect of the processed finished photos will be more in line with the user's aesthetic taste.

[0003] Post-processing includes multiple steps, such as filtering and image recognition. These steps are completely independent and each step needs to be executed independently, resulting in slow imaging speed. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, device, electronic device and storage medium, which can improve the imaging speed of electronic devices.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the image processing method comprising:

[0006] Performing bilateral filtering on the image to be processed to obtain a spatial domain kernel and a pixel domain kernel of the image to be processed, and a filtered image after filtering;

[0007] Perform edge detection on the image content of the filtered image based on the spatial domain kernel and the pixel domain kernel, and detect the area to be processed in the filtered image;

[0008] Perform image processing on the image content in the area to be processed.

[0009] In a second aspect, an embodiment of the present application provides an image processing device, the image processing device comprising:

[0010] A filtering module is used to perform bilateral filtering on the image to be processed to obtain a spatial domain kernel and a pixel domain kernel of the image to be processed, and a filtered image after filtering;

[0011] An edge detection module is used to perform edge detection on the image content of the filtered image based on the spatial domain kernel and the pixel domain kernel, and detect the area to be processed in the filtered image;

[0012] The processing module is used to perform image processing on the image content in the processing area.

[0013] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method of the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method of the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the steps of the method of the first aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method of the first aspect.

[0017] In an embodiment of the present application, a bilateral filter is performed on the input image to be processed. The bilateral filter can effectively reduce noise and smooth the image to be processed. During the bilateral filtering process, a spatial domain kernel and a pixel domain kernel of the image to be processed can be obtained. The spatial domain kernel and the pixel domain kernel are optimized based on the bilateral filter. Therefore, edge detection is performed based on the spatial domain kernel and the pixel domain kernel, which can improve the effect of edge detection. In addition, during the bilateral filtering process, a spatial domain kernel and a pixel domain kernel are generated. Applying these two data to edge detection eliminates the need to recalculate data in the edge detection step, thereby reducing the number of image processing steps, increasing the image processing speed, and increasing the imaging speed of photographing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of an image processing method according to an embodiment of the present application is shown;

[0019] Figure 2 A schematic diagram of a convolution kernel according to an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram of convolution kernel decomposition according to an embodiment of the present application is shown;

[0021] Figure 4 A schematic diagram of an edge detection operator in related art is shown;

[0022] Figure 5 A schematic diagram of edge detection according to an embodiment of the present application is shown;

[0023] Figure 6 shows a structural block diagram of an image processing device according to an embodiment of the present application;

[0024] Figure 7 shows a structural block diagram of an electronic device according to an embodiment of the present application;

[0025] Figure 8A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0027] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] The image processing method, device, electronic device, and storage medium provided by the embodiments of the present application are described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0029] In some embodiments of the present application, an image processing method is provided, which is applied to an electronic device. Figure 1 FIG. 4 shows a flow chart of an image processing method according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0030] Step 102, performing bilateral filtering on the image to be processed to obtain a spatial domain kernel and a pixel domain kernel of the image to be processed, and a filtered image after filtering;

[0031] Step 104 , performing edge detection on the image content of the filtered image based on the spatial domain kernel and the pixel domain kernel, and detecting the area to be processed in the filtered image;

[0032] Step 106: perform image processing on the image content in the area to be processed.

[0033] In an embodiment of the present application, when taking photos using an electronic device such as a mobile phone, in order to achieve better photo effects, the original photo (RAW data) can be automatically processed after being taken. For example, when taking a portrait, the face of the person photographed can be beautified in a targeted manner, so that the final portrait photo looks better.

[0034] Specifically, after capturing an original photo using an image sensor on an electronic device such as a mobile phone, this original photo is used as the image to be processed, and image processing is performed. During this process, the image to be processed first undergoes bilateral filtering. Bilateral filtering not only considers the Euclidean distance between pixels but also the radiometric differences within the pixel area. This effectively reduces noise and smoothes the image to be processed, thereby improving the visual quality of the captured photo.

[0035] At the same time, the photos taken by users include the subject and the background. When performing image processing, users prefer to optimize the subject, such as the face, such as beautify it, while for the background, they hope to retain more original information and thus retain more details.

[0036] Therefore, when performing image processing, it is necessary to perform edge detection on the main part of the image to be processed, such as the face part, so as to detect the area that needs image optimization processing, that is, the area to be processed.

[0037] When performing edge detection, the edge detection methods in related technologies generally use fixed edge detection operators, such as the Sobel operator (a first-order gradient edge detection algorithm) or the Canny operator (a multi-level edge detection algorithm). The non-zero values of these operators are simply opposite in sign according to the column or row, without considering the spatial and grayscale differences between pixels. Therefore, the edge detection effect is poor.

[0038] In this regard, the embodiment of the present application reuses the spatial domain kernel and pixel domain kernel of the image to be processed obtained in the bilateral filtering, and optimizes the edge detection operator through the spatial domain kernel and the pixel domain kernel, so that the optimized edge detection operator is more in line with the spatial domain characteristics and pixel domain characteristics of the current photo, which can effectively improve the edge detection effect.

[0039] In this process, since both the spatial domain kernel and the pixel domain kernel reuse the self-bilateral filtering process, this process can reduce the overhead of the processor performance of the electronic device during image processing, thereby effectively improving the speed and efficiency of image processing, such as photo beautification.

[0040] After identifying the area to be processed, the image content in the area to be processed is processed according to a pre-set photo beautification algorithm, such as a beauty algorithm, to obtain a photo with an automatically beautified effect, thereby improving the photo-taking effect of the electronic device.

[0041] The embodiment of the present application can effectively reduce noise and smooth the captured content by performing bilateral filtering on the captured photos. In addition, during the bilateral filtering process, a spatial domain kernel and a pixel domain kernel are generated. These two data are applied to edge detection, eliminating the need to recalculate data in the edge detection step. While improving the edge detection effect, the image processing steps are reduced, the image processing efficiency is improved, and the imaging speed of the electronic device taking pictures is thereby increased.

[0042] In some embodiments of the present application, the image to be processed includes N first pixels, where N is a positive integer;

[0043] Performing bilateral filtering on the image to be processed to obtain a spatial domain kernel and a pixel domain kernel of the image to be processed, and a filtered image after filtering, including:

[0044] Acquire image data of the image to be processed, the image data including the coordinates of a center point of an initial window within a pixel domain of the image to be processed, first pixel coordinates of N first pixels, and first pixel values of the N first pixels;

[0045] Determine a spatial domain kernel and a pixel domain kernel according to the first pixel point coordinates, the pixel value, and a preset hash table of each first pixel point among the N first pixel points;

[0046] Determine N second pixel values corresponding one-to-one to the N first pixel points according to the spatial domain kernel, the pixel domain kernel, the center point coordinates, and the first pixel value;

[0047] The pixel values of N first pixels in the image to be processed are respectively adjusted to corresponding second pixel values to obtain a filtered image, where the filtered image includes N second pixels, and the N second pixels correspond to the N second pixel values in a one-to-one manner.

[0048] In an embodiment of the present application, the original image captured by the electronic device, that is, the image to be processed includes N first pixels. Taking the resolution of the image to be processed as 4K resolution (4096×2160) as an example, the image to be processed includes 4096×2160=8847360 first pixels, that is, N=8847360.

[0049] When performing bilateral filtering on an image to be processed, the image data of the current image to be processed is first obtained, specifically including the coordinates p of the center point of the initial window (denoted as s) in the pixel domain of the image to be processed, the coordinates of the above-mentioned N first pixel points, denoted as the first coordinates q, and the pixel values of the above-mentioned N first pixel points, denoted as the first pixel values, which are represented by I(p) and I(q).

[0050] After obtaining the image data, the spatial domain kernel G σs and pixel domain kernel G σrDetermine, among which, determine the spatial domain kernel G σs and pixel domain kernel G σr The formula is as follows:

[0051]

[0052]

[0053] Among them, q i and q j is the center point coordinate, p i and p j is the first coordinate, I p and I q is the first pixel value, e is the natural logarithm, σs and σr are preset parameters.

[0054] Since the above calculations need to be performed on each pixel point in the initial window in the pixel domain during the processing, and the calculation process includes exponential operations, the amount of calculation is relatively large.

[0055] In this regard, an embodiment of the present application is pre-set with a hash table, which pre-stores the correspondence between pixel coordinates, pixel values, spatial domain kernels, and pixel domain kernels. Therefore, during the calculation process, the corresponding spatial domain kernels and pixel domain kernels can be read from the hash table through the pixel coordinates and pixel values of the pixel points, thereby effectively reducing the amount of calculation.

[0056] After obtaining the required parameters, bilateral filtering is performed on each first pixel in the image to be processed. Specifically, the formula for bilateral filtering is as follows:

[0057]

[0058]

[0059] Among them, I(p) is the second pixel value after filtering, I(q) is the first pixel value before filtering, G σs is the spatial domain kernel, G σr is the pixel domain kernel, p is the center coordinate of the initial window, and q is the first coordinate.

[0060] In the above formula, each calculation in the initial window can be regarded as a convolution process. Figure 2 A schematic diagram of a convolution kernel according to an embodiment of the present application is shown in FIG. Figure 2 As shown, the convolution kernel 202 can be decomposed into a column vector 204 and a row vector 206 according to rows and columns, so a single two-dimensional convolution can be decomposed into two one-dimensional convolutions. Figure 2 The x items in are all example parameters, which are only used to show the convolution kernel and have no actual meaning.

[0061] Figure 3 FIG. 4 shows a schematic diagram of a convolution kernel decomposition according to an embodiment of the present application, as shown in FIG. Figure 3 As shown in the figure, taking the image to be processed with a width of w pixels and a height of h pixels as an example, assuming that the size of the Gaussian kernel involved in the convolution operation is m×n, the computational complexity of a two-dimensional convolution is w×h×m×n. After decomposing it into two one-dimensional convolutions, the horizontal convolution calculation is performed first, and then the vertical convolution calculation is performed. Therefore, the convolution calculation amount of one pixel is w×h×m and w×h×n, and the total computational complexity is w×h×(n+m). Figure 3 The O(n 2 ) is reduced to Figure 3 O(n) as shown.

[0062] Figure 3 The x items in are all example parameters, which are only used to show the convolution kernel and have no actual meaning.

[0063] Through the above-mentioned bilateral filtering formula, bilateral filtering is performed on each first pixel point in the pixel domain of the image to be processed, that is, the first pixel values of the N first pixel points are adjusted to the second pixel values after bilateral filtering, and N second pixel points adjusted to the second pixel values are obtained. The arrangement of these N second pixel points is consistent with that of the N first pixel points, and a filtered image is formed.

[0064] The embodiment of the present application performs bilateral filtering on the image to be processed, which not only takes into account the Euclidean distance of pixels, but also takes into account the radiation differences within the pixel field. Therefore, it can effectively reduce noise and smooth the image to be processed, thereby improving the image processing effect.

[0065] In some embodiments of the present application, determining N second pixel values corresponding one-to-one to the N first pixel points based on the spatial domain kernel, the pixel domain kernel, the center point coordinates, and the first pixel value includes:

[0066] Generate N filtering tasks corresponding to the N first pixel points one by one according to the spatial domain kernel, the pixel domain kernel, the center point coordinates and the first pixel value;

[0067] Based on the execution results of the N filtering tasks, N second pixel points are obtained.

[0068] In the embodiment of the present application, the image processing speed is improved by utilizing the graphics processing unit (GPU) of the electronic device.

[0069] In the related art, for mobile terminals such as mobile phones, their image processing algorithms need to be executed by the central processing unit (CPU) of the electronic device. When performing image processing on images with higher resolution, the algorithm complexity is large. For example, the bilateral filtering algorithm in the above embodiment needs to repeat the bilateral filtering operation for each pixel point in the initial window. The total amount of calculation is large, which will consume a lot of CPU performance, thereby resulting in slow imaging speed and even slowing down the system operation speed of the electronic device.

[0070] Mobile GPUs typically include multiple computing units, and based on the characteristics of this hardware acceleration structure, they can accelerate image processing. Specifically, assuming the GPU of an electronic device includes M computing units, for an image to be processed with N first pixels, a corresponding filtering task is generated for each of the N first pixels. Therefore, each of the N first pixels corresponds to a total of N filtering tasks.

[0071] These N filtering tasks are evenly distributed to the M computing units of the GPU, so that each of the M computing units of the GPU is responsible for at least one filtering task. Therefore, the GPU can process the filtering tasks of M first pixel points at one time and obtain M second pixel values corresponding to these M first pixel points. That is, the M filtering tasks are processed in parallel by the M computing units of the GPU at the same time, which can effectively improve the image processing speed.

[0072] The embodiments of the present application utilize the graphics processing unit (GPU) of electronic devices such as mobile phones to simultaneously process multiple filtering tasks in parallel, thereby improving the speed of image processing and reducing the calculation time of bilateral filtering, so that the bilateral filtering algorithm can be applied to mobile terminals with low CPU computing power, while improving the photo imaging speed of electronic devices.

[0073] In some embodiments of the present application, edge detection is performed on the image content of the filtered image based on the spatial domain kernel and the pixel domain kernel, including:

[0074] Determine the edge detection operator based on the product of the spatial domain kernel and the pixel domain kernel;

[0075] The edge detection operator is used to perform edge detection on the image content of the filtered image.

[0076] In the embodiment of the present application, when performing edge detection, the practice in the related art is to use a fixed edge detection operator, such as the Sobel operator or the Canny operator, to perform edge detection calculation. Figure 4 A schematic diagram of an edge detection operator in related art is shown, such as Figure 4As shown in FIG, the non-zero values of these operators are just opposite in sign according to the column or row, without considering the difference in space and grayscale values between pixels, so the edge detection effect is poor.

[0077] To address the shortcomings of the above-mentioned related technologies, the embodiments of the present application optimize the edge detection operator by reusing the spatial domain kernel and pixel domain kernel of the image to be processed obtained in bilateral filtering, so that the optimized edge detection operator is more consistent with the spatial domain features and pixel domain features of the current photo, which can effectively improve the effect of edge detection.

[0078] Specifically, the spatial domain kernel G σs and pixel domain kernel G σr The formula is as follows:

[0079]

[0080]

[0081] Among them, q i 、p i ,q j 、p j is the first coordinate, I p and I q is the first pixel value, e is the natural logarithm, σs and σr are preset parameters.

[0082] The spatial domain kernel G σs and pixel domain kernel G σr The product of: (G σs ×G σr ), as a new edge detection operator.

[0083] Taking the beautification of portrait photos as an example, after bilateral filtering is performed on the original portrait photo to obtain the processed filtered image, the beautification algorithm needs to process the facial skin in the portrait photo. Therefore, edge detection is needed to identify the edge part of the face, so as to avoid unnecessary beautification of parts that do not need beautification, such as flowers, plants and trees in the background.

[0084] During edge detection, all pixels within the current pixel range are extracted. Figure 5 FIG. 4 shows a schematic diagram of edge detection according to an embodiment of the present application, as shown in FIG. Figure 5 As shown, taking pixel point P5 as an example, the pixel point P5 and the 8 surrounding pixels are extracted, and the pixel values of this group of pixels are compared with the above edge detection operator (G σs ×G σr ) are multiplied and the true edges of each part of the image, such as the edge of a human face, are detected based on the calculation results.

[0085] Among them, due to the edge detection operator (G σs ×G σr ) in the spatial domain kernel G σs and pixel domain kernel G σr The process of bilateral filtering is reused, thereby reducing repeated calculations and improving image processing efficiency.

[0086] The embodiment of the present application determines the optimized edge detection operator by reusing the spatial domain kernel and the pixel domain kernel in the bilateral filtering process, so that the optimized edge detection operator is more consistent with the spatial domain characteristics and pixel domain characteristics of the current photo, which can effectively improve the effect of edge detection.

[0087] In some embodiments of the present application, performing image processing on image content in the area to be processed includes:

[0088] Determining first image content of the image to be processed in the area to be processed;

[0089] Performing edge detection on the first image content according to the target color information to obtain a target area in the first image content;

[0090] Through preset hyperparameters, feature fusion is performed on the second image content of the filtered image in the target area and the third image content of the image to be processed in the target area to obtain a processed target image.

[0091] In an embodiment of the present application, in order to ensure that the processed image looks natural, when performing image processing, the image content after bilateral filtering and the image content of the original image are combined to ensure that the processed image effect is not too exaggerated and distorted.

[0092] Specifically, taking the beautification of a portrait photo as an example, after obtaining the area to be processed through edge detection, the first image content within the area to be processed is determined in the original image, i.e., the input image to be processed. The area to be processed is the face area obtained based on edge detection. Since the face area includes not only the skin of the face but also areas such as the eyes, eyebrows, and hair, in order to ensure that unnecessary beautification processing is not performed on these non-face skin areas, the embodiment of the present application performs another edge detection on the first image content within the detected area to be processed based on the target color information.

[0093] For portrait beautification scenes, the target color is skin color. For other scenes, such as scenes for clarity processing of photographed text, the target color can be common text colors such as black and white. This application does not impose any restrictions on this.

[0094] A second edge detection is performed on the identified portrait area using the target color, such as skin color, to detect the skin color area, that is, the area of the facial skin, which is recorded as the target area. The second image content within the target area is the facial skin of the portrait taken by the user.

[0095] After the target area is detected, the third image content within the target area is determined in the filtered image obtained after the bilateral filtering process.

[0096] Here, the second image content, that is, the face part in the original photo, is recorded as raw(i,j), and the third image content, that is, the face part after bilateral filtering, is recorded as BilateralFilter(i,j). The preset hyperparameter α is used to adjust the relationship between the two, and the features of raw(i,j) and BilateralFilter(i,j) are fused. The specific formula is as follows:

[0097] Beauty(i,j)=α×(raw(i,j)-BilateralFilter(i,j))+BilateralFilter(i,j);

[0098] Among them, Beauty(i,j) is the processed target image, α is the hyperparameter, BilateralFilter(i,j) is the second image content, and raw(i,j) is the third image content.

[0099] The embodiment of the present application performs feature fusion on the original image and the filtered image, so that the effect of the processed image can be made to be more consistent with that of the original image, thus avoiding distortion of the processed image and improving the effect of image processing.

[0100] In some embodiments of the present application, in order to remove noise from the input image and better retain the details of the image, the input image is first filtered. Bilateral filtering not only considers the Euclidean distance of pixels, but also considers the radiation differences within the pixel field. The effect of noise reduction and smoothing of the input image through bilateral filtering is better.

[0101] Among them, the computational complexity of bilateral filtering is high and it is not suitable for deployment on mobile phones. The embodiment of the present application is based on a solution of GPU acceleration on the mobile terminal, and makes full use of the characteristics of hash tables and separate approximate calculations to improve the computational efficiency of the bilateral filtering algorithm on the mobile terminal while ensuring the beautification effect.

[0102] Specifically, first set p to be the coordinate of the center point of s (initial window) in the pixel domain, q to be the coordinate of the input pixel point, I(p) and I(q) to be the pixel value of the point, representing the spatial domain kernel and the pixel domain kernel respectively. The formula for bilateral filtering can be obtained as follows:

[0103]

[0104]

[0105] Among them, I(p) is the second pixel value after filtering, I(q) is the first pixel value before filtering, G σs is the spatial domain kernel, G σr is the pixel domain kernel, p is the center coordinate of the initial window, and q is the first coordinate.

[0106] Among them, G σs and G σr The formula is as follows:

[0107]

[0108]

[0109] Among them, q i and q j is the center point coordinate, p i and p j is the first coordinate, I p and I q is the first pixel value, e is the natural logarithm, σs and σr are preset parameters.

[0110] During the execution of the computing unit of the mobile GPU, once the input image is determined, G σs and G σr The independent variables are also determined, so the complex exponential operation can be realized efficiently, and the pixel position, pixel value, G σs and G σr Hash table, during the calculation process, according to the key such as pixel position and pixel value, the corresponding G is removed from the hash table σs and G σr The value of is used to participate in the calculation of I(p) to improve the execution efficiency of exponential operations.

[0111] After calculation, we can get the pixel value of p under bilateral filtering. According to the above formula, each pixel in the initial window s in the pixel domain must be calculated as above, and G σs and G σr It involves exponential operations and is computationally intensive.

[0112] In order to reduce the computational complexity, this application adopts the GPU acceleration method by writing Compute Shader (a program that can run on the GPU). Each operation is executed by a separate computing unit of the GPU, and the high concurrency of the mobile GPU is used to improve the computational efficiency.

[0113] In the above calculation process, each calculation process in the initial window is a convolution process. If the current convolution kernel can be decomposed into a column vector and a row vector according to rows and columns, then a single two-dimensional convolution can be decomposed into two one-dimensional convolutions. The decomposition method can realize the reuse of multiplication results. For the entire image with a width of w and a height of h, assuming that the size of the Gaussian kernel involved in the convolution operation is m×n, the calculation amount of a single two-dimensional convolution is w×h×m×n. In the two one-dimensional convolutions, the horizontal convolution operation is calculated first, then the convolution calculation amount of one pixel is w×h×m and w×h×n respectively, and the total calculation amount is w×h×(n+m). Its computational complexity is given by Figure 3 The O(n 2 ) is reduced to Figure 3 O(n) as shown.

[0114] After obtaining the texture image processed by bilateral filtering, the beautification method can process the skin and non-edge parts of the image. Therefore, it is necessary to obtain the edge part of the image. Inaccurate edge detection results will cause beautification processing to be performed on areas that do not need to be beautified. Traditional edge detection methods such as the Sobel operator and the Canny operator have fixed values. The non-zero values of the operators are only opposite in sign by column and row, without taking into account the differences in space and grayscale values between pixels. The embodiment of the present application optimizes the edge detection operator based on bilateral filtering to maximize the effect of edge detection.

[0115] Specifically, edge detection first takes out all pixels within the range of the current pixel, such as Figure 5 As shown, taking pixel point P5 as an example, pixel point P5 and the surrounding 8 pixel points are extracted.

[0116] In order to make full use of the space between pixels and the difference in grayscale values, the pixel values of this group of pixels are compared with the above edge detection operator (G σs ×G σr ) is multiplied and the real edges of each part of the image, such as the edge of a face, are detected based on the calculation results. σs ×G σr ) has been completed, and the results of the previous steps can be reused, thus reducing repeated calculations.

[0117] After obtaining the above results, the final beautification image can be obtained by combining skin color detection and adaptive feature fusion. Specifically, in order to ensure that the beautification is applied to portraits, this application introduces skin color detection. If the image is detected with a skin color area, edge detection will continue.

[0118] After detecting the edges, we obtain the area in the image that needs to be beautified. The result of bilateral filtering is recorded as BilateralFilter(i,j), and the effect of the original image is recorded as raw(i,j). The final beauty effect Beauty(i,j) not only considers the result of bilateral filtering, but also takes into account the result of the original image, thus ensuring that the effect is not distorted.

[0119] Furthermore, the hyperparameter α is introduced to adjust the relationship between the above two, and the features of raw(i,j) and BilateralFilter(i,j) are fused. The specific formula is as follows:

[0120] Beauty(i,j)=α×(raw(i,j)-BilateralFilter(i,j))+BilateralFilter(i,j);

[0121] Among them, Beauty(i,j) is the processed target image, α is the hyperparameter, BilateralFilter(i,j) is the second image content, and raw(i,j) is the third image content.

[0122] The embodiment of the present application accelerates bilateral filtering, takes advantage of the performance advantages of hash tables and separate approximate calculations, and uses bilateral filtering to optimize edge detection operators, thereby improving the effect of edge detection.

[0123] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the method of performing image processing by the image processing device as an example.

[0124] In some embodiments of the present application, an image processing device is provided, which is applied to an electronic device. Figure 6 FIG. 1 shows a structural block diagram of an image processing device according to an embodiment of the present application. Figure 6 As shown, the image processing device 600 includes:

[0125] A filtering module 602 is configured to perform bilateral filtering on the image to be processed to obtain a spatial domain kernel and a pixel domain kernel of the image to be processed, and a filtered image after the filtering process;

[0126] An edge detection module 604 is configured to perform edge detection on the image content of the filtered image based on the spatial domain kernel and the pixel domain kernel, and detect an area to be processed in the filtered image;

[0127] The processing module 606 is configured to perform image processing on the image content in the area to be processed.

[0128] The embodiment of the present application performs bilateral filtering on the captured photos to effectively reduce noise and smooth the captured content, and reuses the parameters in the bilateral filtering process to perform edge detection. While improving the edge detection effect, it reduces the overhead of processor performance, improves image processing efficiency, and thus increases the imaging speed of electronic devices when taking photos.

[0129] In some embodiments of the present application, the image to be processed includes N first pixels, where N is a positive integer;

[0130] The image processing device further includes:

[0131] an acquisition module, configured to acquire image data of an image to be processed, the image data including coordinates of a center point of an initial window within a pixel domain of the image to be processed, first pixel coordinates of N first pixels, and first pixel values of the N first pixels;

[0132] a determination module, configured to determine a spatial domain kernel and a pixel domain kernel based on the first pixel coordinates, the pixel value, and a preset hash table of each of the N first pixel points; and determine N second pixel values corresponding one-to-one to the N first pixel points based on the spatial domain kernel, the pixel domain kernel, the center point coordinates, and the first pixel value;

[0133] The adjustment module is used to adjust the pixel values of N first pixel points in the image to be processed to corresponding second pixel values to obtain a filtered image, where the filtered image includes N second pixel points, and the N second pixel points correspond to the N second pixel values in a one-to-one manner.

[0134] The embodiment of the present application performs bilateral filtering on the image to be processed, which not only takes into account the Euclidean distance of pixels, but also takes into account the radiation differences within the pixel field. Therefore, it can effectively reduce noise and smooth the image to be processed, thereby improving the image processing effect.

[0135] In some embodiments of the present application, the electronic device includes a graphics processor, the graphics processor includes M computing units, where M is a positive integer;

[0136] The image processing device further includes:

[0137] A task generation module, configured to generate N filtering tasks corresponding to the N first pixel points in a one-to-one manner according to the spatial domain kernel, the pixel domain kernel, the center point coordinates, and the first pixel value;

[0138] The task execution module is used to obtain N second pixel points based on the execution results of the N filtering tasks.

[0139] The embodiments of the present application utilize the graphics processing unit (GPU) of electronic devices such as mobile phones to simultaneously process multiple filtering tasks in parallel, thereby improving the speed of image processing and reducing the calculation time of bilateral filtering, so that the bilateral filtering algorithm can be applied to mobile terminals with low CPU computing power, while improving the photo imaging speed of electronic devices.

[0140] In some embodiments of the present application, the edge detection module is specifically configured to: determine an edge detection operator based on the product of a spatial domain kernel and a pixel domain kernel; and perform edge detection on image content of the filtered image using the edge detection operator.

[0141] The embodiment of the present application determines the optimized edge detection operator by reusing the spatial domain kernel and the pixel domain kernel in the bilateral filtering process, so that the optimized edge detection operator is more consistent with the spatial domain characteristics and pixel domain characteristics of the current photo, which can effectively improve the effect of edge detection.

[0142] In some embodiments of the present application, the edge detection module is further configured to determine a first image content of the image to be processed in the area to be processed; perform edge detection on the first image content based on the target color information to obtain a target area in the first image content;

[0143] The processing module is specifically used to perform feature fusion on the second image content of the filtered image in the target area and the third image content of the image to be processed in the target area through preset hyperparameters to obtain a processed target image.

[0144] The embodiment of the present application performs feature fusion on the original image and the filtered image, so that the effect of the processed image can be made to be more consistent with that of the original image, thus avoiding distortion of the processed image and improving the effect of image processing.

[0145] The image processing device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0146] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0147] The image processing device provided in the embodiment of the present application can implement each process implemented in the above method embodiment. To avoid repetition, it will not be described here.

[0148] Optionally, an embodiment of the present application further provides an electronic device, Figure 7 FIG. 1 shows a structural block diagram of an electronic device according to an embodiment of the present application. Figure 7 As shown, the electronic device 700 includes a processor 702, a memory 704, and a program or instruction stored in the memory 704 and executable on the processor 702. When the program or instruction is executed by the processor 702, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they will not be described here.

[0149] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0150] Figure 8 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0151] The electronic device 800 includes but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809 and a processor 810.

[0152] Those skilled in the art will understand that the electronic device 800 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 810 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0153] Among them, the processor 810 is used to perform bilateral filtering on the image to be processed to obtain the spatial domain kernel and pixel domain kernel of the image to be processed, and the filtered image after filtering; based on the spatial domain kernel and pixel domain kernel, edge detection is performed on the image content of the filtered image, and the area to be processed is detected in the filtered image; and image processing is performed on the image content in the area to be processed.

[0154] The embodiment of the present application performs bilateral filtering on the captured photos to effectively reduce noise and smooth the captured content, and reuses the parameters in the bilateral filtering process to perform edge detection. While improving the edge detection effect, it reduces the overhead of processor performance, improves image processing efficiency, and thus increases the imaging speed of electronic devices when taking photos.

[0155] In some embodiments of the present application, the image to be processed includes N first pixels, where N is a positive integer;

[0156] Processor 810 is also used to obtain image data of an image to be processed, the image data including the center point coordinates of an initial window within a pixel domain of the image to be processed, the first pixel point coordinates of N first pixels, and the first pixel values of the N first pixels; determine a spatial domain kernel and a pixel domain kernel based on the first pixel point coordinates, pixel value, and a preset hash table of each of the N first pixels; determine N second pixel values corresponding one-to-one to the N first pixels based on the spatial domain kernel, the pixel domain kernel, the center point coordinates, and the first pixel value; and adjust the pixel values of the N first pixels in the image to be processed to corresponding second pixel values, respectively, to obtain a filtered image, the filtered image including N second pixels, and the N second pixels corresponding one-to-one to the N second pixel values.

[0157] The embodiment of the present application performs bilateral filtering on the image to be processed, which not only takes into account the Euclidean distance of pixels, but also takes into account the radiation differences within the pixel field. Therefore, it can effectively reduce noise and smooth the image to be processed, thereby improving the image processing effect.

[0158] In some embodiments of the present application, the electronic device includes a graphics processor, processor 810, which is also used to generate N filtering tasks corresponding one-to-one to N first pixel points based on the spatial domain kernel, the pixel domain kernel, the center point coordinates and the first pixel value; and obtain N second pixel points based on the execution results of the N filtering tasks.

[0159] The embodiments of the present application utilize the graphics processing unit (GPU) of electronic devices such as mobile phones to simultaneously process multiple filtering tasks in parallel, thereby improving the speed of image processing and reducing the calculation time of bilateral filtering, so that the bilateral filtering algorithm can be applied to mobile terminals with low CPU computing power, while improving the photo imaging speed of electronic devices.

[0160] In some embodiments of the present application, the processor 810 is further configured to determine an edge detection operator based on the product of the spatial domain kernel and the pixel domain kernel; and perform edge detection on the image content of the filtered image using the edge detection operator.

[0161] The embodiment of the present application determines the optimized edge detection operator by reusing the spatial domain kernel and the pixel domain kernel in the bilateral filtering process, so that the optimized edge detection operator is more consistent with the spatial domain characteristics and pixel domain characteristics of the current photo, which can effectively improve the effect of edge detection.

[0162] In some embodiments of the present application, the processor 810 is further used to determine the first image content of the image to be processed in the area to be processed; perform edge detection on the first image content based on the target color information to obtain the target area in the first image content; and perform feature fusion on the second image content of the filtered image in the target area and the third image content of the image to be processed in the target area through preset hyperparameters to obtain a processed target image.

[0163] The embodiment of the present application performs feature fusion on the original image and the filtered image, so that the effect of the processed image can be made to be more consistent with that of the original image, thus avoiding distortion of the processed image and improving the effect of image processing.

[0164] It should be understood that in an embodiment of the present application, the input unit 804 may include a graphics processing unit (GPU) 8041 and a microphone 8042, and the graphics processor 8041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 807 includes a touch panel 8071 and at least one of other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include two parts: a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0165] The memory 809 can be used to store software programs and various data. The memory 809 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 809 may include a volatile memory or a non-volatile memory, or the memory 809 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The memory 809 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0166] Processor 810 may include one or more processing units. Optionally, processor 810 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 810.

[0167] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0168] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0169] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0170] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0171] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0172] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0174] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Performing bilateral filtering on the image to be processed to obtain a spatial domain kernel and a pixel domain kernel of the image to be processed, and a filtered image after the filtering process; performing edge detection on image content of the filtered image according to the spatial domain kernel and the pixel domain kernel, and detecting a to-be-processed area in the filtered image; Performing image processing on the image content in the area to be processed; The performing edge detection on the image content of the filtered image according to the spatial domain kernel and the pixel domain kernel includes: determining an edge detection operator according to the product of the spatial domain kernel and the pixel domain kernel; Edge detection is performed on the image content of the filtered image using the edge detection operator.

2. The image processing method according to claim 1, wherein: The image to be processed includes N first pixels, where N is a positive integer; The method of performing bilateral filtering on the image to be processed to obtain a spatial domain kernel and a pixel domain kernel of the image to be processed, and a filtered image after the filtering process, includes: Acquire image data of the image to be processed, the image data including the center point coordinates of an initial window within a pixel domain of the image to be processed, first pixel point coordinates of the N first pixels, and first pixel values of the N first pixels; Determining the spatial domain kernel and the pixel domain kernel according to the first pixel point coordinates of each of the N first pixel points, the pixel value, and a preset hash table; Determine, according to the spatial domain kernel, the pixel domain kernel, the center point coordinates, and the first pixel value, N second pixel values corresponding one-to-one to the N first pixel points; The pixel values of the N first pixel points in the image to be processed are respectively adjusted to the corresponding second pixel values to obtain the filtered image, wherein the filtered image includes N second pixel points, and the N second pixel points correspond one-to-one to the N second pixel values.

3. The image processing method according to claim 2, wherein: The determining, according to the spatial domain kernel, the pixel domain kernel, the center point coordinates, and the first pixel value, N second pixel values corresponding one-to-one to the N first pixel points includes: Generate N filtering tasks corresponding one-to-one to the N first pixel points according to the spatial domain kernel, the pixel domain kernel, the center point coordinates, and the first pixel value; The N second pixel points are obtained based on the execution results of the N filtering tasks.

4. The image processing method according to any one of claims 1 to 3, characterized in that: The performing image processing on the image content in the area to be processed includes: Determine a first image content of the image to be processed in the area to be processed; Performing edge detection on the first image content according to the target color information to obtain a target area in the first image content; By using preset hyperparameters, feature fusion is performed on the second image content of the filtered image in the target area and the third image content of the image to be processed in the target area to obtain a processed target image.

5. An image processing device, characterized in that: The image processing device comprises: A filtering module is used to perform bilateral filtering on the image to be processed to obtain a spatial domain kernel and a pixel domain kernel of the image to be processed, and a filtered image after the filtering process; an edge detection module, configured to perform edge detection on the image content of the filtered image based on the spatial domain kernel and the pixel domain kernel, and detect an area to be processed in the filtered image; A processing module, configured to perform image processing on the image content in the area to be processed; The edge detection module is specifically used to: determining an edge detection operator according to the product of the spatial domain kernel and the pixel domain kernel; Edge detection is performed on the image content of the filtered image using the edge detection operator.

6. The image processing device according to claim 5, wherein The image to be processed includes N first pixels, where N is a positive integer; The image processing device further includes: an acquisition module, configured to acquire image data of the image to be processed, the image data including the coordinates of a center point of an initial window within a pixel domain of the image to be processed, first pixel coordinates of the N first pixels, and first pixel values of the N first pixels; a determining module, configured to determine the spatial domain kernel and the pixel domain kernel according to the first pixel coordinates of each of the N first pixel points, the pixel value, and a preset hash table; and Determine, according to the spatial domain kernel, the pixel domain kernel, the center point coordinates, and the first pixel value, N second pixel values corresponding one-to-one to the N first pixel points; An adjustment module is used to adjust the pixel values of the N first pixel points in the image to be processed to the corresponding second pixel values, so as to obtain the filtered image, wherein the filtered image includes N second pixel points, and the N second pixel points correspond one-to-one to the N second pixel values.

7. The image processing device according to claim 6, wherein: The image processing device further includes: a task generating module, configured to generate N filtering tasks corresponding one-to-one to the N first pixel points according to the spatial domain kernel, the pixel domain kernel, the center point coordinates, and the first pixel value; A task execution module is used to obtain the N second pixel points based on the execution results of the N filtering tasks.

8. The image processing device according to any one of claims 5 to 7, characterized in that: The edge detection module is further configured to determine a first image content of the image to be processed in the area to be processed; and Performing edge detection on the first image content according to the target color information to obtain a target area in the first image content; The processing module is specifically used to perform feature fusion on the second image content of the filtered image in the target area and the third image content of the image to be processed in the target area through preset hyperparameters to obtain a processed target image.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 4 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 4 are implemented.

Images