Image processing method, device, electronic device and storage medium

By identifying portrait and face areas, calculating the blurring parameters and directly blurring the background, solving the problem of large calculations of mainstream background blurring algorithms, and achieving efficient background blurring and optimization of portrait shooting effects.

CN113313626BActive Publication Date: 2024-12-03GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202110552743.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-20
Publication Date
2024-12-03
Estimated Expiration
2041-05-20

AI Technical Summary

Technical Problem

Mainstream background blur algorithms have a large amount of calculations, which is difficult to meet the real-time requirements of video shooting or shooting previews, especially on platforms with weak computing capabilities, which are difficult to achieve ideal blur effect.

Method used

By identifying the portrait area and face area, and calculating the blurring parameters for background blurring, the calculation steps of image depth estimation are avoided, and the background area is directly blurred using the portrait mask and blurring parameters.

Benefits of technology

It reduces the calculation amount and improves the blurring efficiency, and can achieve background blur without image depth estimation, optimizes the shooting effect of portrait photography, and balances the needs of overall portrait and local faces.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113313626B_ABST
    Figure CN113313626B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses an image processing method, the method comprising: identifying a portrait area in an original image, and obtaining a portrait mask corresponding to the original image according to the portrait area; determining a first area ratio of the portrait area in the original image according to the portrait mask; performing face detection on the original image to determine face area information contained in the original image; determining a blur parameter according to the first area ratio and the face area information, and blurring the background area of ​​the original image according to the portrait mask and the blur parameter. Implementing the embodiment of the present application can reduce the amount of calculation, improve blurring efficiency, and balance the two different shooting requirements of highlighting the overall portrait and highlighting the partial face, and can optimize the shooting effect of portrait shooting.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] In a photograph, a blurred background can well highlight the subject, especially in a portrait shooting scene, where a blurred background can well highlight the portrait. Currently, most electronic devices with photography functions can provide an automatic background blur function. Users can select a focus target (such as a portrait) in the shooting preview interface, and the electronic device will automatically identify the background area in the image except the focus target and blur the background area.

[0003] However, in practice, it is found that mainstream background blur algorithms often need to perform focus target detection and image depth estimation on the image at the same time, and use the results of image depth estimation to guide the blurring of the background area. Image depth estimation often requires a large amount of calculation, which makes it difficult for mainstream background blur algorithms to meet the real-time requirements of scenes such as video shooting or shooting preview, and it is also difficult to achieve the ideal blurring effect on computing platforms with weak computing power. In other words, mainstream background blur algorithms have the problems of large amount of calculation and low blurring efficiency. Summary of the invention

[0004] The embodiments of the present application disclose an image processing method, device, electronic device and storage medium, which can reduce the amount of calculation and improve the blurring efficiency.

[0005] An embodiment of the present application discloses an image processing method, which includes: identifying a portrait area in an original image, and obtaining a portrait mask corresponding to the original image based on the portrait area; determining a first area ratio of the portrait area in the original image based on the portrait mask; performing face detection on the original image to determine face area information contained in the original image; determining a blur parameter based on the first area ratio and the face area information, and blurring the background area of ​​the original image based on the portrait mask and the blur parameter.

[0006] The embodiment of the present application discloses an image processing device, which includes: a portrait recognition module, which is used to recognize a portrait area in an original image, and obtain a portrait mask corresponding to the original image according to the portrait area; and determine a first area ratio of the portrait area in the original image according to the portrait mask; a face recognition module, which is used to perform face detection on the original image and determine face area information contained in the original image; a blurring module, which is used to determine a blurring parameter according to the first area ratio and the face area information, and blur the background area of ​​the original image according to the portrait mask and the blurring parameter.

[0007] An electronic device according to an embodiment of the present application includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements any one of the image processing methods disclosed in the embodiment of the present application.

[0008] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute any one of the image processing methods disclosed in the embodiment of the present application.

[0009] Compared with the related art, the embodiments of the present application have the following beneficial effects:

[0010] The electronic device can perform portrait recognition on the original image. After identifying the portrait area in the original image, the first area ratio of the portrait area in the original image can be determined according to the portrait mask corresponding to the portrait area. In addition, the electronic device can also perform face detection on the original image to obtain the face area information contained in the original image. The first area ratio of the portrait area and the face area information can reflect the physical distance information between the portrait and the face and the imaging device that shoots the original image. Therefore, the electronic device determines the blur parameter according to the first area ratio and the face area information, and blurs the background area of ​​the original image according to the portrait mask and the blur parameter. The background area can be blurred without image depth estimation to achieve the shooting effect of highlighting the portrait subject, which can effectively reduce the amount of calculation and improve the blur efficiency. In addition, since the calculation of the blur parameter integrates the information of the portrait area and the face area, the blur parameter can effectively balance the two different shooting requirements of highlighting the overall portrait and highlighting the local face, thereby optimizing the shooting effect of portrait shooting. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] Figure 1 is a structural schematic diagram of an image processing circuit disclosed in an embodiment of the present application;

[0013] Figure 2 It is a flowchart of an image processing method disclosed in an embodiment of the present application;

[0014] Figure 3 It is a structural schematic diagram of a portrait segmentation model disclosed in an embodiment of the present application;

[0015] Figure 4 is an example diagram of a portrait mask corresponding to an original image disclosed in an embodiment of the present application;

[0016] Figure 5 It is a structural schematic diagram of a face detection model disclosed in an embodiment of the present application;

[0017] Figure 6 is an example diagram of a portrait area and a face area disclosed in an embodiment of the present application;

[0018] Figure 7 It is a flowchart of another image processing method disclosed in an embodiment of the present application;

[0019] Figure 8 is a structural schematic diagram of an image processing device disclosed in an embodiment of the present application;

[0020] Fig. 9 It is a schematic diagram of the structure of another electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0023] In the related art, the background blur method based on depth estimation needs to perform depth estimation on the image first. Image depth estimation can be divided into a hardware-level depth estimation method using a depth measurement sensor and a pure software depth estimation method not using a depth measurement sensor.

[0024] In the hardware-level depth estimation method using a depth measurement sensor, the depth measurement sensor may include a depth camera, a structured light measurement device, etc., but is not limited thereto. Setting a depth measurement sensor on a small or medium-sized mobile terminal such as a smartphone or a smart watch will greatly increase the design difficulty and cost, making it difficult for the background blurring method based on hardware-level depth estimation to be widely used in mobile terminals.

[0025] Although the background blurring method based on pure software depth estimation can avoid the use of depth measurement sensors, the pure software depth estimation method often fits the mapping relationship of mapping the RGB image to the depth map based on the potential correlation between the pixel value and the image depth in the RGB image, so as to estimate the image depth through the mapping relationship. Generally speaking, the computational complexity of the fitting process is high. When the computing power of the electronic device applying the background blurring method is limited, the background blurring method based on pure software depth estimation is difficult to meet the real-time requirements of scenes such as video shooting or shooting preview.

[0026] The embodiments of the present application disclose an image processing method, device, electronic device and storage medium, which can reduce the amount of calculation and improve the blurring efficiency.

[0027] See also Figure 1 , Figure 1 1 is a schematic diagram of the structure of an image processing circuit disclosed in an embodiment of the present application. The image processing circuit can be applied to electronic devices such as smart phones, smart tablets, smart watches, etc., but is not limited thereto. Figure 1 As shown, the image processing circuit may include an imaging device (camera) 110 , a posture sensor 120 , an image memory 130 , an image signal processing (ISP) processor 140 , a control logic 150 , and a display 160 .

[0028] The image processing circuit includes an ISP processor 140 and a control logic 150. The image data captured by the imaging device 110 is first processed by the ISP processor 140, which analyzes the image data to capture image statistics that can be used to determine one or more control parameters of the imaging device 110. The imaging device 110 may include one or more lenses 112 and an image sensor 114. The image sensor 114 may include a color filter array (such as a Bayer filter), and the image sensor 114 may obtain light intensity and wavelength information captured by each imaging pixel and provide a set of raw image data that can be processed by the ISP processor 140. The posture sensor 120 (such as a three-axis gyroscope, a Hall sensor, an accelerometer, etc.) may provide the collected image processing parameters (such as anti-shake parameters) to the ISP processor 140 based on the interface type of the posture sensor 120. The posture sensor 120 interface may use a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

[0029] In addition, the image sensor 114 may also send raw image data to the gesture sensor 120 , and the gesture sensor 120 may provide the raw image data to the ISP processor 140 based on the interface type of the gesture sensor 120 , or the gesture sensor 120 may store the raw image data in the image memory 130 .

[0030] The ISP processor 140 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 140 may perform one or more image processing operations on the raw image data and collect statistical information about the image data. The image processing operations may be performed at the same or different bit depth precisions.

[0031] The ISP processor 140 may also receive image data from the image memory 130. For example, the gesture sensor 120 interface sends raw image data to the image memory 130, and the raw image data in the image memory 130 is then provided to the ISP processor 140 for processing. The image memory 130 may be a part of a memory device, a storage device, or an independent dedicated memory in an electronic device, and may include a DMA (Direct Memory Access) feature.

[0032] When receiving raw image data from the image sensor 114 interface or from the posture sensor 120 interface or from the image memory 130, the ISP processor 140 may perform one or more image processing operations, such as time domain filtering. The processed image data may be sent to the image memory 130 for further processing before being displayed. The ISP processor 140 receives the processed data from the image memory 130 and performs image data processing in the original domain and in the RGB and YCbCr color spaces on the processed data. The image data processed by the ISP processor 140 may be output to the display 160 for viewing by the user and / or further processed by a graphics engine or GPU (Graphics Processing Unit). In addition, the output of the ISP processor 140 may also be sent to the image memory 130, and the display 160 may read the image data from the image memory 130. In one embodiment, the image memory 130 may be configured to implement one or more frame buffers.

[0033] The statistical data determined by the ISP processor 140 may be sent to the control logic 150. For example, the statistical data may include image sensor 114 statistical information such as vibration frequency of a gyroscope, automatic exposure, automatic white balance, automatic focus, flicker detection, black level compensation, lens 112 shading correction, etc. The control logic 150 may include a processor and / or microcontroller that executes one or more routines (such as firmware), and the one or more routines may determine control parameters of the imaging device 110 and control parameters of the ISP processor 140 based on the received statistical data. For example, the control parameters of the imaging device 110 may include attitude sensor 120 control parameters (such as gain, integration time for exposure control, anti-shake parameters, etc.), camera flash control parameters, camera anti-shake displacement parameters, lens 112 control parameters (such as focal length for focus or zoom), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (for example, during RGB processing), and lens 112 shading correction parameters.

[0034] In one embodiment, the original image to be blurred is acquired through the lens 112 and the image sensor 114 in the imaging device (camera) 110, and the original image is sent to the ISP processor 140. The ISP processor 140 obtains the portrait mask corresponding to the original image according to the portrait area, and determines the first area ratio of the portrait area in the original image according to the portrait mask. The ISP processor 140 can simultaneously perform face detection on the original image to determine the face area information contained in the original image. After determining the first area ratio of the portrait area in the original image and the face area information contained in the original image, the ISP processor 140 can determine the blurring parameters according to the first area ratio and the face area information, and blur the background area of ​​the original image according to the portrait mask and the blurring parameters.

[0035] In some embodiments, after the ISP processor 140 determines the blur parameters based on the first area ratio and the face area information, it may also send the blur parameters to the control logic 150, and the control logic 150 adjusts the control parameters of the lens 112 according to the blur parameters to change the focal length of the lens 112, thereby blurring the background area of ​​the original image.

[0036] In some embodiments, after blurring the background area of ​​the original image to obtain a blurred image, the ISP processor 140 may further output the blurred image to the display 160 for viewing by the user.

[0037] See also Figure 2 , Figure 2 is a flowchart of an image processing method disclosed in an embodiment of the present application, Figure 2The method shown can be applied to electronic devices including the above-mentioned image processing circuit, and the electronic devices may include smart phones, smart watches, tablet computers, cameras, etc., but are not limited thereto. Figure 2 As shown, the method may include:

[0038] 210. Identify a portrait region in the original image, and obtain a portrait mask corresponding to the original image according to the portrait region.

[0039] In the embodiment of the present application, the original image may be an image captured by an imaging device of the electronic device, or an image transmitted to the electronic device by other devices through wired or wireless communication, or an image stored in a memory of the electronic device, without specific limitation. The original image may include a variety of different image data, including but not limited to: color data based on the YUV color space, color data based on the RGB color space, and texture data.

[0040] After acquiring the original image, the electronic device can identify the portrait area in the original image through a portrait segmentation method or a portrait matting method. The portrait area may refer to the area in the original image composed of pixel points identified as portraits. The embodiment of the present application does not limit the implementation method of identifying the portrait area from the original image.

[0041] The portrait segmentation method may include but is not limited to: a graph-theory-based segmentation method, a clustering-based segmentation method, a semantic-based segmentation method, an instance-based segmentation method, etc., to identify the portrait area from the original image.

[0042] The portrait mating method may include but is not limited to: a traditional mating method that does not use deep learning (DL) technology and a portrait mating method that uses DL. The segmentation model used in deep learning may include but is not limited to: any segmentation model of the DeepLab series, a U-net, and a fully convolutional network (FCN). Regardless of the model structure used, the segmentation model to be trained can be trained using a portrait data set, so that the segmentation model obtained after training can learn the ability of portrait segmentation, can perform portrait segmentation on the input original image, and identify the portrait area in the original image. Among them, the portrait data set may include multiple sample portrait images annotated with portrait masks; the electronic device may use supervised learning, unsupervised learning, or semi-supervised learning and other training methods to train the segmentation model, which is not limited in the embodiments of the present application.

[0043] For example, see Figure 3 , Figure 3 Schematic diagram of the structure of a portrait segmentation model disclosed in the embodiment of the present application. Figure 3 As shown, the portrait segmentation model 300 may include an encoder 310, a decoder 320 and a normalization (Softmax) layer 330. Among them:

[0044] The encoder 310 may include a plurality of convolutional layers. Optionally, any neural network model in the MobileNet series or the ShuffleNet system may be used as the encoder 310 .

[0045] The decoder 320 may include, but is not limited to: a convolution layer, a deconvolution layer, an upsampling layer, a batch normalization layer (BatchNormalization, BN) layer, and a linear rectification (Rectified Linear Unit, ReLU) layer.

[0046] like Figure 3 As shown, there may also be a jump connection between the network layers included in the encoder 310 and the decoder 320. Based on the above jump connection, the image features extracted by the shallow network included in the encoder 310 and the image features extracted by the deep network included in the decoder 320 can be fused, which is conducive to increasing the generalization ability of the portrait segmentation model.

[0047] After the input original image is sent to the encoder 310, the encoder 310 and the decoder 320 extract features to obtain a prediction value of each pixel in the original image being identified as a portrait area.

[0048] The Softmax layer 330 can be used to normalize the predicted value of each pixel in the original image being identified as a portrait area. That is, it is used to map the predicted value of each pixel in the original image being identified as a portrait area to a probability range of [0,1] to obtain the probability of each pixel in the original image being identified as a portrait area.

[0049] For example, see Figure 4 , Figure 4 is an example diagram of a portrait mask corresponding to an original image disclosed in an embodiment of the present application. Figure 4 As shown, the white pixels correspond to the background area, and the black pixels correspond to the portrait area.

[0050] In the embodiment of the present application, after the electronic device identifies the portrait area in the original image, it can obtain a portrait mask corresponding to the original image according to the identified portrait area, and the portrait mask is used to indicate the portrait area in the original image. After the electronic device generates the portrait mask, if the original image is multiplied by the portrait mask, the portrait area in the original image is retained after the multiplication, and the background area other than the portrait area is eliminated, thereby achieving the effect of cutting out the portrait area from the original image.

[0051] In an embodiment of the present application, the electronic device may compare the probability of each pixel in the original image being identified as a portrait area with the mask threshold to generate a portrait mask of the original image according to the comparison result, and the portrait mask is used to indicate the portrait area in the original image. Exemplarily, the mask threshold may be set to a, a∈[0,1]. The value of the pixel whose probability of being identified as a portrait area in the original image is greater than a in the portrait mask is 255; the value of the pixel whose probability of being identified as a portrait area is less than or equal to a in the portrait mask is 0.

[0052] 220. Determine a first area ratio of the portrait region in the original image according to the portrait mask.

[0053] In an embodiment of the present application, after determining the portrait mask corresponding to the original image, the electronic device can extract the portrait area from the original image through the portrait mask. The electronic device can further calculate the image area of ​​the extracted portrait area, and calculate the ratio between the image area of ​​the extracted portrait area and the area of ​​the original image as the above-mentioned first area ratio. Optionally, the image area of ​​the portrait area can be represented by the number of pixels contained in the portrait area, and the area of ​​the original image can be represented by the number of pixels contained in the original image; accordingly, the ratio between the image area of ​​the portrait area and the area of ​​the original image can be represented by the ratio between the number of pixels contained in the portrait area and the number of pixels contained in the original image.

[0054] In an embodiment of the present application, the electronic device may identify one or more portrait areas in the original image. When the number of portrait areas is greater than 1, the electronic device may add the areas of the portrait areas extracted according to the portrait mask, and use the ratio between the sum of the areas and the area of ​​the original image as the first area ratio of the portrait area. Alternatively, the electronic device may select a portrait area from the extracted multiple portrait areas, and use the ratio between the area of ​​the portrait mask corresponding to the selected portrait area and the area of ​​the original image as the first area ratio of the portrait area. Among them, the selection rules for the electronic device to select a portrait area from multiple portrait areas may include but are not limited to: selecting the portrait area with the largest image area, selecting the portrait area closest to the center area of ​​the original image, and random selection.

[0055] 230. Perform face detection on the original image to determine face region information contained in the original image.

[0056] In an embodiment of the present application, the electronic device can perform face detection through feature matching, machine learning, or deep learning.

[0057] Exemplarily, when performing face detection based on machine learning methods, the electronic device can extract image features such as image texture and gradient, and then use any of the following models to perform face detection: waterfall model, parallel waterfall model, scalar tree model, pyramid model, decision tree model or vector tree model, but not limited to these.

[0058] For example, when performing face detection based on deep learning, the electronic device may use a multi-task convolutional neural network (MT-CNN) face detection method. Figure 5 , Figure 5 Schematic diagram of the structure of a face detection model disclosed in the embodiment of the present application. Figure 5 As shown, the face detection model 500 may include multiple convolutional layers, which may include but are not limited to a batch normalization layer (Batch Normalization, BN) layer and a linear rectification (RectifiedLinear Unit, ReLU) layer. The original image is input to the face detection model 500, and after being processed by each convolutional layer included in the face detection model 500, the face detection model 500 may output one or more 6-dimensional face detection frame information. In each 6-dimensional face detection frame information, 2 dimensions can be used to indicate the probability that the image area contained in the face detection frame is recognized as a real face; the other 4 dimensions can be used to indicate the position of the face frame in the original image, recorded as face_roi[x, y, w, h], x can be the horizontal coordinate of the upper left corner of the face frame in the original image, y can be the vertical coordinate of the upper left corner of the face frame in the original image, w can be used to indicate the width of the face frame, and h can be used to indicate the height of the face frame.

[0059] It should be noted that no matter what method is used for face detection, after the electronic device performs face detection on the original image, the face detection result obtained may include one or more face detection frames for indicating faces. The electronic device may further obtain the second area ratio of the face area in the original image by calculating the ratio between the area of ​​the face detection frame and the area of ​​the original image. Exemplarily, the area of ​​the face detection frame may be determined according to the length and width of the face detection frame included in the face detection frame information. And / or, the electronic device may determine the number of faces in the original image by the number of face detection frames. Therefore, after the electronic device performs face detection on the original image, the face area information obtained may include: the second area ratio of the face area in the original image and / or the number of faces in the original image.

[0060] In an embodiment of the present application, the number of faces in the original image may be greater than 1, and multiple face detection frames may be detected. When the number of faces is greater than 1, the electronic device may add the areas of the face detection frames, and use the ratio between the sum of the areas and the area of ​​the original image as the second area ratio of the face region. Alternatively, the electronic device may select a face detection frame from multiple face detection frames, and use the ratio between the area of ​​the selected face detection frame and the area of ​​the original image as the second area ratio of the portrait region. Among them, the selection rules for the electronic device to select a face detection frame from multiple face detection frames may include but are not limited to: selecting the face detection frame with the largest area, selecting the face detection frame closest to the center area of ​​the original image, and random selection.

[0061] In addition, there is no logical order between the above steps 210 and 230, and the electronic device may also perform the above two steps simultaneously. The portrait area in step 210 may refer to the area occupied by the entire portrait, such as the area occupied by the entire body of a person in a full-body photo. The face area in step 230 may refer to the area occupied by the face, and the face area may include but is not limited to different facial angles such as the full face and the side face. For example, please refer to Figure 6 , Figure 6 is an example diagram of a portrait area and a face area disclosed in an embodiment of the present application. Figure 6 As shown, the portrait area 610 may be the image area occupied by the entire portrait, and the face area 620 may be the area framed by the face detection frame.

[0062] 240. Determine a blur parameter according to the first area ratio and the face region information, and blur the background region of the original image according to the portrait mask and the blur parameter.

[0063] In the embodiment of the present application, the first area ratio of the portrait region in the original image can directly reflect the physical distance information between the portrait and the imaging device that captured the original image. For example, the larger the first area ratio, the closer the physical distance between the portrait and the imaging device; conversely, the smaller the first area ratio, the farther the physical distance between the portrait and the imaging device.

[0064] In an embodiment of the present application, the face region information may include information such as the second area ratio of the face region in the original image and / or the number of faces. The second area ratio of the face region in the original image can directly reflect the physical distance information between the face and the imaging device that captured the original image. For example, the larger the second area ratio, the closer the physical distance between the face and the imaging device; conversely, the smaller the second area ratio, the farther the physical distance between the face and the imaging device. The number of faces in the original image can reflect whether there is a face in the original image.

[0065] In addition, the first area ratio of the portrait area and the number of faces in the original image can also imply information about the angle of the portrait. For example, when the first area ratio is large, but the number of faces in the original image is 0, the portrait captured may be the back or side view.

[0066] In the embodiment of the present application, the blur parameter may include but is not limited to: blur level. The blur level can be used as a parameter indicating the degree of blurring of the background area. The higher the blur level, the more blurred the background area after blurring; conversely, the lower the blur level, the clearer the background area after blurring. The above-mentioned background area may refer to other areas in the original image except the face area or the portrait area.

[0067] In the embodiment of the present application, the blurring process can be implemented by blurring algorithms such as Gaussian blurring, box blurring, and double blurring, but is not limited thereto. For example, when Gaussian blurring is used to blur the background area, the blurring radius of the Gaussian blurring can be used as a measure of the blurring level. The larger the blurring radius, the higher the blurring level.

[0068] In an embodiment of the present application, the electronic device determines the blur parameter based on the first area ratio and the face area information. That is, the blur parameter can be determined by integrating one or more information such as the physical distance information between the portrait and the imaging device, the physical distance information between the face and the imaging device, and the portrait angle information. Therefore, the blur parameter determined by the electronic device can simulate the depth of field law corresponding to the aperture adjustment during portrait shooting. After obtaining the blur parameter, the electronic device can use the portrait mask to segment the background area other than the face area or the portrait area from the original image, and blur the background area through the above-mentioned blur parameters to blur the background area, so as to achieve the shooting effect of highlighting the portrait subject.

[0069] It can be seen that the electronic device does not need to estimate the image depth. It can determine the blur parameters and blur the background area according to the blur parameters only by identifying the portrait area and detecting the face area, thereby reducing the amount of calculation and improving the blur efficiency. In addition, because the calculation of the blur parameters integrates the information of the portrait area and the face area, the blur parameters can effectively balance the two different shooting requirements of highlighting the overall portrait and highlighting the partial face, thereby optimizing the shooting effect of portrait shooting.

[0070] In one embodiment, when the number of faces included in the face region information is not 0, or the second area ratio of the face region included in the face region information in the original image is not 0, there are faces in the original image. It should be noted that when the face region information only includes the above-mentioned second area ratio, the value of the second area ratio may also imply the number of faces. For example, when the second area ratio is not 0, it implies that the number of faces in the original image is not 0. The electronic device may further determine the blur level of the original image based on the first area ratio of the portrait region in the original image and the second area ratio of the face region in the original image. That is, the blur level may be determined jointly based on the area sizes of the portrait region and the face region, and the determined blur level may balance the two different shooting requirements of highlighting the overall portrait and highlighting the partial face.

[0071] Optionally, both the first area ratio and the second area ratio are positively correlated with the blur level of the original image. That is, the larger the first area ratio, the closer the physical distance between the portrait and the imaging device, the higher the blur level, and the more blurred the background area after blurring. The larger the second area ratio, the closer the physical distance between the face and the imaging device, the higher the blur level, and the more blurred the background area after blurring. Based on this, the blur level determined by the electronic device can meet the general rule between the physical distance between the portrait and the face and the imaging device and the depth of field. By using the blur level to blur the background area, the depth of field effect that can be achieved by adjusting the aperture can be simulated.

[0072] Optionally, the electronic device may calculate the first product of the first area ratio of the portrait area in the original image and the first coefficient, and calculate the second product of the second area ratio of the face in the original image and the second coefficient, and then determine the blur level of the original image according to the first product and the second product. Exemplarily, the blur level can be calculated by the following formula: blur level = k1*a+k2*b. Wherein, k1 may be the first coefficient, k2 may be the second coefficient, and k1>0, k2>0; a may be the first area ratio, and b may be the second area ratio. k1 and k2 may be set according to actual business needs and may be empirical values. By adjusting the ratio of k1 and k2, the correlation between the portrait area and the face area and the blur level can be adjusted. For example, when k1>k2, the correlation between the blur level and the first area ratio of the portrait area is greater than the correlation between the blur level and the second area ratio of the face area, and the blur level is more related to the first area ratio of the portrait area; when k1<k2, the blur level is more related to the second area ratio of the face area. It can be seen that by calculating the blur level through the coefficients corresponding to the first area ratio and the second area ratio respectively, the weights of the portrait area and the face area when calculating the blur level can be adjusted according to actual shooting needs, so that the blur effect of the background area can be closer to the blur effect brought by aperture adjustment.

[0073] It can be seen that when the original image includes both a portrait and a face, the electronic device determines the blur level according to the area of ​​the portrait area and the area of ​​the face area in the original image, so that the determined blur level can balance the two different shooting requirements of highlighting the overall portrait and highlighting a partial face.

[0074] In one embodiment, when the number of faces included in the face region information is 0, it means that there is no face in the original image, and the portrait in the original image may be the back of the portrait, the side of the portrait, or the portrait in the original image is not fully photographed. Therefore, the electronic device can further determine the blur level of the original image based on the comparison result of the first area ratio of the portrait region in the original image and the first threshold. The first threshold can be set according to actual business needs and can be an empirical value, for example, it can be set to 0.05.

[0075] Optionally, the electronic device may determine the blur level of the original image to be 0 when the number of faces is equal to 0 and the first area ratio is less than the first threshold. When the first area ratio is less than the first threshold, the area ratio of the portrait area in the original image is too small, and the portrait area identified from the original image may be an erroneous recognition result, or may be an incomplete portrait. Therefore, the electronic device may set the blur level to 0, that is, not blurring the background area, to reduce the amount of calculation increased by unnecessary blur processing.

[0076] Optionally, the electronic device may also determine the blur level of the original image according to the third product of the first area ratio and the third coefficient when the number of faces is equal to 0 and the first area ratio is greater than the first threshold. When the first area ratio is greater than the first threshold, although the face cannot be detected in the original image, the original image still includes a relatively complete portrait, which may be a silhouette or a back view. Therefore, the electronic device may determine the blur level according to the first area ratio of the portrait area, and the blur level and the first area ratio may be positively correlated. Exemplarily, the blur level may be calculated with reference to the following formula: blur level = k3*a; wherein k3 is the third coefficient, k3>0, and a may be the first area ratio. It can be seen that when there is no face in the original image, the electronic device may also calculate the corresponding blur level according to the area of ​​the portrait area to blur the background area, so that when shooting the back or side view of the portrait, the depth of field effect that can be achieved by adjusting the aperture can also be simulated to set off the back or side view of the portrait.

[0077] It can be seen that when the original image only includes a portrait but not a face, the electronic device can also determine the blur level according to the area of ​​the portrait area, so that when shooting scenes such as the side profile or back profile of a portrait, the background area can also be blurred to enhance the shooting effect of the portrait subject.

[0078] In one embodiment, the electronic device may perform a preprocessing operation on the acquired original image before performing the above step 210 or step 230. The preprocessing operation may include but is not limited to: one or more operations of a rotation operation, a scaling operation, and a normalization operation.

[0079] The rotation operation may refer to the operation of rotating a certain pixel point of the original image by a certain angle as the center. The electronic device may determine the shooting direction of the original image according to the width and height of the original image. For example, when the width is greater than the height, the shooting direction of the original image is horizontal; when the height is greater than the width, the shooting direction of the original image is vertical. Alternatively, the shooting direction of the original image may be determined according to the shooting direction value recorded by the imaging device that captured the original image. The shooting direction may include: horizontal or vertical.

[0080] When the electronic device performs step 210 and identifies the portrait area through the portrait segmentation model, and the shooting direction of the input image of the portrait segmentation model is inconsistent with the shooting direction of the original image, the electronic device can perform a rotation operation to make the rotated original image consistent with the shooting direction of the input image of the portrait segmentation model. Alternatively, when the electronic device performs step 230 and performs face detection through the face detection model, and the shooting direction of the input image of the face detection model is inconsistent with the shooting direction of the original image, the electronic device can perform a rotation operation to make the rotated original image consistent with the shooting direction of the input image of the face detection model. The direction of rotation is not limited and can include clockwise rotation or counterclockwise rotation. Exemplarily, the shooting direction of the original image is horizontal, and the horizontal original image is rotated 90° counterclockwise to obtain a vertical original image.

[0081] The scaling operation may refer to the operation of reducing or enlarging the image size of the original image. When the image size of the input image of the above-mentioned portrait segmentation model or face detection model is inconsistent with the image size of the original image, the electronic device may perform a scaling operation on the original image so that the image size of the reduced or enlarged original image is consistent with the image size of the input image of the portrait segmentation model or face detection model. For example, if the image size of the input image of the segmentation model is 640*480, the image size of the original image needs to be reduced or enlarged to 640*480.

[0082] The normalization operation may refer to mapping the image data value of each pixel in the original image to the range of [0,1]. The normalization operation may include: subtracting the mean and then dividing the variance of the values ​​of the RGB three channels corresponding to each pixel in the original image. Exemplarily, assuming that the mean is 127.5, the operation of first subtracting the mean and then dividing the variance for the value X of the RGB channel corresponding to any pixel in the original image can be expressed by the following formula: (X-127.5) / 127.5. Alternatively, the normalization operation may include: directly dividing the values ​​of the RGB three channels corresponding to each pixel in the original image by 255. Exemplarily, for the value X of the RGB channel corresponding to any pixel in the original image, the operation of directly dividing by 255 can be expressed by the following formula: X / 255.

[0083] In an embodiment of the present application, when the original image is processed using a portrait segmentation model or a face detection model, the image specifications of the original image can be first made to conform to the image specifications of the input image of the portrait segmentation model or the face detection model through one or more of the above-mentioned preprocessing operations.

[0084] See also Figure 7 , Figure 7 is a flow chart of another image processing method disclosed in an embodiment of the present application, Figure 7 The method shown can be applied to the electronic device including the above. Figure 7 As shown, the method may include:

[0085] 710. Identify a portrait region in the original image, and obtain a portrait mask corresponding to the original image according to the portrait region.

[0086] In an embodiment of the present application, the electronic device can identify the portrait area from the original image by any one method or a combination of multiple methods included in the portrait segmentation method or the portrait matting method, without specific limitation. The portrait mask obtained by the electronic device based on the portrait area can be a binary image corresponding to the size of the image and the size of the original image, that is, the value of each pixel included in the portrait mask is 0 or 255, and the pixel with a value of 255 is the pixel corresponding to the portrait area.

[0087] 720. Identify connected domains in the portrait mask, and obtain image areas of one or more connected domains included in the portrait mask.

[0088] In the embodiment of the present application, the connected domain in the portrait mask may refer to an area composed of pixels with the same pixel value and adjacent positions in the portrait mask. The electronic device may identify the connected domain in the portrait mask by one or more methods such as seed filling (SeedFilling) and two-pass scanning (Two-Pass), without specific limitation. After identifying one or more connected domains included in the portrait mask, the electronic device may determine the image area of ​​each connected domain according to the number of pixels included in each connected domain.

[0089] 730. Select a connected domain with the largest image area from each connected domain included in the portrait mask, and determine the area ratio of the connected domain with the largest image area in the original image as a first area ratio of the portrait region in the original image.

[0090] In an embodiment of the present application, the electronic device can traverse each connected domain included in the portrait mask, compare the image areas of every two connected domains, and thus select the connected domain with the largest image area. The embodiment of the present application does not limit the specific implementation method of selecting the connected domain with the largest area.

[0091] In some embodiments, in order to reduce the number of comparisons of connected domains and thus reduce the amount of calculation, the electronic device may first screen the connected domains included in the portrait mask, retain the connected domains whose image area is greater than or equal to the second threshold, and then select the connected domain with the largest image area from the retained connected domains. The second threshold may be set according to actual business needs and may be an empirical value, so that the connected domains with smaller areas in the portrait mask are removed by the second threshold and the connected domains with larger areas are retained. Connected domains with smaller areas may be errors caused by misidentification when identifying the portrait area or generating the portrait mask based on the portrait area. Removing the error first and then selecting the largest connected domain can effectively reduce the amount of calculation.

[0092] In an embodiment of the present application, the electronic device may identify one or more portrait areas from the original image, which may be caused by the interference of other portraits in the shooting background. Generally speaking, the physical distance between the portrait as the shooting target and the imaging device is smaller than the physical distance between other portraits and the imaging device. The area of ​​the portrait as the shooting target is larger in the original image and the area in the portrait mask is also larger. Therefore, after selecting the connected domain with the largest image area, the electronic device can determine the area ratio of the connected domain with the largest image area to the original image as the first area ratio of the portrait area, so that the portrait as the shooting target can be screened out from the multiple portraits included in the original image.

[0093] 740. Perform face detection on the original image to obtain one or more face detection frames.

[0094] In the embodiment of the present application, each face detection may carry the probability that the image area contained in the face detection frame is recognized as a real face. The electronic device may perform face detection on the original image by any face detection method or a combination of multiple face detection methods, without specific limitation.

[0095] 750. Screen the face detection frames included in the original image, and retain the face frames whose overlap with the reference face frame is less than a third threshold.

[0096] In the embodiment of the present application, the reference face frame may be a face detection frame with the highest probability of the image area included in each face detection being recognized as a real face. That is, before executing step 750, the electronic device may first select the face detection frame with the highest probability from each face detection frame as the reference face frame based on the probability that the image area included in each face detection frame is recognized as a real face.

[0097] In an embodiment of the present application, after selecting the reference face frame, the electronic device can sequentially calculate the overlap between the reference face frame and the other face frames except the reference face frame in each face detection frame, and retain the face frames with low overlap (less than the third threshold), and remove the face frames with high overlap. The above-mentioned third threshold can be set with reference to actual business needs, and can be an empirical value, which is not specifically limited. The higher the value of the third threshold, the more face frames are retained, and the higher the recall rate of face frame detection; the lower the value of the third threshold, the fewer face frames are retained, and the higher the precision rate of face frame detection. In an embodiment of the present application, the value of the third threshold can be set to balance the recall rate and precision rate of face frame detection.

[0098] Optionally, the Intersection of Union (IoU) between two face detection frames can be used as a measure of the overlap between the two face detection frames. The IoU between two face detection frames may refer to a value obtained by dividing the area of ​​the overlapped portion of the two face detection frames by their combined area. The higher the IoU value, the higher the overlap between the two face detection frames; the lower the IoU value, the lower the overlap between the two face detection frames.

[0099] In some embodiments, after obtaining one or more face detection frames, the electronic device can filter the face frames through any non-maximum suppression (NMS) operation to retain face detection frames whose overlap with the reference face frame is less than a third threshold, without specific limitation.

[0100] 760. Determine the area ratio of the face detection frame with the largest area in the retained face detection frame in the original image as the second area ratio of the face region in the original image, and determine the number of retained face frames as the number of faces in the original image.

[0101] In the embodiment of the present application, the electronic device can traverse the reserved face detection frames, compare the areas of every two reserved face detection frames, and select the face detection frame with the largest area. The embodiment of the present application does not limit the implementation method of selecting the face detection frame with the largest area. After selecting the largest face frame, the electronic device can calculate the area ratio of the face detection frame with the largest area in the original image as the second area ratio of the face area in the original image, so as to screen out the face as the shooting target.

[0102] Furthermore, the electronic device may determine the number of retained face detection frames as the number of faces in the original image, so that the detection of the number of faces is more accurate.

[0103] 770. Determine a blur parameter according to the first area ratio, the second area ratio, and the number of faces.

[0104] In the embodiment of the present application, the blur parameter may include: blur level. Step 770 may include any one or more combinations of the following implementations.

[0105] Method 1: When the number of faces is not 0, the blur level of the original image is determined according to the first area ratio of the portrait area in the original image and the second area ratio of the face area in the original image. Optionally, the blur level can be calculated by the following formula: blur level = k1*a+k2*b. Wherein, k1 can be the first coefficient, k2 can be the second coefficient, and k1>0, k2>0; a can be the first area ratio, and b can be the second area ratio.

[0106] Method 2: When the number of faces is equal to 0 and the first area ratio is less than the first threshold, the blur level is determined to be 0.

[0107] Method 3: When the number of faces is equal to 0 and the first area ratio is greater than or equal to the first threshold, the blur level of the original image is determined according to the third product of the first area ratio and the third coefficient. Optionally, the blur level can be calculated with reference to the following formula: blur level = k3*a; where k3 is the third coefficient, k3>0, and a can be the first area ratio.

[0108] 780. Blur the background area of ​​the original image according to the portrait mask and blur parameters.

[0109] It can be seen that in the embodiment of the present application, the electronic device can accurately identify the portraits and faces as the shooting targets from the original image, and determine the blur parameters based on the information of the portraits and faces as the shooting targets in the original image (such as area ratio and number), thereby eliminating the interference of other portraits in the background, blurring the background area without the need for image depth estimation, and achieving the shooting effect of highlighting the portrait subject as the shooting target, which can effectively reduce the amount of calculation and improve the blur efficiency.

[0110] See also Figure 8 , Figure 8 is a schematic diagram of the structure of an image processing device disclosed in an embodiment of the present application, and the image processing device can be applied to the aforementioned electronic device. Figure 8 As shown, the image processing device 800 may include: a portrait recognition module 810 , a face recognition module 820 , and a blurring module 830 .

[0111] The portrait recognition module 810 may be used to recognize a portrait region in an original image, and obtain a portrait mask corresponding to the original image according to the portrait region; and determine a first area ratio of the portrait region in the original image according to the portrait mask.

[0112] The face recognition module 820 may be used to perform face detection on the original image and determine the face region information contained in the original image.

[0113] The blur module 830 may be used to determine blur parameters according to the first area ratio and the face region information, and blur the background region of the original image according to the portrait mask and the blur parameters.

[0114] In one embodiment, the blur parameter may include a blur level; the face area information may include: a second area ratio of the face area in the original image.

[0115] The blur module 830 may include: a parameter determination unit and a blur processing unit.

[0116] The parameter determination unit may be used to determine the blur level of the original image according to the first area ratio and the second area ratio. Optionally, the first area ratio and the second area ratio are both positively correlated with the blur level of the original image.

[0117] The blur processing unit can be used to blur the background area of ​​the original image according to the portrait mask and blur parameters.

[0118] In one embodiment, the blur module 830 may include: a parameter determination unit and a blur processing unit.

[0119] A parameter determination unit can be used to calculate a first product of a first area ratio of a portrait area and a first coefficient, and to calculate a second product of a second area ratio of a face area and a second coefficient; both the first coefficient and the second coefficient are greater than 0; and, determine the blur level of the original image based on the first product and the second product.

[0120] In one embodiment, the face region information may further include: the number of faces in the original image.

[0121] The parameter determination unit may also be configured to determine the blur level of the original image according to a comparison result between a first area ratio of the portrait region and a first threshold value when the number of faces is equal to 0.

[0122] In one embodiment, the parameter determination unit can also be used to determine the blur level of the original image to 0 when the number of faces is equal to 0 and the first area ratio is less than a first threshold; and / or, when the number of faces is equal to 0 and the first area ratio is greater than or equal to the first threshold, determine the blur level of the original image according to the third product of the first area ratio and the third coefficient.

[0123] In one embodiment, the face recognition module 820 may include: a face detection unit, a first screening unit, and a first determination unit.

[0124] A face detection unit can be used to perform face detection on the original image to obtain one or more face detection frames; each face detection frame carries the probability that the image area contained in the face detection frame is recognized as a real face;

[0125] The first screening unit may be used to screen the face detection frames included in the original image, and retain the face detection frames whose overlap with the reference face frame is less than a third threshold; the reference face frame is the face detection frame whose image area included in the face detection frame has the highest probability of being recognized as a real face;

[0126] The first determining unit may be used to determine an area ratio of a face detection frame with the largest area in the retained face detection frame in the original image as a second area ratio of a face region in the original image.

[0127] Optionally, the first determining unit may also be configured to determine the number of retained face detection frames as the number of faces in the original image.

[0128] In one embodiment, the portrait recognition module 810 may include: a portrait recognition unit and a second determination unit.

[0129] The portrait recognition unit can be used to recognize the portrait area in the original image and obtain the portrait mask corresponding to the original image according to the portrait area.

[0130] The second determination unit can be used to identify the connected domains in the portrait mask to obtain the image area of ​​one or more connected domains included in the portrait mask; and select the connected domain with the largest image area from the connected domains included in the portrait mask; and determine the area ratio of the connected domain with the largest image area in the original image as the first area ratio of the portrait area in the original image.

[0131] In one embodiment, the portrait recognition module 810 may further include: a second screening unit.

[0132] The second screening unit can be used to screen the connected domains included in the portrait mask, retain the connected domains whose image areas are greater than or equal to the second threshold; and select the largest connected domain with the largest image area from the retained connected domains.

[0133] In the embodiment of the present application, the image processing device does not need to estimate the image depth. It can determine the blur parameter and perform blurring of the background area according to the blur parameter only by identifying the portrait area and detecting the face area, thereby reducing the amount of calculation and improving the blurring efficiency. In addition, since the calculation of the blur parameter integrates the information of the portrait area and the face area, the blur parameter can effectively balance the two different shooting requirements of highlighting the overall portrait and highlighting the partial face, thereby optimizing the shooting effect of the portrait.

[0134] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of another electronic device disclosed in the embodiment of the present application. Fig. 9 As shown, the electronic device may include:

[0135] A memory 910 storing executable program codes;

[0136] a processor 920 coupled to the memory 910;

[0137] The processor 920 calls the executable program code stored in the memory 910 to execute any image processing method disclosed in the embodiments of the present application.

[0138] It should be noted that Fig. 9 The electronic device shown may also include a power supply, input buttons, a camera, a speaker, a screen, an RF circuit, a Wi-Fi module, a Bluetooth module, a sensor and other components not shown, which are not described in detail in this embodiment.

[0139] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements any image processing method disclosed in the embodiment of the present application.

[0140] An embodiment of the present application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute any one of the image processing methods disclosed in the embodiments of the present application.

[0141] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present application.

[0142] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0143] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0145] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the above-mentioned methods of various embodiments of the present application.

[0146] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0147] The above is a detailed introduction to an image processing method, device, electronic device and storage medium disclosed in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those of ordinary skill in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Identify a portrait region in an original image, and obtain a portrait mask corresponding to the original image according to the portrait region; Determining a first area ratio of the portrait region in the original image according to the portrait mask; Performing face detection on the original image to determine face region information contained in the original image; the face region information includes a second area ratio of the face region in the original image; Determine a blur parameter according to the first area ratio and the face region information, and perform blur processing on the background region of the original image according to the portrait mask and the blur parameter; The performing face detection on the original image to determine the face region information contained in the original image includes: Performing face detection on the original image to obtain multiple face detection frames; each face detection frame carries a probability that an image area contained in the face detection frame is recognized as a real face; Screening each face detection frame included in the original image, and retaining the face detection frames whose overlap with the reference face frame is less than a third threshold; the reference face frame is a face detection frame whose image area included in each face detection frame has the highest probability of being recognized as a real face; The area ratio of the face detection frame with the largest area among the retained face detection frames in the original image is determined as the second area ratio of the face region in the original image.

2. The method according to claim 1, characterized in that The blur parameter includes a blur level; and determining the blur parameter according to the first area ratio and the face region information includes: The blur level of the original image is determined according to the first area ratio and the second area ratio.

3. The method according to claim 2, characterized in that The first area ratio and the second area ratio are both positively correlated with the blur level of the original image.

4. The method according to claim 3, characterized in that The determining the blur level of the original image according to the first area ratio and the second area ratio includes: Calculating a first product of the first area ratio of the portrait area and a first coefficient, and calculating a second product of the second area ratio of the face area and a second coefficient; the first coefficient and the second coefficient are both greater than 0; The blur level of the original image is determined according to the first product and the second product.

5. The method according to claim 2, characterized in that: The face region information also includes: the number of faces in the original image; the method also includes: When the number of faces is equal to 0, the blur level of the original image is determined according to a comparison result of the first area ratio of the portrait region and a first threshold.

6. The method according to claim 5, characterized in that When the number of faces is equal to 0, determining the blur level of the original image according to a comparison result between a first area ratio of the portrait region and a first threshold value includes: When the number of faces is equal to 0 and the first area ratio is less than a first threshold, determining the blur level of the original image to be 0; and / or, When the number of faces is equal to 0 and the first area ratio is greater than or equal to a first threshold, the blur level of the original image is determined according to a third product of the first area ratio and a third coefficient; the third coefficient is greater than 0.

7. The method according to claim 1, characterized in that After screening the face frames included in the original image and retaining the face frames whose overlap with the reference face frame is less than a third threshold, the method further includes: The number of retained face frames is determined as the number of faces in the original image.

8. The method according to any one of claims 1 to 7, characterized in that: The determining, according to the portrait mask, a first area ratio of the portrait region in the original image includes: Identify a connected domain in the portrait mask, and obtain image areas of one or more connected domains included in the portrait mask; Selecting a connected domain with the largest image area from each connected domain included in the portrait mask; The area ratio of the connected domain with the largest image area in the original image is determined as the first area ratio of the portrait region in the original image.

9. The method according to claim 8, characterized in that The step of selecting the largest connected domain with the largest area from the connected domains included in the portrait mask includes: Screening each connected domain included in the portrait mask, and retaining the connected domain whose image area is greater than or equal to a second threshold; The largest connected domain with the largest image area is selected from the retained connected domains.

10. An image processing device, characterized in that: The device comprises: A portrait recognition module, configured to recognize a portrait region in an original image, and obtain a portrait mask corresponding to the original image according to the portrait region; and determine a first area ratio of the portrait region in the original image according to the portrait mask; A face recognition module, configured to perform face detection on the original image and determine face region information contained in the original image; the face region information includes a second area ratio of the face region in the original image; A blur module, configured to determine a blur parameter according to the first area ratio and the face region information, and blur the background region of the original image according to the portrait mask and the blur parameter; The face recognition module includes: a face detection unit, a first screening unit, and a first determination unit; The face detection unit is used to perform face detection on the original image to obtain multiple face detection frames; each face detection frame carries the probability that the image area contained in the face detection frame is recognized as a real face; The first screening unit is used to screen the face detection frames included in the original image, and retain the face detection frames whose overlap with the reference face frame is less than a third threshold; the reference face frame is the face detection frame whose image area included in the face detection frames has the highest probability of being recognized as a real face; The first determining unit is used to determine the area ratio of the face detection frame with the largest area in the retained face detection frame in the original image as the second area ratio of the face region in the original image.

11. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • Image processing method, apparatus, computer readable storage medium and computer device

    CN107481186A

  • Image virtualization method and mobile terminal

    CN108234882A