Image processing method, device, electronic device and computer readable storage medium

Abstract

The embodiments of the present application disclose an image processing method, device, electronic device and computer-readable storage medium. The method includes: acquiring multiple frames of images to be processed, wherein the multiple frames of images to be processed include at least one bright frame image and at least one dark frame image; performing noise reduction processing on at least one bright frame image, and fusing the bright frame image after noise reduction processing with the at least one dark frame image to obtain a low-light high dynamic range (LLHDR) image; identifying the foreground area in the LLHDR image to obtain a foreground recognition result; blurring the LLHDR image based on the foreground recognition result to obtain a target image. The above-mentioned image processing method, device, electronic device and computer-readable storage medium can improve the blurring effect of the image.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] In the field of imaging technology, in order to highlight the subject in the image, the background area in the image is usually blurred to achieve the effect of making the subject clear. How to improve the blur effect of the image has become a key issue in the field of imaging technology. Summary of the invention

[0003] The embodiments of the present application disclose an image processing method, device, electronic device and computer-readable storage medium, which can improve the blurring effect of an image.

[0004] The embodiment of the present application discloses an image processing method, comprising: acquiring multiple frames of images to be processed, wherein the multiple frames of images to be processed include at least one bright frame image and at least one dark frame image, the bright frame image is an image to be processed whose exposure value is greater than or equal to an exposure threshold, and the dark frame image is an image to be processed whose exposure value is less than the exposure threshold; performing noise reduction processing on at least one frame of the bright frame image, and fusing the bright frame image after the noise reduction processing with the at least one dark frame image to obtain a low-light high dynamic range (LLHDR) image; identifying a foreground area in the LLHDR image to obtain a foreground recognition result; and performing blurring processing on the LLHDR image based on the foreground recognition result to obtain a target image.

[0005] The embodiment of the present application discloses an image processing device, including: an image acquisition module, used to acquire multiple frames of images to be processed, wherein the multiple frames of images to be processed include at least one frame of bright frame image and at least one frame of dark frame image, the bright frame image is an image to be processed whose exposure value is greater than or equal to an exposure threshold, and the dark frame image is an image to be processed whose exposure value is less than the exposure threshold; a high dynamic module, used to perform noise reduction processing on at least one frame of the bright frame image, and fuse the bright frame image after noise reduction processing with the at least one frame of the dark frame image to obtain a low-light high dynamic range (LLHDR) image; a foreground recognition module, used to identify the foreground area in the LLHDR image to obtain a foreground recognition result; and a blur module, used to perform blur processing on the LLHDR image based on the foreground recognition result to obtain a target image.

[0006] An embodiment of the present application discloses an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method described above.

[0007] An embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method described above is implemented.

[0008] The image processing method, device, electronic device and computer-readable storage medium disclosed in the embodiments of the present application obtain multiple frames of images to be processed, wherein the multiple frames of images to be processed include at least one bright frame image and at least one dark frame image, perform noise reduction processing on at least one bright frame image, and fuse the bright frame image after noise reduction processing with the at least one dark frame image to obtain a low-light high dynamic range LLHDR image, identify the foreground area in the LLHDR image, obtain a foreground recognition result, and blur the LLHDR image based on the foreground recognition result to obtain a target image. In the embodiments of the present application, noise reduction and fusion are performed on the image to be processed to obtain an LLHDR image, and foreground recognition and blur processing are performed based on the LLHDR image, which can improve the accuracy of foreground recognition, and make the background area of ​​the blurred target image maintain a high dynamic range, which can improve the blurring effect of the image.

[0009] In addition, the present application provides an image blurring technology from the perspective of the combination of software and hardware. It first uses an exposure meter to control the imaging device through the hardware abstraction layer to collect the image to be processed, and then performs image processing such as fusion and blurring from the software layer, thereby improving the overall expressiveness of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 is a block diagram of an image processing circuit in one embodiment;

[0012] Figure 2 is a flowchart of an image processing method in one embodiment;

[0013] Figure 3 A flowchart of synthesizing an LLHDR image in one embodiment;

[0014] Figure 4 is a schematic diagram of synthesizing an LLHDR image in one embodiment;

[0015] Figure 5 A flowchart of performing point diffusion processing on each first pixel point in the background area of ​​the LLHDR image to obtain a background blurred image in one embodiment;

[0016] Figure 6 is a schematic diagram of superimposing diffusion results of a first pixel in one embodiment;

[0017] Figure 7 A flowchart of performing point diffusion processing on each first pixel point in the background area of ​​the LLHDR image to obtain a background blurred image in another embodiment;

[0018] Figure 8 A flowchart of superimposing a LUT filter on a background blur image in one embodiment;

[0019] Fig. 9 A flowchart of superimposing noise on a background area of ​​a background blur image in one embodiment;

[0020] Fig.10 is a schematic diagram of a normal distribution in one embodiment;

[0021] Fig.11 A schematic diagram of generating a first edge mask in one embodiment;

[0022] Fig.12 is a schematic diagram of a half-pixel value in one embodiment;

[0023] Fig.13 A flowchart of optimizing the edge of the hair region of a portrait mask according to the background complexity of an LLHDR image in one embodiment;

[0024] Fig.14 A schematic diagram of generating a background complexity image in one embodiment;

[0025] Fig.15 is a flowchart of refocusing a portrait on an LLHDR image in one embodiment;

[0026] Fig.16A is a schematic diagram of a clear face area in one embodiment;

[0027] Fig. 16B is a schematic diagram of a weight mask in one embodiment;

[0028] Fig.17 is a block diagram of an image processing device in one embodiment;

[0029] Fig.18 FIG. 4 is a structural block diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present application and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0032] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first pixel may be referred to as a second pixel, and similarly, a second pixel may be referred to as a first pixel. Both the first pixel and the second pixel are pixels, but they do not refer to the same pixel. It should be noted that the terms "multi-frame", "multiple", etc. used in this application refer to "two frames and more than two frames", "two and more than two".

[0033] The embodiment of the present application provides an electronic device, which may include but is not limited to a mobile phone, a smart wearable device, a tablet computer, a PC (Personal Computer), a vehicle terminal, a digital camera, etc., and the embodiment of the present application does not limit this. The electronic device includes an image processing circuit, which can be implemented using hardware and / or software components, and may include various processing units that define an ISP (Image Signal Processing) pipeline. Figure 1 FIG. 1 is a block diagram of an image processing circuit in one embodiment. For ease of explanation, Figure 1 Only various aspects of the image processing technology related to the embodiments of the present application are shown.

[0034] like Figure 1As shown, the image processing circuit includes an ISP processor 140 and a control logic 150. The image data captured by the imaging device 110 is first processed by the ISP processor 140, and the ISP processor 140 analyzes the image data to capture image statistics that can be used to determine one or more control parameters of the imaging device 110. The imaging device 110 may include one or more lenses 112 and an image sensor 114. The image sensor 114 may include a color filter array (such as a Bayer filter), and the image sensor 114 may obtain light intensity and wavelength information captured by each imaging pixel, and provide a set of raw image data that can be processed by the ISP processor 140. The posture sensor 120 (such as a three-axis gyroscope, a Hall sensor, an accelerometer, etc.) can provide the collected image processing parameters (such as anti-shake parameters) to the ISP processor 140 based on the interface type of the posture sensor 120. The posture sensor 120 interface can use a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above interfaces.

[0035] It should be noted that although Figure 1 Only one imaging device 110 is shown in the figure, but in the embodiment of the present application, at least two imaging devices 110 may be included, each imaging device 110 may correspond to one image sensor 114, or multiple imaging devices 110 may correspond to one image sensor 114, which is not limited here. The working process of each imaging device 110 may refer to the above description.

[0036] In addition, the image sensor 114 may also send raw image data to the gesture sensor 120 , and the gesture sensor 120 may provide the raw image data to the ISP processor 140 based on the interface type of the gesture sensor 120 , or the gesture sensor 120 may store the raw image data in the image memory 130 .

[0037] The ISP processor 140 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 140 may perform one or more image processing operations on the raw image data and collect statistical information about the image data. The image processing operations may be performed at the same or different bit depth precisions.

[0038] The ISP processor 140 may also receive image data from the image memory 130. For example, the gesture sensor 120 interface sends raw image data to the image memory 130, and the raw image data in the image memory 130 is then provided to the ISP processor 140 for processing. The image memory 130 may be a part of a memory device, a storage device, or an independent dedicated memory in an electronic device, and may include a DMA (Direct Memory Access) feature.

[0039] When receiving raw image data from the image sensor 114 interface or from the posture sensor 120 interface or from the image memory 130, the ISP processor 140 may perform one or more image processing operations, such as time domain filtering. The processed image data may be sent to the image memory 130 for further processing before being displayed. The ISP processor 140 receives the processed data from the image memory 130 and performs image data processing in the original domain and in the RGB and YCbCr color spaces on the processed data. The image data processed by the ISP processor 140 may be output to the display 160 for viewing by the user and / or further processed by a graphics engine or GPU (Graphics Processing Unit). In addition, the output of the ISP processor 140 may also be sent to the image memory 130, and the display 160 may read the image data from the image memory 130. In one embodiment, the image memory 130 may be configured to implement one or more frame buffers.

[0040] The statistical data determined by the ISP processor 140 may be sent to the control logic 150. For example, the statistical data may include image sensor 114 statistical information such as vibration frequency of a gyroscope, automatic exposure, automatic white balance, automatic focus, flicker detection, black level compensation, lens 112 shading correction, etc. The control logic 150 may include a processor and / or microcontroller that executes one or more routines (such as firmware), and the one or more routines may determine control parameters of the imaging device 110 and control parameters of the ISP processor 140 based on the received statistical data. For example, the control parameters of the imaging device 110 may include attitude sensor 120 control parameters (such as gain, integration time for exposure control, anti-shake parameters, etc.), camera flash control parameters, camera anti-shake displacement parameters, lens 112 control parameters (such as focal length for focus or zoom), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (for example, during RGB processing), and lens 112 shading correction parameters.

[0041] For example, in combination Figure 1The image processing circuit of the present application is used to illustrate the image processing method provided in the embodiment of the present application. The ISP processor 140 can send multiple exposure values ​​contained in the exposure table to the control logic 150 according to the exposure table. The control logic 150 can control the imaging device 110 to collect multiple frames of to-be-processed images corresponding to the multiple exposure values ​​based on the multiple exposure values. The imaging device 110 can send the collected multiple frames of to-be-processed images to the ISP processor 140.

[0042] Among them, the multiple frames of images to be processed may include at least one bright frame image and at least one dark frame image, the bright frame image is an image to be processed whose exposure value is greater than or equal to the exposure threshold, and the dark frame image is an image to be processed whose exposure value is less than the exposure threshold. The ISP processor 140 may perform noise reduction processing on at least one bright frame image, and fuse the bright frame image after noise reduction processing with at least one dark frame image to obtain an LLHDR (low light high dynamic range) image, and then identify the foreground area in the LLHDR image to obtain a foreground recognition result, and blur the LLHDR image based on the foreground recognition result to obtain a target image.

[0043] In some embodiments, the multiple frames of images to be processed collected by the imaging device 110 may be original images in a first image format (such as RAW format). After receiving the multiple frames of original images sent by the imaging device 110, the ISP processor 140 may process the multiple frames of original images to obtain images to be processed in a second image format (such as YUV format or RGB format, etc.). The ISP processor 140 may synthesize the LLHDR image based on the images to be processed in the second image format, and perform foreground recognition and blur processing on the LLHDR image to obtain the target image.

[0044] Optionally, after obtaining the image to be processed in the second image format, the ISP processor 140 may also send the image to be processed in the second image format to other processors in the electronic device, such as a CPU (central processing unit), a GPU (graphics processing unit), and other processors. The other processors may synthesize the LLHDR image based on the image to be processed in the second image format, and perform foreground recognition and blur processing on the LLHDR image to obtain the target image, so as to implement the image processing method provided in the embodiment of the present application.

[0045] like Figure 2 As shown, in one embodiment, an image processing method is provided, which can be applied to the above electronic device. The method may include the following steps:

[0046] Step 210 , obtaining a plurality of frames of images to be processed, wherein the plurality of frames of images to be processed include at least one bright frame image and at least one dark frame image.

[0047] The multiple frames of images to be processed include at least one bright frame image and at least one dark frame image, wherein the bright frame image is an image to be processed whose exposure value is greater than or equal to the exposure threshold, and the dark frame image is an image to be processed whose exposure value is less than the exposure threshold. Optionally, the exposure threshold can be set according to actual needs, for example, the exposure threshold can be an exposure value such as 0.

[0048] In some embodiments, the electronic device may obtain multiple frames of images to be processed from a memory.

[0049] In some embodiments, the electronic device may control the imaging device to collect multiple frames of to-be-processed images corresponding to the multiple exposure values ​​respectively according to the multiple exposure values ​​included in the exposure table. An exposure table may be obtained, and the exposure table may include multiple exposure values, wherein the exposure value (EV) is a numerical value representing the amount of light passing through the camera lens by combining the shutter speed value and the aperture value, and is used to reflect the exposure. According to each exposure value included in the exposure table, the shooting parameters corresponding to each exposure value may be determined, and the shooting parameters may include but are not limited to one or more of the aperture (f value), shutter speed, and exposure time, and the imaging device may be controlled to collect the to-be-processed images corresponding to each exposure value according to the shooting parameters corresponding to each exposure value.

[0050] In an embodiment of the present application, the exposure table may include at least one exposure value greater than or equal to an exposure threshold, and at least one exposure value less than the exposure threshold, so that the multiple frames of images to be processed captured by the imaging device include at least one bright frame image and at least one dark frame image.

[0051] Optionally, when multiple frames of bright frame images are acquired through an imaging device, the exposure values ​​of the multiple frames of bright frame images may be different or the same. Further, the bright frame images in the embodiments of the present application may all be images with an exposure value of 0 (i.e., images with normal exposure). Optionally, when multiple frames of dark frame images are acquired through an imaging device, the exposure values ​​of the multiple frames of dark frame images may be the same or different. For example, the exposure values ​​included in the exposure table are 0, -12, 0, -18, 0, wherein the first, third, and fifth exposure values ​​correspond to images with normal exposure (i.e., bright frame images), and the second and fourth exposure values ​​correspond to dark frame images.

[0052] In some embodiments, the exposure table may be generated based on the scene information that the electronic device is currently in, and the scene information may be acquired and the exposure table may be generated based on the scene information. The scene information may be used to reflect the scene that the electronic device is currently in, and optionally, the scene information may include but is not limited to one or more of the scene type, scene brightness, ISO value, etc.

[0053] Among them, scene types may include night scenes, daytime scenes, indoor scenes, outdoor scenes, etc. Different scene types may correspond to different scene brightness conditions. As an implementation, the current scene type may be determined by a scene classification model. The scene classification model may be trained based on a large number of scene sample images, and each scene sample image may carry a corresponding scene type. A preview image of the current scene may be collected by an imaging device, and the preview image may be input into the scene classification model. The scene classification model may extract image features of the preview image, and determine the scene type corresponding to the preview image based on the image features.

[0054] The scene brightness can be used to characterize the ambient brightness of the scene. As an implementation, a light sensor may be provided in the electronic device, and the ambient light information may be collected by the light sensor, and the scene brightness may be determined based on the ambient light information.

[0055] The ISO value can be used to measure the sensitivity of the film to light. Different ISO values ​​can be used to reflect the brightness of the scene. When the scene is brighter, the ISO value of the imaging device can be smaller, and when the scene is darker, the ISO value of the imaging device can be larger.

[0056] In some embodiments, the exposure value corresponding to each dark frame image may be determined based on the scene information. In a night scene or a dark scene with low ambient brightness, the light source may be overexposed in the image captured by the imaging device. Therefore, it is necessary to fuse the bright frame image with the dark frame image to obtain the LLHDR image. The exposure value corresponding to each dark frame image may be adaptively processed based on the scene information. If the scene information indicates that the current scene is brighter, the exposure value corresponding to the dark frame may be relatively large. If the scene information indicates that the current scene is darker, the exposure value corresponding to the dark frame may be relatively small.

[0057] For example, when the ISO value is greater than 3000, the exposure value corresponding to the dark frame may be -32, and when the ISO value is in the range of 500 to 3000, the exposure value corresponding to the dark frame may be -18, but is not limited thereto. Adaptively adjusting the exposure value corresponding to the dark frame according to the scene information can make the acquired dark frame image adapt to the brightness of the current scene, avoid overexposure or overdarkness of the subsequently synthesized LLHDR image, and improve the image effect of the LLHDR image.

[0058] In some embodiments, after the exposure table is generated, the exposure table can be sent to the HAL (Hardware Abstraction Layer) in the operating system. The HAL layer is located at the interface layer between the operating system kernel and the hardware. Its purpose is to abstract the hardware and hide the hardware interface details of the platform. The image capture function of the imaging device can be called through the HAL layer. After the exposure table is passed to the HAL layer, the HAL layer can control the imaging device to collect the to-be-processed images corresponding to the exposure values ​​contained in the exposure table.

[0059] Step 220 , performing noise reduction processing on at least one bright frame image, and fusing the noise reduced bright frame image with at least one dark frame image to obtain a low light high dynamic range LLHDR image.

[0060] Since the bright frame image has a large image brightness and contains large image noise, at least one bright frame image may be subjected to noise reduction processing first, and then the noise-reduced bright frame image is fused with at least one dark frame image to obtain an LLHDR image with a high dynamic range. The LLHDR image will not have partial image area overexposure, thereby ensuring the accuracy of subsequent foreground recognition and blur processing effect.

[0061] Step 230, identifying the foreground area in the LLHDR image to obtain a foreground identification result.

[0062] The LLHDR image can be subjected to image recognition to obtain the foreground area and background area of ​​the LLHDR image. The foreground area may refer to the image area where the target object in the LLHDR image is located, and the target object refers to the object of interest in the LLHDR image. For example, the LLHDR image may be a person image, and the target object may be a portrait in the person image, and the foreground area may refer to the portrait area. The LLHDR image may be an animal image, and the target object may be an animal in the animal image, and the foreground area may refer to the animal area. The LLHDR image may also be a building image, and the target object may be a building in the building image, and the foreground area may refer to the image area where the building is located, etc., but is not limited thereto. The background area may refer to the image area other than the foreground area in the LLHDR image.

[0063] The electronic device can perform foreground recognition on the LLHDR image to obtain a foreground recognition result, which can be used to determine the image position of the foreground area in the LLHDR image. Optionally, the foreground recognition result can mark each pixel point in the LLHDR image as belonging to the foreground area or the background area.

[0064] As an implementation method, the electronic device may obtain a depth map corresponding to the LLHDR image, and the depth map may include depth information of each pixel in the LLHDR image, and the depth information may be used to characterize the distance between the photographed object and the camera. The greater the depth information, the farther the distance. The depth information corresponding to the foreground area and the background area is quite different. Therefore, the foreground area and the background area in the LLHDR image may be divided according to the depth map. For example, the background area may be an area composed of pixels whose depth information is greater than a first threshold, and the foreground area may be an area composed of pixels whose depth information is less than a second threshold, etc.

[0065] As another embodiment, the electronic device may also use a neural network to determine the foreground area in the LLHDR image, and the LLHDR image may be input into a pre-trained object segmentation model, and the target object contained in the LLHDR image may be identified by the object segmentation model to obtain the foreground area corresponding to the target object. The object segmentation model may be trained based on a large number of sample training images, and each sample training image may be annotated with a foreground area. The object segmentation model may include but is not limited to a network based on a deeplab semantic segmentation algorithm, a U-Net network structure, FCN (Fully Convolutional Networks, Fully Convolutional Neural Network), etc., which are not limited here.

[0066] It should be noted that other methods may also be used to identify the foreground area in the LLHDR image, and the method of identifying the foreground area is not limited in the embodiment of the present application.

[0067] Step 240: blur the LLHDR image based on the foreground recognition result to obtain a target image.

[0068] Among them, the blurring process can be implemented by using a Gaussian filter, mean blurring process, median blurring process, etc., which are not limited here. In some embodiments, the foreground area and the background area in the LLHDR image can be determined according to the foreground recognition result, and the background area in the LLHDR image can be blurred to obtain the target image.

[0069] In some embodiments, the electronic device may divide the background area of ​​the LLHDR image according to the depth map of the LLHDR image, and divide the pixels in the background area with the same or similar depth information into the same image area to obtain multiple background sub-areas. The blur parameters corresponding to each background sub-area may be determined according to the depth information corresponding to each background sub-area. The blur parameters may be used to describe the degree of blur, for example, may include blur strength, blur coefficient and other parameters, and then blur each background sub-area according to the blur parameters corresponding to each background sub-area. Optionally, a background sub-area with a larger depth information may correspond to a larger degree of blur, and a background sub-area with a smaller depth information may correspond to a smaller degree of blur, so that different background sub-areas may be blurred to different degrees, respectively, to improve the blur effect of the image.

[0070] In some embodiments, the electronic device may also first perform a blurring process on the entire LLHDR image, and then fuse the blurred LLHDR image with the LLHDR image before blurring based on the foreground recognition result to obtain a target image. The fusion method may include but is not limited to fusion by taking the mean, fusion by assigning different weight coefficients, Alpha fusion processing, etc. Taking Alpha fusion processing as an example, Alpha fusion processing may assign an Alpha value to each pixel in the LLHDR image before blurring and the LLHDR image after blurring, so that the LLHDR image before blurring and the LLHDR image after blurring have different transparency. For example, the pixel value of the pixel in the background area in the foreground recognition result is 0, and the foreground recognition result can be used as the Alpha value of the LLHDR image before blurring, and the LLHDR image after blurring is fused with the LLHDR image before blurring. Since the foreground area is marked in the foreground recognition result, the foreground area in the fused target image may be the image content of the LLHDR image before blurring, and the background area in the target image may be the image content of the LLHDR image after blurring, which can improve the situation of false blurring of the foreground area or leakage of the background area, and improve the blurring accuracy of the target image.

[0071] In an embodiment of the present application, an image blurring technology is provided from the perspective of the combination of software and hardware. First, an exposure meter is used to control an imaging device at the hardware layer to collect an image to be processed. Then, noise reduction and fusion are performed on the image to be processed at the software layer to obtain an LLHDR image. Foreground recognition and blurring processing are performed based on the LLHDR image. The accuracy of foreground recognition can be improved, and the background area of ​​the blurred target image can maintain a high dynamic range, which can improve the blurring effect of the image.

[0072] In one embodiment, the multiple frames of images to be processed acquired by the imaging device may include multiple frames of original images in a first image format acquired by the image sensor. After the step of controlling the imaging device to acquire the multiple frames of images to be processed corresponding to multiple exposure values, the step may also include: optimizing the multiple frames of original images through the ISP to obtain multiple frames of images to be processed in a second image format.

[0073] The first image format may include a RAW format, such as the original image may be original image data arranged in a Bayer array, etc. The second image format includes a YUV format or an RGB format, etc. After acquiring the original image corresponding to each exposure value, the image sensor may send the original image to the ISP, and the ISP may optimize the multiple frames of original images transmitted by the image sensor to obtain an image to be processed in a YUV (brightness-chrominance) format or an RGB (red-green-blue) format.

[0074] Optionally, the above optimization process may include but is not limited to one or more of the following optimization methods:

[0075] Method 1: Color optimization. ISP can optimize the color of the original image. This color optimization may include improving the contrast of the original image, improving the saturation of the original image, etc. Contrast refers to the measurement of different brightness levels between the brightest white and the darkest black in the light and dark areas of an image, and saturation can be used to characterize the vividness of the color of the image. Color optimization can improve the color expression of the original image.

[0076] Optionally, when the original image contains a portrait, the ISP may also perform color protection on the face area in the original image, and only perform contrast adjustment and saturation adjustment on the image area other than the face area to improve the uniformity of skin color in the face area.

[0077] Method 2: Exposure optimization. ISP can adjust the exposure of the original image to reduce the brightness of the background area, which can avoid overexposure and unnatural appearance of the background area. Optionally, for backlit or dark scenes, the foreground area may be too dark (i.e., the foreground area is too dark). In particular, for portrait images taken in backlit or dark scenes, adjusting the exposure of the original image may cause the foreground face to be too dark.

[0078] Therefore, as an implementation method, when it is recognized that the current scene is backlit or dark (such as a night scene, indoor scene, etc.), the ISP can determine the target exposure based on the brightness histogram of the original image, and then adjust the exposure of the original image to the target exposure. The brightness histogram represents the number of pixels at each brightness level of the original image, which can be used to reflect the overall brightness distribution of the original image, thereby balancing the background area and the foreground area.

[0079] As another implementation, when it is recognized that the current scene is backlit or dark (such as a night scene, an indoor scene, etc.), exposure optimization can be performed only on the image area in the original image whose brightness is greater than a first brightness threshold, which not only reduces the overexposure of the original image, but also avoids adverse effects on the foreground area.

[0080] Optionally, the ISP can determine whether the current scene is a backlit scene or a dark scene based on scene information of the current scene. The ISP can also extract the brightness features of the original image based on the brightness histogram of the original image, and determine whether the current scene is a backlit scene or a dark scene based on the brightness features. For example, the brightness histogram reflects that the pixels in the original image are mainly distributed in two brightness intervals with large differences, which can indicate that the original image contains overly bright image areas and overly dark image areas, and is in a backlit scene. It should be noted that other methods can also be used to identify whether the current scene is a backlit scene or a dark scene, and the embodiments of the present application are not limited to this.

[0081] Method three: noise optimization. The ISP can perform noise reduction processing on the original image, and the noise reduction processing may include but is not limited to median filtering noise reduction, mean filtering noise reduction, Gaussian filtering noise reduction, etc., but is not limited thereto. Optionally, when the ISO value is large, the original image acquired by the imaging device contains more noise. Therefore, the ISP can obtain the ISO value corresponding to the original image and determine the noise reduction parameters according to the ISO value. The noise reduction parameters may include the intensity of noise reduction, etc. The greater the intensity of noise reduction, the better the noise reduction effect, but it may cause the loss of details in the image. The smaller the intensity of noise reduction, the worse the noise reduction effect, but more details in the image can be retained. The noise reduction parameters can be adaptively adjusted according to the ISO value to reduce the image noise in the original image while retaining more image details.

[0082] After optimizing the multiple frames of original images through the ISP to obtain multiple frames of images to be processed in the second image format, the bright frame images in the multiple frames of images to be processed in the second image format can be subjected to noise reduction processing, and the bright frame images after noise reduction processing are fused with the dark frame images in the multiple frames of images to be processed in the second image format to obtain a low-light high dynamic range LLHDR image. Optionally, the synthesis of LLHDR images and subsequent image blurring and other processing can be performed on the ISP, or the ISP can transfer the images to be processed in the second image format to other processors such as the CPU and GPU, and the CPU, GPU and other processors perform the synthesis of LLHDR images and subsequent image blurring and other processing.

[0083] In an embodiment of the present application, after the image sensor captures the original image, the basic image quality can be adjusted by the ISP, and then the LLHDR image can be synthesized, which can improve the image quality of the synthesized LLHDR image and ensure the subsequent image blur effect.

[0084] like Figure 3 As shown, in one embodiment, the multiple frames of images to be processed collected by the imaging device may include at least two bright frame images and at least one dark frame image. The step of performing noise reduction processing on at least one bright frame image and fusing the bright frame image after noise reduction processing with at least one dark frame image to obtain a low-light high dynamic range LLHDR image may include the following steps:

[0085] Step 302: Select a reference frame image from at least two bright frame images.

[0086] A reference frame image can be selected from at least two bright frame images. The reference frame image can be any one of the at least two bright frame images, or a bright frame image with a fixed frame number. For example, three bright frame images are collected each time, and the second bright frame image can be selected as the reference frame image.

[0087] Step 304 , aligning at least two bright frame images to obtain a first aligned region and a first non-aligned region between the reference frame image and other bright frame images.

[0088] In some embodiments, at least two bright frame images may be registered and aligned by detecting feature points of each bright frame image respectively and matching the feature points of each frame image, and obtaining a first alignment region and a first non-alignment region between the reference frame image and other bright frame images according to the matching result. The feature points may be detected using an ORB (Oriented FAST and Rotated BRIEF) algorithm, a SIFT (Scale-invariant feature transform) algorithm, etc., and the specific feature detection algorithm is not limited in the embodiments of the present application.

[0089] It should be noted that other methods can also be used to perform image registration and alignment. For example, at least two bright frame images can be input into a neural network and the images can be registered and aligned through the neural network to obtain a first aligned area and a first non-aligned area between the reference frame image and other bright frame images.

[0090] The first alignment region may refer to an image region in the reference frame image that matches other bright frame images, the similarity between the first alignment region in the reference frame image and some image regions in other bright frame images is greater than a similarity threshold, and the other bright frame images refer to bright frame images other than the reference frame image in at least two bright frame images. Optionally, the first alignment region may be an image region in the reference frame image that matches all other bright frame images, or may be an image region in the reference frame image that matches some other bright frame images.

[0091] The first non-aligned area may be an image area other than the first aligned area in the reference frame image, and may refer to an image area in the reference frame image that does not match other bright frame images.

[0092] Step 306 , based on at least two bright frame images, perform noise reduction processing on the first aligned region in the reference frame image, and perform single-frame noise reduction processing on the first non-aligned region of the reference frame image to obtain a noise-reduced image.

[0093] For the first alignment area in the reference frame image, noise reduction processing can be performed between multiple frame images, and the first alignment area in the reference frame image is subjected to noise reduction processing based on the bright frame image matching the first alignment area. Optionally, the noise reduction processing between the multiple frame images may include mean noise reduction processing, Gaussian noise reduction processing, median noise reduction processing, etc. between the multiple frame images, but is not limited thereto. Taking mean noise reduction processing as an example, for each pixel point X in the first alignment area in the reference frame image, the corresponding pixel point Y can be found in each other bright frame image matching the first alignment area, and the pixel value of each pixel point X in the first alignment area can be added to the pixel value of the corresponding pixel point Y in each other bright frame image to obtain the average value, so as to obtain the pixel value of each pixel point X in the first alignment area in the reference frame image after noise reduction.

[0094] For the first non-aligned area in the reference frame image, a single-frame image denoising process can be performed, and the first non-aligned area can be directly denoised without referring to other bright frame images. Optionally, the denoising process of the single-frame image can include mean denoising process, Gaussian denoising process, median denoising process, etc. in the first non-aligned area, and the inpaint function can also be used to perform denoising on the first non-aligned area. Taking mean denoising process as an example, for each pixel point M in the first non-aligned area in the reference frame image, the pixel value of each pixel point in the neighborhood can be added to the average value to obtain the pixel value of each pixel point M in the first non-aligned area after denoising. For the first aligned area and the first non-aligned area of ​​the reference frame, different denoising strategies are used for denoising, which can effectively reduce noise and improve the accuracy of denoising, thereby improving the denoising effect.

[0095] In some embodiments, before performing noise reduction processing on the reference frame image, the noise reduction parameters corresponding to the reference image may be determined according to the scene information. The noise reduction parameters may include noise reduction strength, etc. When the scene information indicates that the current scene is brighter, the noise generated in the reference image will be less, which may correspond to a smaller noise reduction strength. When the scene information indicates that the current scene is darker, the noise generated in the reference image will be more, which may correspond to a larger noise reduction strength. Further, the noise reduction strength may be positively correlated with the ISO value.

[0096] After determining the noise reduction parameters, noise reduction processing between multiple frames can be performed on the first aligned area in the reference frame image according to the noise reduction parameters, and single-frame noise reduction processing can be performed on the first non-aligned area of ​​the reference frame image according to the noise reduction parameters to obtain a noise-reduced image.

[0097] Taking the mean denoising process between multiple frames as an example, the denoising weights corresponding to the reference frame image and the matched other bright frame images can be allocated according to the denoising parameters. For each pixel point X in the first alignment area in the reference frame image, the pixel values ​​of each pixel point X in the first alignment area and the pixel values ​​of the corresponding pixel points Y in the matched other bright frame images can be weighted averaged according to the denoising weights corresponding to the reference frame image and the matched other bright frame images, to obtain the pixel values ​​of each pixel point X after denoising. Optionally, the denoising strength and the denoising weight corresponding to the reference frame image can be negatively correlated. When the denoising strength is large, the denoising weight corresponding to the reference frame image can be small.

[0098] In an embodiment of the present application, the noise reduction parameters of the bright frame image can be adaptively adjusted according to the scene information to ensure that the obtained noise reduction image retains more image details and has a better noise reduction effect.

[0099] Step 308 , registering and aligning the denoised image and at least one dark frame image to obtain a second aligned region and a second non-aligned region between the denoised image and the at least one dark frame image.

[0100] The noise reduction image may be registered and aligned with each dark frame image in a manner similar to the manner of registering and aligning at least two bright frame images, and will not be described in detail herein.

[0101] The second aligned area of ​​the denoised image refers to the image area in the denoised image that matches the dark frame image, and the second non-aligned area is the image area in the denoised image other than the second aligned area. The second non-aligned area refers to the image area in the denoised image that does not match each dark frame image.

[0102] Step 310, based on the denoised image and at least one dark frame image, a chromaticity adjustment process is performed on the second aligned region in the denoised image, and a single-frame color enhancement process is performed on the second non-aligned region of the denoised image to obtain a LLHDR image.

[0103] For the second aligned area in the denoised image, the chromaticity adjustment processing between multiple frames can be performed with reference to the matched dark frame images, and the second aligned area in the denoised image can be chromaticity-fused with the matched image area in each dark frame image to restore the color of the denoised image, and to suppress the highlight area, thereby avoiding overexposure of the LLHDR image. Optionally, the chromaticity adjustment processing may include chromaticity mean processing between multiple frames of images, etc. Taking the image in which both the bright frame image and the dark frame image are in YUV format as an example, for each pixel point P in the second aligned area in the denoised image, the corresponding pixel point Q can be found in each matched dark frame image, and the UV value (brightness) of each pixel point P in the second aligned area can be added to the UV value of the corresponding pixel point Q in each matched dark frame image to obtain the average value, so as to obtain the UV value of each pixel point P in the second aligned area after chromaticity adjustment processing.

[0104] In some embodiments, before the denoised image is subjected to chromaticity adjustment processing, a first saturation parameter corresponding to the denoised image may be determined according to the scene information, and the first saturation parameter may be used to characterize the color saturation required for LLHDR. Usually, in darker scenes (such as night scenes, etc.), colorful spots are required when blurring the image. Therefore, when the scene information characterizes that the current scene is darker, a larger first saturation parameter may correspond, and when the scene information characterizes that the current scene is brighter, a smaller first saturation parameter may correspond. Furthermore, the first saturation parameter may be positively correlated with the ISO value.

[0105] According to the first saturation parameter determined above, based on the denoised image and at least one dark frame image, the second aligned area in the denoised image can be subjected to chromaticity adjustment processing. Specifically, the chromaticity weights corresponding to the denoised image and each dark frame image can be determined according to the first saturation parameter, and according to the denoised image, each dark frame image, and the chromaticity weights corresponding to the denoised image and each dark frame image, the second aligned area in the denoised image is subjected to chromaticity fusion with the matching image area in each dark frame image. Optionally, the chromaticity weight corresponding to the denoised image can be negatively correlated with the first saturation parameter, and the larger the first saturation parameter, the smaller the chromaticity weight corresponding to the denoised image can be.

[0106] For example, for each pixel point P in the second alignment area in the denoised image, the UV value of each pixel point P in the second alignment area and the UV value of the corresponding pixel point Q in each matched dark frame image can be weighted averaged according to the chromaticity weights corresponding to the denoised image and each dark frame image, to obtain the UV value of each pixel point P in the second alignment area after chromaticity adjustment. By adaptively adjusting the first saturation parameter through scene information, the LLHDR image can be made to meet the needs of different scenes, thereby improving the image quality of LLHDR.

[0107] For the second non-aligned area of ​​the denoised image, a single frame color enhancement process can be performed. As a specific implementation, the second non-aligned area of ​​the denoised image can be color corrected based on a gamma curve. The gamma curve is a special tone curve. Color correction of the second non-aligned area of ​​the denoised image based on the gamma curve can improve the situation where the second non-aligned area is too bright or too dark, so that the second non-aligned area has a richer color effect.

[0108] For example, Figure 4 FIG. 1 is a schematic diagram of synthesizing an LLHDR image in one embodiment. Figure 4 As shown, the imaging device can acquire 3 frames of bright frame images and 2 frames of dark frame images, and the bright frame image 2 can be used as the reference frame image, and the 3 frames of bright frame images are registered and aligned to determine the first alignment area and the first non-alignment area in the bright frame image 2, and the above-mentioned noise reduction processing is performed on the first alignment area and the first non-alignment area in the bright frame image 2, respectively, to obtain a denoised image. Then, the denoised image is registered and aligned with the 2 frames of dark frame images to determine the second alignment area and the second non-alignment area in the denoised image, and the chromaticity adjustment processing is performed on the second alignment area in the denoised image in combination with the 2 frames of dark frame images, and the single-frame color enhancement processing is performed on the second non-alignment area in the denoised image to obtain a LLHDR image.

[0109] In an embodiment of the present application, a reference frame image selected from a plurality of bright frame images is first subjected to denoising processing to obtain a denoised image, and then the denoised image is fused with the dark frame image, and chroma adjustment and color adjustment and other processing are performed on the denoised image. The highlight area in the denoised image can be suppressed to obtain an LLHDR image with denoising and highlight suppression effects, thereby improving the image effect of the LLHDR image, and further improving the image effect of the target image obtained after subsequent blurring of the LLHDR.

[0110] In one embodiment, the step of blurring the LLHDR image based on the foreground recognition result to obtain the target image may include: blurring the background area of ​​the LLHDR image based on the foreground recognition result to obtain a background blurred image; based on the foreground recognition result, fusing the background blurred image with the LLHDR image to obtain the target image.

[0111] like Figure 5 As shown, in one embodiment, the step of blurring the background area of ​​the LLHDR image based on the foreground recognition result to obtain a background blurred image may include the following steps:

[0112] Step 502: Obtain a blur intensity map of the LLHDR image according to the foreground recognition result, and obtain a diffusion weight map corresponding to the LLHDR image.

[0113] The LLHDR image may include a foreground area and a background area, and a pixel in the background area may be defined as a first pixel, and a pixel in the foreground area may be defined as a second pixel. The blur strength map of the LLHDR image may include a blur radius corresponding to each first pixel in the background area of ​​the LLHDR image, and the blur radius may be used to indicate the blur strength. The larger the blur radius, the greater the blur strength, and the more blurred the image after blur processing. In the blur strength map corresponding to the LLHDR image, the blur radius corresponding to different first pixels may be the same or different.

[0114] As an implementation method, the blur intensity map can be generated according to the depth map of the LLHDR image, and the electronic device can perform depth estimation on the LLHDR image through structured light, time of flight (TOF), binocular stereo imaging, monocular phase detection, monocular depth estimation based on deep learning or machine learning, etc., to obtain the depth map of the LLHDR image, and determine the blur radius corresponding to each first pixel point according to the depth information corresponding to each first pixel point in the depth map. Optionally, the depth information of the first pixel point may be positively correlated with the blur radius corresponding to the first pixel point.

[0115] As another embodiment, the blur intensity map can also be generated according to a preset blur intensity parameter. The preset blur intensity parameter can be obtained, the target blur radius can be determined according to the blur intensity parameter, and the blur intensity map of the LLHDR image can be generated according to the target blur radius and the foreground recognition result. Among them, the blur radius of each second pixel point in the foreground area of ​​the LLHDR image in the blur intensity map is 0, and the blur radius of each first pixel point in the background area of ​​the LLHDR image in the blur intensity map is the target blur radius. The blur intensity parameter can be uniformly set by the electronic device before leaving the factory, or it can be set by the user according to actual needs, which is not limited here. Optionally, the target blur radius corresponding to the blur intensity parameter can be determined by looking up a table or the like. It can make the blur radius of all pixel points in the background area of ​​the LLHDR image the same, ensure the consistency of the light spot in the background area during the subsequent blur processing, and can better simulate the image effect of the SLR with a large aperture and extremely narrow depth of field, thereby improving the image blur effect.

[0116] The diffusion weight map of the LLHDR image may include the diffusion weights corresponding to each first pixel in the background area of ​​the LLHDR image. The diffusion weight map may be used to indicate that each first pixel in the background area is subjected to point diffusion processing, and the diffusion weights corresponding to each first pixel may affect the proportion of the first pixel in the point diffusion processing and superposition.

[0117] In some embodiments, the diffusion weight map of the LLHDR image can be generated according to the pixel value of each pixel in the LLHDR image. Taking the LLHDR image as an image in YUV format as an example, it can be generated according to the Y (brightness) component value and UV (chrominance) component value of the pixel of the LLHDR.

[0118] Step 504 , performing point diffusion processing on each first pixel point according to the blur strength map and the diffusion weight map, to obtain a diffusion result corresponding to each first pixel point.

[0119] Each first pixel point in the background area of ​​the LLHDR image can be traversed to obtain the blur radius of each first pixel point in the blur intensity map and the diffusion weight of each first pixel point in the diffusion weight map, and perform point diffusion processing on each first pixel point according to the blur radius and the diffusion weight of each first pixel point to obtain the diffusion result corresponding to each first pixel point.

[0120] Point diffusion processing may refer to simulating real lens blur through a point spread function (PSF). Through point diffusion processing, the pixel value of each first pixel point in the LLHDR image may be diffused according to the diffusion weight and blur radius corresponding to each first pixel point to form a light spot.

[0121] In some embodiments, the diffusion result corresponding to the first pixel point may include the diffusion range corresponding to the first pixel point, the target pixel value of each third pixel point within the diffusion range corresponding to the first pixel point, and the diffusion times of each third pixel point within the diffusion range corresponding to the first pixel point. The diffusion range corresponding to the first pixel point refers to the image area range with the first pixel point as the center and the blur radius as the circumscribed circle radius, and the outer contour of the diffusion range may be a preset aperture shape (such as a circle, a five-pointed star, a square, etc.). The above-mentioned third pixel point may be any pixel point within the diffusion range corresponding to the first pixel point, and the target pixel value of the third pixel point may refer to the pixel value of the third pixel point after the first pixel point is subjected to point diffusion processing.

[0122] Step 506, superimposing the diffusion results corresponding to the first pixel points, and blurring the background area of ​​the LLHDR image according to the superimposed results to obtain a background blurred image.

[0123] The diffusion results corresponding to each first pixel point in the background area of ​​the LLHDR image may be superimposed, and the superposition may include the superposition of pixel values ​​and the superposition of diffusion times. In some embodiments, a pixel value superposition map and a frequency superposition map may be generated, and the pixel value superposition map may be used to record the superimposed pixel values ​​of each first pixel point in the LLHDR image after point diffusion processing, and the frequency superposition map may be used to record the cumulative diffusion times of each first pixel point. The initial pixel value of each first pixel point in the pixel value superposition map may be 0, and the initial value of each first pixel point in the frequency superposition map may also be 0.

[0124] For each first pixel point, after performing point diffusion processing on the current first pixel point, the target pixel values ​​of each third pixel point within the diffusion range of the current first pixel point can be accumulated to the pixel value overlay image, and the number of diffusion times corresponding to each third pixel point can be accumulated to the number overlay map.

[0125] For example, Figure 6 FIG. 1 is a schematic diagram of superimposing the diffusion results of the first pixel in one embodiment. Figure 6As shown, the pixel point X1 and the pixel point X2 can be subjected to point diffusion processing respectively, and their diffusion ranges are both 3*3. The pixel point X1 is subjected to point diffusion processing to obtain a diffusion result 602. From the diffusion result 602, it can be known that the target pixel values ​​of each pixel point within the diffusion range corresponding to the pixel point X1 can be A1-A9, wherein the target pixel value of the pixel point X1 is A5. The pixel point X2 is subjected to point diffusion processing to obtain a diffusion result 604. From the diffusion result 604, it can be known that the target pixel values ​​of each pixel point within the diffusion range corresponding to the pixel point X2 can be B1-B9, wherein the target pixel value of the pixel point X2 is B5. The target pixel values ​​of each pixel point in the diffusion result 602 and the target pixel values ​​of each pixel point in the diffusion result 604 can be accumulated into the pixel value superposition diagram 610, wherein the pixels in the third column are in the diffusion ranges of the pixel point X1 and the pixel point X2 at the same time, so the target pixel values ​​are superimposed.

[0126] After the diffusion results corresponding to each first pixel are superimposed, the background area of ​​the LLHDR image can be blurred according to the superposition results. Further, the blurred pixel value of each first pixel can be calculated according to the pixel value superposition map and the frequency superposition map to achieve a blurred effect. As an implementation method, the total superposition pixel value of each first pixel can be obtained according to the pixel value superposition map, the total number of superpositions of each first pixel can be obtained according to the frequency superposition map, and the total superposition pixel value of each first pixel can be divided by the total number of superpositions to obtain the blurred pixel value of each first pixel, so as to obtain a background blurred image.

[0127] By performing point diffusion processing on each first pixel point in the background area of ​​the LLHDR image with the corresponding diffusion weight, the bright spots with higher brightness in the LLHDR image form bright spots under mutual overlap, while the dark spots with lower brightness in the LLHDR image form hazy diffuse circles.

[0128] In some embodiments, before step 502, the method may further include: downsampling the LLHDR image and the foreground recognition result to obtain the LLHDR downsampled image and the downsampled foreground result of the same size. Step 502 may include: obtaining a blur strength map of the LLHDR downsampled image according to the downsampled foreground result, and obtaining a diffusion weight map corresponding to the LLHDR downsampled image; step 504 may include: blurring the background area of ​​the LLHDR downsampled image according to the superposition result to obtain a downsampled blurred image, and upsampling the downsampled blurred image to obtain a background blurred image of the same size as the LLHDR image.

[0129] Before blurring the LLHDR image, the LLHDR image and the foreground recognition result can be downsampled to obtain the LLHDR downsampled image and downsampled foreground result of the same size. Then, based on the blur strength map and diffusion weight map of the LLHDR downsampled image, the background area of ​​the LLHDR downsampled image is blurred. For the same blur strength parameter, due to the small image size and low resolution of the LLHDR downsampled image, the blur intensity will be higher than that of the LLHDR image, which can solve the problem of limited blur intensity and can more realistically simulate the blur effect of a large aperture.

[0130] In the embodiment of the present application, point diffusion processing can be used to simulate real lens blur, which solves the problem of separation between Gaussian blur and spot mapping, making the overall blur effect of the image harmonious and natural, integrated, and more able to simulate the natural blur and bright and transparent light spot effects of SLR camera photography, greatly improving the texture of the blur effect. In addition, point diffusion processing can be performed on each first pixel point in the background area of ​​the LLHDR image without the need to identify the position of the light source point, thereby avoiding the problem of reduced blur effect due to inaccurate identification of the light source point position, which is conducive to providing a more realistic and natural blur effect.

[0131] like Figure 7 As shown, in some embodiments, generating a background blur image based on point diffusion processing may mainly include the following process steps:

[0132] Step 710: Obtain a diffusion mask.

[0133] The diffusion mask can be used to define the image area in the LLHDR image for point diffusion processing. The foreground area of ​​the LLHDR image can be determined according to the foreground recognition result of the LLHDR image, and the foreground area of ​​the LLHDR image can be removed to obtain the diffusion mask, which can prevent the pixels in the foreground area from diffusing to the background area.

[0134] Step 720: Process the diffusion weight map.

[0135] The diffusion weight map may include a brightness weight map and a chromaticity weight map, the brightness weight map may include at least the brightness weight corresponding to each first pixel in the background area of ​​the LLHDR image, and the chromaticity weight map may include at least the chromaticity weight corresponding to each first pixel in the background area of ​​the LLHDR image. Optionally, the brightness weight corresponding to each first pixel may be positively correlated with the brightness component value of each first pixel, and the larger the brightness component value of the first pixel, the larger the brightness weight corresponding to the first pixel; the chromaticity weight corresponding to each first pixel may be positively correlated with the brightness component value of each first pixel, and the chromaticity weight corresponding to each first pixel may also be positively correlated with the chromaticity component value of each first pixel, and the larger the brightness component value and chromaticity component value of the first pixel, the larger the chromaticity weight corresponding to the first pixel.

[0136] In some embodiments, an initial brightness weight map may be generated according to the brightness component value of each pixel in the LLHDR image, and the brightness weight of each pixel in the initial brightness weight map may be the brightness component value of each pixel. The initial brightness weight map may be tuned to reduce the energy of the dense light source area to avoid the occurrence of spot stacking and overexposure during subsequent point diffusion.

[0137] In some embodiments, an initial chromaticity weight map may be generated according to the brightness component value of each pixel in the background area of ​​the LLHDR image, and the initial chromaticity weight of each pixel in the initial chromaticity weight map may be the brightness component value of each pixel. Chromaticity information may be superimposed on the initial chromaticity weight map, and color diffusion and highlight diffusion may be enhanced to obtain a chromaticity weight map used in subsequent point diffusion processing, so that the light spot in the subsequent background blur can diffuse more colors and enhance the color performance of the light spot.

[0138] Step 730: Generate an aperture kernel.

[0139] One or more aperture templates may be pre-stored in the electronic device. The aperture template can be used to define the shape of the aperture. The aperture template can be any aperture shape of cameras on the market, or a specially designed and customized aperture shape, such as a star shape, a heart shape, a circle, etc. The aperture template can be converted into an aperture kernel, and different aperture kernels can be used to indicate different diffusion ranges. Different aperture kernels can correspond to different blur radii. The same aperture template can generate multiple aperture kernels with different blur radii, that is, the same aperture shape can correspond to multiple aperture kernels with different blur radii.

[0140] The electronic device can obtain aperture parameters, which can be aperture shapes input by users according to actual needs, and can obtain aperture templates corresponding to the aperture parameters, and transform the aperture module according to the blur radius corresponding to each first pixel point in the blur intensity map of the LLHDR image, and generate aperture cores corresponding to each first pixel point. Specifically, the aperture module can be scaled according to the blur radius corresponding to each first pixel point, for example, to a size of 3×3, 4×4...N×N, to obtain an aperture core adapted to the blur radius. Various aperture shapes can be simulated by various aperture templates to enrich the expression of the aperture.

[0141] Step 740: Perform point diffusion processing on each first pixel point in the background area of ​​the LLHDR image.

[0142] According to the above diffusion mask, diffusion weight map and aperture kernel of each first pixel point, point diffusion processing can be performed on each first pixel point in the background area of ​​the preprocessed LLHDR image to obtain a background blurred image.

[0143] In some embodiments, the first pixel point that needs to be diffused in the preprocessed LLHDR image can be determined based on the diffusion mask, and the pixel value of the first pixel point can be multiplied by the corresponding diffusion weight and the corresponding aperture kernel to obtain the target pixel value of each third pixel point within the diffusion range with the first pixel point as the center and the blur strength as the radius, and the target pixel value of each third pixel point is accumulated to the pixel value superposition image. The diffusion weight corresponding to the first pixel point can be multiplied by the corresponding aperture kernel to obtain the diffusion times of each third pixel point within the diffusion range with the first pixel point as the center and the blur strength as the radius, and the diffusion times of each third pixel point are accumulated to the times superposition map.

[0144] The blurred pixel value of each first pixel point can be calculated according to the pixel value overlay map and the frequency overlay map. Specifically, the blurred pixel value of each first pixel point can be calculated according to formula (1):

[0145]

[0146] Among them, kernel represents the aperture kernel, weight represents the diffusion weight, input represents the pixel value before blurring, and Bokeh represents the pixel value after blurring (i.e. the above-mentioned blurred pixel value). Each first pixel point in the background area of ​​the preprocessed LLHDR image can be diffused into a specific aperture shape according to the blurring radius based on the diffusion weight after curve mapping. The diffusion results of all the first pixel points are superimposed to obtain the blurring result, and the bright spots form bright spots, and the dark spots form hazy diffuse circles.

[0147] In some embodiments, for each first pixel point in the background area of ​​the preprocessed LLHDR image, the blurred component value can be calculated in the Y channel, U channel and V channel in the manner described above.

[0148] Exemplarily, the Y component value of each first pixel point can be multiplied by the corresponding brightness weight and the corresponding aperture kernel to obtain the target Y component value of each third pixel point within the diffusion range with the first pixel point as the center and the blur strength as the radius, and the target Y component value of each third pixel point is accumulated to the Y component superposition image. The brightness weight corresponding to the first pixel point can be multiplied by the corresponding aperture kernel to obtain the diffusion times of each third pixel point within the diffusion range with the first pixel point as the center and the blur strength as the radius, and the diffusion times of each third pixel point are accumulated to the times superposition map corresponding to the Y component. The total Y component value of each first pixel point in the Y component superposition image can be divided by the total number of superpositions in the times superposition map corresponding to the Y component to obtain the Y component value of each first pixel point after blurring. The U component value of the U channel of each first pixel point and the V component value of the V channel can also be determined in a similar manner based on the chrominance weight to determine the U component value and V component value of each first pixel point after blurring.

[0149] In the embodiment of the present application, point diffusion processing is used to simulate the lens blur of the camera, which solves the problem of the sense of separation between the blur and the light spot, making the overall blur harmonious and natural, and integrated into one, achieving the effect of natural blur and bright and transparent light spot similar to that of a SLR, and improving the texture of the blur.

[0150] like Figure 8 As shown, in one embodiment, after obtaining the background blurred image, the method may further include the following steps:

[0151] Step 802: Obtain the pixel value corresponding to each pixel point in the background blur image.

[0152] In some embodiments, the LLHDR image may be an image in YUV format, and the YUV value corresponding to each pixel in the background blur image may be obtained.

[0153] Step 804 , mapping the pixel value corresponding to each pixel point in the background blur image to the target lookup table LUT to obtain a mapping result.

[0154] LUT (Look-Up-Table) can be understood as a conversion model or a combination of multiple conversion models. LUT can be considered as a function. After the pixel value of each pixel is repositioned by LUT, a new pixel value can be obtained. LUT can include 1D LUT (one-dimensional LUT), 2D LUT (two-dimensional LUT) and 3D LUT (three-dimensional LUT). Among them, the values ​​of the pixel points in each color channel in 1DLUT are independent of each other. When the value of a color channel changes, it will only affect the value of the color channel output in 1D LUT; in 3D LUT, the values ​​of each color channel of the pixel point will affect each other. When the value of a color channel changes, the values ​​of each color channel output by 3D LUT may change. LUT can be used to convert the background blur image, and the LUT filter can be superimposed on the light spot of the background blur image to enhance the overall expression of the background blur image.

[0155] Optionally, the target LUT may be a standard LUT of a YUV color space. The electronic device may pre-generate a standard LUT corresponding to the YUV color space, wherein the Y component may be mapped to a block in the standard LUT, the UV component may be mapped to rows and columns in the standard LUT respectively, and the three channels may be filled in accordance with 0 to 255 respectively. By inputting the coordinates corresponding to the YUV, the corresponding mapping results may be found. The standard LUT may be a 3D LUT. The mapped YUV results may be directly found in the standard LUT according to the YUV values ​​corresponding to each pixel in the blurred background image, and the mapped YUV results may be fused with the YUV values ​​corresponding to each pixel in the blurred background image to obtain the superimposed result of the LUT filter superimposed in the blurred background image to enhance the color, contrast and atmosphere of the blurred background image.

[0156] As an implementation method, the target LUT may also be a LUT after the standard LUT is downsampled. The standard LUT may be downsampled to obtain a downsampled target LUT. For example, the standard 256LUT may be downsampled to a 64LUT or a 16LUT, wherein the 64LUT range is to downsample the standard LUT to 1 / 4, and the 16LUT range is to downsample the standard LUT to 1 / 16. The YUV values ​​corresponding to each pixel in the background blur image (referring to the values ​​in the Y channel, U channel, and V channel) may be downsampled so that the downsampled YUV values ​​match the downsampled target LUT. For example, if the target LUT is a 64LUT, the YUV values ​​corresponding to the pixels may be divided by 4 to obtain the downsampled YUV values. The downsampled YUV values ​​may be rounded up or down, and the LUT coordinates in the downsampled target LUT may be determined based on the rounding result, and then the mapping result may be obtained based on the coordinates.

[0157] Specifically, the coordinates of each pixel point in the LUT range can be calculated by formulas (2) to (3):

[0158] idx u =y u ·bw 2 +u u bw+v u Formula (2);

[0159] idx d =y d ·bw 2 +u d bw+v d Formula (3);

[0160] Among them, y u Indicates the result of rounding up the Y component value after downsampling; u u Indicates the result of rounding up the U component value after downsampling; v u Indicates the result of rounding up the V component value after downsampling; bw indicates the width of each grid in the target LUT; idx u Indicates the LUT coordinate corresponding to the rounded-up result; y d Indicates the result of rounding down the Y component value after downsampling; u d Indicates the result of rounding down the U component value after downsampling; v d Indicates the result of rounding down the V component value after downsampling; idx d Indicates the LUT coordinates corresponding to the rounded-down result.

[0161] The interpolation weight corresponding to the downsampled YUV value of each pixel can be calculated, and the mapped LUT result can be calculated based on the interpolation weight and the corresponding LUT coordinate. Specifically, the interpolation weight of the pixel can be the result of rounding up the downsampled YUV value minus the downsampled YUV value. The interpolation weight can be calculated by formula (4):

[0162] a (y,u,v) =(y,u,v) u -(y,u,v) Formula (4);

[0163] Among them, (y,u,v) represents the YUV value after downsampling, (y,u,v) u Indicates the result of rounding up the downsampled YUV value, α (y,u,v) Represents the interpolation weight. Then, the interpolation calculation is performed based on the interpolation weight of each pixel and the LUT coordinate to obtain the mapping result. The LUT result can be calculated using formula (5):

[0164] (y,u,v) 1 =(1-α(t,u,v) )·LUT[idx u ]+α (y,u,v) ·LUT[idx d ] Formula (5);

[0165] Among them, (y,u,v) 1 Indicates the calculated mapping result (i.e., LUT result), LUT[idx u ] indicates based on LUT coordinate idx u The YUV result found in the target LUT, LUT[idx d ] indicates based on LUT coordinate idx d The YUV result found in the target LUT.

[0166] Step 806: Based on the LUT mask, the mapping result and the background blur image are fused.

[0167] In some embodiments, the LUT mask can be used as the Alpha value of the mapping result to perform Alpha fusion on the mapping result and the background blur image. Specifically, the mapping result and the background blur image can be fused using formula (6):

[0168] (y,u,v) 2 =(1-mask 1 )·(Y,U,V)+mask 1 (y,u,v) 1 Formula (6);

[0169] Among them, (y,u,v) 2 Indicates the result after fusion, that is, the result of superimposing the LUT filter on the background blur image, mask 1 represents the LUT mask, and (Y,U,V) represents the background blur image. The LUT mask can be used to determine the image area that needs to be superimposed with the LUT filter. Furthermore, the LUT mask can also be used to determine the degree of superposition of the image area that needs to be superimposed with the LUT filter. The LUT mask can be defined according to the actual image processing requirements.

[0170] In an embodiment of the present application, by superimposing a LUT filter on the light spot in the background blur image, the color, contrast and atmosphere of the background blur image can be enhanced, and the blur processing effect can be improved. By downsampling the standard LUT of 256 and then mapping the YUV value of each pixel in the background blur image, the amount of calculation can be reduced and the image processing efficiency can be improved.

[0171] like Fig. 9 As shown, in one embodiment, after obtaining the background blurred image, the method may further include the following steps:

[0172] Step 902: determine a noise estimation region in the foreground region of the LLHDR image.

[0173] The foreground area of ​​the image is usually in-focus, that is, the area within the plane where the camera is focused, and the background area of ​​the image is usually out-of-focus, that is, the area outside the plane where the camera is focused. In order to improve the blurred texture of the image, it is necessary to ensure that the noise levels of the in-focus area and the out-of-focus area in the background blurred image are consistent or close, and the noise level can be used to represent the intensity of the noise.

[0174] In an embodiment of the present application, noise estimation may be performed on the foreground area of ​​the LLHDR image, and noise with the same noise level as that of the foreground area may be superimposed on the background area of ​​the background blur image to enhance the image texture of the background blur image.

[0175] The electronic device may first determine a noise estimation area in the foreground area of ​​the LLHDR image, and the noise estimation area may be at least a portion of the image area in the foreground area, and the noise estimation area may be an image area that can accurately assess the noise level of the foreground area. Optionally, the shape of the noise estimation area may be a rectangle, a square, a circle, etc., which is not limited here. As an implementation, the focus point coordinates in the LLHDR image may be obtained, and the focus point coordinates are used as the center of the noise estimation area, and a noise estimation area of ​​a preset size (for example, the length and width are both preset side length parameters) is determined in the foreground area.

[0176] In some embodiments, the LLHDR image may be a portrait image. Since the focus point in the portrait image is usually on the face, the portrait area of ​​the LLHDR image may be used as the noise estimation area. The foreground recognition result of the LLHDR image may include face information, and the face information may include the image position of the face area in the LLHDR image. The face area may be determined based on the face information, and the face area may be used as the noise estimation area. Optionally, when there are multiple face areas in the LLHDR image, multiple face areas may be used as noise estimation areas at the same time, or one of the face areas may be used as a noise estimation image. For example, the face area with the largest image area may be used as the noise estimation area, or the face area with the smallest depth information (i.e., the face area closest to the camera device) may be used as the noise estimation area, etc. The noise estimation area may be determined in a variety of ways, and the present application does not limit this.

[0177] Step 904: determine the noise parameter corresponding to the noise estimation area, where the noise parameter is used to characterize the noise level of the noise estimation area.

[0178] The noise estimation area may be filtered to obtain a target filtering result, and noise parameters corresponding to the noise estimation area may be determined according to the target filtering result. The noise parameters may include but are not limited to parameters such as noise standard deviation and noise variance that can reflect the distribution of image noise in the noise estimation area. The filtering process may include but is not limited to Laplace filtering, mean filtering, median filtering, Gaussian filtering, etc., which are not limited here.

[0179] Taking Laplace filtering as an example, a first filtering operator and a second filtering operator can be obtained, both of which are Laplace operators, and the noise estimation area is filtered according to the first filtering algorithm and the second filtering operator to obtain a target filtering result. For example, the first filtering operator and the second filtering operator can be as follows:

[0180]

[0181] Among them, La 1 represents the first filter operator, La 2 represents the second filtering operator.

[0182] As an implementation method, the first filter operator and the second filter operator can be used to perform convolution operations with the noise estimation area to obtain the first filter result and the second filter result, and then the first filter result and the second filter result are subtracted to obtain the target filter result.

[0183] As another implementation, the first filtering algorithm and the second filtering operator may be subtracted to obtain a third filtering algorithm, and then the third filtering operator and the noise estimation area may be convolved to obtain the target filtering result. Specifically, the third filtering operator may be calculated by formula (7), and the target filtering result may be calculated by formula (8):

[0184] N=2(La 2 -La 1 ) formula (7);

[0185] I laplacian =I src *N formula (8);

[0186] Where, N represents the third filter operator, I src represents the noise estimation area, I laplacian Indicates the target filtering result. This method only requires one filtering process, which can reduce the amount of calculation.

[0187] In some embodiments, in order to avoid inaccurate target filtering results when the noise estimation area includes background areas or strong edges, edge pixels in the noise estimation area can be extracted, and the target filtering results can be adjusted according to the proportion of all edge pixels in the noise estimation area to obtain a more accurate target filtering result.

[0188] After determining all edge pixels included in the noise estimation area, the proportion of all edge pixels in the noise estimation area can be calculated, and the value of each edge pixel in the target filtering result can be reduced according to the proportion to reduce the weight of the edge area in the target filtering result. The proportion of all edge pixels in the noise estimation area can be positively correlated with the reduced value of each edge pixel. The greater the proportion of all edge pixels in the noise estimation area, the more the value of each edge pixel in the target filtering result is reduced. By reducing the weight of the edge area in the target filtering result, the influence of the edge area on the target filtering result can be reduced, and the accuracy of the target filtering result can be improved.

[0189] The noise can be estimated based on the adjusted target filtering result to obtain the noise parameter corresponding to the noise estimation area. Taking the noise parameter as the noise standard deviation as an example, the noise standard deviation corresponding to the noise estimation area can be calculated by formula (9):

[0190]

[0191] Among them, σ represents the noise standard deviation corresponding to the noise estimation area, H represents the height of the noise estimation area, W represents the width of the noise estimation area, and I laplacian ′ represents the adjusted target filtering result. It can be understood that if there is no need to adjust the weight of the edge area of ​​the target filtering result (for example, in the above scheme of subtracting the first filtering result and the second filtering result to obtain the target filtering result, no adjustment is required because the edge area will cancel each other out), then the target filtering result obtained in the above can be directly substituted into formula (9) to obtain the noise standard deviation corresponding to the noise estimation area.

[0192] Optionally, after calculating the noise parameter corresponding to the noise estimation area, the noise parameter can be adjusted according to the ISO value, and the noise parameter can be globally balanced to improve the accuracy of the noise parameter. Optionally, when the ISO value is large (i.e., the scene is darker), the noise parameter can be increased, and when the ISO value is small (i.e., the scene is brighter), the noise parameter can be reduced. It should be noted that adjusting the noise parameter is not a necessary process. For example, when the ISO value = 3000, the noise parameter may not be adjusted, but it is not limited to this.

[0193] Step 906: superimpose noise on the background area of ​​the background blur image according to the noise parameter.

[0194] In some embodiments, random noise corresponding to each pixel point in the background area of ​​the background blurred image may be generated according to the noise parameters corresponding to the noise estimation area and the foreground recognition result. Optionally, the random noise may include but is not limited to Gaussian random noise, which is a random noise whose instantaneous value distribution is a Gaussian distribution. According to the noise weights and random noise corresponding to each pixel point in the background area of ​​the background blurred image, target noise corresponding to each pixel point in the background area of ​​the background blurred image may be generated, and the pixel values ​​of each pixel point in the background area of ​​the background blurred image may be superimposed with the corresponding target noise.

[0195] As an implementation method, random noise corresponding to each pixel point in the background area of ​​the background blurred image can be generated according to the noise standard deviation of the noise estimation area. The generated random noise obeys a normal distribution. Optionally, the random noise can obey a normal distribution with a mathematical expectation of 0 and a standard deviation of the noise standard deviation of the noise estimation area.

[0196] In some embodiments, a target numerical range may be determined from a normal range, and random noise corresponding to each pixel point in the background area of ​​the blurred background image may be generated within the target numerical range, using a pseudo-random method to reduce the amount of calculation.

[0197] After generating the random noise corresponding to each pixel point in the background area of ​​the background blur image, the target noise corresponding to each pixel point in the background area of ​​the background blur image can be generated according to the noise weight corresponding to each pixel point in the background area of ​​the background blur image and the random noise, and then the pixel value corresponding to each pixel point in the background area of ​​the background blur image is superimposed with the target noise. The pixel value of each pixel point in the background area of ​​the background blur image after superposition can be calculated by formula (10):

[0198] Y′ (x,y) =Y (x,y) +noise*weight′ (x,y) Formula (10);

[0199] Among them, Y (x,y) represents the pixel value of the pixel point (x, y) in the background blur image, noise represents the random noise of the pixel point (x, y), weight′ (x,y) Represents the noise weight of the pixel (x, y), Y′ (x,y) Represents the pixel value after the noise is superimposed on the pixel point (x, y), where noise*weight′ (x,y) This is the target noise.

[0200] Optionally, the noise weight mentioned above may be a preset default weight value, and the default weight value may be fixed, and the noise weights of the pixels in the background area of ​​the blurred background image may be the same.

[0201] Optionally, the above-mentioned noise weight can also be determined based on the brightness component value of each pixel point in the background area of ​​the background blurred image, and different noise weights can be assigned to different brightness component values. As an implementation method, the noise weight corresponding to each pixel point in the background area of ​​the background blurred image can be determined by searching, and a noise weight table can be pre-constructed, which can define the noise weight table corresponding to each brightness component value. When it is necessary to superimpose noise on the background blurred image, the corresponding noise weight can be directly found in the noise weight table according to the brightness component value of each pixel point in the background area of ​​the background blurred image. It can simplify the process of obtaining noise weights and improve image processing efficiency.

[0202] As another implementation, the noise weight can also be calculated by formula (11):

[0203]

[0204] Among them, y (x,y) Represents the brightness component value of the pixel (x, y), weight′ (x,y) Represents the noise weight of the pixel (x, y). The noise weight of each pixel in the background area of ​​the background blurred image is calculated based on formula (11). The noise weight can be locally balanced according to the brightness, so as to enhance the noise weight of the darker image area and reduce the noise weight of the brighter image area. This can improve the image quality of the background blurred image with superimposed noise and improve the blurring effect of the image.

[0205] In the embodiment of the present application, based on the noise level of the foreground area of ​​the LLHDR image, noise of equal intensity is superimposed on the background area of ​​the background blur image to achieve consistency between the foreground and background noise, thereby improving the blur texture and blur expression of the background blur image. In addition, the noise level is balanced by considering the influence of the global scene and the local area at the same time, solving the problem of consistency of in-focus and out-of-focus noise from a global and local perspective, further ensuring the accuracy of the superimposed noise, and improving the image quality of the superimposed noise in the background blur image.

[0206] After obtaining the background blurred image, the foreground recognition result can be optimized to improve the accuracy of the foreground recognition result, and the background blurred image and the LLHDR image can be fused based on the optimized foreground recognition result to obtain a target image with a better blurring effect. In some embodiments, the LLHDR image is an image including a portrait, the foreground recognition result can include a portrait mask and a hair recognition result, and optimizing the foreground recognition result can include optimizing the edge of the portrait area in the portrait mask and / or optimizing the edge of the hair area in the hair recognition result.

[0207] Among them, the portrait mask can be used to characterize the image position of the portrait area of ​​the LLHDR image, and can mark the pixels belonging to the portrait area in the LLHDR image. Optionally, the portrait mask can use different pixel values ​​to represent the pixels belonging to the portrait area and the pixels of the non-portrait area (i.e., the background area). For example, a pixel value of 255 indicates that the pixel belongs to the portrait area, and a pixel value of 0 indicates that the pixel belongs to the background area; or a pixel value of 1 indicates that the pixel belongs to the portrait area, and a pixel value of 0 indicates that the pixel belongs to the background area, etc.; a value between 0 and 1 can also be used to represent the probability of each pixel belonging to the portrait area, and the closer the value is to 1, the higher the probability that the pixel belongs to the portrait area, etc., but it is not limited to this.

[0208] In some embodiments, identifying the foreground area in the LLHDR image and obtaining the foreground recognition result may include: performing portrait recognition on the LLHDR image to obtain a portrait mask. The manner of performing portrait recognition on the LLHDR image may include but is not limited to the following manners:

[0209] Method 1: Identify the portrait area of ​​the LLHDR image based on the depth map of the LLHDR image to obtain a portrait mask. The portrait area of ​​the LLHDR image can be identified based on the depth map. For example, the area composed of pixels whose depth information in the LLHDR image is less than a first depth threshold can be determined as the portrait area, and the area composed of pixels whose depth information is greater than a second depth threshold can be determined as the background area.

[0210] The method for the electronic device to perform depth estimation on the LLHDR image may be a software depth estimation method, or a method of calculating depth information in combination with hardware devices, etc.

[0211] Method 2: The LLHDR image can be processed for portrait segmentation to obtain a portrait mask. The portrait segmentation processing method may include but is not limited to a portrait segmentation method based on graph theory, a portrait segmentation method based on clustering, a portrait segmentation method based on semantics, a portrait segmentation method based on an instance, a portrait segmentation method based on a deeplab series network model, a segmentation method based on a U-net network, or a portrait segmentation method based on a fully convolutional network (FCN).

[0212] Taking the example of an electronic device performing portrait segmentation processing on an LLHDR image through a portrait segmentation model to obtain a portrait mask, the portrait segmentation model may be a model of a U-Net structure, and the portrait segmentation model may include an encoder and a decoder, the encoder may include multiple downsampling layers, and the decoder may include multiple upsampling layers. The portrait segmentation model may first perform multiple downsampling convolution processing on the LLHDR image through multiple downsampling layers of the encoder, and then perform multiple upsampling processing through multiple upsampling layers of the decoder to obtain a portrait mask. In the portrait segmentation model, jump connections can be implemented between downsampling layers and upsampling layers with the same resolution, and the features of the downsampling layers and upsampling layers with the same resolution are fused to make the upsampling process more accurate.

[0213] Optionally, the portrait segmentation model can be trained based on a first portrait sample set, which may include multiple portrait sample images carrying portrait labels. The portrait label can be used to annotate the portrait area in the portrait sample image. For example, the portrait label may include a sample portrait mask, etc.

[0214] The hair recognition result can be used to characterize the image position of the hair area in the LLHDR image, and can mark the pixel points belonging to the hair area in the LLHDR image. Optionally, the hair recognition result may include a hair segmentation result and / or a hair cutout result, wherein the hair segmentation result is obtained by performing a segmentation process on the hair area of ​​the LLHDR image, and the hair cutout result is obtained by performing a cutout process on the hair area of ​​the LLHDR image.

[0215] In some embodiments, identifying the foreground area in the LLHDR image and obtaining the foreground identification result may include: performing a segmentation process on the hair area of ​​the LLHDR image to obtain a hair segmentation result. The method of performing a segmentation process on the hair area of ​​the LLHDR image may be similar to the method of performing a portrait segmentation process, such as performing a segmentation process on the hair area of ​​the LLHDR image through a hair segmentation module to obtain a hair segmentation result. The hair segmentation module may be trained based on a second portrait sample set, and the second portrait sample set may include multiple portrait sample images marked with hair areas, etc., and reference may be made to the above description of portrait segmentation, which will not be repeated here. Optionally, the hair segmentation result may include a first hair mask, and the first hair mask may also use different pixel values ​​to represent the pixel points belonging to the hair area and the pixel points of the non-hair area (the image area other than the hair area). For example, the pixel value of the pixel points of the non-hair area in the first hair mask is 0, and the pixel value of the pixel points of the hair area is 255, but it is not limited thereto.

[0216] Optionally, the same segmentation model may be used to perform portrait segmentation processing and hair region segmentation processing on the LLHDR image. The segmentation model may be obtained by training with the first portrait sample set and the second portrait sample set.

[0217] In some embodiments, identifying the foreground area in the LLHDR image and obtaining the foreground identification result may include: performing a matting process on the hair area of ​​the LLHDR image to obtain a hair matting result. The matting process of the hair area may include but is not limited to a Poisson Matting method, a Bayesian matting method based on Bayesian theory, a data-driven machine learning matting method, or a closed surface matting method, etc., which are traditional matting methods that do not use deep learning, or a deep learning-based matting method using an artificial neural network such as a convolutional neural network (CNN).

[0218] As a specific implementation, the electronic device may input the LLHDR image into the hair cutout model, extract the features of the LLHDR image through the hair cutout model, and determine the hair area in the LLHDR image according to the features to obtain the hair cutout result. The hair cutout model may be obtained by training based on a third portrait sample set, and the third portrait sample set includes a plurality of portrait sample images with hair areas marked with high precision. Compared with the hair segmentation model, the hair cutout model has a higher precision in identifying the hair area. Optionally, in order to ensure the accuracy of the hair cutout model, the hair cutout model may be trained according to a preset accuracy condition so that the hair cutout result output by the hair cutout model meets the accuracy condition. Optionally, the accuracy condition may be set using one or more accuracy indicators. For example, the accuracy indicator may include the sum of absolute differences (SAD), mean squared error (MSE), gradient error, etc. between the obtained portrait segmentation result and the real portrait segmentation result; the accuracy condition may include that the error between the obtained hair cutout result and the real hair area is less than one or more of a SAD threshold, less than an MSE threshold, less than a gradient error threshold, etc. This will produce more precise and accurate hair-level hair clipping results.

[0219] In some embodiments, the third portrait sample set may further include a portrait segmentation result obtained by performing portrait segmentation processing on the portrait sample image, and a hair segmentation result obtained by performing hair region segmentation processing on the portrait sample image. The hair cutout model is trained using the portrait segmentation result, hair segmentation result and annotated hair region corresponding to the portrait sample image, and adding portrait segmentation and hair segmentation prior semantic information allows the hair cutout model to first identify the portrait region, then locate the hair region and finally perform fine hair-level cutout on the hair region when performing cutout processing on the hair region, so as to improve the accuracy of the hair cutout result obtained by the hair cutout model.

[0220] In some embodiments, the hair cutout model may be a model that adopts a network architecture such as U-Net, or a model architecture of a generative adversarial network, etc. The third portrait sample set may include simple background sample images and complex background sample images. The hair cutout model that adopts the generative adversarial network model architecture is trained by using complex background sample images. This can improve the network robustness, improve the expressiveness of the hair cutout model in complex background images, and improve the accuracy of the hair cutout results.

[0221] Optionally, the LLHDR image can be subjected to portrait segmentation processing, hair region segmentation processing, and hair region cutout processing respectively by the same image processing model to obtain a portrait mask, a hair segmentation result, and a hair cutout result. The image processing model can be trained according to the first portrait sample set, the second portrait sample set, and the third portrait sample set respectively.

[0222] Optionally, the hair cutout result may include a second hair mask, which may mark the pixel points belonging to the hair area in the LLHDR image. Optionally, the second hair mask may use a value of 0 to 255 to represent the probability that the pixel point belongs to the hair area. The closer the pixel value is to 255, the higher the probability that it belongs to the hair area.

[0223] The following first introduces the relevant methods for optimizing the edges of the portrait area.

[0224] Optimizing the edge of the portrait area may include one or more of alignment processing, smoothing processing, and transition processing of the edge of the portrait area in the portrait mask. The alignment processing may be based on the edge of the portrait in the LLHDR image to align the edge of the portrait area of ​​the portrait mask. Smoothing processing can be used to eliminate protrusions and depressions in the edge of the portrait area and improve burrs or jagged edges on the edge of the portrait area. Transition processing may refer to blurring processing, which is used to make the edge of the portrait area transition naturally and avoid the problem of texture feeling caused by too sharp edges.

[0225] In some embodiments, the edge of the portrait area in the portrait mask can be extracted to generate a first edge mask, and the contour edge in the first edge mask is aligned, smoothed, and transitioned to obtain a second edge mask. The portrait mask is then updated according to the contour edge in the second edge mask to achieve optimization of the edge of the portrait area of ​​the portrait mask.

[0226] As an implementation method, the portrait mask may be dilated, and the portrait mask before dilation may be subtracted from the portrait mask after dilation to obtain a portrait outer contour line, and then a first edge mask may be generated based on the portrait outer contour line. The dilation process is a local maximum operation, and the first convolution kernel may be used to convolve the portrait area in the portrait mask, and the pixel points covered by the first convolution kernel may be calculated to expand the portrait area in the portrait mask.

[0227] For example, Fig.11 FIG. 1 is a schematic diagram of generating a first edge mask in one embodiment. Fig.11As shown, the portrait mask 1110 may be expanded to obtain the expanded portrait mask 1120, and then the portrait mask 1110 may be subtracted from the expanded portrait mask 1120 to obtain the portrait outer contour line, and a first edge mask 1130 may be generated based on the portrait outer contour line.

[0228] Optionally, generating a first edge mask based on the portrait outer contour line may include: traversing each pixel point on the portrait outer contour line, calculating the absolute difference value between each pixel point on the portrait outer contour line and other pixel points in the N*N (such as 3*3, 5*5, 4*4, etc.) neighborhood, and determining other pixel points whose absolute difference value is greater than a difference threshold (which can be set according to actual needs, such as 1, 2, etc.) as pixel points on the initial edge contour. The portrait outer contour line can be subtracted from the initial edge contour to obtain an initial gradient contour, and the area on the initial gradient contour whose image area is less than a second area threshold is deleted to obtain a target gradient contour, wherein the second area threshold can be set according to actual needs and is not limited in this application. By deleting the small area in the initial edge contour, the accuracy of the obtained target gradient contour can be improved. After obtaining the target gradient contour, the target gradient contour can be expanded to generate a first edge mask, and the contour edge in the first edge mask is the target gradient contour after the expansion process. The first edge mask generated in this way can ensure that the edge of the portrait in the first edge mask is accurate and has a suitable width, which is beneficial to improving the accuracy of subsequent optimization processing of the first edge mask.

[0229] After the first edge mask is generated, the first edge mask may be aligned. The electronic device may perform global smooth filtering on the contour edge in the first edge mask based on the LLHDR image in RGB format, wherein the global smooth filtering may be implemented by a global smooth filter (Fast Global Smoother Filter, FGS Filter), etc. The FGSFilter may establish an energy function model for the global smooth problem of the first edge mask as a whole, and achieve global smoothing by minimizing the energy function, which is a globally optimal solution. Optionally, if the LLHDR image is in YUV format, the LLHDR image may be converted from YUV format to RGB format first.

[0230] The electronic device can also use the brightness component map of the LLHDR image as a guide map, and perform guided filtering (Guided Filter) on the first edge mask after global smooth filtering to obtain an edge-aligned first edge mask. GuidedFilter can transfer the smoothness of the guide map to the first edge mask through a local linear relationship, and smooth it by minimizing the difference between the edge mask and the guide map. It is a locally optimal solution. Through FGS Filter and GuidedFilter, the first edge mask is filtered from the global and local perspectives using the LLHDR image, and the error between the contour edge in the first edge mask and the actual contour edge in the LLHDR image is aligned.

[0231] Since the contour edge will spread out with burrs and jagged edges when the first edge mask is aligned, the edge-aligned first edge mask may be smoothed to obtain a first edge mask with smooth edges. In some embodiments, the smoothing process may include edge narrowing and edge smoothing:

[0232] 1. Narrowing of the edge

[0233] Compared with the first edge mask without alignment, the contour edge in the edge-aligned first edge mask will diffuse outward (i.e., diffuse toward the background area), resulting in inaccurate portrait edges. Therefore, the contour edge in the edge-aligned first edge mask can be narrowed first, and the contour edge in the first edge mask can be retracted inward toward the direction close to the portrait.

[0234] Based on the mapping curve, the edge contour in the edge-aligned first edge mask can be narrowed to obtain the edge-narrowed first edge mask. Based on the mapping curve, each pixel point of the contour edge in the edge-aligned first edge mask can be mapped, so that some pixel points that are classified as the contour edge before mapping can be reclassified as belonging to the background area after mapping, thereby narrowing the contour edge.

[0235] For example, the mapping curve may include but is not limited to a Sigmoid curve, and the Sigmoid curve may be expressed by formula (12):

[0236]

[0237] Among them, x 8 Represents the pixel value of the pixel before mapping, y 8 Indicates the pixel value of the mapped pixel point, a indicates the smoothness, and b indicates the center position. The smoothness a and the center position b can be empirical values ​​and can be set according to actual business needs, and there is no specific limitation. The center position b can be positively correlated with the narrowing of the contour edge. The larger the center position b, the more the contour edge narrows inward.

[0238] The interval of the pixel values ​​of the pixel points after the curve mapping usually changes, therefore, the mapped first edge mask may be normalized. Optionally, the pixel values ​​of each pixel point included in the first edge mask may be normalized to a value range of 0 to 255.

[0239] 2. Edge Smoothing

[0240] A weighted interpolation smoothing process may be performed on each pixel point of the contour edge in the first edge mask with narrowed edge to obtain a first edge mask with smoothed edge. Optionally, the weighted interpolation smoothing process may include an Inpaint edge smoothing process, and the Inpaint edge smoothing process is an iterative weighted filtering algorithm, which performs weighted interpolation on the pixel values ​​in the neighborhood of each pixel point in an iterative manner to smooth the pixel values ​​of each pixel point of the contour edge and the protrusions or depressions in the neighborhood.

[0241] After the edge-aligned first edge mask is smoothed to obtain the edge-smoothed first edge mask, the edge-smoothed first edge mask may be subjected to transition processing to obtain a second edge mask. A transition parameter corresponding to the edge-smoothed first edge mask may be calculated, and the edge-smoothed first edge mask may be subjected to transition processing according to the transition parameter. Optionally, the transition parameter may include a blur radius, and the blur radius may be used to indicate the degree of blur. The larger the blur radius, the higher the degree of blur. In some embodiments, the ROI (Region Of Interest) corresponding to the portrait area may be determined in the LLHDR image according to the portrait area in the portrait mask. Further, the ROI may be an area formed by a bounding box of the portrait area, wherein the bounding box may be an external rectangular box of the portrait area. The ratio of the image area of ​​the ROI to the total image area of ​​the LLHDR image may be calculated, and the blur radius may be determined according to the ratio. Optionally, the blur radius may be positively correlated with the ratio. The larger the ratio of the ROI in the LLHDR image, the larger the blur radius may be.

[0242] Optionally, the narrowed first edge mask may be normalized so that the pixel value of each pixel point is within a normal value range, for example, between 0 and 255. Further, the contour edge of the normalized first edge mask may be blurred again according to the blur radius to obtain a second edge mask.

[0243] After obtaining the second edge mask, the portrait mask and the second edge mask may be merged, and the edge of the portrait region in the portrait mask may be updated according to the contour edge of the second edge mask to obtain a portrait mask with optimized portrait edge. As an implementation, the pixel values ​​of each pixel point in the portrait mask corresponding to the contour edge of the second edge mask may be replaced with the pixel values ​​in the second edge mask, and the pixel values ​​of each pixel point outside the contour edge may be adjusted to the pixel values ​​corresponding to the background region (e.g., adjusted to 0), etc.

[0244] In an embodiment of the present application, a first edge mask can be generated based on the portrait mask, and the first edge mask can be optimized by alignment processing, smoothing processing, transition processing, etc., to obtain a second edge mask with a more accurate, smoother contour edge and a natural transition, and the portrait edge of the portrait mask is updated based on the second edge mask, so that the portrait edge of the portrait mask is more accurate, more refined and smooth, and can avoid problems such as the sense of mapping caused by too sharp edges, and can improve the blurring effect of the target image obtained subsequently.

[0245] The following will introduce the relevant implementation solutions for optimizing the edges of the hair area in the hair recognition results.

[0246] The electronic device can optimize the edge of the hair region in the hair recognition result, and fuse the optimized hair recognition result with the portrait mask to correct the edge of the hair region in the portrait mask to obtain a more accurate portrait mask. Fig.13 As shown, correcting the edge of the hair area in the portrait mask may include the following steps:

[0247] Step 1310: determine the background complexity of the LLHDR image according to the LLHDR image, the first portrait mask and the hair segmentation result.

[0248] In the embodiment of the present application, the first portrait mask may be a portrait mask included in the foreground recognition result, or may be a portrait mask after the above-mentioned portrait edge optimization.

[0249] Background complexity can be used to describe the complexity of the background area in the LLHDR image. The richer the content contained in the background area, the higher the corresponding background complexity can be. As an implementation method, the background area of ​​the LLHDR image can be determined based on the first portrait mask and the hair segmentation result, and edge detection is performed on the background area to extract the edge features of the background area, and the background complexity corresponding to the LLHDR image is determined based on the edge features. Optionally, the more edge features contained in the background area, the richer the content details of the background area, and the higher the background complexity can be. Canny edge detection operator, Laplacian detection operator, DoG detection operator, Sophie detection operator, etc. can be used to perform edge detection on the LLHDR image, but are not limited to this.

[0250] In some embodiments, step 1310 may include: determining a hair extension region of interest in the LLHDR image according to the first portrait mask and the hair segmentation result; calculating first gradient information of the hair extension region of interest; and calculating background complexity according to the first gradient information.

[0251] The hair extended region of interest may include the hair region and part of the background region. As an implementation method, the hair region in the LLHDR image may be determined according to the hair segmentation result, and the bounding box (boundingBox) of the hair region may be determined. Optionally, the bounding box may be a circumscribed rectangular box of the hair region. The image region included in the bounding box may be defined as the hair region of interest. The hair region of interest may be extended to obtain the hair extended region of interest. For example, the hair region of interest may be enlarged according to a preset magnification (such as 1x, 2x, etc.) to obtain the hair extended region of interest. The central pixel point of the hair extended region of interest may be the same as the central pixel point of the hair region of interest. The length and width of the hair region of interest may also be increased by a preset boundary value to obtain the hair extended region of interest.

[0252] Optionally, the hair segmentation result may include a first hair mask, and the hair region in the first hair mask may be dilated, and the portrait region in the first portrait mask may be eroded, and then the first hair mask after the dilation process is subtracted from the first portrait mask after the erosion process to obtain the hair outer contour, and the hair extension region of interest is determined based on the hair outer contour. The erosion process is a local minimum operation, and the second convolution kernel may be used to convolve with the portrait region in the first portrait mask, and the pixels covering the second convolution kernel may be retained to reduce the portrait region in the first portrait mask.

[0253] Optionally, a connected domain detection may be performed on the image area indicated by the outer contour of the hair. The image area indicated by the outer contour of the hair may be an image area in the first hair mask or an image area in the LLHDR image. A noise area whose area of ​​the connected domain is less than a third area threshold may be determined, and the noise area may be deleted to obtain a target hair area. By deleting a small area of ​​noise in the hair area, the accuracy of hair area recognition may be improved, and the accuracy of subsequent calculation of background complexity may be improved. It should be noted that the embodiment of the present application does not limit the method of connected domain detection. After obtaining the target hair area, a first bounding box corresponding to the target hair area may be determined, and the first bounding box may be expanded to obtain a hair extension region of interest. For example, the first bounding box may be enlarged according to a preset multiple to obtain a hair extension region of interest; or the length and width of the first bounding box may be increased by a preset boundary value to obtain a hair extension region of interest.

[0254] After determining the hair extension region of interest in the LLHDR image, the gradient value of each pixel in the hair extension region of interest may be calculated to obtain first gradient information. The first gradient information may be calculated by, but is not limited to, calculating the first gradient information based on a Sobel operator, calculating the first gradient information based on a Scharr operator, calculating the first gradient information based on a Laplacian operator, and the like.

[0255] Taking the calculation of the first gradient information based on the Sobel operator as an example, the Sobel operator may include a first matrix corresponding to the horizontal direction and a second matrix corresponding to the vertical direction. The extended region of interest may be convolved with the first matrix and the second matrix respectively to obtain the first gradient value of each pixel point in the extended region of interest in the horizontal direction and the second gradient value in the vertical direction, and then the target gradient value of each pixel point in the extended region of interest is calculated based on the first gradient value and the second gradient value of each pixel point in the extended region of interest, that is, the first gradient information is obtained. Specifically, the first gradient information may be calculated by formula (13) and formula (14):

[0256]

[0257] G=|G x |+|G y |Formula (14);

[0258] Among them, A represents the extended area of ​​interest, G x Represents the gradient information of the extended region of interest in the horizontal direction, G y represents the gradient information of the extended region of interest in the vertical direction, and G represents the first gradient information.

[0259] Optionally, the overall background complexity of the hair extension region of interest may be used as the background complexity of the LLHDR image. As an implementation, the target gradient value of each pixel point belonging to the background area in the hair extension region of interest may be used as the complexity corresponding to each pixel point, and the overall background complexity of the hair extension region of interest may be obtained according to the complexity of each pixel point belonging to the background area in the hair extension region of interest.

[0260] In some embodiments, the second gradient information can be deleted from the first gradient information according to the first portrait mask and the hair segmentation result to obtain the third gradient information, wherein the second gradient information can include the gradient information corresponding to the portrait area of ​​the first portrait mask and the gradient information corresponding to the hair area in the hair segmentation result, and the third gradient information is the gradient information of the background area in the hair extension region of interest. Further, the third gradient information can be dilated and blurred to obtain a background complexity image corresponding to the hair extension region of interest, and the overall background complexity of the hair extension region of interest is calculated according to the background complexity image. The blurring process can adopt Gaussian blurring, mean blurring, median blurring and other processing methods, and the specific dilation processing method and blurring processing method are not limited in the embodiment of the present application. The background complexity image can be used to characterize the complexity of the background area in the hair extension region of interest at different positions.

[0261] For example, Fig.14 FIG. 1 is a schematic diagram of generating a background complexity image in one embodiment. Fig.14 As shown, the first gradient information 1410 of the hair extension region of interest can be calculated, and the second gradient information can be deleted from the first gradient information 1410, that is, the gradient information of the portrait region is deleted from the first gradient information 1410 to obtain the third gradient information 1420, and the third gradient information 1420 is the gradient information of the background region of the hair extension region of interest. The third gradient information 1420 can be dilated and blurred to obtain a background clutter image 1430.

[0262] Optionally, after the background complexity image is calculated, the average complexity of the background complexity image may be calculated, and the average complexity may be used as the overall background complexity corresponding to the hair extension region of interest. As another embodiment, the overall background complexity corresponding to the hair extension region of interest may also be the complexity of the most frequently occurring pixels belonging to the background region in the hair extension region of interest. A plurality of complexity intervals may be pre-divided, and the complexity intervals in which the complexity of each pixel in the background complexity image is located are counted. A target complexity interval in which the complexity of the most pixels is located may be determined, and the overall background complexity of the hair extension region of interest may be determined based on the target complexity interval. For example, the middle complexity of the target complexity interval may be used as the overall background complexity, etc., but is not limited thereto.

[0263] Step 1320 , determine whether the background complexity is greater than a first complexity threshold, if so, execute step 1340 , if not, execute step 1330 .

[0264] It can be determined whether the background complexity corresponding to the LLHDR image is greater than a first complexity threshold. If the background complexity corresponding to the LLHDR image is greater than the first complexity threshold, the background area is a complex background, and the LLHDR image can be determined to be a complex background image; if the background complexity corresponding to the LLHDR image is less than or equal to the first complexity threshold, the background area is a simple background, and the LLHDR image can be determined to be a simple background image.

[0265] Since the hair cutout result is more expressive and accurate when the background area of ​​the LLHDR image is a simple background, if the LLHDR image is a simple background image, the electronic device can obtain the hair cutout result of the LLHDR image, and generate a target portrait mask based on the hair cutout result and the first portrait mask. When the background area of ​​the LLHDR image is a complex background, the hair cutout result obtained by cutting out the hair area of ​​the LLHDR image has poor expressiveness, and it is easy to mistake the background area for the hair area, and the accuracy is low. Therefore, if the LLHDR image is a complex background image, the target portrait mask can be generated based on the first portrait mask and the hair segmentation result. According to the background complexity of the LLHDR image, selecting an appropriate hair recognition result can improve the accuracy of the generated target portrait mask.

[0266] It should be noted that the above-mentioned first complexity threshold can be obtained based on statistics of multiple test results. For example, the hair area of ​​multiple sample images with different background complexities can be cut out, and the hair cutout results corresponding to each sample image can be obtained. The accuracy of the hair cutout results corresponding to each sample image can be statistically analyzed, and the first complexity threshold is determined based on the analysis results, so that the accuracy of the hair cutout results of most sample images with background complexity less than or equal to the first complexity threshold is high, while the accuracy of the hair cutout results of most sample images with background complexity greater than the first complexity threshold is low.

[0267] Step 1330 , generating a target portrait mask according to the hair cutout result and the first portrait mask.

[0268] In some embodiments, step 1330 may include: performing a first optimization process on the hair cutout result to obtain a target cutout result, and fusing the target cutout result with the first portrait mask to obtain a target portrait mask. By performing the first optimization process on the hair cutout result, a target cutout result with higher accuracy and fineness may be obtained to improve the accuracy of the edge of the hair region of the target portrait mask.

[0269] In some embodiments, the first optimization process may include but is not limited to at least one of the following processing methods:

[0270] Method 1: Although the LLHDR image is a simple background image, there may still be some background sub-regions with rich image content in the background area of ​​the LLHDR image. Therefore, the hair area around the complex background sub-region in the hair cutout result can be eroded and blurred. The complex background sub-region is a background sub-region in the hair cutout result whose complexity is greater than the second complexity threshold, and the second complexity threshold is less than the first complexity threshold.

[0271] Optionally, a background complexity image corresponding to the hair extension region of interest may be obtained, and the background region in the hair extension region of interest may be divided according to the background complexity image. If there is a background sub-region whose complexity is greater than the second complexity threshold, the background sub-region may be determined as a complex background sub-region. The complex background sub-region may be a background sub-region whose pixel complexity is greater than the second complexity threshold, or a background sub-region whose average complexity is greater than the second complexity threshold, etc., which is not limited here.

[0272] When the image content of the background sub-region is relatively rich, the background sub-region may be mistakenly identified as the hair region when the LLHDR image is cut out for the hair region. Therefore, the hair region around the complex background sub-region in the hair cutout result can be eroded and blurred to reduce the hair region around the complex background sub-region and achieve a natural transition from the complex background sub-region to the hair region, thereby improving the accuracy of the hair region.

[0273] Method 2: improving the confidence of the hair region around the simple background sub-region in the hair cutout result, wherein the simple background sub-region is a background sub-region in the hair cutout result whose complexity is less than or equal to the second complexity threshold.

[0274] The background area in the hair extension region of interest may be divided according to the background complexity image. If there is a background sub-area whose complexity is less than or equal to the second complexity threshold, the background sub-area may be determined as the first simple sub-area. The simple background sub-area may be a background sub-area whose pixel complexity is less than or equal to the second complexity threshold, or a background sub-area whose average complexity is less than or equal to the second complexity threshold, etc., which is not limited here.

[0275] Since the hair cutout results have better expressiveness and more accurate results in simple backgrounds, the confidence of the hair area around the simple background sub-area in the hair cutout results can be improved. This confidence can be used to characterize the credibility (also understood as probability) of the pixel belonging to the hair area. The higher the confidence of the pixel, the greater the possibility of belonging to the hair area, and the more reliable the result. By improving the confidence of the hair area around the simple background sub-area in the hair cutout results, the hair can be made more prominent in the simple background image and the accuracy of the hair area can be improved.

[0276] Method 3: Upsampling and filtering the hair cutout result based on the guided filter. In some embodiments, the hair cutout result may be obtained by processing the cropped hair extension region of interest through a hair cutout model. The hair cutout model may be a small-scale network model, and the resolution of the obtained hair cutout result may be relatively small. Therefore, the hair cutout result may be first enlarged to the same size as the cropped hair extension region of interest (or enlarged to the same size as the LLHDR image), and then the cropped hair extension region of interest (or LLHDR image) is used as the guided image of the guided filter. The guided filter may filter the hair cutout result based on the image information of the cropped hair extension region of interest (or LLHDR image), so that the texture and edge features of the filtered hair cutout result are similar to those of the hair extension region of interest, thereby improving the fineness and accuracy of the hair cutout result.

[0277] It should be noted that the hair cutout model can also be a large-scale network model, which can obtain high-resolution hair cutout results to improve the accuracy of the hair cutout results. Upsampling and filtering the hair cutout results based on the guided filter is not a necessary processing step.

[0278] Method 4: If the image scene corresponding to the LLHDR image is the target scene, the hair area of ​​the hair cutout result is corroded and blurred. The target scene may be a scene whose scene brightness value is lower than the brightness threshold, such as a night scene, a dimly lit indoor scene, etc. The hair cutout result has a clearer effect in bright scenes, but in darker scenes, the hair cutout result has a poorer expressiveness and may result in inaccurate hair areas. Therefore, in the embodiment of the present application, it may be determined whether the image scene corresponding to the LLHDR image is the target scene. If it is the target scene, the hair area of ​​the hair cutout result may be corroded and blurred, so that the edge of the hair area shrinks inward and the transition is blurred, thereby improving the accuracy of the hair area.

[0279] In some embodiments, the ISO value corresponding to the LLHDR image can be obtained, and it can be determined whether the photosensitivity value corresponding to the LLHDR image is greater than the photosensitivity threshold. If the ISO value is greater than the photosensitivity threshold, it means that the ISO value of the LLHDR image is high. The higher the ISO value, the higher the sensitivity of the film to weak light, and more weak light can be captured, which is suitable for use in scenes with dim light. Therefore, if the ISO value corresponding to the LLHDR image is greater than the photosensitivity threshold, it can be determined that the image scene corresponding to the LLHDR is the target scene. The photosensitivity threshold can be an empirical value obtained through multiple experimental tests.

[0280] It should be noted that the first optimization processing method may include only one of the above-mentioned methods, or may be any combination of multiple methods, and the order of execution of various methods is not limited in the embodiments of the present application.

[0281] In some embodiments, fusing the target cutout result with the first portrait mask to obtain the target portrait mask may include: for each pixel point in the target cutout result, comparing the pixel value of the pixel point in the target cutout result with the pixel value in the first portrait mask, and taking the maximum value, which is the pixel value of the pixel point in the target portrait mask. Specifically, the target cutout result and the first portrait mask may be fused according to formula (15):

[0282] T_MASK (x,y) =MAX(Matting (x,y) ,pe_mask (x,y) ) formula (15);

[0283] Among them, Matting (x,y) Indicates the pixel value of the pixel point (x, y) in the target matting result, pe_mask (x,y) Represents the pixel value of the pixel point (x, y) in the first portrait mask, T_MASK (x,y) Represents the value of the pixel (x, y) in the target portrait mask.

[0284] Step 1340: If the background complexity is greater than the first complexity threshold, a target portrait mask is generated according to the first portrait mask and the hair segmentation result.

[0285] When the background complexity corresponding to the LLHDR image is greater than the first complexity threshold, the LLHDR image is a complex background image, and the hair cutout result has poor expressiveness. A target portrait mask can be generated according to the first portrait mask and the hair segmentation result. In some embodiments, step 1340 may include: performing a second optimization process on the hair segmentation result to obtain a target segmentation result, and fusing the target segmentation result with the first portrait mask to obtain a target portrait mask. By performing a second optimization process on the hair segmentation result, a target segmentation result with higher accuracy and precision is obtained to improve the accuracy of the target portrait mask.

[0286] In some embodiments, the second optimization process may include but is not limited to at least one of the following processing methods:

[0287] Method 1: Corrosion processing is performed on the hair region of the hair segmentation result. By corroding the hair region of the hair segmentation result, the problem of too many hair regions in the hair segmentation can be improved, thereby improving the accuracy of the hair segmentation result.

[0288] Method 2: Blurring the hair region of the hair segmentation result. By blurring the hair region of the hair segmentation result, the edge transition of the hair region in the hair segmentation result can be made natural.

[0289] Method 3: Guided filtering is performed on the hair segmentation result based on the guided filter. The LLHDR image can be used as a guided image of the guided filter, and the guided filter can perform guided filtering on the hair segmentation result based on the image information of the LLHDR image to obtain a more refined hair segmentation result.

[0290] In some embodiments, when the area of ​​the hair region is too small, a pixel-level grid may appear when the hair segmentation result is subjected to guided filtering. Therefore, the hair segmentation result after guided filtering may be blurred to improve the pixel-level grid situation and enhance the accuracy and precision of the hair segmentation result.

[0291] Method 4: gamma correction is performed on the hair segmentation result based on the background complexity. The gamma parameter can be calculated according to the background complexity corresponding to the LLHDR image, and the gamma correction is performed on the hair segmentation result according to the gamma parameter. Specifically, the gamma parameter can be calculated according to formula (16), and the gamma correction is performed according to formula (17):

[0292]

[0293] Among them, gamma 3 represents the third gamma parameter, complexity represents the background complexity, x 9 Indicates the value before gamma correction processing, y 9 Indicates the value after gamma correction. By performing gamma correction on the hair segmentation result, adaptive inward transition processing can be performed to improve the accuracy of the hair segmentation result.

[0294] The second optimization processing method may include only one of the above methods, or any combination of multiple methods, and the execution order of the various methods is not limited in the embodiment of the present application. For example, the hair segmentation result may be first corroded, and then the corroded hair segmentation result may be blurred, and then the blurred hair segmentation result may be guided filter processed based on the guided filter, and then the hair segmentation result may be gamma corrected based on the background complexity to obtain the target segmentation result.

[0295] In some embodiments, the step of fusing the target segmentation result with the first portrait mask to obtain the target portrait mask may include: for each pixel point included in the target segmentation result, the pixel value of the pixel point in the target segmentation result may be compared with the pixel value of the pixel point in the first portrait mask, and the maximum value thereof is taken, and the maximum value is the pixel value of the pixel point in the target portrait mask. Specifically, the target cutout result may be fused with the first portrait mask according to formula (18):

[0296] T_MASK (x,y) =MAX(hair_mask (x,y) ,pe_mask (x,y) ) formula (18);

[0297] Among them, hair_mask (x,y) Indicates the pixel value of the pixel point (x, y) in the target segmentation result, pe_mask (x,y) Represents the pixel value of the pixel point (x, y) in the first portrait mask, T_MASK (x,y) Represents the value of the pixel (x, y) in the target portrait mask.

[0298] After obtaining the target portrait mask, the background blur image and the LLHDR image can be fused based on the target portrait mask to obtain the target image. Since the target portrait mask accurately marks the hair area of ​​the LLHDR image, it can avoid the situation where the hair area is mistakenly blurred or the background area is missed, and the blur effect of the target image can be improved.

[0299] In the embodiment of the present application, when the LLHDR image is a simple background image, the target portrait mask is generated by using a hair cutout result with higher precision and better expression, which can improve the clarity of the hair in the hair area. When the LLHDR image is a complex background image, the hair cutout result has poor expression, and the target portrait mask is generated by using the hair segmentation result, which can accurately obtain the portrait mask of the LLHDR image, thereby improving the recognition accuracy of the portrait area of ​​the LLHDR image. In addition, by optimizing the hair cutout result or the hair segmentation result, a target cutout result or a target segmentation result with higher precision and higher accuracy is obtained, which can improve the accuracy of the hair area in the generated target portrait mask, and make the target portrait mask contain more abundant hair details, which can improve the blurring effect of the target image obtained subsequently.

[0300] In some embodiments, in order to avoid a sense of pasting caused by overly sharp edges of the hair region, before step 1310, the hair region of the first portrait mask may be subjected to an internal transition process to obtain a new first portrait mask, and then based on the background complexity of the LLHDR image, the hair cutout result or the hair segmentation result may be selected to be merged with the new first portrait mask to obtain a target portrait mask.

[0301] The inner transition processing may at least include blur processing. In some embodiments, the step of performing inner transition processing on the hair region of the first portrait mask to obtain a new first portrait mask may include: determining inner transition parameters according to the first portrait mask and the hair segmentation result; performing blur processing on the hair region of the first portrait mask according to the inner transition parameters to obtain a second portrait mask; and fusing the second portrait mask with the first portrait mask to obtain a new first portrait mask.

[0302] The inner transition parameter may be used to characterize the degree of inner transition processing performed on the hair area. Optionally, the inner transition parameter may include an inner transition width. Further, the inner transition width may be a blur radius of a filter for blur processing. The larger the blur radius, the greater the blur degree and the stronger the blur effect. The edge of the hair area of ​​the first portrait mask may be blurred according to the blur radius to achieve a blurred transition effect.

[0303] In some embodiments, the hair region of interest in the first portrait mask may be determined according to the hair segmentation result, wherein the hair region of interest may be an image region formed by a boundary box of the hair region of the first portrait mask. Optionally, the inner transition parameter may be determined according to the ratio of the image area of ​​the hair region of interest to the total image area of ​​the first portrait mask. Specifically, when the ratio is large and the number of portraits included in the first portrait mask is small, the inner transition parameter may be large, and when the ratio is small or the number of portraits is large, the inner transition parameter may be small. Optionally, the ratio may be divided by the number of portraits in the first portrait mask to obtain an average ratio corresponding to each portrait, and the average ratio may be positively correlated with the inner transition width. When the average ratio is large, the inner transition width may be large. By adaptively determining the inner transition parameter according to the ratio of the image area of ​​the hair region of interest to the total image area of ​​the first portrait mask and the number of portraits, the accuracy of the inner transition processing may be improved, and the accuracy of the hair region may be further ensured.

[0304] In some embodiments, after the inner transition parameter is determined, the hair region in the hair region of interest may be blurred according to the inner transition parameter to obtain a second portrait mask, and the second portrait mask is merged with the first portrait mask to obtain a new first portrait mask. The blurring process may include but is not limited to mean blurring. Optionally, before blurring the hair region in the hair region of interest, the hair region in the hair region of interest may be eroded, and then the hair region in the hair region of interest after the erosion process is blurred based on the determined inner transition width, so as to avoid the problem of too wide transition area at the edge of the hair.

[0305] As an implementation method, the two portrait masks and the first portrait mask can be fused according to the edge weight map corresponding to the LLHDR image to obtain a new first portrait mask. The edge weight map can be generated according to the outer contour of the hair. Optionally, the portrait area in the first portrait mask can be eroded, and the hair segmentation result is subtracted from the eroded first portrait mask to obtain the outer contour of the hair, and the outer contour of the hair is expanded and blurred to obtain a smoother and softer edge weight map.

[0306] As an implementation method, the second portrait mask and the first portrait mask can be alpha-fused, the edge weight map can be used as the alpha value of the second portrait mask, and based on the edge weight map, the second portrait mask and the first portrait mask are alpha-fused to obtain a new first portrait mask.

[0307] When the background complexity corresponding to the LLHDR image is less than or equal to the first complexity threshold, a target portrait mask may be generated according to the hair cutout result of the LLHDR image and the new first portrait mask. Further, a first optimization process may be performed on the hair cutout result, and the target cutout result obtained after the first optimization process may be fused with the new first portrait mask to obtain the target portrait mask.

[0308] When the background complexity corresponding to the LLHDR image is greater than the first complexity threshold, a target portrait mask may be generated according to the first portrait mask and the new first portrait mask. Further, a second optimization process may be performed on the hair segmentation result, and the target segmentation result obtained after the second optimization process is merged with the new first portrait mask to obtain the target portrait mask.

[0309] In an embodiment of the present application, the first portrait mask can be firstly subjected to internal transition blurring based on the hair segmentation result, so that the hair edge of the first portrait mask is more accurate and the transition is more natural, and then the hair details of the outer contour of the hair area in the first portrait mask are supplemented based on the background complexity of the LLHDR image, thereby improving the accuracy of the portrait area in the target portrait mask.

[0310] In some embodiments, the LLHDR image is an image containing a portrait. In related blurring technologies, when blurring the background of an image containing a portrait, the entire portrait area in the image is usually kept clear, resulting in a problem of not highlighting the key points, affecting the blurring effect. In the embodiment of the present application, in order to improve the blurring effect of the image, the LLHDR image can be refocused on the portrait, so that the target image has the effect of keeping each face clear and blurring other portrait areas, thereby highlighting the key points.

[0311] like Fig.15 As shown, in one embodiment, the process of refocusing the LLHDR image for a portrait may include the following steps:

[0312] Step 1510, generating a weight mask according to the portrait mask and the face information, wherein the weight mask is used to determine a clear face area corresponding to each face in the LLHDR image, and a blurred portrait transition area corresponding to each face.

[0313] The foreground recognition result of the LLHDR image may include the face information of each face contained in the LLHDR image and the portrait mask of the LLHDR image. The face information may include but is not limited to one or more of the face area of ​​each face, the center point coordinates of each face area, the radius of each face area, etc.

[0314] In some embodiments, the step of identifying the foreground area in the LLHDR image and obtaining the foreground recognition result may include: performing face recognition on the LLHDR image to obtain the face information of each face contained in the LLHDR image. Optionally, the method of performing face recognition on the LLHDR image may include but is not limited to a method of performing recognition based on a face template, a method of performing face recognition based on a classifier, and a method of performing face recognition through a deep neural network. For example, the electronic device may use a convolutional neural network to perform face recognition on the LLHDR image. The convolutional neural network may be trained based on a face sample set, and the face sample set may include multiple face images marked with face areas. The convolutional neural network can extract facial feature points in the LLHDR image, and determine the face detection frame corresponding to each face in the LLHDR image based on the facial feature points. The image area where the face detection frame corresponding to each face is located can be used as the face area.

[0315] The shape of the face detection frame may include but is not limited to square, rectangle, circle, etc. If the face detection frame is a quadrilateral such as a square or rectangle, the radius of the corresponding face area may be the radius of the circumscribed circle of the face detection frame. If the face detection frame is a circle, the radius of the face area is the radius of the face detection frame. The coordinates of the center point of the face area are the coordinates of the center pixel point of the face detection frame. If the face detection frame is a quadrilateral such as a square or rectangle, the horizontal coordinate of the center pixel point may be the horizontal coordinate of the lower left corner point of the face detection frame plus half of the width, and the vertical coordinate of the center pixel point may be the vertical coordinate of the lower left corner point of the face detection frame plus half of the height. If the face detection frame is a circle, the center point of the face area is the center of the face detection frame.

[0316] A weight mask can be generated according to the portrait mask of the LLHDR image and the facial information of each face contained therein, and the weight mask can mark the face clear area corresponding to each face in the LLHDR image and the portrait blur transition area corresponding to each face. The face clear area refers to the area in the portrait area of ​​the LLHDR image that needs to be kept clear, and the portrait blur transition area refers to the area in the portrait area that changes from clear to blurry.

[0317] In some embodiments, the step of generating a weight mask according to the portrait mask of the LLHDR image and the facial information of each face contained therein may include: determining the facial clear area corresponding to each face in the portrait mask according to the facial information; respectively determining the first distances between the center points of the target facial clear area corresponding to each fourth pixel point of the portrait area in the portrait mask; normalizing the first distances of each fourth pixel point according to the portrait transition range of the target facial clear area corresponding to each fourth pixel point to obtain the normalized value corresponding to each fourth pixel point, and generating a weight mask according to the normalized value and the portrait mask.

[0318] Optionally, the image area formed by the circumscribed circle of each face area can be used as the clear face area of ​​the face area, and the clear face area corresponding to each face in the portrait mask is a circular area with the center point coordinates of the face area corresponding to each face as the center point and the radius of the circumscribed circle of the face area corresponding to each face as the area radius. Taking the first face in the LLHDR image as an example, the first face is any face in the LLHDR image, and the clear face area of ​​the first face can be a circular area with the center point coordinates of the face area corresponding to the first face as the center point and the radius of the circumscribed circle of the face area corresponding to the first face as the area radius, that is, the clear face area of ​​the first face can be the circumscribed circular area of ​​the face area of ​​the first face.

[0319] For example, Fig.16A FIG. 1 is a schematic diagram of a clear face area in an embodiment. Fig.16A As shown, the white area in the portrait mask is the portrait area, the black area is the background area, and the portrait mask includes two human faces. The face clear area 1602 and the face clear area 1604 can be determined based on the face area of ​​each face.

[0320] In the embodiment of the present application, the pixel points belonging to the portrait area in the portrait mask are defined as fourth pixel points. After determining the face clear area corresponding to each face in the portrait mask, the first distance between each fourth pixel point and the center point of the corresponding target face clear area can be determined respectively. The target face clear area can be the target face clear area to which each fourth pixel point is closest.

[0321] In some embodiments, the number of faces included in the LLHDR image can be obtained. If the LLHDR image contains only one face, the face clear area corresponding to the face is the target face clear area corresponding to all fourth pixels in the portrait area of ​​the portrait mask, and the distance between each fourth pixel and the center point of the face clear area corresponding to the face can be determined as the first distance. If the LLHDR image contains at least two faces, the target face clear area closest to each fourth pixel can be determined, and the first distance between each fourth pixel and the center point of the corresponding target face clear area can be calculated respectively.

[0322] In some embodiments, respectively calculating the first distance between each fourth pixel point and the center point of the corresponding target face clear area may include: calculating the third distance from the target pixel point to each face clear area based on the second distance between the target pixel point and the center point of each face clear area, and the area radius of each face clear area; determining the portrait clear area with the smallest third distance as the target face clear area corresponding to the target pixel point; determining the first distance between the target pixel point and the center point of the target face clear area based on the third distance corresponding to the target pixel point to the target face clear area, and the area radius of the target face clear area.

[0323] Among them, the target pixel point can be any pixel point in the portrait area of ​​the portrait mask, that is, the target pixel point can be any fourth pixel point. The coordinates of the center point of each clear face area can be obtained, and the second distance from the target pixel point to the center point of each clear face area can be calculated based on the coordinates of the target pixel point and the coordinates of the center point of each clear face area using the distance formula between two points. The second distance from the target pixel point to the center point of each clear face area can be subtracted from the area radius of the corresponding clear face area to obtain the third distance from the target pixel point to each clear portrait area. Specifically, the third distance from the target pixel point to each clear portrait area can be calculated using formula (19):

[0324] d 3 _ k(x,y) =d 2 _ k(x,y) -r k Formula (19);

[0325] Among them, d 3 _ k(x,y) Indicates the third distance from the pixel point (x, y) to the clear area of ​​the kth portrait, d 2 _ k(x,y) represents the second distance between the pixel point (x, y) and the center point of the k-th portrait clear area, r k represents the area radius of the k-th portrait clear area, where k can be a positive integer greater than or equal to 2.

[0326] After obtaining the third distances from the target pixel to each clear portrait area, the smallest third distance can be selected, and the clear portrait area corresponding to the smallest third distance is used as the target clear portrait area corresponding to the target pixel. The third distance from the target pixel to the target clear portrait area can be added to the area radius of the target face area to obtain the first distance between the target pixel and the center point of the target clear face area. Specifically, the first distance between the target pixel and the center point of the target clear face area can be calculated by formula (20):

[0327]

[0328] in, The smallest third distance among the third distances from the pixel point (x, y) to each portrait clear area, r k′ The radius of the portrait clear area (i.e., the target portrait clear area) corresponding to the minimum third distance, d 1(x,y) Represents the first distance from the pixel point (x, y) to the center point of the target portrait clear area. All fourth pixel points in the portrait area in the portrait mask can be traversed, and each fourth pixel point can be calculated to obtain the first distance to the center point of the target portrait clear area in the manner described in the above embodiment.

[0329] After obtaining the first distance between each fourth pixel and the center point of the corresponding target portrait clear area, the first distance corresponding to each fourth pixel can be normalized to obtain the normalized value corresponding to each fourth pixel, and a weight mask is generated based on the normalized value corresponding to each fourth pixel and the portrait mask. The normalization process may refer to mapping the numerical value to a value within the range of 0 to 1.

[0330] In some embodiments, taking the target pixel point of the portrait area in the portrait mask as an example, the target pixel point can be any fourth pixel point, and the first distance corresponding to each fourth pixel point is normalized to obtain the normalized value corresponding to each fourth pixel point, which may include: determining the portrait transition range corresponding to the target clear face area according to the area radius of the target clear face area corresponding to the target pixel point; calculating the first distance from the target pixel point to the center point of the target clear face area, and the difference between the area radius of the target clear face area, and determining the ratio between the difference and the portrait transition range, and then normalizing the ratio to obtain the normalized value of the target pixel point.

[0331] The portrait transition range may be a pre-set range of a portrait blur transition area, and the area shape and size of the portrait blur transition area may be set according to actual needs. For face clear areas of different sizes, portrait transition ranges of different sizes may correspond respectively, and the size of the portrait transition range may be related to the area radius of the corresponding face clear area. Exemplarily, the portrait transition range may be a circular ring range outside the face clear area and close to the face clear area, and the small circle radius of the circular ring range is the area radius of the face clear area, and the large circle radius may be twice the area radius of the face clear area. That is, the portrait transition range may be the range formed by the area radius to twice the area radius from the center point of the face clear area, such as the area radius of the target face clear area corresponding to the target pixel point is r k′ , then the portrait transition range can be [r k′ , 2r k′ ].

[0332] Specifically, the transition range of the portrait can be [r k′ , 2r k′ ] as an example, the ratio corresponding to the target pixel can be calculated by formula (21):

[0333]

[0334] Among them, d 1(x,y) Indicates the first distance from the pixel point (x, y) to the center point of the target portrait clear area, r k′ Indicates the radius of the target face clear area corresponding to the target pixel, F (x,y) Indicates the ratio corresponding to the pixel point (x, y).

[0335] The ratio corresponding to the target pixel can be normalized. As an implementation method, it can be determined whether the ratio is less than 0 or greater than 1. If the ratio is less than 0, it means that the target pixel belongs to the clear face area, and the normalized value of the target pixel can be determined to be 0; if the ratio is greater than 1, it means that the target pixel is not in the portrait transition range corresponding to the target portrait clear area, and does not belong to the portrait blur transition area or the face clear area, and the normalized value of the target pixel can be determined to be 1. If the ratio is greater than or equal to 0 and less than or equal to 1, it means that the target pixel is in the portrait transition range corresponding to the target portrait clear area and belongs to the portrait blur transition area, and the ratio can be used as the normalized value of the target pixel.

[0336] After the normalized values ​​of the fourth pixels in the portrait area of ​​the portrait mask are calculated according to the above method, the pixel values ​​of the fourth pixels in the portrait mask can be multiplied by the corresponding normalized values ​​to obtain the weight values ​​corresponding to the fourth pixels, so as to generate a weight mask. The weight values ​​corresponding to the fourth pixels are the pixel values ​​corresponding to the fourth pixels in the weight mask. For example, Fig. 16B FIG. 4 is a schematic diagram of a weight mask in one embodiment. Fig. 16B As shown, Fig.16A The weighted mask includes two faces, wherein the left face corresponds to the face clear area 1602 and the portrait blurred transition area 1612 (the area in the left portrait area that gradually changes from black to white), and the right face corresponds to the face clear area 1604 and the portrait blurred transition area 1614 (the area in the right portrait area that gradually changes from black to white). Each face has a corresponding face clear area and portrait blurred transition area. It should be noted that Fig. 16B The two circles and one straight line in the figure are only used to help illustrate the area in the weight mask, and do not mean that the actual weight mask carries the figure.

[0337] In some embodiments, after generating a weight mask based on the normalized values ​​corresponding to each pixel point in the portrait area of ​​the portrait mask and the portrait mask, the weight mask can also be blurred. The blurring process may include median blurring, etc., so that the blurred weight mask is smoother, which can make the blurring effect of the subsequent synthesized blurred portrait image more natural.

[0338] Step 1520, obtaining a blurred image corresponding to the LLHDR image, wherein the blurriness of the blurred image is less than the blurriness of the background area in the background blur image.

[0339] The blurred image is an image obtained by blurring the LLHDR image. The blurring process may include but is not limited to mean blurring process, median blurring process, etc.

[0340] Step 1530, the blurred image and the LLHDR image are fused according to the weight mask to obtain a blurred portrait image.

[0341] Since the weighted mask marks the face clear area corresponding to each face and the portrait blurred transition area corresponding to each face, the face clear area in the obtained portrait blurred image is a clear image area, the portrait area other than the face clear area and the portrait blurred transition area is a blurred image area, and the portrait blurred transition area is a transition area from clear to blurred.

[0342] In some embodiments, the weight mask can be used as the Alpha value of the blurred image, and the blurred image and the LLHDR image can be Alpha fused to obtain a blurred portrait image. Further, assuming that the pixel value of each fourth pixel point in the portrait area in the portrait mask is 255, and the pixel value of the pixel point in the background area is 0, then the weight value of the portrait clear area in the weight mask is 0, and the weight value of the portrait blurred transition area is a value that gradually changes in the range of 0 to 255. The weight values ​​of other portrait areas except the portrait clear area and the portrait blurred transition area are 255, and the pixel values ​​of the pixels in the background area in the weight mask can be 0. Therefore, the face clear area and the background area in the fused blurred portrait image correspond to the image content in the LLHDR image, and the other portrait areas except the portrait clear area and the portrait blurred transition area correspond to the image content in the blurred image. The portrait blurred transition area is the fused image content of the LLHDR image and the blurred image, and the portrait blurred transition area can present a transition effect from clear to blurred.

[0343] Step 1540: Based on the portrait mask, the blurred portrait image and the blurred background image are merged to obtain a target image.

[0344] In some embodiments, the portrait mask can be used as the Alpha value of the blurred portrait image, and the blurred portrait image and the blurred background image can be Alpha-fused to obtain a target image. Each face in the obtained target image remains clear, and other portrait areas except the face remain blurred to a lesser extent. There is a transition area from clear to blurred, which can avoid the problem of no prominent focus caused by all portrait areas in the image being clear, and the image blurring effect is more natural, improving the visual display effect of the image. In addition, when the LLHDR image contains multiple faces, it can ensure that each face in the obtained target image is clear, which can avoid the situation where only one person is in focus while other faces are blurred, thereby improving the image blurring effect.

[0345] In some embodiments, in order to improve the performance of the portrait in the target image, before the blurred image and the LLHDR image are fused, the face of the LLHDR image may be re-lit to brighten the portrait area of ​​the LLHDR image. Re-lighting the face of the LLHDR image may include: determining the brightness information of the clear face area corresponding to each face in the LLHDR image according to the weight mask, and calculating the average brightness of the face according to the brightness information; determining the face brightening parameter according to the average brightness of the face; and brightening the LLHDR image according to the face brightening parameter and the weight mask.

[0346] The weight mask marks the face clear area and portrait blurred transition area corresponding to each face. The face clear area can be used as the face brightening area, and the portrait blurred transition area can be used as the brightening transition area. In this way, the portrait area closer to the face in the brightened LLHDR image is brightened more, achieving a gradual brightening effect from the center of the face to the outside, to simulate soft fill light.

[0347] The brightness information of the clear face area corresponding to each face in the LLHDR image can be determined according to the weight mask. The brightness information includes the brightness value (which can be a brightness component value) of each pixel point belonging to the clear portrait area, and the average value is calculated according to the brightness value of each pixel point belonging to the clear portrait area in the LLHDR image to obtain the average brightness of the face.

[0348] After calculating the average brightness of the face, the face brightening parameter can be determined according to the average brightness of the face. The face brightening parameter can be used to characterize the degree of brightening. The larger the face brightening parameter, the more brightening it is. Optionally, the face brightening parameter and the average brightness of the face may be negatively correlated. When the average brightness of the face is small, the face brightening parameter may be larger, and the brightening effect is more obvious. In some embodiments, the face brightening parameter can be determined according to the average brightness of the face and the preset brightening parameter. The preset brightening parameter can be set according to the actual brightness adjustment needs of the user, or it can be determined by the research and development personnel based on multiple image processing experiments. When the average brightness of the face is determined, the preset brightening parameter and the face brightening parameter may be positively correlated. The larger the preset brightening parameter, the larger the face brightening parameter.

[0349] Specifically, the face brightening parameter can be determined by formula (22):

[0350]

[0351] Among them, FaceBright represents the preset brightening parameter, faceavg represents the average brightness of the face, and FaceBright 1 Indicates the determined face brightening parameters.

[0352] After determining the face brightening parameters, the portrait area of ​​the LLHDR image can be brightened according to the face brightening parameters and the weight mask, wherein the clear face area in the LLHDR image corresponds to a first brightening degree, and other portrait areas in the LLHDR image except the clear face area and the portrait blurred transition area correspond to a second brightening degree, the first brightening degree is greater than the second brightening degree, and the portrait blurred transition area gradually transitions from the first brightening degree to the second brightening degree.

[0353] In some embodiments, the brightness improvement value corresponding to each pixel in the portrait area of ​​the LLHDR image can be calculated according to the face brightening parameter and the pixel value of each pixel in the portrait area of ​​the LLHDR image in the weight mask, and the sum of the brightness value of each pixel in the portrait area of ​​the LLHDR image and the corresponding brightness improvement value can be calculated to obtain the LLHDR image after brightening. Specifically, the LLHDR image can be brightened by formula (23):

[0354]

[0355] Among them, Y (x,y) WeightMask represents the brightness value of the pixel point (x, y) in the portrait area of ​​the LLHDR image. (x,y) Indicates the pixel value (i.e. weight value) of the pixel point (x, y) in the weight mask, Y′ (x,y)Represents the brightness value of the pixel (x, y) after brightening, that is, the brightness value in the LLHDR image after brightening. Optionally, since the pixel value of the clear face area in the weight mask is 0, the pixel points belonging to the clear portrait area are brightened to the greatest extent, while the pixel values ​​of other portrait areas except the clear face area and the blurred portrait transition area in the weight mask are 255, so there is no brightening, and the blurred portrait transition area gradually transitions from the brightening degree of the clear face area to other portrait areas that are not brightened, achieving a gradual brightening effect from the center of the face to the outside. By relighting the portrait of the LLHDR image, the image performance of the face in the LLHDR image can be improved.

[0356] After brightening the LLHDR image, the blurred image and the brightened LLHDR image can be fused according to the weight mask to obtain a blurred portrait image. The blurred image can be obtained by blurring the LLHDR image before brightening or by blurring the LLHDR image after brightening.

[0357] It should be noted that the portrait mask involved in the portrait refocusing and portrait relighting process can be the portrait mask obtained from the foreground recognition result, or the portrait mask after the portrait area edge optimization in the above embodiment, or the portrait mask after the hair area edge optimization (that is, the above-mentioned target portrait mask), or the portrait mask after the portrait area edge optimization and the hair area edge optimization.

[0358] In the embodiment of the present application, by refocusing the LLHDR image for the portrait, the face of each portrait subject in the target image can be kept clear, and the portrait area other than the face can be kept blurred to a small extent, which can avoid the problem of no prominent focus caused by all portrait areas in the image being clear, and can avoid the situation where only one person is focused while other faces are blurred, thereby improving the image blur effect. In addition, by relighting the LLHDR image for the portrait, a gradual brightening effect from the center of the face to the outside can be achieved to simulate soft fill light, improve the visual effect of the portrait in the target image, and improve the image effect of the target image.

[0359] In one embodiment, an image processing method is provided, which may include the following steps:

[0360] 1. Obtain scene information and generate an exposure table based on the scene information;

[0361] 2. According to the multiple exposure values ​​included in the exposure table, control the imaging device to collect multiple frames of original images in RAW format corresponding to the multiple exposure values, wherein the multiple frames of original images include at least one bright frame image and at least one dark frame image;

[0362] 3. Optimize and process multiple frames of original images through ISP to obtain multiple frames of images to be processed in YUV format;

[0363] 4. Perform noise reduction on the bright frame image in YUV format, and fuse the bright frame image after noise reduction with the dark frame image in YUV format to obtain the LLHDR image;

[0364] 5. Perform face recognition on the LLHDR image to obtain the face information of each face contained in the LLHDR image, and use the face information to perform portrait segmentation processing, hair area segmentation processing and hair area cutout processing on the LLHDR image to obtain a portrait mask, a hair segmentation result and a hair cutout result;

[0365] 6. Generate a blur intensity map corresponding to the LLHDR image based on the portrait mask and hair segmentation results, and generate a diffusion weight map based on the LLHDR image;

[0366] 7. Performing point diffusion processing on each pixel point in the background area of ​​the LLHDR image based on the blur intensity map and the diffusion weight map to obtain a first background blur image;

[0367] 8. Superimposing the 3DLUT filter on the first background blur image to obtain a second background blur image;

[0368] 9. Superimposing noise on the background area of ​​the second background blur image to obtain a third background blur image;

[0369] 10. Optimize the edge of the portrait area in the portrait mask to obtain an edge-optimized portrait mask;

[0370] 11. According to the edge-optimized portrait mask and the hair segmentation result, the inner transition parameter is determined, and the hair area in the edge-optimized portrait mask is blurred according to the inner transition parameter to obtain a new portrait mask;

[0371] 11. Determine the background complexity of the LLHDR image. When the background complexity is less than or equal to the first complexity threshold, perform a first optimization process on the hair cutout result to obtain a target cutout result, and fuse the target cutout result with the new portrait mask to obtain a target portrait mask. When the background complexity is greater than the first complexity threshold, perform a second optimization process on the hair segmentation result to obtain a target segmentation result, and fuse the target segmentation result with the new portrait mask to obtain a target portrait mask.

[0372] 12. Relight the LLHDR image to obtain a brightened LLHDR image;

[0373] 13. Refocus the portrait on the brightened LLHDR image to obtain a blurred portrait image;

[0374] 14. Based on the target portrait mask, the blurred portrait image is fused with the third background blurred image to obtain the target image.

[0375] The description of each step in the embodiment of the present application is described in detail in the above embodiment, and the relevant description in the above embodiment can be referred to, and will not be repeated here. It should be noted that the execution order of the above steps 1 to 14 can be changed or some steps can be deleted, and other image processing steps can be added, provided that they comply with the processing logic, and this is not limited here.

[0376] In the embodiment of the present application, an image blurring technology is provided from the perspective of the combination of software and hardware. The ISP is first used to adjust the basic image quality, and then noise reduction and highlight compression are performed from the software layer to obtain an LLHDR image, so that the background area of ​​the target image obtained by subsequent blurring maintains a high dynamic range; portrait recognition and point diffusion blurring processing are performed based on the LLHDR image to simulate the blurring effect of a large aperture; by optimizing the edge of the portrait area and the edge of the hair area, a more accurate target portrait mask can be obtained, so that the hair part of the target image obtained based on the target portrait mask is accurately blurred and the transition is natural; the LLHDR image is re-lit for the portrait and re-focused for the portrait, and the portrait area in the target image is adjusted to make the portrait in the target image have a better visual effect. The embodiment of the present application optimizes the blurring effect of the image from multiple aspects, and improves the blurring expressiveness of the portrait image.

[0377] like Fig.17 As shown, in one embodiment, an image processing device 1700 is provided, which can be applied to the above electronic device. The image processing device 1700 may include an image acquisition module 1710 , a high dynamic module 1720 , a foreground recognition module 1730 and a blur module 1740 .

[0378] The image acquisition module 1710 is used to acquire multiple frames of images to be processed, wherein the multiple frames of images to be processed include at least one bright frame image and at least one dark frame image, the bright frame image is an image to be processed whose exposure value is greater than or equal to an exposure threshold, and the dark frame image is an image to be processed whose exposure value is less than the exposure threshold.

[0379] The high dynamic range module 1720 is used to perform noise reduction processing on at least one bright frame image, and fuse the bright frame image after noise reduction processing with at least one dark frame image to obtain a low-light high dynamic range LLHDR image.

[0380] The foreground recognition module 1730 is used to recognize the foreground area in the LLHDR image and obtain a foreground recognition result.

[0381] The blur module 1740 is used to blur the LLHDR image based on the foreground recognition result to obtain a target image.

[0382] In one embodiment, the image acquisition module 1710 is also used to acquire scene information and generate an exposure table based on the scene information; wherein the scene information includes one or more of scene type, scene brightness, and sensitivity ISO value; and is used to control the imaging device to collect multiple frames of to-be-processed images corresponding to the multiple exposure values ​​respectively according to the multiple exposure values ​​included in the exposure table.

[0383] In one embodiment, the high dynamic module 1720 includes a noise reduction unit and a highlight suppression unit.

[0384] A denoising unit is used to select a reference frame image from at least two bright frame images; align the at least two bright frame images to obtain a first aligned area and a first non-aligned area between the reference frame image and other bright frame images; based on the at least two bright frame images, perform denoising on the first aligned area in the reference frame image, and perform single-frame denoising on the first non-aligned area of ​​the reference frame image to obtain a denoised image.

[0385] A highlight compression unit is used to align the denoised image and at least one dark frame image to obtain a second aligned area and a second non-aligned area between the denoised image and the at least one dark frame image; and based on the denoised image and the at least one dark frame image, perform chromaticity adjustment processing on the second aligned area in the denoised image, and perform single-frame color enhancement processing on the second non-aligned area of ​​the denoised image to obtain an LLHDR image.

[0386] In one embodiment, the image to be processed includes an original image in a first image format acquired by an image sensor. The above-mentioned image processing device 1700 may also include a basic tuning module, which is used to optimize the multiple frames of original images through ISP to obtain multiple frames of images to be processed in a second image format. The first image format includes a RAW format, the second image format includes a YUV format or an RGB format, and the optimization processing includes one or more of color optimization, exposure optimization, and noise optimization.

[0387] The high dynamic module 1720 is also used to perform noise reduction processing on at least one bright frame image in the second image format, and to fuse the bright frame image after noise reduction processing with at least one dark frame image in the second image format to obtain a low-light high dynamic range LLHDR image.

[0388] In one embodiment, the blur module 1740 includes a blur unit and a fusion unit.

[0389] The blurring unit is used to blur the background area of ​​the LLHDR image based on the foreground recognition result to obtain a background blurred image.

[0390] The fusion unit is used to fuse the background blur image with the LLHDR image based on the foreground recognition result to obtain the target image.

[0391] In one embodiment, the blurring unit is further used to obtain a blurring intensity map of the LLHDR image according to the foreground recognition result, and to obtain a diffusion weight map corresponding to the LLHDR image, wherein the blurring intensity map includes a blurring radius corresponding to each first pixel in the background area of ​​the LLHDR image, and the diffusion weight map includes a diffusion weight corresponding to each first pixel in the background area of ​​the LLHDR image; according to the blurring intensity map and the diffusion weight map, point diffusion processing is performed on each first pixel to obtain a diffusion result corresponding to each first pixel; and the diffusion results corresponding to each first pixel are superimposed, and the background area of ​​the LLHDR image is blurred according to the superimposed result to obtain a background blurred image.

[0392] In one embodiment, the blurring unit is further used to determine a target blurring radius according to a blurring intensity parameter, and to generate a blurring intensity map of the LLHDR image according to the target blurring radius and a foreground recognition result, wherein the blurring radius of each second pixel point in the foreground area of ​​the LLHDR image in the blurring intensity map is 0, and the blurring radius of each first pixel point in the background area of ​​the LLHDR image in the blurring intensity map is the target blurring radius.

[0393] In one embodiment, the blurring unit is further used to downsample the LLHDR image and the foreground recognition result to obtain a LLHDR downsampled image and a downsampled foreground result of the same size; obtain a blur strength map of the LLHDR downsampled image according to the downsampled foreground result, and obtain a diffusion weight map corresponding to the LLHDR downsampled image; and to blur the background area of ​​the LLHDR downsampled image according to the superposition result after superimposing the diffusion results corresponding to each first pixel point in the background area of ​​the LLHDR downsampled image to obtain a downsampled blurred image, and upsample the downsampled blurred image to obtain a background blurred image of the same size as the LLHDR image.

[0394] In one embodiment, the blur module 1740 further includes a LUT filter overlay unit.

[0395] The LUT filter overlay unit is used to obtain the pixel value corresponding to each pixel point in the background blur image; map the pixel value corresponding to each pixel point in the background blur image to the lookup table LUT range to obtain the mapping result; and based on the LUT mask, fuse the mapping result and the background blur image, the LUT mask is used to determine the image area that needs to be overlaid with the LUT filter.

[0396] In one embodiment, the blurring module 1740 further includes a noise superposition unit.

[0397] The noise superposition unit is used to determine a noise estimation area in the foreground area of ​​the LLHDR image; determine a noise parameter corresponding to the noise estimation area, the noise parameter is used to characterize the noise level of the noise estimation area; and superimpose noise in the background area of ​​the background blur image according to the noise parameter.

[0398] In one embodiment, the foreground recognition result includes a portrait mask. The above-mentioned image processing module 1700 also includes an edge optimization module.

[0399] The edge optimization module is used to extract the edge of the portrait area in the portrait mask to generate a first edge mask; align, smooth and transition the contour edge in the first edge mask to obtain a second edge mask; and update the portrait mask according to the contour edge in the second edge mask.

[0400] In one embodiment, the foreground recognition result includes a first portrait mask, a hair segmentation result, and a hair cutout result, wherein the first portrait mask is used to characterize the position of the portrait area in the LLHDR image, the hair segmentation result is obtained by performing a segmentation process on the hair area of ​​the LLHDR image, and the hair cutout result is obtained by performing a cutout process on the hair area of ​​the LLHDR image.

[0401] In one embodiment, an edge optimization module is used to determine the background complexity of the LLHDR image according to the LLHDR image, the first portrait mask and the hair segmentation result; if the background complexity is less than or equal to the first complexity threshold, a target portrait mask is generated according to the hair cutout result and the first portrait mask; if the background complexity is greater than the first complexity threshold, a target portrait mask is generated according to the first portrait mask and the hair segmentation result.

[0402] In one embodiment, the edge optimization module is further used to determine an inner transition parameter based on the first portrait mask and the hair segmentation result; blur the hair area of ​​the first portrait mask according to the inner transition parameter to obtain a second portrait mask; merge the second portrait mask with the first portrait mask to obtain a new first portrait mask; and generate a target portrait mask based on the hair cutout result and the new first portrait mask if the background complexity is less than or equal to the first complexity threshold, and generate a target portrait mask based on the new first portrait mask and the hair segmentation result if the background complexity is greater than the first complexity threshold.

[0403] The fusion unit is further used to fuse the background blur image with the LLHDR image based on the target portrait mask to obtain the target image.

[0404] In one embodiment, the fusion unit is further used to generate a weight mask according to the portrait mask and the face information, the weight mask is used to determine the face clear area corresponding to each face in the LLHDR image, and the portrait blurred transition area corresponding to each face; obtain a blurred image corresponding to the LLHDR image, the blur degree of the blurred image is less than the blur degree of the background area in the background blurred image; fuse the blurred image and the LLHDR image according to the weight mask to obtain a portrait blurred image; based on the portrait mask, fuse the portrait blurred image with the background blurred image to obtain a target image. Among them, the face clear area refers to the area in the portrait area of ​​the LLHDR image that needs to be kept clear, and the portrait blurred transition area refers to the change area from clear to blurred in the portrait area.

[0405] In one embodiment, the fusion unit is further used to determine the brightness information of the clear face area corresponding to each face in the LLHDR image according to the weight mask, and calculate the average brightness of the face according to the brightness information; determine the face brightening parameter according to the average brightness of the face; brighten the LLHDR image according to the face brightening parameter and the weight mask, wherein the clear face area in the LLHDR image corresponds to a first brightening degree, and other portrait areas in the LLHDR image except the clear face area and the portrait blurred transition area correspond to a second brightening degree, the first brightening degree is greater than the second brightening degree, and the portrait blurred transition area gradually transitions from the first brightening degree to the second brightening degree; and fuse the blurred image and the brightened LLHDR image according to the weight mask to obtain a blurred portrait image.

[0406] Fig.18 FIG. 1 is a structural block diagram of an electronic device in an embodiment. Fig.18As shown, the electronic device 1800 may include one or more of the following components: a processor 1810, and a memory 1820 coupled to the processor 1810, wherein the memory 1820 may store one or more computer programs, and the one or more computer programs may be configured to implement the methods described in the above embodiments when executed by one or more processors 1810.

[0407] The processor 1810 may include one or more processing cores. The processor 1810 uses various interfaces and lines to connect various parts of the entire electronic device 1800, and executes various functions and processes data of the electronic device 1800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1820, and calling data stored in the memory 1820. Optionally, the processor 1810 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1810 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1810, but may be implemented separately through a communication chip.

[0408] The memory 1820 may include a random access memory (RAM) or a read-only memory (ROM). The memory 1820 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 1820 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created by the electronic device 1800 during use, etc.

[0409] It is understandable that the electronic device 1800 may include more or fewer structural elements than those in the above structural block diagram, for example, a power module, physical buttons, a WiFi (Wireless Fidelity) module, a speaker, a Bluetooth module, a sensor, etc., and is not limited here.

[0410] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program implements the method described in the above embodiment when executed by a processor.

[0411] An embodiment of the present application discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program can be executed by a processor to implement the methods described in the above embodiments.

[0412] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a ROM, etc.

[0413] As used herein, any reference to memory, storage, database, or other medium may include nonvolatile and / or volatile memory. Suitable nonvolatile memory may include ROM, programmable ROM (Programmable ROM, PROM), erasable PROM (Erasable PROM, EPROM), electrically erasable PROM (Electrically Erasable PROM, EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. By way of illustration and not limitation, RAM may be in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (RDRAM) and direct memory bus dynamic RAM (DRDRAM).

[0414] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present application.

[0415] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0416] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0417] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0418] The above is a detailed introduction to an image processing method, device, electronic device, and computer-readable storage medium disclosed in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those of ordinary skill in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. An image processing method, characterized in that: include: Acquire multiple frames of images to be processed, wherein the multiple frames of images to be processed include at least one bright frame image and at least one dark frame image, the bright frame image is an image to be processed whose exposure value is greater than or equal to an exposure threshold, and the dark frame image is an image to be processed whose exposure value is less than the exposure threshold; Performing noise reduction processing on at least one frame of the bright frame image, and fusing the bright frame image after the noise reduction processing with the at least one frame of the dark frame image to obtain a low-light high dynamic range (LLHDR) image; Identify the foreground area in the LLHDR image to obtain a foreground identification result; the foreground identification result includes a first portrait mask, a hair segmentation result and a hair cutout result, wherein the first portrait mask is used to characterize the position of the portrait area in the LLHDR image, the hair segmentation result is obtained by performing a segmentation process on the hair area of ​​the LLHDR image, and the hair cutout result is obtained by performing a cutout process on the hair area of ​​the LLHDR image; Based on the foreground recognition result, a background area of ​​the LLHDR image is blurred to obtain a background blurred image; Based on the foreground recognition result, the background blur image is fused with the LLHDR image to obtain a target image; Before fusing the background blur image with the LLHDR image based on the foreground recognition result to obtain the target image, the method further includes: Determining the background complexity of the LLHDR image according to the LLHDR image, the first portrait mask and the hair segmentation result; If the background complexity is less than or equal to a first complexity threshold, generating a target portrait mask according to the hair cutout result and the first portrait mask; If the background complexity is greater than the first complexity threshold, generating a target portrait mask according to the first portrait mask and the hair segmentation result; Based on the foreground recognition result, the background blur image is fused with the LLHDR image to obtain a target image, including: Based on the target portrait mask, the background blur image is fused with the LLHDR image to obtain a target image.

2. The method according to claim 1, characterized in that: Before acquiring the multiple frames of images to be processed, the method further includes: Acquire scene information, and generate an exposure table according to the scene information; wherein the scene information includes one or more of scene type, scene brightness, and ISO value; The step of acquiring multiple frames of images to be processed comprises: According to the multiple exposure values ​​included in the exposure table, the imaging device is controlled to collect multiple frames of to-be-processed images respectively corresponding to the multiple exposure values.

3. The method according to claim 1, characterized in that The multiple frames of images to be processed include at least two frames of bright frame images; and the noise reduction process on at least one frame of the bright frame image includes: Selecting a reference frame image from the at least two bright frame images; Aligning the at least two bright frame images to obtain a first aligned region and a first non-aligned region between the reference frame image and other bright frame images; Based on the at least two bright frame images, the first aligned region in the reference frame image is subjected to noise reduction processing, and the first non-aligned region of the reference frame image is subjected to single-frame noise reduction processing to obtain a noise-reduced image.

4. The method according to claim 3, characterized in that The bright frame image after noise reduction processing is merged with the at least one dark frame image to obtain a low-light high dynamic range (LLHDR) image, including: Aligning the denoised image and the at least one dark frame image to obtain a second aligned region and a second non-aligned region between the denoised image and the at least one dark frame image; Based on the denoised image and the at least one dark frame image, a chromaticity adjustment process is performed on the second aligned area in the denoised image, and a single-frame color enhancement process is performed on the second non-aligned area of ​​the denoised image to obtain a LLHDR image.

5. The method according to claim 1, characterized in that The blurring the background area of ​​the LLHDR image based on the foreground recognition result to obtain a background blurred image includes: Acquire a blur intensity map of the LLHDR image according to the foreground recognition result, and acquire a diffusion weight map corresponding to the LLHDR image, wherein the blur intensity map includes a blur radius corresponding to each first pixel point in the background area of ​​the LLHDR image, and the diffusion weight map includes a diffusion weight corresponding to each first pixel point in the background area of ​​the LLHDR image; Performing point diffusion processing on each of the first pixel points according to the blur strength map and the diffusion weight map to obtain a diffusion result corresponding to each of the first pixel points; The diffusion results corresponding to the first pixel points are superimposed, and the background area of ​​the LLHDR image is blurred according to the superimposed results to obtain a background blurred image.

6. The method according to claim 5, characterized in that The acquiring the blur intensity map of the LLHDR image according to the foreground recognition result includes: A target blur radius is determined according to a blur intensity parameter, and a blur intensity map of the LLHDR image is generated according to the target blur radius and the foreground recognition result, wherein the blur radius of each second pixel point in the foreground area of ​​the LLHDR image in the blur intensity map is 0, and the blur radius of each first pixel point in the background area of ​​the LLHDR image in the blur intensity map is the target blur radius.

7. The method according to claim 5, characterized in that Before acquiring the blur intensity map of the LLHDR image according to the foreground recognition result and acquiring the diffusion weight map corresponding to the LLHDR image, the method further includes: Downsampling the LLHDR image and the foreground recognition result to obtain a LLHDR downsampled image and a downsampled foreground result of the same size; The step of obtaining a blur intensity map of the LLHDR image according to the foreground recognition result and obtaining a diffusion weight map corresponding to the LLHDR image includes: Obtaining a blur intensity map of the LLHDR downsampled image according to the downsampled foreground result, and obtaining a diffusion weight map corresponding to the LLHDR downsampled image; The method of performing blurring processing on the background area of ​​the LLHDR image according to the superposition result to obtain a background blur image includes: The background area of ​​the LLHDR downsampled image is blurred according to the superposition result to obtain a downsampled blurred image, and the downsampled blurred image is upsampled to obtain a background blurred image of the same size as the LLHDR image.

8. The method according to any one of claims 5 to 7, characterized in that: After obtaining the background blurred image, the method further includes: Obtaining pixel values ​​corresponding to each pixel point in the background blur image; Mapping the pixel value corresponding to each pixel point in the background blur image to the range of the lookup table LUT to obtain a mapping result; Based on the LUT mask, the mapping result and the background blur image are fused, and the LUT mask is used to determine the image area where the LUT filter needs to be superimposed.

9. The method according to any one of claims 5 to 7, characterized in that: After obtaining the background blurred image, the method further includes: determining a noise estimation region in a foreground region of the LLHDR image; Determine a noise parameter corresponding to the noise estimation area, where the noise parameter is used to characterize a noise level in the noise estimation area; Noise is superimposed on the background area of ​​the background blur image according to the noise parameter.

10. The method according to claim 1, characterized in that The foreground recognition result includes a portrait mask; before fusing the background blur image with the LLHDR image based on the foreground recognition result to obtain a target image, the method further includes: Extracting the edge of the portrait area in the portrait mask to generate a first edge mask; Performing alignment processing, smoothing processing and transition processing on the contour edge in the first edge mask to obtain a second edge mask; The portrait mask is updated according to the contour edge in the second edge mask.

11. The method according to claim 1, characterized in that Before determining the background complexity of the LLHDR image according to the LLHDR image, the first portrait mask and the hair segmentation result, the method further includes: Determining an inner transition parameter according to the first portrait mask and the hair segmentation result; Performing blur processing on the hair region of the first portrait mask according to the inner transition parameter to obtain a second portrait mask; Merging the second portrait mask with the first portrait mask to obtain a new first portrait mask; Generating a target portrait mask according to the hair cutout result and the first portrait mask includes: generating a target portrait mask according to the hair cutout result and the new first portrait mask; The generating a target portrait mask according to the first portrait mask and the hair segmentation result includes: generating a target portrait mask according to the new first portrait mask and the hair segmentation result.

12. The method according to claim 1, characterized in that The foreground recognition result includes facial information of each face contained in the LLHDR image and a portrait mask of the LLHDR image; Based on the foreground recognition result, the background blur image is fused with the LLHDR image to obtain a target image, including: Generate a weight mask according to the portrait mask and the face information, wherein the weight mask is used to determine a face clear area corresponding to each face in the LLHDR image, and a portrait blurred transition area corresponding to each face; wherein the face clear area refers to an area in the portrait area of ​​the LLHDR image that needs to be kept clear, and the portrait blurred transition area refers to an area in the portrait area that changes from clear to blurred; Acquire a blurred image corresponding to the LLHDR image, wherein the blur degree of the blurred image is less than the blur degree of the background area in the background blur image; fusing the blurred image and the LLHDR image according to the weight mask to obtain a blurred portrait image; Based on the portrait mask, the blurred portrait image and the blurred background image are fused to obtain a target image.

13. The method according to claim 12, characterized in that Before acquiring the blurred image corresponding to the LLHDR image, the method further includes: Determine the brightness information of the clear face area corresponding to each face in the LLHDR image according to the weight mask, and calculate the average brightness of the face according to the brightness information; Determining a face brightening parameter according to the average brightness of the face; According to the face brightening parameter and the weight mask, the LLHDR image is brightened, wherein the clear face area in the LLHDR image corresponds to a first brightening degree, and other portrait areas in the LLHDR image except the clear face area and the portrait blur transition area correspond to a second brightening degree, the first brightening degree is greater than the second brightening degree, and the portrait blur transition area gradually transitions from the first brightening degree to the second brightening degree; The step of fusing the blurred image and the LLHDR image according to the weight mask to obtain a blurred portrait image includes: The blurred image and the brightened LLHDR image are fused according to the weight mask to obtain a blurred portrait image.

14. The method according to claim 1, characterized in that The image to be processed includes an original image in a first image format acquired by an image sensor; after acquiring multiple frames of images to be processed, the method further includes: Performing optimization processing on multiple frames of the original image through ISP to obtain multiple frames of images to be processed in a second image format, wherein the first image format includes a RAW format, the second image format includes a YUV format or an RGB format, and the optimization processing includes one or more of color optimization, exposure optimization, and noise optimization; The step of performing noise reduction processing on at least one bright frame image and fusing the bright frame image after noise reduction processing with the at least one dark frame image to obtain a low-light high dynamic range (LLHDR) image comprises: Noise reduction processing is performed on at least one bright frame image in the second image format, and the bright frame image after the noise reduction processing is fused with at least one dark frame image in the second image format to obtain a low-light high dynamic range LLHDR image.

15. An image processing device, characterized in that: include: An image acquisition module, used to acquire multiple frames of images to be processed, wherein the multiple frames of images to be processed include at least one bright frame image and at least one dark frame image, the bright frame image is an image to be processed whose exposure value is greater than or equal to an exposure threshold, and the dark frame image is an image to be processed whose exposure value is less than the exposure threshold; A high dynamic range module is used to perform noise reduction processing on at least one frame of the bright frame image, and fuse the bright frame image after the noise reduction processing with the at least one frame of the dark frame image to obtain a low light high dynamic range LLHDR image; a foreground recognition module, configured to recognize a foreground area in the LLHDR image and obtain a foreground recognition result; the foreground recognition result includes a first portrait mask, a hair segmentation result, and a hair cutout result, wherein the first portrait mask is used to characterize the position of the portrait area in the LLHDR image, the hair segmentation result is obtained by performing a segmentation process on the hair area of ​​the LLHDR image, and the hair cutout result is obtained by performing a cutout process on the hair area of ​​the LLHDR image; A blur module, used for blurring the LLHDR image based on the foreground recognition result to obtain a target image; The blur module includes a blur unit and a fusion unit; The blurring unit is used to blur the background area of ​​the LLHDR image based on the foreground recognition result to obtain a background blurred image; The fusion unit is used to fuse the background blur image with the LLHDR image based on the foreground recognition result to obtain a target image; an edge optimization module, configured to determine the background complexity of the LLHDR image according to the LLHDR image, the first portrait mask and the hair segmentation result; if the background complexity is less than or equal to a first complexity threshold, generate a target portrait mask according to the hair cutout result and the first portrait mask; if the background complexity is greater than the first complexity threshold, generate a target portrait mask according to the first portrait mask and the hair segmentation result; The fusion unit is further used to fuse the background blur image with the LLHDR image based on the target portrait mask to obtain a target image.

16. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor implements the method according to any one of claims 1 to 14.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images