Image processing method, image processing device and storage medium

Abstract

The invention relates to an image processing method, an image processing device and a storage medium. The image processing method comprises the following steps: in response to an image processing instruction, separating a to-be-processed image into a target main body image and a background image; and respectively carrying out fuzzy rendering processing on the target main body image and the background image. And determining a target main body weight map corresponding to the target main body image according to the target main body image, and determining a background weight map corresponding to the background image. And according to the target main body weight map and the background weight map, fusing the target main body image and the background image which are subjected to fuzzy rendering processing to obtain a target image. According to the invention, the terminal which is limited in photographing hardware and chip performance and has a photographing function can obtain the target image which has a good blurring effect and is close to an optical blurring effect based on an optical element based on a software algorithm.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, an image processing device and a storage medium. Background Art

[0002] Given the limited camera hardware of mobile phones, it is impossible to achieve the blurring effect based on optical hardware like large-aperture SLR cameras, and blurring needs to be achieved through software algorithms.

[0003] The weighted fusion blur technology, aggregate blur and dispersion blur in the related technologies are limited by the limited terminal performance and the inherent defects of the algorithm. They cannot achieve the optical blur effect based on optical hardware. There are artifacts in the boundary between the main area and the background area of ​​the target object. The rendering method based on deep learning is limited by the limitation of training data and is only suitable for camera terminals that obtain training data. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, an image processing device and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising: in response to an image processing instruction, separating an image to be processed into a target subject image and a background image; performing blur rendering processing on the target subject image and the background image respectively; determining a target subject weight map corresponding to the target subject image and a background weight map corresponding to the background image according to the target subject image, wherein each pixel in the target subject weight map corresponds one-to-one to each pixel in the target subject image and includes a fusion weight of each pixel in the target subject image, and each pixel in the background weight map corresponds one-to-one to each pixel in the background image and includes a fusion weight of each pixel in the background image; and according to the target subject weight map and the background weight map, the target subject image and the background image after blur rendering processing are fused to obtain a target image.

[0006] In one embodiment, the image to be processed is acquired in the following manner: acquiring the original image, determining the blur radius of each pixel in the original image when blurring is performed, setting a statistical histogram containing the blur radius of each pixel in the original image when blurring is performed, wherein the horizontal axis data of the statistical histogram corresponds to different blur radii, and the vertical axis data corresponds to the proportion of pixels with different blur radii in the entire original image; determining the starting blur radius of the pixel corresponding to the starting point of the valley area of ​​the statistical histogram when blurring is performed, determining the minimum blur radius among the different blur radii when each pixel of the background image in the original image is processed, and determining the absolute value of the difference between the starting blur radius and the minimum blur radius; in response to the absolute value being greater than a length threshold, determining the original image as the image to be processed.

[0007] In one embodiment, the starting fuzzy radius is determined in the following manner: in response to the presence of a section in the statistical histogram whose vertical axis data is less than a proportion threshold, the fuzzy radius corresponding to the starting node in the statistical histogram whose vertical axis data is less than the proportion threshold is determined as the starting fuzzy radius; in response to the absence of a section in the statistical histogram whose vertical axis data is less than the proportion threshold, the maximum fuzzy radius in the statistical histogram is determined as the starting fuzzy radius.

[0008] In one embodiment, the separation of the image to be processed into the target subject image and the background image includes: obtaining an object mask of the image to be processed, the object mask corresponding to the same category of object images in the image to be processed, the same category of objects including the target subject and objects of the same category as the target subject, the grayscale value of the area corresponding to the same category of object images in the object mask is the maximum grayscale value, and the grayscale value of the area corresponding to the background image is the minimum grayscale value; extracting the same category of object images in the image to be processed through the object mask, and separating the image to be processed into the same category of object images and non-same category of object images. an image of an object; in response to the presence of multiple objects of the same category in the image to be processed, determining depth maps corresponding to the images of the objects of the same category, and determining an image of the same category whose image depth is less than an image threshold as the target subject image, and determining a union of images other than the target subject image in the images of the same category of objects and the images of objects of different categories as the background image; in response to the absence of other objects of the same category as the target subject in the image to be processed, determining the image of the object of the same category extracted by the object mask as the target subject image, and determining the image of the object of different categories as the background image.

[0009] In one embodiment, determining the target subject weight map of the target subject image includes: obtaining a target subject mask based on the image to be processed and the target subject image, or obtaining a target subject mask based on the image to be processed and the background image, wherein the target subject mask corresponds to the target subject image; setting the grayscale value of each pixel in the target subject mask as a weight value, and determining the target subject mask that has been set as the target subject weight map.

[0010] In one embodiment, determining the background weight map of the background image includes: inverting the grayscale value of each pixel in the target subject mask, converting the maximum grayscale value into the minimum grayscale value, and converting the minimum grayscale value into the maximum grayscale value, and determining the inverted target subject mask as the initial background weight map; determining the boundary of the background area in the initial background weight map as the first boundary, and determining the maximum blur radius when the target subject image is blurred, and determining the boundary after the first boundary is extended to the target subject image by the maximum blur radius value as the second boundary; determining the fused weight map between the first boundary and the second boundary according to the first boundary, the second boundary, the maximum blur radius and a preset weight determination method; and determining the fused image of the initial background weight map and the fused weight map as the background weight map.

[0011] In one embodiment, the target subject image and the background image that have been blurred and rendered are fused according to the target subject weight map and the background weight map to obtain the target image, including: multiplying the pixel value of each pixel in the target subject image by the weight of each pixel corresponding to the target subject weight map and dividing the result by the maximum pixel value to obtain a first weighted image; multiplying the pixel value of each pixel in the background image by the weight of each pixel corresponding to the background weight map and dividing the result by the maximum pixel value to obtain a second weighted image; and determining the fused image of the first weighted image and the second weighted image as the target image.

[0012] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing device, comprising: a processing unit, for separating an image to be processed into a target subject image and a background image in response to an image processing instruction; a rendering unit, for performing blur rendering processing on the target subject image and the background image respectively; a determination unit, for determining a target subject weight map corresponding to the target subject image and a background weight map corresponding to the background image according to the target subject image, each pixel in the target subject weight map corresponds one-to-one to each pixel in the target subject image, including a fusion weight of each pixel in the target subject image, and each pixel in the background weight map corresponds one-to-one to each pixel in the background image, including a fusion weight of each pixel in the background image; a fusion unit, for fusing the target subject image and the background image after blur rendering processing according to the target subject weight map and the background weight map to obtain a target image.

[0013] In one embodiment, the image to be processed is acquired by the processing unit in the following manner: acquiring the original image, determining the blur radius of each pixel in the original image when blurring is performed, setting a statistical histogram containing the blur radius of each pixel in the original image when blurring is performed, wherein the horizontal axis data of the statistical histogram corresponds to different blur radii, and the vertical axis data corresponds to the proportion of pixels with different blur radii in the entire original image; determining the starting blur radius of the pixel corresponding to the starting point of the valley area of ​​the statistical histogram when blurring is performed, determining the minimum blur radius among the different blur radii when each pixel of the background image in the original image is processed, and determining the absolute value of the difference between the starting blur radius and the minimum blur radius; in response to the absolute value being greater than a length threshold, determining the original image as the image to be processed.

[0014] In one embodiment, the starting fuzzy radius is determined by the processing unit in the following manner: in response to the presence of a section in the statistical histogram whose vertical axis data is less than a proportion threshold, the fuzzy radius corresponding to the starting node in the statistical histogram whose vertical axis data is less than the proportion threshold is determined as the starting fuzzy radius; in response to the absence of a section in the statistical histogram whose vertical axis data is less than the proportion threshold, the maximum fuzzy radius in the statistical histogram is determined as the starting fuzzy radius.

[0015] In one embodiment, the processing unit separates the image to be processed into a target subject image and a background image in the following manner: obtaining an object mask of the image to be processed, the object mask corresponding to the same category of object images in the image to be processed, the same category of objects including the target subject and objects of the same category as the target subject, the grayscale value of the area corresponding to the same category of object images in the object mask is the maximum grayscale value, and the grayscale value of the area corresponding to the background image is the minimum grayscale value; extracting the same category of object images in the image to be processed through the object mask, and separating the image to be processed into the same category of object images and the background image. images of objects of different categories; in response to the presence of multiple objects of the same category in the image to be processed, determining depth maps corresponding to the images of the objects of the same category, and determining images of the objects of the same category whose image depth is less than an image threshold as the target subject image, and determining a union of images other than the target subject image in the images of the objects of the same category and the images of objects of different categories as the background image; in response to the absence of other objects of the same category as the target subject in the image to be processed, determining the image of the object of the same category extracted by the object mask as the target subject image, and determining the image of the objects of different categories as the background image.

[0016] In one embodiment, the determination unit determines the target subject weight map of the target subject image in the following manner: obtaining a target subject mask based on the image to be processed and the target subject image, or obtaining a target subject mask based on the image to be processed and the background image, wherein the target subject mask corresponds to the target subject image; setting the grayscale value of each pixel in the target subject mask as a weight value, and determining the target subject mask that has been set as the target subject weight map.

[0017] In one embodiment, the determination unit determines the background weight map of the background image in the following manner: inverting the grayscale value of each pixel in the target subject mask, converting the maximum grayscale value into the minimum grayscale value, and converting the minimum grayscale value into the maximum grayscale value, and determining the inverted target subject mask as the initial background weight map; determining the boundary of the background area in the initial background weight map as the first boundary, and determining the maximum blur radius when the target subject image is blurred, and determining the boundary after the first boundary is extended to the target subject image by the maximum blur radius value as the second boundary; determining the fused weight map between the first boundary and the second boundary according to the first boundary, the second boundary, the maximum blur radius and a preset weight determination method; and determining the fused image of the initial background weight map and the fused weight map as the background weight map.

[0018] In one embodiment, the fusion unit fuses the target subject image and the background image that have been blurred and rendered according to the target subject weight map and the background weight map to obtain a target image in the following manner: multiplying the pixel value of each pixel in the target subject image by the weight of each pixel corresponding to the target subject weight map and dividing the result by the maximum pixel value to obtain a first weighted image; multiplying the pixel value of each pixel in the background image by the weight of each pixel corresponding to the background weight map and dividing the result by the maximum pixel value to obtain a second weighted image; and determining a fused image of the first weighted image and the second weighted image as the target image.

[0019] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the image processing method described in the first aspect or any one of the embodiments of the first aspect.

[0020] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the image processing method described in the first aspect or any one of the embodiments of the first aspect.

[0021] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: separating the image to be processed, blurring the separated target subject image and background image, setting fusion weights for the target subject image and background image respectively (including the fusion weight of each pixel in the image to be fused), and then weightedly fusion the target subject image and the background image to obtain the target image. Through the present disclosure, a terminal with a camera function with limited camera hardware and chip performance can obtain a target image with good blurring effect based on a software algorithm, which is close to the optical blurring effect based on optical elements.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] Figure 1 The figure is a schematic diagram showing a blur processing image according to an exemplary embodiment of the present disclosure.

[0025] Figure 2 It is a schematic diagram showing a comparison of blurring processing results according to an exemplary embodiment.

[0026] Figure 3 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0027] Figure 4 It is a block diagram showing an image processing method according to an exemplary embodiment of the present disclosure.

[0028] Figure 5 The figure is a flowchart of a method for obtaining an image to be processed according to an exemplary embodiment.

[0029] Figure 6 The invention is a flow chart showing a method for determining a starting blur radius according to an exemplary embodiment.

[0030] Figure 7 A statistical histogram is shown according to an exemplary embodiment of the present disclosure.

[0031] Figure 8 The present invention is a flow chart of a method for separating an image to be processed into a target subject image and a background image according to an exemplary embodiment.

[0032] Fig. 9 It is a schematic diagram of a method for separating an image to be processed into a target subject image and a background image according to an exemplary embodiment of the present disclosure.

[0033] Fig.10 The invention is a flowchart of a method for determining a target subject weight map of a target subject image according to an exemplary embodiment.

[0034] Fig.11 The invention is a flowchart of a method for determining a background weight map of a background image according to an exemplary embodiment.

[0035] Fig.12 It is a schematic diagram showing an image blurring principle based on an optical element according to an exemplary embodiment of the present disclosure.

[0036] Fig.13 The figure is a schematic diagram showing a comparison between a blurred image obtained based on a software algorithm and a blurred image obtained based on an optical element according to an exemplary embodiment of the present disclosure.

[0037] Fig.14 It is a schematic diagram showing a comparison between a blurred image obtained based on a software algorithm and a blurred image obtained based on an optical element according to yet another exemplary embodiment of the present disclosure.

[0038] Fig.15 It is a schematic diagram showing a comparison between a blurred image obtained based on a software algorithm and a blurred image obtained based on an optical element according to yet another exemplary embodiment of the present disclosure.

[0039] Fig.16 It is a curve diagram showing a formula for determining a fusion weight spectrum according to an exemplary embodiment of the present disclosure.

[0040] Fig.17 The present invention is a flowchart of a method for obtaining a target image by fusing a target subject image and a background image that have been subjected to fuzzy rendering processing according to an exemplary embodiment.

[0041] Fig.18 The figure is a schematic diagram showing a comparison of imaging effects of different image processing methods according to an exemplary embodiment of the present disclosure.

[0042] Fig.19 is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0043] Fig. 20 The invention is a block diagram of a device for image processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0045] The image processing method provided by the embodiment of the present disclosure is applied to the scenario of blurring an image.

[0046] The depth-of-field (DOF) effect (also known as portrait blur) is an important optical characteristic of camera lens imaging, which generates a blurred image with a clear target subject and a blurred background, achieving the artistic effect of highlighting the target subject and weakening the background. The extremely shallow depth-of-field blur effect is a type of image depth-of-field effect, specifically when the focus subject is very close to the camera lens, the depth of field range is very narrow, and part of the subject area will also be blurred. Figure 1As shown in the schematic diagram of the image processed with medium blur, when focusing on the center of the target subject (orange peel), due to the very shallow depth of field, only the central area of ​​the orange peel (i.e., the focus of the shooting device) is the clearest, while the edge area of ​​the orange peel (a certain distance from the focus of the shooting device) will also produce a slight blur. The extremely shallow depth of field blur effect of the camera is heavily dependent on optical elements such as large aperture and large focal length. On portable devices with camera functions with limited hardware conditions, due to the limited space of the terminal imaging module, the limited aperture and focal length of the device, it is difficult to produce the optical blur effect of devices such as SLR cameras that are specifically used for taking pictures. Limited by hardware conditions, image depth of field rendering based on software algorithms is an important means to achieve extremely shallow depth of field blur effects on devices such as terminals. It can be understood that since the image processing method in the present disclosure is implemented based on image depth of field rendering based on software algorithms, it does not have to rely on hardware facilities such as apertures. The image processing method involved in the present disclosure is also applicable to the simulation of blur effects in scenes such as movies and games.

[0047] In the extremely shallow depth of field blur rendering, image blur is the main step to achieve extremely shallow depth of field blur. Extremely shallow depth of field blur rendering requires that the blur algorithm that implements image blur can meet the requirements of anisotropic filtering and progressive smooth image blur as the depth changes. In addition, given the requirements of portability and long battery life for portable devices with camera functions such as terminals, the blur processing of the blur algorithm on the corresponding device requires fast completion and low power consumption.

[0048] In the related art, the image blur algorithms involved in the commonly used image depth of field blur rendering methods can be roughly divided into three categories: weighted fusion blur techniques (Splatting Techniques), gathering blur (Gathering Blur) and scattering blur (Scattering Blur).

[0049] The weighted fusion blur technology requires the Circle of Confusion (COC) calculated based on the depth as the weight to fuse the blurred image and the clear image to achieve a depth-layered blur effect and obtain an image with a clear target subject and a blurred background. When the radius of the circle of confusion is small, the weighted fusion blur technology has limited computational complexity and low requirements for algorithm performance. Therefore, when the radius of the circle of confusion is small, the weighted fusion blur technology has a real-time rendering effect and can quickly acquire the target image. However, as the radius of the circle of confusion increases, the computational complexity increases dramatically, and the demand for device algorithm performance increases dramatically, resulting in a significant delay in imaging, and the fused high-frequency texture will appear in the blurred background area, affecting the blur effect of the final imaging.

[0050] Aggregate blur is a common convolution blur, which adds the image pixels corresponding to the current blur kernel to obtain the blur value of the current pixel. Since the blur value of the current pixel only depends on a single blur kernel sliding window calculation, the amount of calculation is small, the demand for device computing performance is low, and it has good parallelization characteristics, which can meet the performance requirements of portable devices such as terminals. However, in principle, it is difficult to avoid blur stratification at different depths in the aggregate blur algorithm, especially in mean blur, which will lead to blur stratification artifacts, resulting in artifacts on the boundary between the target main area and the background area in the final imaging, resulting in unsatisfactory imaging effects.

[0051] Scattered blur is a blur method with an imaging effect that is closer to optical blur based on hardware devices such as optical elements. Its rendering principle is opposite to the above-mentioned gathered blur. It diffuses the value of the current pixel into the surrounding pixel area according to the shape of the blur kernel, and achieves a smooth blur transition at different depth layers, with high blur quality. However, the principle of scattered blur causes the blur values ​​between each pixel to be associated with each other, making it difficult to implement parallel operations, and the requirements on device performance are higher than those of gathered blur. And due to the inherent characteristics of the algorithm itself, there is still a gap between the extremely shallow depth of field blur effect of the rendering method based on scattered blur and the extremely shallow depth of field blur effect based on optical elements, and there is an unnatural transition problem at the boundary between the target subject and the background area.

[0052] In the related art, in order to better simulate the real optical extremely shallow depth of field blur effect based on optical elements in the camera, some terminals with high chip computing power will use parallelized mean scatter blur to obtain the best blur effect relative to other rendering methods. Considering the performance and power consumption limitations of smart terminals, parallelized mean scatter blur refers to the mean scatter of each pixel in the image to be processed. In addition, the image can also be processed in blocks, the image to be processed is divided into blocks, each block of the image is rendered separately, and all blocks are fused after rendering to obtain the final rendering result. The mean scatter blur implemented by parallelized mean scatter blur is still different from the real optical blur model based on optical elements. For example, in areas with discontinuous depth, there will be artifacts of distant color penetrating into the near area. In images with extremely shallow depth of field blur effect, the above situation occurs more frequently, affecting the user's photo experience. Even if the strategy of suppressing the blur weight of the distant area is adopted, such artifacts cannot be fundamentally solved, and the imaging effect problem of the near area color leaking into the distant area will also occur.

[0053] There are also blur processing methods based on deep learning in related technologies. When blur processing is performed based on the deep learning method, the error model of traditional mean scatter blur and deep learning rendering is established to repair the error between traditional mean scatter blur and optical blur rendering, thereby obtaining a high-quality blur effect close to that achieved based on optical elements, such as Figure 2 As shown in the schematic diagram of the comparison of the blur processing results, the person in the figure is the target subject, the full-focus image in the figure is the original image used for blur processing, and the blurred images obtained based on the traditional mean scattering rendering, the blurred images obtained based on the deep learning method, and the blurred images based on the optical blur effect are respectively. By observing the boundary between the person and the background area in each image, it can be seen that the boundary between the person and the background area in the image obtained by the blurring method based on deep learning is relatively clear compared to the traditional mean scattering method, which is close to the optical blur effect. However, this method currently obtains the target image (Ground truth) that meets the machine learning standard through the rendering engine based on the algorithm, and there is still a difference with the effect of SLR cameras and other devices based on optical elements. At the same time, the neural network model used to obtain the blurred image in the deep learning method is trained based on a large amount of training data, which is mainly used in terminal devices that obtain training data. For other terminal devices, the processing effect will be significantly reduced, so the generalization of the deep learning method is a difficulty in its application on the terminal.

[0054] In summary, the blur rendering methods involved in the related technologies have their own characteristics and are suitable for different application scenarios. However, the above-mentioned methods based on algorithm models cannot produce high-quality, extremely shallow depth of field blur effects close to those achieved by blurring based on optical components on terminals with limited hardware and insufficient performance. The blur rendering methods based on deep learning are mainly used in terminal devices that obtain training data and lack versatility.

[0055] In view of this, the present disclosure proposes an image processing method based on layered rendering to achieve an extremely shallow depth of field blur effect. The method is mainly used to shoot close-up target subjects (the target subject is about 1m away from the terminal lens, the focus is on the target subject, and the background is far away from the target subject). First, according to the characteristics of optical blur, a streamlined and parallelizable extremely shallow depth of field rendering model is constructed. The image to be rendered is divided into two layers using the object mask and the depth map, one layer is the foreground target subject area, and the other layer is the distant view area. The foreground target subject area and the distant view area are respectively mean-blurred, and a fusion weight map of the two layers of rendering results is generated. Finally, the rendering results of the two layers of images are fused to obtain a high-quality, realistic extremely shallow depth of field blur effect.

[0056] Figure 3 FIG. 1 is a flowchart of an image processing method according to an exemplary embodiment. Figure 3As shown, the method includes steps S101 to S104.

[0057] In step S101 , in response to an image processing instruction, the image to be processed is separated into a target subject image and a background image.

[0058] In step S102, blur rendering is performed on the target subject image and the background image respectively.

[0059] In step S103, according to the target subject image, a target subject weight map corresponding to the target subject image is determined, and a background weight map corresponding to the background image is determined.

[0060] Among them, each pixel in the target subject weight map corresponds one-to-one to each pixel in the target subject image, including the fusion weight of each pixel in the target subject image, and each pixel in the background weight map corresponds one-to-one to each pixel in the background image, including the fusion weight of each pixel in the background image.

[0061] In step S104, the target subject image and the background image after fuzzy rendering are fused according to the target subject weight map and the background weight map to obtain a target image.

[0062] In the embodiment of the present disclosure, while rendering the target subject image area and the background area respectively, a corresponding fusion weight map for fusing the target subject image area and the background area (including the fusion weight map corresponding to the background area and the fusion weight map corresponding to the target subject image area) is generated, and the blurred rendering of the target subject image area and the background area is completed by fusing the fusion weight map, thereby obtaining a very shallow depth of field blur effect close to that of a SLR camera. It is understandable that when the target subject image area and the background area are fused, they need to be fused pixel by pixel, so the fusion weight map in the present disclosure includes the weight corresponding to each pixel in the image.

[0063] In the disclosed embodiment, a depth map and a photographed object mask corresponding to the image to be processed are obtained, the image to be processed is separated by the depth map and the photographed object mask, and a target subject image and a background image are obtained respectively. The separated target subject image and background image are blurred respectively, and a fusion weight map (including the fusion weight of each pixel in the image to be fused) is set for the target subject image and the background image respectively. Then, the target subject image and the background image are fused by the obtained fusion weight to obtain the target image. Through the present disclosure, a terminal with a camera function having limited camera hardware and chip performance can obtain a target image with a good blur effect based on a software algorithm, which is close to the optical blur effect based on optical elements.

[0064] In an exemplary embodiment of the present disclosure, Figure 4As shown in the block diagram of the image processing method, when a terminal with a camera function takes a close-up of a person in portrait mode, the close-up person is the target subject. After obtaining the image to be rendered, a portrait mask map and a depth map corresponding to the image to be rendered are obtained. The portrait mask map corresponds to the person in the image to be processed, and the depth map contains the depth information of each pixel in the image to be rendered. The depth information can be used as the distance between each point in the image to be rendered and the terminal lens. The present disclosure processes the image to be processed with the portrait mask and the depth map, separates the target portrait area in the image to be rendered from the background area outside the target portrait area, and performs independent blur rendering processing on each.

[0065] It is understandable that the image processing method in the present disclosure is mainly used to obtain the target subject where the focus is, and blur the non-target subject area (background area). Based on this, there are certain restrictions on the image to be processed for obtaining the target image in this application, requiring that the shooting focus is on the target subject, and there is a certain depth difference between the target subject and the background area to ensure that the target subject can be highlighted after blurring.

[0066] Figure 5 FIG. 1 is a flow chart showing a method for obtaining an image to be processed according to an exemplary embodiment. Figure 5 As shown, the method includes steps S201 to S203.

[0067] In step S201, an original image is acquired, a blur radius of each pixel in the original image when blur processing is performed is determined, and a statistical histogram containing the blur radius of each pixel in the original image when blur processing is performed is set.

[0068] The horizontal axis data of the statistical histogram corresponds to different blur radii, and the vertical axis data corresponds to the proportion of pixels with different blur radii in the entire original image.

[0069] In step S202, the starting blur radius of the pixel corresponding to the starting point of the valley area of ​​the statistical histogram when blurring is performed is determined, the minimum blur radius among the different blur radii when each pixel of the background image in the original image is processed is determined, and the absolute value of the difference between the starting blur radius and the minimum blur radius is determined.

[0070] In step S203, in response to the absolute value being greater than the length threshold, the original image is determined as the image to be processed.

[0071] In the disclosed embodiment, when acquiring a target image to produce an extremely shallow depth of field effect for the target subject image, there are some implicit prior conditions: the focus point must be on the target subject; the overall depth of the target subject is basically continuous, and the depth of the whole body varies little. After acquiring the original image, the blur radius of each pixel in the original image when blurring is performed is determined, and a statistical histogram containing the blur radius of the above-mentioned different pixels and the corresponding pixel ratio data is established. The depth distribution of pixels in the target subject image area, background area, and transition area between the two in the original image is determined by the statistical histogram to determine whether the original image meets the above implicit conditions.

[0072] It can be understood that when performing image blurring, the blur radius of pixels in the image is affected by the image depth. The greater the depth difference from the imaging focus, the larger the blur radius. Therefore, the blur radius of the target subject image at the imaging focus is smaller, and the target subject image after blurring is relatively clear. The data area with a smaller blur radius in the histogram is the area corresponding to the target subject image; while the background image is farther from the imaging focus, the blur radius is larger, and the background area after blurring is relatively blurred. The data area with a larger target radius in the histogram is the area corresponding to the background image; in addition to the target subject image, the original image also includes a non-target subject image area similar to the target subject image or a transition area between other objects of the same category and the background. The depth of the above area is discontinuous with the target subject image, and the depth is closer to the imaging focus than the background area. The corresponding area is a transition area with a smaller data distribution in the histogram. By judging the absolute value of the difference between the maximum blur radius (i.e., the starting blur radius) of the portrait area and the minimum blur radius of the background area, the depth difference between the portrait area and the background area can be judged, and then it can be judged whether the portrait area is continuous, and then it can be judged whether the original image can be used as the image to be processed to obtain the target image.

[0073] It is understandable that there are multiple data distribution forms for the data distribution of the histogram. Different starting blur radiuses need to be used for calculation and judgment for different types of distribution forms. The following embodiments of the present disclosure illustrate the method for determining the starting blur radius.

[0074] Figure 6 FIG. 1 is a flow chart showing a method for determining a starting blur radius according to an exemplary embodiment. Figure 6 As shown, the method includes step S301, step S302A and step S302B.

[0075] In step S301, a statistical histogram containing the blur radius of each pixel in the original image when blur processing is performed is set.

[0076] In step S302A, in response to the presence of a segment in the statistical histogram where the vertical axis data is less than the proportion threshold, the fuzzy radius corresponding to the starting node in the statistical histogram where the vertical axis data is less than the proportion threshold is determined as the starting fuzzy radius.

[0077] In step S302B, in response to the fact that there is no segment in the statistical histogram whose vertical axis data is less than the percentage threshold, the maximum blur radius in the statistical histogram is determined as the starting blur radius.

[0078] In the embodiment of the present disclosure, different methods for determining the starting blur radius are set for different data distribution modes of the statistical histogram: the presence of valley areas and the absence of valley areas, respectively, so that the image processing method in the present disclosure is more versatile and can process a variety of images with different layouts of the same category of objects and background areas.

[0079] In an exemplary embodiment of the present disclosure, data judgment is performed on a statistical histogram with valley values ​​to determine whether the image of the statistical histogram can be used to obtain a target image. Figure 7 As shown in a statistical histogram, the present disclosure first calculates the blur radius Radius of each pixel in the original image, and obtains the blur radius histogram corresponding to the original image (proportion [blur radius], Hist[Rad]), where the blur radius Rad=[0,1,2,…,Rcmax], Rcmax represents the maximum blur radius in the original image, which is less than or equal to the maximum blur radius MAX_Radius set in the rendering algorithm. Then analyze the acquired blur radius histogram, starting from a radius of 0 to the maximum blur radius Rcmax, to determine whether the histogram has a valley (valley value area). When it is determined that the proportion data Hist[i] corresponding to a certain blur radius is <= the first proportion threshold th1, the Hist[i] represents a breakpoint, and the continuous th2 breakpoints form a valley. Record the corresponding starting blur radius Rseg of the first valley. If there is no valley in the histogram, the maximum blur radius of the pixel when the original image is blurred is used as the starting blur radius: Rseg=Rcmax. Then calculate the blur radius data distribution of the non-same type of shooting object area (background area), and count the minimum blur radius Rfmin therein. When the absolute value of the difference between the initial blur radius and the minimum blur radius is less than the second threshold th2: |Rfmin-Rseg|>=th2, the implicit condition of the present disclosure for blur processing is met, and the part of the same type of object area with a blur radius greater than the initial blur radius is moved to the distant view area. In other cases, the assumption of the present disclosure is not met, and the image processing method of the present disclosure is directly exited and no longer used. In the exemplary embodiment of the present disclosure, the first threshold th1 is 0.001, the second threshold th2 is 10, and the maximum blur radius MAX_Radius set in the rendering algorithm is 40.

[0080] In the embodiment of the present disclosure, the image to be rendered is separated into a target subject image and a background image by using a depth map corresponding to the rendered image and a mask of the photographed object. The following embodiment of the present disclosure describes the method of separating the images to be rendered.

[0081] Figure 8 FIG. 1 is a flow chart of a method for separating an image to be processed into a target subject image and a background image according to an exemplary embodiment. Figure 8 As shown, the method includes step S401, step S402, step S403A and step S403B.

[0082] In step S401, an object mask of an image to be processed is obtained.

[0083] Among them, the object mask corresponds to the same category of object images in the image to be processed, and the same category of objects includes the target subject and the same category of objects as the target subject. The grayscale value of the area in the object mask corresponding to the same category of object images is the maximum grayscale value, and the grayscale value of the area corresponding to the background image is the minimum grayscale value.

[0084] In step S402, the same category of object images in the to-be-processed images are extracted by using the object mask, and the to-be-processed images are separated into the same category of object images and the different category of object images.

[0085] In step S403A, in response to the presence of multiple objects of the same category in the image to be processed, depth maps corresponding to images of the objects of the same category are determined, and images of the objects of the same category whose image depth is less than an image threshold are determined as target subject images, and the union of images other than the target subject images of the objects of the same category and images of objects of different categories are determined as background images.

[0086] In step S403B, in response to the absence of other objects of the same category as the target subject in the image to be processed, the image of the object of the same category extracted through the object mask is determined as the target subject image, and the image of the object of a different category is determined as the background image.

[0087] In the disclosed embodiment, while acquiring the image to be processed, the object mask and the depth image corresponding to the image to be processed are acquired, and the image to be processed is separated by the object mask and the depth image. It is understandable that when extracting the image by the object mask, all the images of the same category of objects in the image to be processed will be extracted. When the image to be rendered contains only a single object of the same category, the extracted single image of the same category of objects can be used as the target subject image, and the image other than the single image of the same category of objects can be directly used as the background image. When there are multiple objects of the same category in the image to be processed, the object mask will extract all the multiple images of the same category of objects, and the processed image will be divided into images of the same category of objects and images of objects of different categories. At this time, it is necessary to extract the target subject area of ​​the depth connection where the imaging focus is located in the image of the same category of objects through the depth map as the target subject image, and then merge the images other than the target subject image in the image of the same category of objects and the images of objects of different categories as the background image.

[0088] In the disclosed embodiment, the image to be processed is separated by using the object mask and depth image corresponding to the image to be processed, so as to ensure that the target main image is completely separated from the background image, and there is no jaggedness in the boundary part, so as to ensure that the target main image and the background image are processed separately and are not interfered by each other's pixels, thereby ensuring the final imaging effect.

[0089] In the embodiment of the present disclosure, the object mask used to extract the object image only includes a maximum grayscale value of 255 and a minimum grayscale value of 0, wherein the maximum grayscale value corresponds to the portrait area, and the minimum grayscale value corresponds to the non-portrait area. Therefore, when the present disclosure uses the object mask to extract the same category of object images, it is necessary to pre-process the object mask in advance, that is, to perform binarization processing on the object mask to eliminate the grayscale value gradient area in the conventional object mask, so that the grayscale value of the object image area is 255, and the grayscale value of the non-object image area is 0.

[0090] In an exemplary embodiment of the present disclosure, when the target subject image is a portrait, the image to be processed containing only a single portrait is separated and processed, such as Fig. 9 1 is a schematic diagram of a method for separating an image to be processed into a target person image and a background image, extracting a target person image region (close person image region) in the image to be rendered through a close-up portrait mask, making the grayscale values ​​of other regions 0; and obtaining an image containing only a background region (far region), wherein the grayscale values ​​of non-background regions are 0.

[0091] In the embodiment of the present disclosure, after obtaining the background image and the target subject image, the two need to be blurred respectively, and then fused based on their respective weights. The following embodiment of the present disclosure describes a method for determining a target subject weight map of a target subject image.

[0092] Fig.10 FIG. 1 is a flow chart showing a method for determining a target subject weight map of a target subject image according to an exemplary embodiment. Fig.10 As shown, the method includes step S501 to step S502.

[0093] In step S501, a target subject mask is obtained according to the image to be processed and the target subject image, or a target subject mask is obtained according to the image to be processed and the background image.

[0094] Among them, the target subject mask corresponds to the target subject image.

[0095] In step S502, the grayscale value of each pixel in the target subject mask is set as a weight value, and the target subject mask having been set is determined as a target subject weight map.

[0096] In the disclosed embodiment, after obtaining the target subject image, the object mask corresponding to the target subject image can be obtained based on the target subject image, that is, the grayscale value of the target subject image area is 255, and the grayscale value of other parts is 0. Then, the grayscale value in the target subject image mask is used as the weight, that is, the weight of each pixel in the target subject image area is 255, and the weight of other areas is 0, so as to obtain the target subject weight map.

[0097] In the disclosed embodiment, the target subject image Close, the background image Far and the image to be rendered Ori_Img satisfy the following relationship: Ori_Img = Close + Far. According to the target subject image, the target subject mask Close_Mask is obtained, and the grayscale value of the target subject area in the mask = 255, and the grayscale value of the distant view area = 0. They satisfy the formula:

[0098] Ori_Img×Close_Mask / 255=Close

[0099] Ori_Img×(255-Close_Mask) / 255=Far

[0100] The following embodiments of the present disclosure illustrate a method for determining a background weight map of a background image.

[0101] Fig.11 FIG. 1 is a flow chart showing a method for determining a background weight map of a background image according to an exemplary embodiment. Fig.11 As shown, the method includes steps S601 to S604.

[0102] In step S601, the grayscale value of each pixel in the target subject mask is inverted, the maximum grayscale value is converted to the minimum grayscale value, and the minimum grayscale value is converted to the maximum grayscale value. The inverted target subject mask is determined as the initial background weight map.

[0103] In step S602, the boundary of the background area in the initial background weight map is determined as the first boundary, and the maximum blur radius when the target subject image is blurred is determined, and the boundary after the first boundary is extended toward the target subject image by the maximum blur radius value is determined as the second boundary.

[0104] In step S603, a fusion weight map between the first boundary and the second boundary is determined according to the first boundary, the second boundary, the maximum blur radius and a preset weight determination method.

[0105] In step S604, a fused image of the initial background weight map and the fused weight map is determined as the background weight map.

[0106] In the present disclosure, Fig.12 As shown in the schematic diagram of the principle of image blur based on optical elements, the 3D model in the left figure can be regarded as the depth distribution of a close-up scene. Part 2 represents the part of the subject to be blurred, with a small blur and a small blur radius; Part 1 represents the distant background, with a large blur and a large blur radius. Shoot the 3D model in the left figure, draw a schematic diagram of the light path, and unfold it into a plane to obtain the real optical blur model in the right figure (real blur model based on optical elements). According to the principle of blurring, it can be seen that the centers of the arcs 4 and 3 coincide.

[0107] In the present disclosure, Fig.13 The comparison diagram of the blurred image obtained based on the software algorithm and the blurred image obtained based on the optical element is shown in the figure. At a pixel point A near the depth discontinuity area, there is a big difference in the rendering impact area of ​​point A using the ordinary mean scattering model and the real optical scattering model (the difference is mainly reflected in the different areas of the three arcs, and the ordinary mean scattering model will obviously have an extra part). This is also the reason why the ordinary mean scattering model causes artifacts in the depth discontinuity area. The present invention uses an ordinary mean scattering model to render two layers of images, but optimizes the model and creates a fusion weight map so that it can eliminate artifacts in the near and far depth discontinuity areas, and can perform parallel calculations.

[0108] In the present disclosure, Fig.14As shown in the schematic diagram of the comparison between the blurred image obtained based on the software algorithm and the blurred image obtained based on the optical element. First, a comparative analysis is performed on the real optical scattering model and the commonly used ordinary mean scattering model (the model adopted in this application) to obtain the difference between the two. As shown in the figure, the figure is a rendering comparison diagram of four different points A to be blurred in the real optical scattering model and the ordinary mean scattering model. The four pictures in the left column are the rendering methods of the real optical scattering model, and the four pictures in the right column are the rendering methods of the ordinary mean scattering model. First, observe the left image in the first row. According to the triangle similarity principle, we can get h1 / h2=r1 / r2, where r1 represents the radius of the dark gray striped part of the circle (which can be regarded as the blur radius of the pixel in the target main image), r2 represents the radius of the light gray striped part of the circle (which can be regarded as the blur radius of the pixel in the background image), h1 can represent the distance between the blur center of the pixel in the target main image and the boundary of the target main image, h2 can represent the distance between the blur center of the pixel in the target main image and the blur area of ​​the background image, θ is a variable, which is the angle between the straight line where r1 is located in the figure and the horizontal line segment where point A is located in the figure, and the value range of θ is [0,π / 2]. When θ=π / 2, the image in the first row can be obtained. When θ is between (0,π / 2), the image in the second row can be obtained. When θ=0 and h1=r1, the image in the third row can be obtained. When θ=0 and h1=r2, the image in the fourth row can be obtained. It can be seen that in the case of the first and fourth rows, the real optical scattering model is exactly the same as the ordinary mean scattering model. From the figure, it can be found that the real scattering model is related to r1, r2, h1, and h2. In real scenes, the size of r1 and r2 near the junction of the near view and the distant view (the area of ​​depth discontinuity) always changes. If a real scattering model is constructed for each point of these junctions, the calculation complexity will increase dramatically, and it cannot be applied on terminals with high chip computing power.

[0109] Further, yes Fig.14 We quantitatively analyze the universality of the second row in Fig.15The diagram shows a comparison of the blurred image obtained by the software algorithm and the blurred image obtained by the optical element. The variables with the same size in the left and right figures use the same symbol. The radius of the arcuate part of the dark gray stripe and the arcuate part of the light gray stripe are equal in the left and right figures, which are r1 and r2 respectively. S2r represents the area of ​​the arcuate part of the light gray stripe in the left figure (which can be regarded as the area of ​​a certain pixel in the background area after blurring), S2n represents the area of ​​the arcuate part of the light gray stripe in the right figure (which can be regarded as the area of ​​a certain pixel in the background area after blurring), and the areas of the arcuate part of the dark gray stripe in the left and right figures are equal (which can be regarded as the area of ​​a certain pixel in the target image area after blurring), and are both represented by S1. The center angle corresponding to the arcuate part of the light gray stripe in the left figure is 2*θr, and the center angle corresponding to the arcuate part of the light gray stripe in the right figure is 2*θn. h1 is the distance from the center of the arcuate to the boundary of the dark gray object. Assume that the pixel value of the light gray distant view (background area) is V2, and the pixel value of the dark gray near view (target main image area) is V1. Based on the above parameters, the values ​​of the center point (black dot) in the left and right scattered models can be calculated respectively:

[0110]

[0111]

[0112] Among them, Br can be regarded as the value of the pixel in the background image in the optical scattering model, and Bn can be regarded as the value of the pixel in the target main image in the optical scattering model.

[0113] Subtracting the above two values ​​gives the difference ΔB between the two models:

[0114]

[0115] Among them, it can be found If r1, r2, V1, and V2 are fixed, ΔB is a function that is only related to h1 (i.e., the distance between the blur center of the pixel in the target subject image and the boundary of the target subject image), and the value range of h1 is [0, r2]. By taking two derivatives, we can find that within [0, r2], ΔB is a function that first increases and then decreases; and when h1 = 0, ΔB = 0; when ΔB reaches its maximum value; when h1=r2, ΔB=0, and Fig.14 This is consistent with the differences described in . Fig.14It can be seen that in the real scattering model, when h1≥r1, the center point does not receive the scattering of distant pixels, but is only formed by the scattering of near pixels. In summary, the difference between the hardware-based optical scattering model and the software-based general scattering model is mainly that the value range of h1 is [0, r2], and the scattering blur effect of pixels in this range, where h1 is the distance between the blur center of the pixel in the target subject image and the boundary of the target subject image, and r2 is the blur radius of the pixel in the target subject image.

[0116] In the embodiments of the present disclosure, in real scenes, the depth discontinuity area is not a horizontal and vertical area as shown in 14 and 15, and the blurriness (blur radius of each pixel) on both sides of the depth discontinuity area will not be completely consistent. If a real optical rendering model is used, each pixel point in the depth discontinuity area and its vicinity needs to recalculate the scattering formula, which is very complex and cannot be parallelized. Due to the limitation of terminal performance, it is almost impossible to apply it on the terminal. In order to make the blurring effect based on the software algorithm closer to the real optical blurring based on optical elements, and to ensure that the performance of the blurring algorithm can meet the requirements of the terminal, the present disclosure selects an ordinary mean scattering model and optimizes it, which is specially used for terminal close-up shooting of the same category of objects to obtain a high-quality extremely shallow depth of field effect.

[0117] In the disclosed embodiment, after obtaining the background image and the target subject image, the target task image (close target subject area Close) and the background image (far area Far) are respectively subjected to mean scatter blur. When scattering, the two images are independent of each other and can be calculated in parallel. All pixels are scattered and blurred using a uniform value, and can also be calculated in blocks and in parallel.

[0118] In an exemplary embodiment, after the mean blurring in the related art is completed, the target task blurred image (Close_Blur) and the background blurred image (Far_Blur) are obtained. The two areas need to be fused. It is assumed that the fused weight image is the target subject weight map Close_FusionAlpha* and the background weight map Far_FusionAlpha*. In the original ordinary mean blurring model, the grayscale values ​​in the Close_FusionAlpha* and Far_FusionAlpha* images are both 255, and the blurring result is:

[0119] Blur_Result = (Close_Blur×Close_FusionAlpha+Far_Blur×Far_FusionAlpha) / 255. In the above case, the pixels belonging to the distant view near the depth discontinuity area will disperse the energy to the area where the near view target body is located, thereby causing artifacts.

[0120] In the embodiment of the present disclosure, based on the embodiment of the above-mentioned related technology, in order to make the designed scattering model closer to the real optical scattering model, when the two areas are fused, the present disclosure will produce a background weight map Far_FusionAlpha_Ref containing the fusion weight of each pixel in the background image to control the fusion range of the background image (distant area). The initial background weight map Far_FusionAlpha is generated by the inverted target subject image mask Close_Mask and the maximum blur radius r1c of the pixels in the target subject image Close.

[0121] In the disclosed embodiment, first, the initial background weight map Far_FusionAlpha is set to 255-Close_Mask, and the pixel value of each pixel in the target subject mask is flipped and used as the weight value. Then, the distance calculation formula is used for the initial background weight map Far_FusionAlpha to calculate the distance from the area with a grayscale value of 0 to the area with a grayscale value of 255 in the initial background weight map Far_FusionAlpha (i.e., the distance between the background image and the target subject image), and the maximum distance is truncated with the above-mentioned maximum blur radius r1c, so that the boundary of the background image (the first boundary) is synchronously extended to the target subject image area by a distance of r1c to obtain the second boundary Far_FusionAlpha_Dist. The weight is set for the area between the first boundary and the second boundary so that when the two images are fused, there will be no abrupt transition problem at the fusion boundary.

[0122] In the embodiment of the present disclosure, the second boundary Far_FusionAlpha_Dist and the maximum blur radius r1c are used to generate a smooth transition area at the fusion boundary (the area between the first boundary and the second boundary). The specific operation is as follows: the blur radius [0, 1, 2, ..., r1c] is mapped to [255, 0]. The commonly used mapping methods are linear mapping and nonlinear mapping. Fig.16 As shown in the figure, assuming that the weight is y and the distance is x, the formula of the linear mapping curve is: y = -255 / r1c×x+255. The formula of the nonlinear mapping curve is: y = 255×[(r1c-x) / r1c]gamma; when gamma = 0.5, the corresponding Fig.16 The upward curvature in the graph; when gamma = 2.0, it corresponds to Fig.16 The downward curved curve in the gamma value is determined while obtaining the image to be rendered. The distance value in the second boundary Far_FusionAlpha_Dist is determined according to Fig.16The curve in is mapped to obtain Far_FusionAlpha_DistMap (the fusion weight map between the first boundary and the second boundary). After the mapping is completed, these values ​​are pasted into Far_FusionAlpha_Ref together with the initial background weight map Far_FusionAlpha, that is:

[0123] Background weight map Far_FusionAlpha_Ref = Far_FusionAlpha + Far_FusionAlpha_DistMap

[0124] The following embodiments of the present disclosure illustrate a method for obtaining a target image.

[0125] Fig.17 FIG. 1 is a flow chart showing a method for fusing a target subject image and a background image that have been subjected to fuzzy rendering processing to obtain a target image according to an exemplary embodiment. Fig.17 As shown, the method includes steps S701 to S703.

[0126] In step S701, the pixel value of each pixel in the target subject image is multiplied by the weight of each pixel corresponding to the target subject weight map and divided by the maximum pixel value to obtain a first weighted image.

[0127] In step S702, the pixel value of each pixel in the background image is multiplied by the weight of each pixel corresponding to the background weight map and divided by the maximum pixel value to obtain a second weighted image.

[0128] In step S703, a fused image of the first weighted image and the second weighted image is determined as a target image.

[0129] In the disclosed embodiment, the pixel value of each pixel in the target subject image is weighted based on the weight map of the same category of objects to obtain a first weighted image; the pixel value of each pixel in the image of the same category of objects is weighted based on the background weight map to obtain the first weighted image. The first weighted image and the second weighted image are combined to obtain a rendered target image.

[0130] In the disclosed embodiment, the final blur result Blur_Result_Ref is:

[0131] Blur_Result_Ref=(Close_Blur×Close_FusionAlpha+Far_Blur×Far_FusionAlpha_Ref) / 255

[0132] Among them, Close_Blur is the blurred image of the target subject, Close_FusionAlpha is the weight map of the same category of objects, Far_Blur is the blurred background image, and Far_FusionAlpha_Ref is the background weight map.

[0133] In the present disclosure, Fig.18 The imaging effect comparison diagram is shown. The left side of the figure is a blur effect diagram obtained by blur rendering using a general mean scattering model, and the right side is a blur effect diagram obtained by the image processing method in the present disclosure. By comparison, it can be seen that compared with the effect of blur rendering using a general mean scattering model, the rendering effect of the image processing method in the present disclosure is better, and there will be no interference between the pixels of the target person image and the background image in the boundary area between the target person and the background image, and there will be no artifacts.

[0134] In the embodiment of the present disclosure, the image to be processed is separated, the separated images of the same category of objects and the background image are blurred, and fusion weights (including the fusion weight of each pixel in the image to be fused) are set for the images of the same category of objects and the background image respectively, and then the images of the same category of objects and the background image are weightedly fused to obtain the target image. Through the present disclosure, a terminal with a camera function with limited camera hardware and chip performance can obtain a target image with good blurring effect based on a software algorithm, which is close to the optical blurring effect based on optical elements, and the image processing method disclosed in the disclosure is versatile and applicable to a variety of camera terminals.

[0135] Based on the same concept, the embodiment of the present disclosure further provides an image processing device 100 .

[0136] It is understandable that, in order to realize the above functions, the image processing device 100 provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0137] Fig.19 1 is a block diagram of an image processing apparatus 100 according to an exemplary embodiment. Fig.19 The device includes a processing unit 101, a rendering unit 102, a determination unit 103 and a fusion unit 104.

[0138] The processing unit 101 is used for separating the image to be processed into a target subject image and a background image in response to an image processing instruction.

[0139] The rendering unit 102 is used to perform blur rendering processing on the target main image and the background image respectively.

[0140] The determination unit 103 is used to determine the target subject weight map corresponding to the target subject image and determine the background weight map corresponding to the background image according to the target subject image.

[0141] Among them, each pixel in the target subject weight map corresponds one-to-one to each pixel in the target subject image, including the fusion weight of each pixel in the target subject image, and each pixel in the background weight map corresponds one-to-one to each pixel in the background image, including the fusion weight of each pixel in the background image.

[0142] The fusion unit 104 is used to fuse the target subject image and the background image after the fuzzy rendering process according to the target subject weight map and the background weight map to obtain the target image.

[0143] In one embodiment, the image to be processed is obtained by the processing unit 101 in the following manner: obtaining the original image, determining the blur radius of each pixel in the original image when blurring is performed, setting a statistical histogram containing the blur radius of each pixel in the original image when blurring is performed, wherein the horizontal axis data of the statistical histogram corresponds to different blur radii, and the vertical axis data corresponds to the proportion of pixels with different blur radii in the entire original image. Determine the starting blur radius of the pixel corresponding to the starting point of the valley area of ​​the statistical histogram when blurring is performed, determine the minimum blur radius among the different blur radii when each pixel of the background image in the original image is processed, and determine the absolute value of the difference between the starting blur radius and the minimum blur radius. In response to the absolute value being greater than the length threshold, the original image is determined as the image to be processed.

[0144] In one implementation, the initial fuzzy radius is determined by the processing unit 101 in the following manner: in response to the presence of a segment in the statistical histogram where the vertical axis data is less than a percentage threshold, the fuzzy radius corresponding to the starting node in the statistical histogram where the vertical axis data is less than the percentage threshold is determined as the initial fuzzy radius. In response to the absence of a segment in the statistical histogram where the vertical axis data is less than the percentage threshold, the maximum fuzzy radius in the statistical histogram is determined as the initial fuzzy radius.

[0145] In one embodiment, the processing unit 101 separates the image to be processed into a target subject image and a background image in the following manner: obtain an object mask of the image to be processed, the object mask corresponds to the same category of object images in the image to be processed, the same category of objects includes the target subject and the same category of objects as the target subject, the grayscale value of the area corresponding to the same category of object images in the object mask is the maximum grayscale value, and the grayscale value of the area corresponding to the background image is the minimum grayscale value. Through the object mask, the same category of object images in the image to be processed are extracted, and the image to be processed is separated into the same category of object images and non-same category of object images. In response to the presence of multiple same category of objects in the image to be processed, the depth map corresponding to the same category of object images is determined, and the image whose image depth is less than the image threshold in the same category of object images is determined as the target subject image, and the union of the image other than the target subject image and the non-same category of object images in the same category of object images is determined as the background image. In response to the absence of other objects of the same category as the target subject in the image to be processed, an image of an object of the same category extracted through the object mask is determined as the target subject image, and an image of an object of a different category is determined as the background image.

[0146] In one implementation, the determination unit 103 determines the target subject weight map of the target subject image in the following manner: a target subject mask is obtained according to the image to be processed and the target subject image, or a target subject mask is obtained according to the image to be processed and the background image, and the target subject mask corresponds to the target subject image. The gray value of each pixel in the target subject mask is set as the weight value, and the target subject mask that has been set is determined as the target subject weight map.

[0147] In one embodiment, the determination unit 103 determines the background weight map of the background image in the following manner: the grayscale value of each pixel in the target subject mask is inverted, the maximum grayscale value is converted to the minimum grayscale value, and the minimum grayscale value is converted to the maximum grayscale value, and the inverted target subject mask is determined as the initial background weight map. The boundary of the background area in the initial background weight map is determined as the first boundary, and the maximum blur radius when the target subject image is blurred is determined, and the boundary after the first boundary is extended to the target subject image by the maximum blur radius value is determined as the second boundary. According to the first boundary, the second boundary, the maximum blur radius and the preset weight determination method, the fusion weight map between the first boundary and the second boundary is determined. The fused image of the initial background weight map and the fusion weight map is determined as the background weight map.

[0148] In one embodiment, the fusion unit 104 fuses the target subject image and the background image after blur rendering according to the target subject weight map and the background weight map to obtain the target image in the following manner: multiply the pixel value of each pixel in the target subject image by the weight of each pixel corresponding to the target subject weight map and divide by the maximum pixel value to obtain a first weighted image. Multiply the pixel value of each pixel in the background image by the weight of each pixel corresponding to the background weight map and divide by the maximum pixel value to obtain a second weighted image. The fused image of the first weighted image and the second weighted image is determined as the target image.

[0149] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0150] Fig. 20 2 is a block diagram of an apparatus 200 for image processing according to an exemplary embodiment. The apparatus 200 may be provided as a terminal. For example, the apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0151] Reference Fig. 20 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0152] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0153] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0154] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0155] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0156] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0157] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0158] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0159] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0160] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0161] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0162] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0163] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0164] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.

[0165] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0166] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0167] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present solution, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0168] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. An image processing method, characterized in that: include: In response to the image processing instruction, the image to be processed is separated into a target subject image and a background image; Performing fuzzy rendering processing on the target subject image and the background image respectively; According to the target subject image, determine a target subject weight map corresponding to the target subject image, and determine a background weight map corresponding to the background image, wherein each pixel in the target subject weight map corresponds one-to-one to each pixel in the target subject image, including a fusion weight of each pixel in the target subject image, and each pixel in the background weight map corresponds one-to-one to each pixel in the background image, including a fusion weight of each pixel in the background image; According to the target subject weight map and the background weight map, the target subject image and the background image after fuzzy rendering are fused to obtain a target image.

2. The method according to claim 1, characterized in that The image to be processed is obtained in the following manner: Acquire an original image, determine a blur radius of each pixel in the original image when blur processing is performed, and set a statistical histogram containing the blur radius of each pixel in the original image when blur processing is performed, wherein the horizontal axis data of the statistical histogram corresponds to different blur radii, and the vertical axis data corresponds to the proportion of pixels with different blur radii in the entire original image; Determine a starting blur radius when blurring is performed on pixels corresponding to a starting point of a valley region of the statistical histogram, determine a minimum blur radius among different blur radii when image processing is performed on each pixel of a background image in the original image, and determine an absolute value of a difference between the starting blur radius and the minimum blur radius; In response to the absolute value being greater than a length threshold, the original image is determined as the image to be processed.

3. The method according to claim 2, characterized in that The initial blur radius is determined in the following manner: In response to the presence of a section in the statistical histogram where the vertical axis data is less than a percentage threshold, determining the fuzzy radius corresponding to the starting node in the statistical histogram where the vertical axis data is less than the percentage threshold as the starting fuzzy radius; In response to the absence of a segment in the statistical histogram whose vertical axis data is less than a percentage threshold, the maximum blur radius in the statistical histogram is determined as the starting blur radius.

4. The method according to claim 1, characterized in that The step of separating the image to be processed into a target main image and a background image comprises: Acquire an object mask of the image to be processed, wherein the object mask corresponds to an object image of the same category in the image to be processed, the objects of the same category include the target subject and objects of the same category as the target subject, the grayscale value of an area in the object mask corresponding to the object image of the same category is a maximum grayscale value, and the grayscale value of an area corresponding to the background image is a minimum grayscale value; Extracting the same category of images of the objects in the to-be-processed images through the object mask, and separating the to-be-processed images into the same category of images of the objects and images of objects of different categories; In response to the presence of a plurality of objects of the same category in the image to be processed, depth maps corresponding to the images of the objects of the same category are determined, and images whose image depths are less than an image threshold among the images of the objects of the same category are determined as the target subject images, and a union of images other than the target subject images among the images of the objects of the same category and the images of objects of different categories is determined as the background image; In response to the absence of other objects of the same category as the target subject in the image to be processed, the image of the object of the same category extracted through the object mask is determined as the target subject image, and the image of the object of a different category is determined as the background image.

5. The method according to claim 1, characterized in that The determining of the target subject weight atlas of the target subject image comprises: Acquire a target subject mask according to the image to be processed and the target subject image, or acquire a target subject mask according to the image to be processed and the background image, wherein the target subject mask corresponds to the target subject image; The grayscale value of each pixel in the target subject mask is set as a weight value, and the target subject mask after setting is determined as the target subject weight atlas.

6. The method according to claim 1, characterized in that The step of determining a background weight map of the background image comprises: Inverting the grayscale value of each pixel in the target subject mask, converting the maximum grayscale value into the minimum grayscale value, and converting the minimum grayscale value into the maximum grayscale value, and determining the inverted target subject mask as the initial background weight map; Determine the boundary of the background area in the initial background weight map as the first boundary, determine the maximum blur radius when blurring the target subject image, and determine the boundary after the first boundary is extended toward the target subject image by the maximum blur radius value as the second boundary; Determine a fusion weight map between the first boundary and the second boundary according to the first boundary, the second boundary, the maximum blur radius and a preset weight determination method; A fused image of the initial background weight map and the fused weight map is determined as the background weight map.

7. The method according to claim 1, characterized in that The step of fusing the target subject image and the background image after fuzzy rendering according to the target subject weight map and the background weight map to obtain the target image includes: Multiplying the pixel value of each pixel in the target subject image by the weight of each pixel corresponding to the target subject weight map and dividing the result by the maximum pixel value to obtain a first weighted image; Multiplying the pixel value of each pixel in the background image by the weight of each pixel corresponding to the background weight map and dividing the result by the maximum pixel value to obtain a second weighted image; A fused image of the first weighted image and the second weighted image is determined as the target image.

8. An image processing device, characterized in that: include: A processing unit, configured to separate the image to be processed into a target subject image and a background image in response to an image processing instruction; A rendering unit, used for performing fuzzy rendering processing on the target main image and the background image respectively; A determination unit, used to determine, according to the target subject image, a target subject weight map corresponding to the target subject image, and determine a background weight map corresponding to the background image, wherein each pixel in the target subject weight map corresponds one-to-one to each pixel in the target subject image, and includes a fusion weight of each pixel in the target subject image, and each pixel in the background weight map corresponds one-to-one to each pixel in the background image, and includes a fusion weight of each pixel in the background image; The fusion unit is used to fuse the target subject image and the background image after the fuzzy rendering process according to the target subject weight map and the background weight map to obtain the target image.

9. The device according to claim 8, characterized in that The image to be processed is obtained by the processing unit in the following manner: Acquire an original image, determine a blur radius of each pixel in the original image when blur processing is performed, and set a statistical histogram containing the blur radius of each pixel in the original image when blur processing is performed, wherein the horizontal axis data of the statistical histogram corresponds to different blur radii, and the vertical axis data corresponds to the proportion of pixels with different blur radii in the entire original image; Determine a starting blur radius when blurring is performed on pixels corresponding to a starting point of a valley region of the statistical histogram, determine a minimum blur radius among different blur radii when image processing is performed on each pixel of a background image in the original image, and determine an absolute value of a difference between the starting blur radius and the minimum blur radius; In response to the absolute value being greater than a length threshold, the original image is determined as the image to be processed.

10. The device according to claim 9, characterized in that The initial blur radius is determined by the processing unit in the following manner: In response to the presence of a section in the statistical histogram where the vertical axis data is less than a percentage threshold, determining the fuzzy radius corresponding to the starting node in the statistical histogram where the vertical axis data is less than the percentage threshold as the starting fuzzy radius; In response to the absence of a segment in the statistical histogram whose vertical axis data is less than a percentage threshold, the maximum blur radius in the statistical histogram is determined as the starting blur radius.

11. The device according to claim 8, characterized in that The processing unit separates the image to be processed into a target main image and a background image in the following manner: Acquire an object mask of the image to be processed, wherein the object mask corresponds to an object image of the same category in the image to be processed, the objects of the same category include the target subject and objects of the same category as the target subject, the grayscale value of an area in the object mask corresponding to the object image of the same category is a maximum grayscale value, and the grayscale value of an area corresponding to the background image is a minimum grayscale value; Extracting the same category of images of the objects in the to-be-processed images through the object mask, and separating the to-be-processed images into the same category of images of the objects and images of objects of different categories; In response to the presence of a plurality of objects of the same category in the image to be processed, depth maps corresponding to the images of the objects of the same category are determined, and images whose image depths are less than an image threshold among the images of the objects of the same category are determined as the target subject images, and a union of images other than the target subject images among the images of the objects of the same category and the images of objects of different categories is determined as the background image; In response to the absence of other objects of the same category as the target subject in the image to be processed, the image of the object of the same category extracted through the object mask is determined as the target subject image, and the image of the object of a different category is determined as the background image.

12. The device according to claim 8, characterized in that The determination unit determines the target subject weight map of the target subject image in the following manner: Acquire a target subject mask according to the image to be processed and the target subject image, or acquire a target subject mask according to the image to be processed and the background image, wherein the target subject mask corresponds to the target subject image; The grayscale value of each pixel in the target subject mask is set as a weight value, and the target subject mask after setting is determined as the target subject weight atlas.

13. The device according to claim 8, characterized in that The determining unit determines the background weight map of the background image in the following manner: Inverting the grayscale value of each pixel in the target subject mask, converting the maximum grayscale value into the minimum grayscale value, and converting the minimum grayscale value into the maximum grayscale value, and determining the inverted target subject mask as the initial background weight map; Determine the boundary of the background area in the initial background weight map as the first boundary, determine the maximum blur radius when blurring the target subject image, and determine the boundary after the first boundary is extended toward the target subject image by the maximum blur radius value as the second boundary; Determine a fusion weight map between the first boundary and the second boundary according to the first boundary, the second boundary, the maximum blur radius and a preset weight determination method; A fused image of the initial background weight map and the fused weight map is determined as the background weight map.

14. The device according to claim 8, characterized in that The fusion unit fuses the target subject image and the background image after fuzzy rendering according to the target subject weight map and the background weight map to obtain the target image in the following manner: Multiplying the pixel value of each pixel in the target subject image by the weight of each pixel corresponding to the target subject weight map and dividing the result by the maximum pixel value to obtain a first weighted image; Multiplying the pixel value of each pixel in the background image by the weight of each pixel corresponding to the background weight map and dividing the result by the maximum pixel value to obtain a second weighted image; A fused image of the first weighted image and the second weighted image is determined as the target image.

15. An electronic device, characterized in that: include: processor: a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the image processing method according to any one of claims 1 to 7.

16. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor, the processor is enabled to execute the image processing method according to any one of claims 1 to 7.

Images