Triplot production method, triplot production device, and storage medium

Abstract

The present disclosure relates to a trinary graph production method, a trinary graph production device and a storage medium. The production method comprises: obtaining an original image to be produced into a trinary graph, the original image comprising at least a portrait mask after a portrait blur processing, a face frame and a foreground region mask; obtaining an initial trinary graph according to the portrait mask, the face frame and the foreground region mask, the initial trinary graph comprising a head region trinary graph and a non-head region trinary graph; performing region segmentation based on unknown regions in the head region trinary graph, updating the unknown regions in the head region trinary graph according to the complexity of the unknown regions in each segmented region after segmentation, and obtaining an updated head region trinary graph; and merging the non-head region trinary graph and the updated head region trinary graph to obtain a final trinary graph of the original image. The present disclosure can reasonably determine the range of unknown regions and improve the accuracy of trinary graph production.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the multimedia technology, and in particular, to a trimap generation method, a trimap generation device and a storage medium. BACKGROUND

[0002] Image matting can extract foreground objects from images or videos, and achieve accurate separation of foreground and background, which is the basis of digital image editing and digital image synthesis technology. When image matting is performed, an original image and a corresponding trimap need to be input. The trimap is an image that divides the original image into foreground to be retained, background to be removed, and unknown region to be solved.

[0003] In related technologies, there are two ways to generate a trimap. One way is manual generation, which needs to be combined with morphological processing by manual annotation, and has the problems of long time consumption and difficult interactive operation. The other way is to generate by algorithm operation. However, both manual generation and algorithm operation have the problem that the range of unknown region to be solved is unreasonable, resulting in unsatisfactory trimap generation effect. SUMMARY

[0004] To overcome the problems in the related art, the present disclosure provides a trimap generation method, a trimap generation device and a storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a trimap generation method is provided, comprising:

[0006] An original image to be generated into a trimap is obtained, the original image comprising at least a portrait mask after portrait blur processing, a face frame and a foreground region mask. An initial trimap is obtained according to the portrait mask, the face frame and the foreground region mask, the initial trimap comprising a head region trimap and a non-head region trimap. Region segmentation is performed based on unknown regions in the head region trimap. The unknown regions in the head region trimap are updated according to the complexity of the unknown regions in each segmented region after segmentation, to obtain an updated head region trimap. The non-head region trimap and the updated head region trimap are merged to obtain a final trimap of the original image.

[0007] In an implementation, obtaining an initial trimap according to the portrait mask, the face frame and the foreground region mask comprises:

[0008] According to the face frame, the original image is divided into a head region and a non-head region; the foreground region mask is subjected to morphological processing to obtain a first tripartite graph, and a non-head region tripartite graph in the first tripartite graph is extracted; based on pixel gray values of the portrait mask and a pixel gray value threshold, a second tripartite graph is obtained, and a head region tripartite graph in the second tripartite graph is extracted; the non-head region tripartite graph in the first tripartite graph and the head region tripartite graph in the second tripartite graph are merged to obtain an initial tripartite graph.

[0009] In an implementation, based on the pixel gray values of the portrait mask and the pixel gray value threshold, the second tripartite graph is obtained, including:

[0010] The pixel points in the portrait mask with a gray value greater than or equal to a first pixel gray value threshold are determined as a portrait mask foreground region; the pixel points in the portrait mask with a gray value less than a second pixel gray value threshold are determined as a portrait mask background region; the pixel points in the portrait mask other than the portrait mask foreground region and the portrait mask background region are determined as a portrait mask unknown region; the portrait mask foreground region, the portrait mask background region and the portrait mask unknown region are merged to obtain the second tripartite graph.

[0011] In an implementation, according to the face frame, the original image is divided into a head region and a non-head region, including:

[0012] The face frame is enlarged by a set multiple with the center pixel point of the face frame as the center to obtain an enlarged face frame; a region in the original image located in the enlarged face frame is determined as the head region, and a region in the original image located outside the enlarged face frame is determined as the non-head region.

[0013] In an implementation, based on the unknown region in the head region tripartite graph, region segmentation is performed, including:

[0014] The original unknown region in the head region tripartite graph is taken as a first region of interest; the first region of interest is subjected to morphological dilation processing to obtain a second region of interest; a region to be segmented is determined based on the region in the enlarged face frame, and region segmentation is performed on the region to be segmented to obtain a plurality of segmented regions, each segmented region in the plurality of segmented regions includes an unknown region, and the unknown region located in the segmented region includes a partial region of the first region of interest and a partial region of the second region of interest.

[0015] In an implementation, the determination of the region to be segmented includes:

[0016] Taking the center pixel point of the face frame as a reference position, the portrait mask is horizontally divided, and the portrait mask grayscale values included in each region after horizontal division are determined; taking the center pixel point of the face frame as a reference position, the portrait mask is vertically divided, and the portrait mask grayscale values included in each region after vertical division are determined; the number of pixel points with a portrait mask grayscale value greater than a third pixel grayscale value threshold in each region is counted, and the region with the largest number of pixel points is determined as a body region; other regions in the portrait mask except the body region are determined, and the region in the other regions located in the enlarged face frame is determined as a to-be-divided region.

[0017] In an implementation, the region division of the to-be-divided region includes:

[0018] The edges of the face frame are extended to the edges of the enlarged face frame to divide the to-be-divided head region into a plurality of divided regions, wherein the plurality of divided regions after division include a first type of divided region and a second type of divided region, the first type of divided region is a region composed of the extended edges and the edges of the enlarged face frame, and the second type of divided region is a region composed of the edges of the face frame, the extended edges and the edges of the enlarged face frame; for the first type of divided region, a straight line with a set angle with the extended edges is used for division, and for the second type of divided region, a straight line perpendicular to the edges of the face frame is used for division.

[0019] In an implementation, the unknown region in the head region trimap is updated according to the complexity of the unknown region in each divided region after division, including:

[0020] For each divided region in the plurality of divided regions, the unknown region in the head region trimap in each divided region is updated in the following manner respectively: the number of color vertices in the second region of interest is determined, and the number of color vertices is determined as the complexity of the unknown region in the divided region; if the complexity of the unknown region in the divided region is less than or equal to a complexity threshold, the size of the unknown region located in the divided region is maintained unchanged; if the complexity of the unknown region in the divided region is greater than the complexity threshold, the unknown region located in the divided region is reduced to a set pixel width outside the foreground region.

[0021] In an implementation, before the initial trimap is obtained according to the portrait mask, the face frame and the foreground region mask, the trimap making method further includes:

[0022] It is determined that the portrait after the portrait blur processing is in the foreground region.

[0023] In an implementation, the determination that the portrait after the portrait blur processing is in the foreground region includes:

[0024] a center pixel point of the face frame as a center, a window region with a set radius is constructed; if there are pixel points greater than a first pixel quantity threshold in the window region located in the foreground region, and a quantity of pixel points with pixel gray scale values greater than a fourth pixel gray scale threshold in the window region is greater than a second pixel quantity threshold, it is determined that the portrait after the portrait blur processing is in the foreground region.

[0025] According to a second aspect of the embodiments of the present disclosure, a trinary graph production device is provided, comprising:

[0026] an acquisition module, configured to acquire an original image to be produced into a trinary graph, the original image comprising at least a portrait mask after portrait blur processing, a face frame, and a foreground region mask;

[0027] a processing module, configured to obtain an initial trinary graph according to the portrait mask, the face frame, and the foreground region mask, the initial trinary graph comprising a head region trinary graph and a non-head region trinary graph, perform region segmentation based on an unknown region in the head region trinary graph, update the unknown region in the head region trinary graph according to a complexity of the unknown region in each segmented region after segmentation, obtain an updated head region trinary graph, and combine the non-head region trinary graph and the updated head region trinary graph to obtain a final trinary graph of the original image.

[0028] In an implementation, the processing module is configured to:

[0029] divide the original image into a head region and a non-head region according to the face frame, perform morphological processing on the foreground region mask to obtain a first trinary graph, and extract a non-head region trinary graph in the first trinary graph; obtain a second trinary graph based on pixel gray scale values of the portrait mask and a pixel gray scale value threshold, and extract a head region trinary graph in the second trinary graph; and combine the non-head region trinary graph in the first trinary graph and the head region trinary graph in the second trinary graph to obtain an initial trinary graph.

[0030] In an implementation, the processing module is configured to:

[0031] determine, as a portrait mask foreground region, a pixel point with a gray scale value greater than or equal to a first pixel gray scale value threshold in the portrait mask; determine, as a portrait mask background region, a pixel point with a gray scale value less than a second pixel gray scale value threshold in the portrait mask; determine, as a portrait mask unknown region, a pixel point other than the portrait mask foreground region and the portrait mask background region in the portrait mask; and combine the portrait mask foreground region, the portrait mask background region, and the portrait mask unknown region to obtain a second trinary graph.

[0032] In an embodiment, the processing module is configured to:

[0033] enlarge the face frame by a set multiple of the center pixel point of the face frame to obtain an enlarged face frame; the region determination unit is configured to determine a region in the original image located within the enlarged face frame as a head region, and determine a region in the original image located outside the enlarged face frame as a non-head region.

[0034] In an embodiment, the processing module is configured to:

[0035] the original unknown region in the head region tripartite graph is taken as a first region of interest; the first region of interest is subjected to a morphological dilation process to obtain a second region of interest; a region to be segmented is determined based on the region within the enlarged face frame, and the region to be segmented is subjected to region segmentation to obtain a plurality of segmented regions, each of the plurality of segmented regions includes an unknown region, and the unknown region located within the segmented region includes part of the first region of interest and part of the second region of interest.

[0036] In an embodiment, the processing module is configured to:

[0037] the portrait mask is horizontally divided with the center pixel point of the face frame as a reference position, and the portrait mask grayscale value included in each region after horizontal division is determined; the portrait mask is vertically divided with the center pixel point of the face frame as a reference position, and the portrait mask grayscale value included in each region after vertical division is determined; the number of pixel points with a portrait mask grayscale value greater than a third pixel grayscale value threshold in each region is counted, and the region with the largest number of pixel points is determined as a body region; other regions of the portrait mask except the body region are determined, and a region in the other regions located within the enlarged face frame is determined as a region to be segmented.

[0038] In an embodiment, the processing module is configured to:

[0039] the edges of the face frame are extended to the edges of the enlarged face frame to divide the head region to be segmented into a plurality of segmented regions, wherein the plurality of segmented regions after division include a first type of segmented region and a second type of segmented region, the first type of segmented region is a region composed of the extended edges and the edges of the enlarged face frame, and the second type of segmented region is a region composed of the edges of the face frame, the extended edges, and the edges of the enlarged face frame; the first type of segmented region is segmented using a straight line having a set angle with the extended edges, and the second type of segmented region is segmented using a straight line perpendicular to the edges of the face frame.

[0040] In an embodiment, the processing module is configured to:

[0041] For each of the plurality of segmentation regions, the unknown region in the head region trimap in each segmentation region is updated by determining a number of color vertices in the second region of interest, and determining a complexity of the unknown region in the segmentation region based on the number of color vertices; if the complexity of the unknown region in the segmentation region is less than or equal to a complexity threshold, maintaining the size of the unknown region in the segmentation region unchanged; and if the complexity of the unknown region in the segmentation region is greater than the complexity threshold, reducing the unknown region in the segmentation region to a set pixel width outside the foreground region.

[0042] In an embodiment, the trimap generation apparatus further comprises:

[0043] The portrait position determination module is configured to determine that the portrait after the portrait blur processing is in the foreground region before the processing module generates the initial trimap according to the portrait mask, the face bounding box, and the foreground region mask.

[0044] In an embodiment, the portrait position determination module is configured to:

[0045] The window region with a set radius is constructed with the center pixel point of the face bounding box as the center; if there are more than a first pixel number threshold of pixel points in the window region that are in the foreground region, and the number of pixel points in the window region with pixel gray values greater than a fourth pixel gray threshold in the portrait mask is greater than a second pixel number threshold, it is determined that the portrait after the portrait blur processing is in the foreground region.

[0046] According to a third aspect of the embodiments of the present disclosure, a trimap generation apparatus is provided, comprising:

[0047] a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the trimap generation method in the first aspect or any one of the embodiments of the first aspect.

[0048] According to a fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, when the instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to execute the trimap generation method in the first aspect or any one of the embodiments of the first aspect.

[0049] The technical scheme provided by the embodiment of the present disclosure can have the following beneficial effects: based on the unknown region in the head region tri-map, region segmentation is performed, and the unknown region in the head region tri-map is updated according to the complexity of the unknown region in each segmented region after segmentation, so that the size of the unknown region range can be reasonably determined, thereby providing the accuracy of the tri-map production.

[0050] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0052] Figure 1 is a flowchart of a tri-map production method according to an exemplary embodiment.

[0053] Figure 2 is a schematic diagram of a calculation result of an original image using a portrait virtual algorithm according to an embodiment of the present disclosure;

[0054] Figure 3 is a flowchart of a tri-map production method according to an exemplary embodiment;

[0055] Figure 4 is a flowchart of a tri-map production method according to an exemplary embodiment;

[0056] Figure 5 is a schematic diagram of a divided original image according to an exemplary embodiment;

[0057] Figure 6 is a schematic diagram of a first tri-map according to an exemplary embodiment;

[0058] Figure 7 is a flowchart of a tri-map production method according to an exemplary embodiment;

[0059] Figure 8 is a schematic diagram of a second tri-map according to an exemplary embodiment;

[0060] Figure 9 is a schematic diagram of an initial tri-map according to an exemplary embodiment;

[0061] Figure 10 is a flowchart of a region segmentation according to an exemplary embodiment;

[0062] Figure 11 is a flowchart of a region segmentation according to an exemplary embodiment;

[0063] Figure 12 is a flowchart of a region segmentation according to an example embodiment;

[0064] Figure 13 is a schematic diagram of a region to be segmented according to an example embodiment of the present disclosure;

[0065] Figure 14 is a schematic diagram of a region to be segmented according to an example embodiment of the present disclosure;

[0066] Figure 15 is a flowchart of a tri-map making method according to an example embodiment;

[0067] Figure 16 is a flowchart of a tri-map making method according to an example embodiment;

[0068] Figure 17 is a flowchart of a tri-map making method according to an example embodiment;

[0069] Figure 18 is a schematic diagram of a tri-map made by a related art according to an example embodiment of the present disclosure;

[0070] Figure 19 is a schematic diagram of a tri-map made by the present disclosure according to an example embodiment of the present disclosure;

[0071] Figure 20 is a schematic diagram of a first original image according to an example embodiment of the present disclosure;

[0072] Figure 21 is a schematic diagram of a tri-map transparency mask of the first original image obtained by a related art according to an example embodiment of the present disclosure;

[0073] Figure 22 is a schematic diagram of a tri-map transparency mask of the first original image obtained by the present disclosure according to an example embodiment of the present disclosure;

[0074] Figure 23 is a schematic diagram of a second original image according to an example embodiment of the present disclosure;

[0075] Figure 24 is a schematic diagram of a tri-map transparency mask of the second original image obtained by a related art according to an example embodiment of the present disclosure;

[0076] Figure 25 is a schematic diagram of a tri-map transparency mask of the second original image obtained by the present disclosure according to an example embodiment of the present disclosure;

[0077] Figure 26 is a block diagram of a tri-map making apparatus according to an example embodiment;

[0078] Figure 27 is a block diagram for making a tripartite device according to an exemplary embodiment. DETAILED DESCRIPTION

[0079] The exemplary embodiments will be described in detail with reference to the accompanying drawings. In the following description, unless otherwise denoted, the same numbers in different drawings denote the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they only represent examples of devices and methods consistent with some aspects of the present disclosure, as detailed in the appended claims.

[0080] In the film and television industry, in order to improve the presentation effect of the picture and meet the artistic needs of the film and television works, image processing technology needs to be used to process images or videos. This process uses the chroma key matting method in image matting to accurately extract the foreground object from the image or video, obtains a high-precision transparent mask (alpha matte), separates the foreground and background accurately, and then performs operations such as replacing the background for the foreground and highlighting the foreground and weakening the background, thereby improving the presentation effect of the picture and meeting the artistic needs of the film and television works.

[0081] In order to improve the shooting effect of the terminal, especially to highlight the effect of the portrait in the portrait mode, it is necessary to provide a transparent mask for the portrait through image matting technology. However, the chroma key matting in the film and television industry requires a large amount of manpower and material resources to build a pure color scene as a background, which is not common in real life. Therefore, the chroma key matting method cannot be applied to the terminal to provide services for users to take pictures in the portrait mode. In order to realize high-precision matting in a natural scene, related technologies have proposed natural image matting. At present, natural image matting methods are mainly divided into three categories: matting algorithm based on optimal pixel pairs, matting algorithm based on deep learning, and matting algorithm based on propagation. The matting algorithm based on optimal pixel pairs converts the transparent mask estimation problem into finding a pair of optimal foreground and background pixel points for each unknown pixel point. The matting algorithm based on deep learning is a process of learning the matting algorithm through a large amount of training data, thereby realizing the estimation of the transparent mask. The matting algorithm based on propagation is to utilize the similarity of adjacent pixels to propagate the alpha value from the known pixel area to the unknown pixel area, thereby obtaining all the transparent mask estimates.

[0082] But the color information of the image in the natural scene is very rich, in order to improve the accuracy and effect of image matting, prior information needs to be provided for each image, if the prior information of the image is missing, it will greatly increase the difficulty of matting. Therefore, in the image matting process, not only an original image needs to be input, but also a trimap corresponding to the original image needs to be input as prior information, the trimap is to divide the color image into the foreground which needs to be reserved, the background which needs to be removed and the unknown region which needs to be determined. The original image in the embodiment of the disclosure can be a color image (rgb).

[0083] There are two ways to make the trimap at present, one is manual making, which needs manual annotation and morphological processing, and has the problems of long time consumption and difficult interactive operation, and is not suitable for real-time and automatic application scenarios, and the other is generated by algorithm operation, which mainly extracts information in the image (image segmentation, target detection, etc.), uses the information as priori, selects the foreground and the background, and uses the morphological method at the junction of the foreground and the background to obtain the unknown region, thereby generating the trimap automatically. However, the method of generating by algorithm operation is very dependent on the accuracy of the prior information, and the unknown region obtained by the morphological method is not fine enough. Generally, too small unknown region will cause confusion of the foreground and the background and loss of more details, and too large unknown region will cause rapid increase of error estimation and performance of the matting algorithm.

[0084] The disclosure provides a trimap making method to solve the problem of inaccurate division of unknown region in the related art. In the making method, an initial trimap is obtained according to a portrait mask after portrait blur processing, a face frame and a foreground region mask, the initial trimap includes a head region trimap and a non-head region trimap. Then, based on the unknown region in the head region trimap, region segmentation is performed, and the unknown region in the head region trimap is updated according to the complexity of the unknown region in each segmented region after segmentation, to obtain an updated head region trimap. The update process can reasonably determine the range of the unknown region, not only overcoming the confusion of the foreground and the background caused by too small unknown region and preserving more details of the original image, but also solving the problems of rapid increase of error estimation and performance of the matting algorithm caused by too large unknown region. Finally, the non-head region trimap and the updated head region trimap are merged to obtain the final trimap of the original image, and the image matting based on the final trimap improves the image matting effect.

[0085] The trimap making method disclosed in the disclosure can make a fine trimap in real time in a natural scene when taking a photo by portrait blur, not only can maintain the effect of the image matting algorithm, but also greatly expands the application range of the image matting algorithm.

[0086] The method of making a trimap according to the present disclosure is applicable to any scenario that utilizes a portrait blur algorithm for processing, such as image fusion, automatic foreground extraction, video synthesis and film making, and cameras in terminals.

[0087] Figure 1 is a flowchart of a method of making a trimap according to an example embodiment, as Figure 1 The method of making a trimap is used in a terminal and includes the following steps.

[0088] In step S11, an original image to be made into a trimap is obtained.

[0089] In the embodiment of the present disclosure, the original image to be made into a trimap includes at least a portrait mask after portrait blur processing, a face frame, and a foreground region mask.

[0090] Figure 2 is a schematic diagram of a calculation result obtained using a portrait virtual algorithm according to an embodiment of the present disclosure. As Figure 2 shown, when the terminal is running in a portrait mode, the original image obtained is calculated in real time according to a portrait blur algorithm to obtain a portrait mask (portrait mask), a face frame, and a depth map. The original image in the embodiment of the present disclosure is a clear image. Then, the depth map is calculated by the depth of the focus point (binary processing) to obtain a clear area (Depth of Field, dof), and a foreground region mask (dof mask) is generated according to the dof. Finally, the portrait mask, the face frame, and the dof mask of the clear image are obtained. For example, in a mobile phone camera, when the mobile phone is in a portrait mode, the focus is set to the human body by default, so the human body belongs to the dof region. First, a clear image is obtained in the camera, and the dof region, the portrait mask, and the face frame are calculated using the portrait blur algorithm. The dof mask is obtained by binary processing the dof region.

[0091] In step S12, an initial trimap is obtained according to the portrait mask, the face frame, and the foreground region mask.

[0092] In the embodiment of the present disclosure, the initial trimap obtained according to the portrait mask, the face frame, and the foreground region mask includes a head region trimap and a non-head region trimap.

[0093] The following embodiment will describe the implementation process of obtaining an initial trimap according to a portrait mask, a face frame, and a foreground region mask with reference to the accompanying drawings.

[0094] In the embodiment of the present disclosure, Figure 3 is a flowchart of a method of making a trimap according to an example embodiment, as Figure 3As shown, according to the portrait mask, the face frame and the foreground region mask, an initial trimap is obtained, including the following steps.

[0095] In step S21, the original image is divided into a head region and a non-head region according to the face frame.

[0096] Figure 4 is a flowchart of a trimap making method according to an example embodiment, as shown in Figure 4 As shown, the original image is divided into a head region and a non-head region according to the face frame, including the following steps.

[0097] In step S41, the face frame is enlarged by a set multiple with the center pixel point of the face frame as the center to obtain an enlarged face frame.

[0098] In step S42, the region in the original image located in the enlarged face frame is determined as the head region, and the region in the original image to be made into a trimap located outside the enlarged face frame is determined as the non-head region.

[0099] In the embodiment of the present disclosure, considering that the boundary of the hair region in the human body is the most complex, the original image to be made into a trimap is divided into a head region and a non-head region according to the face frame, Figure 5 is a schematic diagram of the original image after division according to an example embodiment, as shown in Figure 5 As shown, the center pixel point frect_center(xc,yc) of the face frame is calculated using the coordinates and the length and width size of the face frame, and the face frame frect is enlarged by a scale multiple with frect_center as the center to obtain an enlarged face frame large_frect(xl,yl,wl,hl). The image is divided into two regions by large_frect, one part belongs to large_frect and is called the head region; the other part is not in large_frect and is called the non-head region. In the embodiment of the present disclosure, scale is set to 3, and in practice, scale can be set according to the size of the face. The enlarged face frame needs to satisfy that the head region includes the hair region. Because the boundary of the hair region is complex, determining the unknown region of the hair region has a great influence on the matting effect, and the head region needs to be finely processed to improve the making effect of the head trimap.

[0100] In step S22, the foreground region mask is subjected to morphological processing to obtain a first trimap, and a non-head region trimap in the first trimap is extracted.

[0101] In the embodiments of the present disclosure, the foreground region mask dof mask is subjected to a morphological erosion operation according to a preset erosion radius, to obtain an eroded foreground region mask erode_dof_mask; in the embodiments of the present disclosure, the preset erosion radius is width_e1 pixel width, and width_e1 can be set to 5.

[0102] The foreground region mask dof mask is subjected to a morphological dilation operation according to a preset dilation radius, to obtain a dilated foreground region mask dilate_dof_mask; in the embodiments of the present disclosure, the preset dilation radius is width_d1 pixel width, and width_d1 can be set to 5.

[0103] Figure 6 is a schematic diagram of a first tripartite graph according to an example embodiment, as shown in Figure 6 erode_dof_mask and dilate_dof_mask are subjected to a merging operation, the gray value of a point with the same pixel gray value is retained, and the gray value of a point with a different pixel gray value is set to a set pixel gray value, to obtain a first tripartite graph. In the embodiments of the present disclosure, the set pixel gray value is 128.

[0104] A non-head region tripartite graph in the first tripartite graph obtained in step S22 is extracted.

[0105] In step S23, a second tripartite graph is obtained based on the pixel gray value of the portrait mask and a pixel gray value threshold, and a head region tripartite graph in the second tripartite graph is extracted.

[0106] In the embodiments of the present disclosure, since the gray value in the portrait mask represents the probability that the pixel point belongs to the human body, there is often a transition zone from white to black at the boundary of the human body in the portrait mask. In particular, in the hair region of the human body, there is often a large-area transition zone. In order to improve the tripartite graph making effect on the hair region of the human body, a second tripartite graph is obtained based on the pixel gray value of the portrait mask and a pixel gray value threshold, and a head region tripartite graph in the second tripartite graph is extracted.

[0107] In the embodiments of the present disclosure, Figure 7 is a flowchart of a tripartite graph making method according to an example embodiment, as shown in Figure 7 The making of the second tripartite graph includes the following steps.

[0108] In step S31, the pixel points in the portrait mask whose gray value is greater than or equal to a first pixel gray value threshold are determined as foreground regions of the portrait mask; in the embodiments of the present disclosure, the first pixel gray value threshold can be set to 220.

[0109] In step S32, the pixel points in the portrait mask whose gray values are less than the second pixel gray value threshold are determined as the background region of the portrait mask; in the embodiment of the present disclosure, the second pixel gray value threshold can be set to 10.

[0110] In step S33, the pixel points in the portrait mask other than the foreground region of the portrait mask and the background region of the portrait mask are determined as the unknown region of the portrait mask.

[0111] In step S34, the foreground region of the portrait mask, the background region of the portrait mask and the unknown region of the portrait mask are merged to obtain a second tri-map.

[0112] In the embodiment of the present disclosure, Figure 8 is a schematic diagram of the second tri-map according to an exemplary embodiment, as Figure 8 shown, the second tri-map includes a white foreground region of the portrait mask, a black background region of the portrait mask and a gray unknown region of the portrait mask.

[0113] Compared with a real-time body segmentation system provided in the related art, in the related art, face detection and body detection are first performed on an image, the region where the body is located is preliminarily determined, the boundary of the body region is processed by using erosion and expansion to obtain a tri-map, and finally a matting algorithm is used to obtain a fine body region. However, the face detection and body detection technology described in the scheme in the related art cannot completely obtain all the information of the human body, such as hair and decoration. Moreover, the tri-map made by only using erosion and expansion is relatively rough, and the unknown region is too small, the foreground and the background are confused, the image matting algorithm effect is poor, and the unknown region is too large, the performance of the matting algorithm is sharply increased, and the like. The embodiment of the present disclosure performs threshold processing operation based on the pixel gray value of the portrait mask and the pixel gray value threshold, which lays a foundation for subsequent fine processing of the unknown region of the head region.

[0114] In step S24, the non-head region tri-map in the first tri-map and the head region tri-map in the second tri-map are merged to obtain an initial tri-map.

[0115] In the embodiment of the present disclosure, Figure 9 is a schematic diagram of the initial tri-map according to an exemplary embodiment, as Figure 9 shown, the non-head region tri-map in the first tri-map and the head region tri-map in the second tri-map are merged to obtain an initial tri-map. By performing fine processing on the head region in the initial tri-map, the accuracy of the final tri-map is improved.

[0116] In step S13, based on the unknown region in the head region ternary graph, region segmentation is performed, and according to the complexity of the unknown region in each segmented region after segmentation, the unknown region in the head region ternary graph is updated to obtain an updated head region ternary graph.

[0117] In the embodiment of the present disclosure, considering that the hair of a person in a natural scene is obviously different from that of an individual, at this time, the size of the unknown region in the head region is significantly different. In order to reduce the performance of the image matting algorithm and alleviate the effect problem caused by the alpha error estimation of the image matting algorithm when the color is confused, the embodiment of the present disclosure performs fine processing on the unknown region of the head region.

[0118] In the embodiment of the present disclosure, Figure 10 is a flowchart of region segmentation according to an exemplary embodiment, as shown in Figure 10 Based on the unknown region in the head region ternary graph, region segmentation is performed, including the following steps.

[0119] In step S51, the original unknown region in the head region ternary graph is taken as a first region of interest (ROI1).

[0120] In step S52, the first region of interest is subjected to morphological dilation processing to obtain a second region of interest (ROI2).

[0121] In step S53, based on the region in the enlarged face frame, a region to be segmented is determined, and region segmentation is performed on the region to be segmented to obtain a plurality of segmented regions, each of the plurality of segmented regions includes an unknown region, and the unknown region located in the segmented region includes part of the first region of interest and part of the second region of interest.

[0122] In the embodiment of the present disclosure, according to the similarity of the pixel points in the small neighborhood in the natural image, whether the background color in which the hair is located in the first region of interest is simple is expressed by the complexity of the color in the second region of interest, and then the range of the unknown region is determined according to the judgment result.

[0123] In the embodiment of the present disclosure, Figure 11 is a flowchart of region segmentation according to an exemplary embodiment, as shown in Figure 11 Determining the region to be segmented includes the following steps.

[0124] In step S61, the portrait mask is horizontally segmented with the center pixel point of the face frame as a reference position, and the gray value of the portrait mask included in each region after horizontal segmentation is determined.

[0125] In step S62, the portrait mask is vertically divided with the center pixel point of the face frame as the reference position, and the portrait mask grayscale values in each region after vertical division are determined.

[0126] In step S63, the number of pixel points with the portrait mask grayscale value greater than the third pixel grayscale value threshold in each region is counted, and the region with the largest number of pixel points is determined as the body region.

[0127] In step S64, other regions in the portrait mask except the body region are determined, and the region in the other regions located in the enlarged face frame is determined as the region to be divided.

[0128] For example, in the embodiment of the present disclosure, since the posture of the mobile phone cannot be predicted when the user takes a picture with the mobile phone, the approximate position of the body part (not including the head) in the portrait mask needs to be determined first. A straight line is drawn horizontally with the center pixel point frect_center in the face frame as the reference to divide the image into two regions farea_Up and farea_Down. Then, a straight line is drawn vertically with frect_center as the reference to divide the image into two regions farea_Left and farea_Right. The number of pixel points with the portrait mask grayscale value greater than the third pixel grayscale value threshold in the four regions is calculated. In the embodiment of the present disclosure, the third pixel grayscale value threshold is set to 220. The region with the largest number of pixel points is determined as the position of the body part, and is marked as direction_body. Other regions in the portrait mask except the body region are determined, and the region in the other regions located in the enlarged face frame is determined as the region to be divided.

[0129] In the embodiment of the present disclosure, Figure 12 is a flowchart of region division according to an exemplary embodiment, as Figure 12 shown, the region division of the region to be divided includes the following steps.

[0130] In step S71, the edges of the face frame are extended to the edges of the enlarged face frame to divide the head region to be divided into a plurality of division regions.

[0131] In the embodiment of the present disclosure, the plurality of division regions after division include first type division regions and second type division regions, the first type division regions are regions composed of the extended edges and the edges of the enlarged face frame, and the second type division regions are regions composed of the edges of the face frame, the extended edges and the edges of the enlarged face frame.

[0132] In the embodiment of the present disclosure, Figure 13 is a schematic diagram of the region to be divided according to the embodiment of the present disclosure. As Figure 13As shown, the part of the non-foreground region in the large_frect is defined as the region to be segmented, the a side, the b side and the c side of the face frame are extended to the sides of the enlarged face frame, and the extended sides d, e, f, g, h and i obtained by extending the a side, the b side and the c side of the face frame to the sides of the enlarged face frame divide the region to be segmented into five parts as shown, wherein the segmentation regions numbered 1 and 2 are the first type of segmentation region, and the segmentation regions numbered 3, 4 and 5 are the second type of segmentation region. Figure 13 As shown, the segmentation region numbered 1 includes the extended side d, the extended side e and the side of the enlarged face frame, the segmentation region numbered 2 includes the extended side f, the extended side g and the side of the enlarged face frame, the segmentation region numbered 3 includes the b side of the face frame, the extended side e, the extended side f and the side of the enlarged face frame, the segmentation region numbered 4 includes the a side of the face frame, the extended side d, the extended side h and the side of the enlarged face frame, and the segmentation region numbered 5 includes the c side of the face frame, the extended side g, the extended side i and the side of the enlarged face frame. Figure 13 As shown, the segmentation region numbered 1 includes the extended side d, the extended side e and the side of the enlarged face frame, the segmentation region numbered 2 includes the extended side f, the extended side g and the side of the enlarged face frame, the segmentation region numbered 3 includes the b side of the face frame, the extended side e, the extended side f and the side of the enlarged face frame, the segmentation region numbered 4 includes the a side of the face frame, the extended side d, the extended side h and the side of the enlarged face frame, and the segmentation region numbered 5 includes the c side of the face frame, the extended side g, the extended side i and the side of the enlarged face frame.

[0133] In step S72, for the first type of segmentation region, a straight line having a set angle with the extended side is used for segmentation, and for the second type of segmentation region, a straight line perpendicular to the side of the face frame is used for segmentation.

[0134] As shown, Figure 14 is a schematic diagram of the segmented region to be segmented according to an example embodiment of the present disclosure. As shown, Figure 14 In order to ensure that the adjacent first region of interest and the second region of interest are one-to-one corresponding and to make the calculation more simple, the first type of segmentation region is segmented by using a straight line having a set angle with the extended side, as shown, Figure 14 As shown, the segmentation region numbered 4 and the segmentation region numbered 5 are segmented by using a horizontal straight line, and the segmentation region numbered 3 is segmented by using a vertical straight line, and the width of each segmented region is set to width_seg in the segmentation process, and the width of the last segmented region in each segmented region should be less than or equal to width_seg, and width_seg is set to 32 in the example embodiment of the present disclosure. In addition, the second type of segmentation region numbered 1 and the second type of segmentation region numbered 2 are segmented by using a 45° straight line, and are directly divided into two segmented regions. As shown, Figure 14 As shown, each segmented region has a part of the first region of interest and a part of the second region of interest.

[0135] In the example embodiment of the present disclosure, Figure 15 is a flowchart of a tripartite graph making method according to an example embodiment, as shown, Figure 15 The updating of the unknown region in the head region tripartite graph includes the following steps.

[0136] In step S81, for each of the plurality of segmentation regions, the unknown region in the head region ternary graph in each segmentation region is updated respectively in the same way.

[0137] In the embodiment of the present disclosure, the complexity of all pixel points in the second region of interest is used to determine the complexity of the background in the first region of interest. If the complexity in each segmentation region is less than a complexity threshold, it indicates that the color in the segmentation region is simple, otherwise it indicates that the color in the segmentation region is complex. In the embodiment of the present disclosure, the complexity threshold is set to 3.

[0138] In step S82, the number of color vertices in the second region of interest is determined, and the complexity of the unknown region in the segmentation region is determined based on the number of color vertices.

[0139] In the embodiment of the present disclosure, for each segmentation region, the color vertex of each pixel point in the second region of interest is calculated, and the number of color vertices is counted, and the complexity of the unknown region in the segmentation region is determined based on the number of color vertices. In one implementation, the complexity of the unknown region in the segmentation region is determined based on the color vertex hash value. In the embodiment of the present disclosure, taking the YUV image as an example, the color vertex is represented by a hash value, for example, the color of a pixel point is (y, u, v), and the corresponding color vertex hash value is index = ((y >> a) << 8) + ((u >> a) << 4) + (v >> a), in the embodiment of the present disclosure, a is set to 4; the number of color vertex hash values in each segmentation region is counted, and the number of color vertex hash values is taken as the complexity of the segmentation region.

[0140] In step S83, if the complexity of the unknown region in the segmentation region is less than or equal to the complexity threshold, the size of the unknown region located in the segmentation region is maintained unchanged.

[0141] In step S84, if the complexity of the unknown region in the segmentation region is greater than the complexity threshold, the unknown region located in the segmentation region is reduced to a set pixel width outside the foreground region.

[0142] In the embodiment of the present disclosure, if the color in the segmentation region is simple, it is inferred that the complexity of the background in the first region of interest in the segmentation region is low, and retaining the unknown region can obtain greater benefits. If the color in the segmentation region is complex, it is inferred that the complexity of the background in the first region of interest in the segmentation region is high, in order to reduce the wrong estimation of the image matting algorithm and reduce the algorithm calculation time, the unknown region is reduced to a set pixel width outside the foreground region, in the embodiment of the present disclosure, the unknown region is reduced to a position of 5 pixel widths outside the foreground region.

[0143] In the embodiments of the present disclosure, the calculation complexity is reduced by region segmentation, and the complexity of the unknown region in the segmented region is compared with the complexity threshold, so that the range of the unknown region can be quickly determined. Tests prove that the time consumed for making the trimap by using the present disclosure is about 30 ms, which is better than the time consumed for making the trimap by using related technologies.

[0144] In step S14, the non-head region trimap and the updated head region trimap are merged to obtain the final trimap of the original image to be made.

[0145] In the embodiments of the present disclosure, the head region in the initial trimap is finely processed, so that the trimap effect of the head details such as hair and headwear in the finally made trimap is better.

[0146] In the embodiments of the present disclosure, Figure 16 is a flowchart of a trimap making method according to an exemplary embodiment, as shown in Figure 16 Before obtaining the initial trimap according to the portrait mask, the face frame and the foreground region mask, the trimap making method further includes:

[0147] In step S91, it is determined that the portrait after the portrait blurring processing is in the foreground region.

[0148] In the embodiments of the present disclosure, before obtaining the initial trimap according to the portrait mask, the face frame and the foreground region mask, it is determined that the portrait after the portrait blurring processing is in the foreground region. The purpose of the determination in the embodiments of the present disclosure is to ensure the accuracy in the operation process of the matting algorithm.

[0149] In the embodiments of the present disclosure, Figure 17 is a flowchart of a trimap making method according to an exemplary embodiment, as shown in Figure 17 The determination that the portrait after the portrait blurring processing is in the foreground region includes the following steps.

[0150] In step S101, a window region with a set radius is constructed with the center pixel point of the face frame as the center.

[0151] In step S102, if there are pixel points greater than the first pixel number threshold in the window region located in the foreground region, and the number of pixel points with pixel gray values greater than the fourth pixel gray threshold in the window region in the portrait mask is greater than the second pixel number threshold, it is determined that the portrait after the portrait blurring processing is in the foreground region.

[0152] For example, in the embodiment of the present disclosure, the process of determining that the portrait after the portrait blur processing is in the foreground region is as follows: first, a window region with a certain radius is selected with the center pixel point frect_center as the center, then it is judged whether the pixels in the window region are located in the dof region, and if 3 / 4 of the pixel points are located in the dof region, the judgment is true; then the number of pixel gray values of the portrait mask that are >=200 in the window region is counted, and if the number exceeds 3 / 4 of all pixel points in the window region, the judgment is true. If the above judgments are all true, it is concluded that the human body is in the dof region, otherwise, the step of making a trimap is directly exited.

[0153] A method for automatically generating a trimap is provided in the related art. The method first performs superpixel segmentation on an image to divide the image into a plurality of superpixel blocks; second, uses an oriented texture curve (OTC) feature to describe each superpixel block, then processes these features to obtain foreground and background superpixel blocks, and finally obtains a trimap through erosion and expansion. When multiple objects appear in an image, errors are prone to occur when using this scheme. Obviously, each object can be foreground or background, and the feature processing provided by this scheme cannot obtain correct results. In the embodiment of the present disclosure, it is determined that the portrait after the portrait blur processing is in the foreground region before making a trimap, which ensures the accuracy of the trimap finally made.

[0154] The trimap making method disclosed in the embodiment of the present disclosure can provide a trimap input for the related art. For example, in the process of trimap adaptation and alpha estimation for image matting, the known trimap is fine processed by using a deep learning method, a part of unknown points in the image are classified as foreground or background points, and a transparent mask is accurately estimated by using a deep learning method. Meanwhile, in a natural scene, trimaps are various, and the trimap making method disclosed in the embodiment of the present disclosure can make up for the scenes that cannot be correctly processed by the deep learning method.

[0155] In the embodiment of the present disclosure, the trimap made by the related art is compared with the trimap made by the present disclosure to verify the accuracy of the trimap making method of the embodiment of the present disclosure. Figure 18 FIG. 1 is a schematic diagram of a trimap made by the related art according to an embodiment of the present disclosure, as shown in FIG. 1, the unknown region of the head region trimap is very narrow, and the accuracy is low compared with the head region of the portrait in the original image. Figure 18 FIG. 1 is a schematic diagram of a trimap made by the related art according to an embodiment of the present disclosure, as shown in FIG. 1, the unknown region of the head region trimap is very narrow, and the accuracy is low compared with the head region of the portrait in the original image. Figure 19 FIG. 2 is a schematic diagram of a trimap made by the present disclosure according to an embodiment of the present disclosure, as shown in FIG. 2, the unknown region of the head region trimap is very close to the hairstyle of the portrait in the original image. Figure 19 FIG. 2 is a schematic diagram of a trimap made by the present disclosure according to an embodiment of the present disclosure, as shown in FIG. 2, the unknown region of the head region trimap is very close to the hairstyle of the portrait in the original image. Figure 18 FIG. 2 is a schematic diagram of a trimap made by the present disclosure according to an embodiment of the present disclosure, as shown in FIG. 2, the unknown region of the head region trimap is very close to the hairstyle of the portrait in the original image.Figure 19 It can be found that, for the same original image, the effect of the three-part image made by the technical solution of the present disclosure is obviously better than that of the three-part image made by the related technology.

[0156] In the embodiments of the present disclosure, Figure 20 is a schematic diagram of a first original image shown in the embodiments of the present disclosure, Figure 21 is a schematic diagram of a three-part image transparent mask of the first original image obtained by the related technology shown in the embodiments of the present disclosure, Figure 22 is a schematic diagram of a three-part image transparent mask of the first original image obtained by the present disclosure shown in the embodiments of the present disclosure; comparison Figure 21 and Figure 22 It can be found that, for the same first original image, the three-part image transparent mask obtained by the present disclosure can clearly show the hair, while the three-part image transparent mask obtained by the related technology hardly shows the hair, so the present disclosure improves the accuracy of making the three-part image compared with the related technology.

[0157] In the embodiments of the present disclosure, Figure 23 is a schematic diagram of a second original image shown in the embodiments of the present disclosure; Figure 24 is a schematic diagram of a three-part image transparent mask of the second original image obtained by the related technology shown in the embodiments of the present disclosure; Figure 25 is a schematic diagram of a three-part image transparent mask of the second original image obtained by the present disclosure shown in the embodiments of the present disclosure; comparison Figure 24 and Figure 25 It can be found that, for the same second original image, as Figure 24 shown, the hair area in the three-part image transparent mask obtained by the related technology is very blurred, only showing the outline of the head shape of the second original image, but Figure 25 as shown, the effect of the three-part image transparent mask obtained by the present disclosure in the hair area is obvious.

[0158] In the embodiments of the present disclosure, the three-part image making method provided by the present disclosure can improve the image processing efficiency and increase the resolution. For example, on a mobile phone high pass Snapdragon 888 platform, the resolution of the captured image is 1530x2040 after downsampling. The present disclosure can make the three-part image in real time without human intervention, and the three-part image made by the present disclosure can improve the image matting technology effect when used for image matting, and promote the application of image matting technology in mobile phones.

[0159] Based on the same concept, the embodiments of the present disclosure also provide a three-part image making device.

[0160] It can be understood that the tripartite graph making apparatus provided by the embodiments of the present disclosure comprises a hardware structure and / or a software module corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of the examples disclosed in the embodiments of the present disclosure, the embodiments of the present disclosure can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is realized by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present disclosure.

[0161] Figure 26 is a block diagram of a tripartite graph making apparatus according to an example embodiment. Referring to Figure 26 The apparatus 100 comprises an acquisition module 101 and a processing module 102.

[0162] The acquisition module 101 is configured to acquire an original image to be made into a tripartite graph, the original image to be made into a tripartite graph comprising at least a portrait mask after a portrait blur processing, a face frame and a foreground region mask;

[0163] The processing module 102 is configured to obtain an initial tripartite graph according to the portrait mask, the face frame and the foreground region mask, the initial tripartite graph comprising a head region tripartite graph and a non-head region tripartite graph, perform region segmentation based on an unknown region in the head region tripartite graph, update the unknown region in the head region tripartite graph according to the complexity of the unknown region in each segmented region after segmentation, obtain an updated head region tripartite graph, and merge the non-head region tripartite graph and the updated head region tripartite graph to obtain a final tripartite graph of the original image.

[0164] In the embodiments of the present disclosure, the processing module 102 is configured to:

[0165] divide the original image into a head region and a non-head region according to the face frame, perform morphological processing on the foreground region mask to obtain a first tripartite graph, extract a non-head region tripartite graph in the first tripartite graph, obtain a second tripartite graph based on a pixel gray value of the portrait mask and a pixel gray value threshold, and extract a head region tripartite graph in the second tripartite graph, and merge the non-head region tripartite graph in the first tripartite graph and the head region tripartite graph in the second tripartite graph to obtain an initial tripartite graph.

[0166] In the embodiments of the present disclosure, the processing module 102 is configured to:

[0167] The pixel points in the portrait mask with a gray value greater than or equal to a first pixel gray value threshold are determined as a foreground region of the portrait mask, the pixel points in the portrait mask with a gray value less than a second pixel gray value threshold are determined as a background region of the portrait mask, then the pixel points in the portrait mask other than the foreground region and the background region of the portrait mask are determined as an unknown region of the portrait mask, and finally the foreground region, the background region and the unknown region of the portrait mask are merged to obtain a second tripartite graph.

[0168] In the embodiment of the present disclosure, the processing module 102 is configured to:

[0169] The face frame is enlarged by a set multiple with the center pixel point of the face frame as the center to obtain an enlarged face frame, the region in the original image located in the enlarged face frame is determined as a head region, and the region in the original image located outside the enlarged face frame is determined as a non-head region.

[0170] In the embodiment of the present disclosure, the processing module 102 is configured to:

[0171] The original unknown region in the head region tripartite graph is taken as a first region of interest, and the first region of interest is subjected to a morphological dilation process to obtain a second region of interest; then the region to be segmented is determined based on the region in the enlarged face frame, and the region to be segmented is subjected to region segmentation to obtain a plurality of segmented regions, each of the plurality of segmented regions includes an unknown region, and the unknown region located in the segmented region includes a part of the first region of interest and a part of the second region of interest.

[0172] In the embodiment of the present disclosure, the processing module 102 is configured to:

[0173] The portrait mask is horizontally segmented with the center pixel point of the face frame as a reference position, and the portrait mask gray value included in each region after horizontal segmentation is determined; the portrait mask is vertically segmented with the center pixel point of the face frame as the reference position, and the portrait mask gray value included in each region after vertical segmentation is determined; the number of pixel points with a portrait mask gray value greater than a third pixel gray value threshold in each region is counted, and the region with the largest number of pixel points is determined as a body region; other regions in the portrait mask other than the body region are determined, and the region in the other regions located in the enlarged face frame is determined as the region to be segmented.

[0174] In the embodiment of the present disclosure, the processing module 102 is configured to:

[0175] extend the edges of the face frame to edges of the enlarged face frame to divide the head region to be segmented into a plurality of segmentation regions, wherein the plurality of segmentation regions after division include a first type of segmentation region and a second type of segmentation region, the first type of segmentation region is a region composed of the extended edges and the edges of the enlarged face frame, and the second type of segmentation region is a region composed of the edges of the face frame, the extended edges and the edges of the enlarged face frame; for the first type of segmentation region, a straight line having a set angle with the extended edges is used for segmentation, and for the second type of segmentation region, a straight line perpendicular to the edges of the face frame is used for segmentation.

[0176] In the embodiments of the present disclosure, the processing module 102 is configured to:

[0177] For each segmentation region in the plurality of segmentation regions, the unknown region in the head region trinary graph in each segmentation region is updated in the following manner: the number of color vertices of the second region of interest is determined, and the complexity of the unknown region in the segmentation region is determined based on the number of color vertices; if the complexity of the unknown region in the segmentation region is less than or equal to the complexity threshold, the size of the unknown region located in the segmentation region is maintained unchanged; if the complexity of the unknown region in the segmentation region is greater than the complexity threshold, the unknown region located in the segmentation region is reduced to a set pixel width outside the foreground region.

[0178] In the embodiments of the present disclosure, the trinary graph making device further comprises:

[0179] The portrait position determination module is configured to determine that the portrait after the portrait blurring processing is located in the foreground region before the processing module obtains the initial trinary graph according to the portrait mask, the face frame and the foreground region mask.

[0180] In the embodiments of the present disclosure, the portrait position determination module is configured to:

[0181] A window region with a set radius is constructed with the center pixel point of the face frame as the center; if there are pixel points greater than a first pixel number threshold located in the foreground region in the window region, and the number of pixel points with pixel gray value greater than a fourth pixel gray threshold in the window region is greater than a second pixel number threshold, it is determined that the portrait after the portrait blurring processing is located in the foreground region.

[0182] As to the device in the above embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments relating to the methods, and will not be described in detail here.

[0183] Figure 27 is a block diagram of a device for making a trinary graph according to an exemplary embodiment. For example, the device 200 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0184] Referring to Figure 27 The device 200 can include one or more of the following components: a processing component 202, a memory 204, a power component 206, a multimedia component 208, an audio component 210, an input / output (I / O) interface 212, a sensor component 214, and a communication component 216.

[0185] The processing component 202 generally controls the overall operation of the device 200 such as the operation of the display, the telephone call, the data communication, the camera operation and the recording operation. The processing component 202 can include one or more processors 220 to execute instructions to complete all or a part of steps of the above methods. In addition, the processing component 202 can include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 can include a multimedia module to facilitate the interaction between the multimedia component 208 and the processing component 202.

[0186] The memory 204 is configured to store various types of data to support the operations of the device 200. Examples of these data include instructions for any applications or methods operating on the device 200, contact data, phonebook data, messages, pictures, videos, and so on. The memory 204 can be realized by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0187] The power component 206 provides power to the various components of the device 200. The power component 206 can include a power management system, one or more power sources, and other components associated with generating, managing and distributing power for the device 200.

[0188] The multimedia component 208 includes a screen providing an output interface between the device 200 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensors can not only sense a boundary of a touching or swiping action, but also detect duration and pressure related to the touching or swiping action. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operation mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0189] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC) to receive an external audio signal when the device 200 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 204 or transmitted via the communication component 216. In some embodiments, the audio component 210 also includes a speaker to output audio signals.

[0190] The I / O interface 212 provides an interface between the processing component 202 and peripheral interface modules, such as a keypad, a click wheel, buttons, and so on. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0191] The sensor component 214 includes one or more sensors to provide various state assessments for the device 200. For example, the sensor component 214 can detect an open / closed position of the device 200, relative positioning of components, such as a display and a keypad of the device 200, a change in position of the device 200 or a component of the device 200, presence or absence of user contact with the device 200, a change in orientation of the device 200 or acceleration / deceleration of the device 200, and temperature changes of the device 200. The sensor component 214 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 214 can also include a light sensor, such as a CMOS or CCD image sensor, for use in an imaging application. In some embodiments, the sensor component 214 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0192] The communication component 216 is configured to facilitate wired or wireless communication between the apparatus 200 and other devices. The apparatus 200 can access a wireless network based on a communication standard, such as WiFi, 4G, or 5G, or a combination thereof. In an example embodiment, the communication component 216 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 216 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0193] In an example embodiment, the apparatus 200 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic components, for performing the above-described methods.

[0194] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 204 including instructions, is also provided, which can be executed by the processor 220 of the apparatus 200 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0195] It can be understood that, in the present disclosure, “multiple” refers to two or more, and other quantifiers are similar thereto. “And / or” describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character “ / ” generally represents that the associated objects before and after it are in an “or” relationship. The singular form “a”, “said” and “the” are also intended to include the plural form, unless the context clearly indicates otherwise.

[0196] It can be further understood that the terms “first”, “second”, and the like are used to describe various information, but the information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a particular order or importance. In fact, the expressions “first”, “second”, and the like can be completely interchangeable. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the present disclosure.

[0197] It will be further understood that "connected" can include direct connection between two members or indirect connection between two members through other members.

[0198] It will be further understood that, unless otherwise specified, "connected" includes direct connection or indirect connection through others.

[0199] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations contained in the full scope of the appended claims along with the full scope of equivalents to which such claims are entitled. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0200] It is to be understood that the present disclosure is not limited to the precise construction described and as shown in the attached drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A method of making a tripartite graph, characterized by, The method comprises the following steps: obtaining an original image to be processed into a trimap, wherein the original image at least comprises a portrait mask after a portrait blur processing, a face frame and a foreground region mask; obtaining an initial trimap according to the portrait mask, the face frame and the foreground region mask, wherein the initial trimap comprises a head region trimap and a non-head region trimap; based on an unknown region in the head region trimap, performing region segmentation, and for each segmented region after the segmentation, the unknown region in the head region trimap in each segmented region is updated in the following manner: determining a number of color vertices in a second region of interest, and based on the number of color vertices, determining a complexity of the unknown region in the segmented region; if the complexity of the unknown region in the segmented region is less than or equal to a complexity threshold, maintaining the size of the unknown region located in the segmented region unchanged; if the complexity of the unknown region in the segmented region is greater than the complexity threshold, reducing the unknown region located in the segmented region to a set pixel width outside the foreground region to obtain an updated head region trimap; merging the non-head region trimap and the updated head region trimap to obtain a final trimap of the original image.

2. The tripartite map production method according to claim 1, characterized in that, obtaining an initial trimap according to the portrait mask, the face frame and the foreground region mask comprises: dividing the original image into a head region and a non-head region according to the face frame; performing morphological processing on the foreground region mask to obtain a first trimap, and extracting a non-head region trimap in the first trimap; obtaining a second trimap based on pixel gray values of the portrait mask and a pixel gray value threshold, and extracting a head region trimap in the second trimap; merging the non-head region trimap in the first trimap and the head region trimap in the second trimap to obtain the initial trimap.

3. The triplot production method of claim 2, wherein obtaining a second trimap based on pixel gray values of the portrait mask and a pixel gray value threshold comprises: determining, as a portrait mask foreground region, a pixel point in the portrait mask whose gray value is greater than or equal to a first pixel gray value threshold; determining, as a portrait mask background region, a pixel point in the portrait mask whose gray value is less than a second pixel gray value threshold; determining, as a portrait mask unknown region, a pixel point in the portrait mask other than the portrait mask foreground region and the portrait mask background region; merging the portrait mask foreground region, the portrait mask background region and the portrait mask unknown region to obtain the second trimap.

4. The triplot production method of claim 2, wherein dividing the original image into a head region and a non-head region according to the face frame comprises: enlarging the face frame by a set multiple with a center pixel point of the face frame as the center to obtain an enlarged face frame; determining, as the head region, a region in the original image located in the enlarged face frame, and determining, as the non-head region, a region in the original image located outside the enlarged face frame.

5. The triplot production method of claim 4, wherein, based on the unknown region in the head region trimap, performing region segmentation comprises: taking an original unknown region in the head region trimap as a first region of interest; performing morphological dilation processing on the first region of interest to obtain a second region of interest; determine a region to be segmented based on the region in the face frame after zooming in, and perform region segmentation on the region to be segmented to obtain a plurality of segmented regions, each of the plurality of segmented regions includes an unknown region, and the unknown region located in the segmented region includes part of the first region of interest and part of the second region of interest.

6. The triplot production method of claim 5, wherein, The method further comprises: reference to the center pixel point of the face frame, the portrait mask is horizontally segmented, and the portrait mask gray value included in each region after horizontal segmentation is determined; reference to the center pixel point of the face frame, the portrait mask is vertically segmented, and the portrait mask gray value included in each region after vertical segmentation is determined; statistically count the number of pixel points in each region whose portrait mask gray value is greater than a third pixel gray value threshold, and determine the region with the largest number of pixel points as the body region; determine other regions in the portrait mask except the body region, and determine the region in the other regions located in the face frame after zooming in as the region to be segmented.

7. The triplot production method according to claim 5 or 6, characterized in that, The method further comprises: extend the edges of the face frame to the edges of the face frame after zooming in to divide the head region to be segmented into a plurality of segmented regions, wherein the plurality of segmented regions after division include first type segmented regions and second type segmented regions, the first type segmented regions are regions composed of the extended edges and the edges of the face frame after zooming in, and the second type segmented regions are regions composed of the edges of the face frame, the extended edges and the edges of the face frame after zooming in; for the first type segmented regions, a straight line with a set angle with the extended edges is used for segmentation, and for the second type segmented regions, a straight line perpendicular to the edges of the face frame is used for segmentation.

8. The triplot production method of claim 1, wherein, The method further comprises: determine that the portrait after the portrait blur processing is in the foreground region.

9. The triplot production method of claim 8, wherein, The method further comprises: construct a window region with a set radius with the center pixel point of the face frame as the center; if there are pixel points greater than a first pixel number threshold in the window region located in the foreground region, and the number of pixel points in the window region whose portrait mask gray value is greater than a fourth pixel gray threshold is greater than a second pixel number threshold, then determine that the portrait after the portrait blur processing is in the foreground region.

10. A triplot making apparatus characterized by comprising: The method further comprises: an acquisition module configured to acquire an original image to be used for making a trimap, the original image at least including a portrait mask after portrait blur processing, a face frame and a foreground region mask; The processing module is configured to: obtain an initial trimap according to the portrait mask, the face frame, and the foreground region mask, the initial trimap including a head region trimap and a non-head region trimap; based on an unknown region in the head region trimap, perform region segmentation; and for each segmented region after the segmentation, update the unknown region in the head region trimap in each segmented region by: determining a number of color vertices in a second region of interest, and based on the number of color vertices, determining a complexity of the unknown region in the segmented region; if the complexity of the unknown region in the segmented region is less than or equal to a complexity threshold, maintaining the size of the unknown region located in the segmented region unchanged; and if the complexity of the unknown region in the segmented region is greater than the complexity threshold, reducing the unknown region located in the segmented region to a set pixel width outside the foreground region to obtain an updated head region trimap, and merging the non-head region trimap and the updated head region trimap to obtain a final trimap of the original image.

11. The triplot producing apparatus according to claim 10, wherein The processing module is configured to: divide the original image into a head region and a non-head region according to the face frame; perform morphological processing on the foreground region mask to obtain a first trimap, and extract a non-head region trimap in the first trimap; obtain a second trimap based on pixel grayscale values of the portrait mask and a pixel grayscale value threshold, and extract a head region trimap in the second trimap; merge the non-head region trimap in the first trimap and the head region trimap in the second trimap to obtain an initial trimap.

12. The triplot producing apparatus according to claim 11, wherein The processing module is configured to: determine, as a portrait mask foreground region, a pixel point in the portrait mask whose grayscale value is greater than or equal to a first pixel grayscale value threshold; determine, as a portrait mask background region, a pixel point in the portrait mask whose grayscale value is less than a second pixel grayscale value threshold; determine, as a portrait mask unknown region, a pixel point in the portrait mask other than the portrait mask foreground region and the portrait mask background region; merge the portrait mask foreground region, the portrait mask background region, and the portrait mask unknown region to obtain a second trimap.

13. The triplot producing apparatus according to claim 11, wherein The processing module is configured to: enlarge the face frame by a set multiple with a center pixel point of the face frame as a center to obtain an enlarged face frame; determine, as a head region, a region in the original image located in the enlarged face frame, and determine, as a non-head region, a region in the original image located outside the enlarged face frame.

14. The triplot producing apparatus according to claim 13, wherein The processing module is configured to: take an original unknown region in the head region trimap as a first region of interest; perform morphological dilation processing on the first region of interest to obtain a second region of interest; and perform morphological erosion processing on the second region of interest to obtain a third region of interest. Determine a region to be segmented based on the region in the face frame after amplification, and perform region segmentation on the region to be segmented to obtain a plurality of segmented regions, each of the plurality of segmented regions including an unknown region, and the unknown region located in the segmented region including a partial region of the first region of interest and a partial region of the second region of interest.

15. The triplot making apparatus according to claim 14, wherein The processing module is configured to: divide the portrait mask horizontally with the center pixel point of the face frame as a reference position, and determine the portrait mask grayscale value included in each region after horizontal division; divide the portrait mask vertically with the center pixel point of the face frame as a reference position, and determine the portrait mask grayscale value included in each region after vertical division; count the number of pixel points with a portrait mask grayscale value greater than a third pixel grayscale value threshold in each region, and determine the region with the largest number of pixel points as a body region; determine other regions in the portrait mask except the body region, and determine the region located in the face frame after amplification in the other regions as a region to be segmented.

16. The triplot making apparatus according to claim 14 or 15, wherein The processing module is configured to: extend the edges of the face frame to the edges of the face frame after amplification to divide the head region to be segmented into a plurality of segmented regions, wherein the plurality of segmented regions after division include a first type of segmented region and a second type of segmented region, the first type of segmented region is a region composed of the extended edges and the edges of the face frame after amplification, and the second type of segmented region is a region composed of the edges of the face frame, the extended edges and the edges of the face frame after amplification; for the first type of segmented region, a straight line with a set angle with the extended edges is used for segmentation, and for the second type of segmented region, a straight line perpendicular to the edges of the face frame is used for segmentation.

17. The triplot producing apparatus according to claim 10, wherein The three-part graph making device further includes: A portrait position determination module is configured to determine that the portrait after the portrait blur processing is located in the foreground region before the processing module obtains the initial three-part graph according to the portrait mask, the face frame and the foreground region mask.

18. The triplot producing apparatus according to claim 17, wherein The portrait position determination module is configured to: construct a window region with a set radius with the center pixel point of the face frame as the center; if the pixel points greater than a first pixel number threshold in the window region are located in the foreground region, and the number of pixel points with a pixel grayscale value greater than a fourth pixel grayscale threshold in the window region is greater than a second pixel number threshold, it is determined that the portrait after the portrait blur processing is located in the foreground region.

19. A triplot making apparatus characterized by comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the three-part graph making method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, the mobile terminal is enabled to perform the three-part graph making method of any one of claims 1-9.

Images