Method and system for generating a trimap for image matting

Abstract

A system and method of performing image matting on an image is provided. The method includes detecting, by an image processing system, one or more objects in the image; determining, by the image processing system, a confidence map associated with the image for each of the one or more objects using one or more image segmentation techniques; and generating, by the image processing system, a trimap for each of the one or more objects in the image from the confidence map based on at least one of a size of each of the one or more objects in the image and a distance between a first pixel in the image and a second pixel in at least one of the one or more objects in the image, wherein the trimap is used to perform the image matting.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an image processing system and a method thereof, and more particularly, to a method and system for generating a tri-map for image matting. BACKGROUND

[0002] Generally, in the field of image processing, a technique called image matting is used to accurately estimate foreground objects in images and videos. The estimated foreground objects are extracted from the images and videos. Image matting is an important technique in image and video editing applications, such as film making for creating visual effects, color pop on the background, masking or applying blur effects on the background, perspective zoom, combining two or more images and videos, etc. The image matting technique generally generates a "foreground alpha matte" or "matte" for separating foreground from background in an image or video. In addition, image matting is different from the image segmentation problem, where image segmentation generates a binary image in which pixels in the image belong to either foreground or background. However, image matting is different from image segmentation, where some pixels can belong to both foreground and background, such pixels are called unknown pixels or mixed pixels. Estimating unknown pixels in an image is performed using a tri-map. Using a tri-map of an image or video, a "foreground alpha matte" is produced in image matting. SUMMARY

[0003] TECHNICAL PROBLEM

[0004] Existing image matting methods or currently state-of-the-art segmentation algorithms have limitations in introducing finer boundary details in images including objects such as hair strands, fur clothing, etc. In addition, existing image matting methods generally use separate tri-map input or user intervention to identify regions including unknown pixels.

[0005] One of the existing face matting techniques is to automatically generate a tri-map using face points and grab cut. Another existing interactive image matting technique requires human interaction to mark unknown regions with a bush stroke. However, these techniques require too much input or human intervention, thereby complicating the matte generation process and making it prone to human errors. Therefore, it is necessary to automatically generate a fine foreground matte from an image or video without any user intervention.

[0006] The information provided above is only intended to strengthen the understanding of the general background of the present disclosure, and should not be considered as acknowledging or in any form suggesting that this information constitutes prior art known to those skilled in the art.

[0007] SOLUTION TO THE PROBLEM

[0008] In the following description, additional aspects will be set forth and described in part in greater detail. The following description is not intended to limit in any way the aspects described, but is intended to provide a description of one or more implementations of the described aspects.

[0009] According to embodiments, a method of performing image matting on an image is provided, the method comprising: detecting, by an image processing system, one or more objects in the image; determining, by the image processing system, a confidence map associated with the image for each of the one or more objects using one or more image segmentation techniques; and generating, by the image processing system, a trimap for each of the one or more objects in the image from the confidence map based on at least one of a size of each of the one or more objects in the image and a distance between a first pixel in the image and a second pixel in at least one of the one or more objects in the image, wherein the trimap is used to perform the image matting.

[0010] Detecting the one or more objects in the image comprises: identifying the one or more objects in the image using one or more object detection techniques.

[0011] Determining the confidence map comprises: computing the confidence map for the one or more objects in the image based on at least one of a pre-trained deep learning model, a depth sensor, or a time-of-flight (TOF) sensor; and modifying the confidence map using a boundary correction technique based on color similarity.

[0012] The method further comprises: determining a transformation value for each of the one or more objects using a ratio of a size of the image to the size of each of the one or more objects in the image and a predetermined value as inputs to a transformation function.

[0013] Generating the trimap for each of the one or more objects in the image further comprises: computing a distance from a first pixel in the image that is outside of the one or more objects to a second pixel in at least one of the one or more objects in the image; comparing the distance to the transformation value corresponding to each of the one or more objects; and based on the distance being less than the transformation value, labeling the first pixel as an unknown pixel; or based on the distance being greater than or equal to the transformation value, labeling the first pixel as at least one of a foreground pixel or a background pixel using the confidence map.

[0014] Performing the image matting comprises: providing the image and the trimap of the image to a first pre-trained deep learning model; and generating a foreground mask of the image, wherein the foreground and the background in the image are separated using the foreground mask.

[0015] According to an embodiment, there is provided an image processing system for performing image matting on an image, the image processing system comprising: a memory; and a processor configured to: detect one or more objects in the image; determine a confidence map associated with the image for each of the one or more objects using one or more image segmentation techniques; and generate a trimap for each of the one or more objects in the image from the confidence map based on at least one of a size of each of the one or more objects in the image and a distance between a first pixel in the image and a second pixel in at least one of the one or more objects in the image, wherein the trimap is used to perform the image matting.

[0016] The processor is further configured to identify the one or more objects in the image using one or more object detection techniques.

[0017] The processor is further configured to compute the confidence map for the one or more objects in the image based on at least one of a pre-trained deep learning model, a depth sensor, or a time-of-flight (TOF) sensor; and modify the confidence map using a boundary correction technique based on color similarity.

[0018] The processor is further configured to determine a transformation value for each of the one or more objects using a ratio of a size of the image to the size of each of the one or more objects in the image and a predetermined value as inputs to a transformation function.

[0019] The processor is further configured to compute a distance from a first pixel in the image that is outside of the one or more objects to a second pixel in at least one of the one or more objects in the image; compare the distance to the transformation value corresponding to each of the one or more objects; and based on the distance being less than the transformation value, label the first pixel as an unknown pixel; or based on the distance being greater than or equal to the transformation value, label the first pixel as at least one of a foreground pixel or a background pixel using the confidence map.

[0020] The processor is further configured to provide the image and the trimap of the image to a first pre-trained deep learning model; and generate a foreground mask of the image, wherein the foreground and the background in the image are separated using the foreground mask.

[0021] According to an embodiment, a method of performing image matting on an image is provided, the method comprising: detecting, by an image processing system, one or more objects in the image; providing, by the image processing system, the image and the one or more objects in the image to a second pre-trained deep learning model to generate a trimap for each of the one or more objects in the image, the second pre-trained deep learning model generating the trimap based on at least one of a size of the image and a size of each of the one or more objects in the image; and performing, by the image processing system, image matting on the image using the trimap for the image.

[0022] The method further comprises training the second pre-trained deep learning model, wherein training the second pre-trained deep learning model comprises: providing one or more sample images as input to an initial deep learning model, wherein the initial deep learning model is based on an encoder-decoder model with skip connections; computing, based on an output of the initial deep learning model, a loss value for each pixel in the one or more sample images using a loss function, wherein the output represents a label for each pixel in the sample image; aggregating the loss value based on the label associated with each pixel in the one or more sample images, wherein the label comprises one of a foreground pixel, a background pixel, or an unknown pixel; and modifying, based on the aggregated loss value, one or more weights of the initial deep learning model using a supervised learning technique.

[0023] The second pixel in at least one of the one or more objects in the image is determined based on a bounding box applied to each of the one or more objects in the image, wherein the second pixel is located at a center of the bounding box.

[0024] Generating the trimap for each of the one or more objects in the image comprises generating the trimap using a morphological operation, wherein a kernel size of the morphological operation is determined based on a ratio of the size of the image to the size of each of the one or more objects in the image.

[0025] The one or more object detection techniques comprise at least one of a bounding box technique using a machine learning based Viola-Jones object detection framework utilizing Haar features, a Scale-Invariant Feature Transform (SIFT), a Histogram of Oriented Gradients (HOG) feature, a deep learning based region proposal, a single shot refinement neural network for object detection, or a single shot multibox detector. BRIEF DESCRIPTION OF DRAWINGS

[0026] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0027] Figure 1An exemplary environment showing a trimap for generating an image matting according to embodiments is shown;

[0028] Figure 2 A detailed block diagram of an image processing system according to embodiments is shown;

[0029] Figure 3 A flowchart showing method steps for generating a trimap for an image matting based on a transformation value and a confidence map according to embodiments is shown;

[0030] Figure 4A An exemplary detection of one or more objects in an image according to embodiments is shown;

[0031] Figure 4B An exemplary determination of a confidence map using a depth sensor according to embodiments is shown;

[0032] Figure 4C An exemplary determination of a confidence map using a time-of-flight sensor according to embodiments is shown;

[0033] Figure 4D An exemplary determination of a confidence map using a pre-trained deep learning model and a modification of the confidence map using a color similarity based boundary correction technique according to embodiments is shown;

[0034] Figure 4E An exemplary trimap generated for an image according to embodiments is shown;

[0035] Figure 4F An exemplary foreground mask generated from a trimap of an image according to embodiments is shown;

[0036] Figure 5 A flowchart showing method steps for generating a trimap for an image matting using a second deep learning model according to embodiments is shown;

[0037] Figure 6 An exemplary generation of a trimap and a foreground mask for an image using a second deep learning model and a first deep learning model according to embodiments is shown; and

[0038] Figure 7 A computer system for generating a trimap for an image matting according to embodiments is shown. DETAILED DESCRIPTION

[0039] In this disclosure, the term“exemplary” is used herein to mean“serving as an example, instance, or illustration.” Any implementation of the description herein that is described as“exemplary” is not necessarily to be construed as preferred or advantageous over other implementations.

[0040] While the embodiments of the disclosure can take many different forms, specific embodiments thereof are shown in the drawings and are described herein in detail with the understanding that the present disclosure is to be considered as an exemplification and is not intended to limit the aspects of the disclosure to that as described herein. The embodiments of the disclosure described herein are not intended to be limited to the preferred embodiments unless so limit by the claims.

[0041] The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a setup, device or method that comprises a list of components or steps does not include only those components or steps but can include other components or steps not expressly listed or inherent to the setup or device or method. In other words, without further restriction, one or more elements of a given setup or system or apparatus developed by "comprising" are not excluded in the setup or system or apparatus.

[0042] In the following description of embodiments, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments in which the disclosure can be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments can be utilized and that changes can be made without departing from the scope of the present disclosure.

[0043] Figure 1 An exemplary environment for generating a trimap for image matting according to embodiments is shown.

[0044] The present disclosure relates to a method of generating a trimap (107) to perform image matting. Image matting techniques generate a "foreground mask" (109). The foreground mask (109) is used to separate the foreground from the background in an image (103) or video. The image (103) or video can be an RGB image or video, a CMYK image or video, etc. In the image (103), the foreground represents the portion of the image (103) that is closer in distance to the camera (110) that captured the image (103), and the background represents the portion of the image (103) that is farther in distance to the camera (110) that captured the image (103). For example, three people in the image (103) can be included in the foreground, and trees in the image (103) can be included in the background. Further, an image processing system (102) is used to perform image matting. The image processing system (102) can be on a user device, such as a smartphone, a tablet, a laptop, a desktop, a server, etc. Figure 1at least one of a smartphone (101), a smartwatch, a notebook computer, a desktop computer, a server, a tablet computer, etc. is shown. The image processing system (102) can perform image matting on an image (103) or a video in real-time. The image processing system (102) receives the image (103) from an image capture device, such as a camera (110), or from a storage medium. The image processing system (102) detects one or more objects (104A, 104B, 104C, collectively referred to as “104”) in the image (103). The one or more objects (104) represent portions of the image (103) that are to be identified as foreground in the image (103). For example, the one or more objects (104) can include at least one of one or more people, one or more animals, one or more items, one or more animated characters, one or more vehicles, etc. The image processing system (102) detects the one or more objects (104) in the image (103) using one or more object detection techniques. For example, the one or more object detection techniques can include a machine learning based viola-jones object detection framework that utilizes haar features, scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG) features, and deep learning based region proposal, single shot refi ne neural network for object detection, single shot multi-box detector, etc. For example, as shown in FIG. 1, the image processing system (102) detects three people’s faces as the one or more objects (104) in the image (103). In one embodiment, the image processing system (102) can receive user input that indicates the one or more objects (104) in the image (103). Figure 1

[0045] In addition, the image processing system (102) determines a confidence map (105) associated with the image (103) for each of the one or more objects (104) in the image (103) using one or more image segmentation techniques. The one or more image segmentation techniques can include at least one of a pre-trained deep learning model, a depth sensor, and a time-of-flight (TOF) sensor. The confidence map (105) represents a probability that a pixel belongs to the foreground or the background in the image (103).

[0046] ​In an embodiment, the image processing system (102) uses a transformation function to calculate a transformation value (106) for each of one or more objects (104). Here, at least one of the size of the image (103) and / or the size of each of one or more objects (104) in the image (103) can be provided as input to the transformation function. The transformation function can include at least one of a linear transformation function or a nonlinear transformation function. For example, the transformation function can be a hyperbolic tangent function (tanh), sigmoid, exponential, Gaussian, etc. The transformation value (106) is calculated to be proportional to the size of each of one or more objects (104) in the image (103). For example, for three people in the image (103), the transformation value (106) could be 14 for the first object (104A), 5 for the second object (104B), and 8 for the third object (104C).

[0047] In this embodiment, after calculating the transformation value (106), the image processing system (102) generates a value for... Figure 1 A triangulation (107) of each of one or more objects (104) in the image (103) shown. The triangulation (107) is generated from the confidence map (105) based on the distance between each pixel in image (103) and each of the one or more objects (104) in image (103) and the corresponding transformation value (106) for each of the one or more objects (104). The triangulation (107) of image (103) represents a defined background (i.e., black), a defined foreground (i.e., white), and an unknown region (i.e., gray). In an embodiment, the image processing system (102) can generate the triangulation (107) based on image (103) and one or more objects (104) in image (103) using a pre-trained second deep learning model. The second deep learning model can be pre-trained to learn: to determine the confidence map (105), to compute the transformation value (106), and to generate the triangulation (107) of image (103). In an embodiment, the image processing system (102) generates a tripartite image (107) using morphological operations and determines the kernel size of the morphological operations based on the ratio of the size of the image (103) to the size of each of one or more objects (104) in the image (103) (e.g., face proportions).

[0048] Furthermore, the tri-image (107) and the image (103) are provided as input to a first deep learning model for performing image matting on the image (103). For example, the first deep learning model can be a neural network based on an encoder-decoder architecture. The first deep learning model generates... Figure 1An output of a foreground mask (109) of the illustrated image (103). By using the foreground mask (109), the foreground and the background can be separated from each other from the image (103).

[0049] Figure 2 A detailed block diagram of the image processing system (102) according to an embodiment is illustrated.

[0050] The image processing system (102) can include a central processing unit ("CPU" or "processor") (203) and a memory (202) storing instructions executable by the processor (203). The processor (203) can include at least one data processor for executing program components to accomplish a request generated by a user or a system. The memory (202) can be communicatively coupled to the processor (203). The image processing system (102) further includes an input / output (I / O) interface (201). The I / O interface (201) can be coupled to the processor (203) through which an input signal or / and an output signal can be transmitted. In one embodiment, the image processing system (102) can receive an image (103) and a user input through the I / O interface (201).

[0051] In some implementations, the image processing system (102) can include Figure 2 Data (204) and modules (209) are illustrated. As an example, the data (204) and the modules (209) can be stored in the memory (202) of the image processing system (102). At least one module from the plurality of modules (209) can be implemented through an artificial intelligence (AI) model. The memory (202) can be a non-volatile memory, a volatile memory, and a function associated with the AI can be executed through the non-volatile memory, the volatile memory, and the processor (203).

[0052] Further, the processor (203) can include one or more processors. At this time, the one or more processors can be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), etc.), a graphics-specialized processing unit (such as a graphics processing unit (GPU), a visual processing unit (VPU)), and / or an AI-specialized processor (such as a neural processing unit (NPU)). The one or more processors control the processing of input data according to a pre-defined operation rule or an artificial intelligence (AI) model stored in a non-volatile memory and a volatile memory. The pre-defined operation rule or the artificial intelligence model is provided through training or learning. Here, the learning refers to generating a pre-defined operation rule or an artificial intelligence model having a desired characteristic by applying a learning algorithm to a plurality of learning data. The learning can be performed in a device that executes an artificial intelligence according to an embodiment, and / or can be implemented through a separate server / system.

[0053] The AI model can include a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through a computation of a previous layer and an operation of the plurality of weights. Examples of the neural network include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN), and a deep Q network. A learning algorithm is a method for training a predetermined target device (e.g., a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or a prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0054] In an embodiment, the data (204) can include, for example, object data (205), confidence map data (206), trimap data (207), and other data (208). The data (204) is described in detail herein.

[0055] In an embodiment, the object data (205) includes one or more objects (104) detected in the image (103). In an embodiment, one or more pixel values associated with each of the one or more objects (104) and coordinates of the one or more pixels can be stored in the object data (205). In another embodiment, the object data (205) can include coordinates of a bounding box associated with each of the one or more objects (104) in the image (103). For example, a shape of the bounding box can be a square, a rectangle, a circle that contains each of the one or more objects (104).

[0056] In an embodiment, the confidence map data (206) includes a confidence map (105) determined for each of the one or more objects (104) in the image (103). The confidence map (105) represents a probability that a pixel belongs to a foreground or a background in the image (103).

[0057] In an embodiment, the trimap data (207) includes a trimap (107) generated for the image (103). The trimap (107) divides the image (103) into 3 classes, i.e., explicit foreground, explicit background, and unknown region. The explicit foreground is represented by white pixels, the explicit background is represented by black pixels, and the unknown region is represented by gray pixels.

[0058] In an embodiment, the other data (208) can include a transformation value (106) associated with each of the one or more objects (104) in the image (103), a foreground mask (109) of the image (103), a first deep learning model, a second deep learning model, etc.

[0059] In some embodiments, the data (204) can be stored in the memory (202) in various data structures. Further, the data (204) can be organized using a data model, such as a relational or hierarchical data model. Other data (208) can store data, including temporary data and temporary files generated by the modules (209) to perform various functions of the image processing system (102). Further, in embodiments, the memory (202) can include at least one of a static random access memory (SRAM), a dynamic random access memory (DRAM), a global buffer, and the like.

[0060] In some embodiments, the data (204) stored in the memory (202) can be processed by the modules (209) of the image processing system (102). The modules (209) can be stored within the memory (202) that is communicatively coupled to the processor (203), or can be stored externally to the memory (202) as shown. As used herein, the term module (209) can refer to an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), an electronic circuit, a processor (203) (shared, dedicated, or group) and memory (202) that execute one or more software or firmware programs, a combinational logic circuit, and / or other suitable components that provide the described functionality. In some other embodiments, the modules (209) can be implemented using at least one of an ASIC or an FPGA. Figure 2

[0061] In one implementation, the modules (209) can include, for example, an object detection module (210), a confidence map determination module (211), a computation module (212), a trimap generation module (213), an image matting module (214), and other modules (215). It can be appreciated that these aforementioned modules (209) can represent a single module or a combination of different modules.

[0062] In embodiments, the object detection module (210) is used to detect one or more objects (104) in the image (103). The one or more objects (104) in the image (103) are detected by identifying the one or more objects (104) in the image (103) using one or more object detection techniques. For example, the one or more object detection techniques can include a machine learning based Viola-Jones object detection framework that utilizes Haar features, Scale-Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG) features, and deep learning based region proposal, single shot Refine Neural Network for object detection, single shot MultiBox Detector, and the like.

[0063] ​In an embodiment, the confidence map determination module (211) is used to determine a confidence map (105) associated with the image (103) for each of the one or more objects (104) using one or more image segmentation techniques. The confidence map determination module (211) computes the confidence map (105) for the one or more objects (104) in the image (103) based on at least one of a pre-trained deep learning model, a depth sensor, or a time-of-flight (TOF) sensor. Further, the confidence map determination module (211) modifies the confidence map (105) using a boundary correction technique based on color similarity.

[0064] In an embodiment, the computation module (212) computes a transformation value (106) for each of the one or more objects (104) using a transformation function. The computation module (212) determines the transformation value (106) for each of the one or more objects (104) using a ratio of a size of the image (103) and a size of each of the one or more objects (104) in the image (103) and a predetermined value as inputs to the transformation function. The transformation function can include at least one of a linear transformation function or a non-linear transformation function.

[0065] In an embodiment, the trimap generation module (213) is used to generate a trimap (107) for each of the one or more objects (104) in the image (103). The trimap generation module (213) computes a distance from a pixel in the image (103) to each of the one or more objects (104) in the image (103). Further, the distance is compared with the transformation value (106) corresponding to each of the one or more objects (104). Further, the trimap generation module (213) labels the pixel as an unknown pixel when the distance is less than the transformation value (106). Alternatively, the trimap generation module (213) labels the pixel as at least one of a foreground pixel or a background pixel using the confidence map (105) when the distance is greater than or equal to the transformation value (106).

[0066] In an embodiment, the image matting module (214) is used to perform image matting of the image (103). The image matting module (214) provides the image (103) and the trimap (107) of the image (103) to a pre-trained first deep learning model. Further, the image matting module (214) generates a foreground mask (109) of the image (103) based on an output of the pre-trained first deep learning model. The foreground and the background in the image (103) are separated using the foreground mask (109).

[0067] In an embodiment, the other modules (215) are used to generate the trimap (107) using a pre-trained second deep learning model.

[0068] Figure 3 A flow diagram showing a method of performing image matting of an image (103) is shown, in accordance with an embodiment.

[0069] Figure 3 The order or sequence of the method 300 shown should not be construed as a limitation, and the method 300 can be arranged in any order, and any number of the described method blocks can be combined. Furthermore, individual blocks can be deleted from the method without departing from the scope of the disclosure. Additionally, the method can be implemented in any suitable hardware, software, firmware, or any combination thereof.

[0070] In operation 301, the image processing system (102) detects one or more objects (104) in the image (103).

[0071] In an embodiment, detecting one or more objects (104) in the image (103) includes identifying one or more objects (104) in the image (103) using one or more object detection techniques. The one or more object detection techniques can include a machine learning based Viola-Jones object detection framework with Haar features, Scale-Invariant Feature Transform (SIFT), Histogram of Oriented Gradients (HOG) features, and deep learning based region proposal, single shot RefineNet for object detection, single shot MultiBox Detector, etc. The one or more objects (104) can include at least one of one or more people, one or more faces, one or more animals, one or more articles, one or more animated characters, one or more vehicles, one or more buildings, etc. For example, in Figure 4A In an embodiment, detecting one or more faces of one or more people as the one or more objects (104) in the image (103) is shown.

[0072] In operation 302, the image processing system (102) determines a confidence map (105) associated with the image (103) for each of the one or more objects (104) using one or more image segmentation techniques.

[0073] In an embodiment, the image processing system (102) determines the confidence map (105) by computing the one or more objects (104) in the image (103) based on at least one of a pre-trained deep learning model using a depth sensor or a Time of Flight (TOF) sensor. The confidence map (105) represents a probability of a pixel belonging to a foreground or a background in the image (103). In an embodiment, a user can select at least one object from the one or more objects (104) to generate the confidence map (105). Specifically, as Figure 4B As shown, a depth sensor (401) is used to generate a depth map (402) for the image (103) captured by the camera (110). As Figure 4BAs shown, using the depth map (402), the image processing system (102) generates the confidence map (105) based on the depth of the one or more objects (104) in the image (103) using the detected one or more objects (104). In another embodiment, as shown in FIG. 4B, the image processing system (102) generates the confidence map (105) based on the depth of the one or more objects (104) in the image (103) using the depth map (402) and the one or more objects (104) detected in the image (103). Figure 4C As shown, the TOF sensor (403) can be used to generate the depth map (402) for the image (103) captured by the camera (110). As shown in FIG. 4A, the image processing system (102) can use the depth map (402) to generate the confidence map (105) based on the depth of the one or more objects (104) in the image (103) using the detected one or more objects (104). Figure 4C As shown, using the depth map (402), the image processing system (102) generates the confidence map (105) based on the depth of the one or more objects (104) in the image (103) using the detected one or more objects (104). In another embodiment, as shown in FIG. 4B, the image processing system (102) generates the confidence map (105) based on the depth of the one or more objects (104) in the image (103) using the depth map (402) and the one or more objects (104) detected in the image (103). Figure 4D As shown, the image processing system (102) uses a pre-trained deep learning model to determine the confidence map (105) for the image (103). Further, as shown in FIG. 4B, the image processing system (102) uses the confidence map (105) to determine the transformation value (106) for each of the one or more objects (104) in the image (103). Figure 4E As shown, the image processing system (102) uses a color similarity based boundary correction technique to modify the confidence map (105). Further, one or more image enhancement techniques can be used to perform color similarity based boundary correction in the image (103) based on histogram, wavelet transform, gradient information, watershed based depth map, region based color averaging.

[0074] In an embodiment, after generating the confidence map (105), the image processing system (102) calculates the transformation value (106) for each of the one or more objects (104) using a transformation function, wherein at least one of the size of the image (103) and the size of each of the one or more objects (104) in the image (103) is provided as an input to the transformation function.

[0075] In an embodiment, the image processing system (102) calculates the transformation value (106) by determining the transformation value (106) for each of the one or more objects (104) using the ratio of the size of the image (103) and the size of each of the one or more objects (104) in the image (103) and a predetermined value as an input to the transformation function. In an embodiment, the predetermined value can be 31, 51, etc. However, the predetermined value is not limited to this. The image processing system (102) can receive the predetermined value as a user input. The transformation value (106) is directly proportional to the ratio of the size of the one or more objects (104) in the image (103) and the size of the image (103). For example, the transformation value (106) can be calculated using the following equation:

[0076]

[0077] For example, the transform value (106) can be a value such as 5, 12, 17, 35, etc. The transform value (106) is computed for each object from the one or more objects (104) in the image (103).

[0078] In operation 303, the image processing system (102) generates a trimap (107) for each of the one or more objects (104) in the image (103) from the confidence map (105) based on at least one of a size of each of the one or more objects (104) in the image (103) and a distance of each pixel in the image (103) from each of the one or more objects (104) in the image (103). Here, the trimap (107) is used to perform image matting.

[0079] In an embodiment, the image processing system (102) generates the trimap (107) for each of the one or more objects (104) in the image (103) by computing a distance from a pixel in the image (103) to each of the one or more objects (104) in the image (103). The distance is computed for each pixel in the image (103). The distance can be computed using Euclidean distance technique. For example, a pixel located at a center of a bounding box applied to each of the one or more objects (104) in the image (103) is used to compute a distance to each pixel outside the bounding box. Further, the image processing system (102) can compare the distance of each pixel and the transform value (106) corresponding to each of the one or more objects (104) and label each pixel as an unknown pixel when the distance between the pixel located at the center of the bounding box and the pixel outside the bounding box is less than the transform value (106). For example, if the distance between the pixel located at the center of the bounding box and the pixel outside the bounding box is 7 and the transform value (106) is 12, the pixel is labeled as an unknown pixel. The unknown pixels in the trimap (107) constitute an unknown region. Alternatively, when the distance between the pixel located at the center of the bounding box and the pixel outside the bounding box is greater than or equal to the transform value (106), the image processing system (102) labels each pixel as at least one of a foreground pixel or a background pixel using the confidence map (105). For example, if the distance between the pixel located at the center of the bounding box and the pixel outside the bounding box is 22 and the transform value (106) is 9, the pixel is labeled as one of a foreground pixel or a background pixel using the confidence map (105). The foreground pixels in the trimap (107) constitute a foreground region and the background pixels in the trimap (107) constitute a background region. Figure 4F The trimap (107) shown is generated by the image processing unit, corresponding to Figure 4E The confidence map (105) shown corresponds.

[0080] In an embodiment, asFigure 4F As shown, the image processing system (102) performs image matting of the image (103) by providing the image (103) and the tri-map (107) of the image (103) to the pre-trained first deep learning model. Further, as shown, a foreground mask (109) of the image (103) is generated using the output of the pre-trained first deep learning model, wherein the foreground region and the background region in the image (103) are separated using the foreground mask (109). The pre-trained first deep learning model can be based on an encoder-decoder model, a convolutional neural network, a recurrent neural network, and the like. Figure 4F As shown, the image processing system (102) performs image matting of the image (103) by providing the image (103) and the tri-map (107) of the image (103) to the pre-trained first deep learning model. Further, as shown, a foreground mask (109) of the image (103) is generated using the output of the pre-trained first deep learning model, wherein the foreground region and the background region in the image (103) are separated using the foreground mask (109). The pre-trained first deep learning model can be based on an encoder-decoder model, a convolutional neural network, a recurrent neural network, and the like.

[0081] Figure 5 A flowchart showing a method of performing image matting of the image (103) using the pre-trained second deep learning model, according to an embodiment, is shown. Figure 6 An exemplary generation of a tri-map and a foreground mask for an image, according to an embodiment, using the second deep learning model and the first deep learning model, is shown.

[0082] In operation 501, the image processing system (102) detects one or more objects (104) in the image (103). In the above Figure 3 The detection of the one or more objects (104) in the image (103) using one or more object detection techniques is described in detail in step 301 of the above

[0083] In operation 502, the image processing system (102) provides the image (103) and the one or more objects (104) in the image (103) to the pre-trained second deep learning model to generate a tri-map (107) for each of the one or more objects (104) in the image (103). The pre-trained second deep learning model generates the tri-map (107) based on at least one of a size of the image (103) and a size of each of the one or more objects (104) in the image (103).

[0084] In an embodiment, the second deep learning model is based on an encoder-decoder model with skip connections. The image processing system (102) trains the second deep learning model by providing one or more sample images as input to the initial deep learning model. Further, the image processing system (102) computes a loss value for each pixel in the one or more sample images using a loss function based on an output of the initial deep learning model. The output represents a label for each pixel in the image (103). The label indicates at least one of an unknown pixel, a foreground pixel, or a background pixel. The loss function can include at least one of a cross-entropy function, a mean squared error function, and the like. The loss value represents an error between the output of the initial deep learning model and an expected output. The expected output can be a foreground mask (109) associated with the one or more images determined by the user. Further, the image processing system (102) aggregates the loss values based on the label associated with each pixel in the one or more sample images. Thereafter, the image processing system (102) modifies at least one of the one or more weights associated with the initial deep learning model using a supervised learning technique based on the aggregated loss values. Based on the modified one or more weights, the second deep learning model learns to generate the trimap (107) for the image (103) based on at least one of a size of the image (103) and a size of each of the one or more objects (104) in the image (103).

[0085] In operation 503, the image processing system (102) performs image matting for the image (103) using the trimap (107) of the image (103). In the above, the generation of the trimap (107) using the pre-trained first deep learning model is described in detail with reference to step 304 of the method 300. Figure 3 The image matting generates a foreground mask (109) for the image (103), wherein the foreground regions and the background regions in the image (103) are separated using the foreground mask (109).

[0086] According to the method for performing image matting of the image (103), the image processing unit (102) can obtain output data, i.e., the foreground mask (109), the trimap (107), the confidence map (105) of the image (103) by using the image data (i.e., pixel values) as input data for the artificial intelligence model. The artificial intelligence model can be obtained by training. That is, the pre-defined operation rule or the artificial intelligence model configured to perform the desired feature (or purpose) is obtained by training the base artificial intelligence model with a plurality of training data by a training algorithm. The artificial intelligence model can include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values, and performs neural network computation by performing computation between the computation result of the previous layer and the plurality of weight values. Examples of the neural network include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN), and a deep Q-network.

[0087] The method of generating the trimap (107) to perform image matting of the image (103) includes generating an adaptive trimap (107) for each of the one or more objects (104) in the image (103) based on the size and depth of the one or more objects (104) in the image (103). The transformation values (106) computed for each object (104) from the one or more objects (104) reduce the unknown regions in the trimap (107) of the image (103). The first deep learning model produces a high-quality foreground mask (109) that can be used in a portrait mode to introduce finer details (hair strands, fur, etc.) for the focused object (104) and produce a DSLR (digital single-lens reflex camera)-like bokeh effect. In addition, the finer foreground mask (109) can be used to extract the one or more objects (104) in the image (103) and blend the one or more objects (104) naturally with different backgrounds. The generation of the foreground mask (109) does not require a person to interact with the image (103) to label the unknown regions in the trimap (107) for image matting. The foreground mask (109) is generated for a low-quality or low-resolution image captured by a mobile phone or a smart phone. The foreground mask (109) can be used for 3D (three-dimensional) modeling of the one or more objects (104) in the image (103).

[0088] Figure 7A block diagram illustrating an exemplary computer system (700) according to embodiments is shown. The computer system (700) can be used to implement the method of generating a trimap (107) to perform image matting. The computer system (700) can include a central processing unit ("CPU" or "processor") (702). The processor (702) can include at least one data processor for executing program components for monitoring physical fitness of a user. The processor (702) can include special purpose processing units such as an integrated system (bus) controller, a memory management control unit, a floating point unit, a graphics processing unit, a digital signal processing unit, etc.

[0089] The processor (702) can be communicatively coupled to one or more I / O devices via an input / output (I / O) interface (701). The I / O interface (701) can employ a communication protocol / method such as, but not limited to, audio, analog, digital, mono, RCA connector, stereo, IEEE- 1394, serial bus, Universal Serial Bus (USB), infrared, PS / 2, BNC (coaxial cable connector), coaxial, component, composite, Digital Visual Interface (DVI), High-Definition Multimedia Interface (HDMI), radio frequency (RF) antenna, S-Video (S terminal output), Video Graphics Array (VGA), IEEE 802.n / b / g / n / x, Bluetooth, cellular (e.g., Code Division Multiple Access (CDMA), High-Speed Packet Access (HSPA+), Global System for Mobile Communications (GSM), Long-Term Evolution (LTE), WiMAX, etc.), etc.

[0090] The computer system (700) can communicate with one or more I / O devices through the use of the I / O interface (701). For example, the input device (710) can be an antenna, a keyboard, a mouse, a joystick, an (infrared) remote control, a camera, a card reader, a facsimile machine, an adapter, a biometric reader, a microphone, a touch screen, a touchpad, a trackball, a stylus, a scanner, a storage device, a transceiver, a video device / source, etc. The output device (711) can be a printer, a facsimile machine, a video display (e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), a light-emitting diode (LED), a plasma, a plasma display panel (PDP), an organic light-emitting diode display (OLED), etc.), an audio speaker, etc.

[0091] In some embodiments, the computer system (700) is connected to a service provider via a communication network (709). The processor (702) may be connected to the communication network (709) via a network interface (703). The network interface (703) may communicate with the communication network (709). The network interface (703) may employ connection protocols, including but not limited to direct connection, Ethernet (e.g., twisted pair 10 / 100 / 1000Base T), Transmission Control Protocol / Internet Protocol (TCP / IP), Token Ring, IEEE 802.11a / b / g / n / x, etc. The communication network (709) may include, but is not limited to, direct interconnect, e-commerce networks, peer-to-peer (P2P) networks, local area networks (LANs), wide area networks (WANs), wireless networks (e.g., using Wireless Application Protocol), the Internet, Wi-Fi, etc. By using the network interface (703) and the communication network (709), the computer system (700) may communicate with one or more service providers.

[0092] In some embodiments, the processor (702) may be communicatively coupled to a memory (705) (e.g., random access memory (RAM), read-only memory (ROM), etc.) via a storage interface (704). Using connection protocols such as Serial Advanced Technology Attachment (SATA), Integrated Drive Electronics (IDE), IEEE-1394, Universal Serial Bus (USB), Fibre Channel, Small Computer System Interface (SCSI), etc., the storage interface (704) may be connected to the memory (705) (including but not limited to storage drives, removable disk drives, etc.). The memory drive may also include magnetic drums, disk drives, magneto-optical drives, optical drives, redundant arrays of independent disks (RAID), solid-state storage devices, solid-state drives, etc.

[0093] The memory (705) can store a collection of program or database components, including but not limited to a user interface (706), an operating system (707), a network server (708), etc. In some embodiments, the computer system (700) can store user / application data (706), such as data, variables, records, etc. Such a database can be implemented as a fault-tolerant, relational, scalable, and secure database, such as Oracle or Sybase.

[0094] An operating system (707) facilitates resource management and operation of a computer system (700). Examples of operating systems include, but are not limited to, those mentioned above. UNIX-like system distributions (e.g., BERKELEY SOFTWARE) (BSD) OPENBSD, etc. Release (e.g., etc.), ( 10, etc.), GOOGLE TM ANDROID TM , OS, etc.

[0095] In some embodiments, computer system (700) can implement program components that store a web browser. The web browser can be a hypertext browsing application, such as INTERNET GOOGLE TM CHROME TM , etc. Secure web browsing can be provided using the Hypertext Transfer Protocol Secure (HTTPS), Secure Sockets Layer (SSL), Transport Layer Security (TLS), etc. The web browser can utilize tools such as AJAX (Asynchronous JavaScript and XML), HTML (Hypertext Markup Language), application programming interfaces (APIs), etc. In some embodiments, computer system (700) can implement program components that store a mail server. The mail server can be an Internet mail server, such as Microsoft Exchange, etc. The mail server can utilize facilities such as Active Server Pages (ASP), Dynamic HTML (DHTML), etc. In some embodiments, computer system (700) can implement program components that store a mail server. The mail server can be an Internet mail server, such as Microsoft Exchange, etc. The mail server can utilize facilities such as Active Server Pages (ASP), Dynamic HTML (DHTML), etc. C++ / C#, .NET, CGI SCRIPTS, PHP, etc. The mail server can utilize communication protocols such as the Internet Message Access Protocol (IMAP), the Messaging Application Programming Interface (MAPI), Exchange, the Post Office Protocol (POP), the Simple Mail Transfer Protocol (SMTP), etc. In some embodiments, computer system (700) can implement program components that store a mail client. The mail client can be a mail browsing application, such as MAIL, etc.

[0096] Furthermore, one or more computer-readable storage media can be used to implement embodiments consistent with this disclosure. A computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor can be stored. Therefore, a computer-readable storage medium can store instructions for execution by one or more processors, including instructions for causing the processor to perform steps or stages consistent with the embodiments described herein. The term "computer-readable medium" should be understood to include tangible articles and exclude carrier waves and transient signals (i.e., non-transient). Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard disks, optical disc (CD) ROMs, digital video discs (DVDs), flash drives, magnetic disks, and any other known physical storage media.

[0097] The image (103) can be received from a remote device (712) connected to the communication network (709).

[0098] The terms “an embodiment,” “an embodiment,” “multiple embodiments,” “the embodiment,” “one or more embodiments,” “some embodiments,” and “an embodiment” refer to “one or more (but not all) embodiments of this disclosure” unless otherwise expressly stated.

[0099] The terms “including,” “comprising,” “having,” and variations thereof mean “including, but not limited to,” unless otherwise expressly stated.

[0100] The items listed do not imply that any or all of them are mutually exclusive unless otherwise expressly stated. The terms “a,” “an,” and “the” mean “one or more” unless otherwise expressly stated.

[0101] Singular expressions may include plural expressions unless the context clearly indicates otherwise.

[0102] When this document describes a single device or item, it is obvious that a single device / item can be used in place of more than one device / item (whether or not they cooperate). Similarly, in the case of this document describing more than one device or item, it is obvious that a single device / item can be used in place of more than one device or item, or that a different number of devices / items can be used in place of the number of devices or programs shown. The functionality and / or features of a device may be included in different ways by one or more other devices.

[0103] Figure 3 and Figure 5 The operation displays specific events occurring in a specific order. In some embodiments, specific operations can be executed, modified, or deleted in different orders. Furthermore, various operations can be added to the reference. Figure 3 and Figure 5The described embodiments. Moreover, the operations described herein can be processed in an sequential order, or in parallel, or in a combination thereof. Further, the operations can be processed by a single processing unit or by distributed processing units. Although the subject matter has been described in a language specific to the software art, it is

[0104] Embodiments of the present disclosure have been shown and described above, however, the embodiments of the present disclosure are not limited to the previously described specific embodiments, and it is obvious that those skilled in the art in the technical field to which the present disclosure belongs can make various modifications, replacements and improvements without departing from the scope of the present disclosure. Furthermore, it should be understood that such modifications, replacements and improvements should fall within the scope of protection of the present disclosure, and should not be independently interpreted as departing from the technical concept or expectation of the present disclosure.

Claims

1. A method of performing image matting on an image, the method comprising: detecting, by an image processing system, one or more objects in the image; determining, by the image processing system, a confidence map associated with the image for each of the one or more objects using one or more image segmentation techniques; and generating, by the image processing system, a trimap for each of the one or more objects in the image from the confidence map based on a distance between a first pixel in the image and a second pixel in at least one of the one or more objects in the image, wherein the trimap is used to perform the image matting, wherein generating the trimap for each of the one or more objects in the image further comprises: calculating the distance from the first pixel in the image that is outside of the one or more objects to the second pixel in at least one of the one or more objects in the image; comparing the distance to a transformation value corresponding to each of the one or more objects; and based on the distance being less than the transformation value, labeling the first pixel as an unknown pixel, or based on the distance being greater than or equal to the transformation value, labeling the first pixel as one of a foreground pixel or a background pixel using the confidence map, wherein the first pixel is outside of a bounding box applied to each of the one or more objects in the image and the second pixel is at a center of the bounding box. Detecting the one or more objects in the image comprises identifying the one or more objects in the image using one or more object detection techniques.

2. The method of claim 1, wherein, Determining the confidence map comprises:

3. The method of claim 1, wherein, calculating the confidence map for the one or more objects in the image based on at least one of a pre-trained deep learning model, a depth sensor, or a time-of-flight (TOF) sensor; and modifying the confidence map using a boundary correction technique based on color similarity. Performing the image matting comprises:

4. The method of claim 1, wherein, providing the image and the trimap of the image to a first pre-trained deep learning model; and generating a foreground mask of the image, wherein the foreground and the background in the image are separated using the foreground mask.

5. An image processing system for performing image matting on an image, the image processing system comprising: a memory; and a processor configured to: detect one or more objects in the image; determine a confidence map associated with the image for each of the one or more objects using one or more image segmentation techniques; and generate a trimap for each of the one or more objects in the image from the confidence map based on at least one of a size of each of the one or more objects in the image and a distance between a first pixel in the image and a second pixel in at least one of the one or more objects in the image, wherein the trimap is used to perform the image matting, wherein the processor is further configured to: calculate the distance from the first pixel in the image that is outside of the one or more objects to the second pixel in at least one of the one or more objects in the image; compare the distance to a transformation value corresponding to each of the one or more objects; and based on the distance being less than the transformation value, label the first pixel as an unknown pixel, or based on the distance being greater than or equal to the transformation value, label the first pixel as one of a foreground pixel or a background pixel using the confidence map, ​ ​ wherein the first pixel is located outside of a bounding box applied to each of the one or more objects in the image and the second pixel is located at a center of the bounding box.

6. The image processing system of claim 5, wherein, The processor is further configured to identify the one or more objects in the image using one or more object detection techniques.

7. The image processing system of claim 5, wherein, The processor is further configured to: compute a confidence map for the one or more objects in the image based on at least one of a pre-trained deep learning model, a depth sensor, or a time-of-flight (TOF) sensor; and modify the confidence map using a boundary correction technique based on color similarity.

8. The image processing system of claim 5, wherein, The processor is further configured to: provide the image and the tri-map of the image to the first pre-trained deep learning model; and generate a foreground mask of the image, wherein the foreground and the background in the image are separated using the foreground mask.

9. The method of claim 1, wherein, The second pixel in at least one of the one or more objects in the image is determined based on the bounding box.

10. The method of claim 2, wherein, The one or more object detection techniques include at least one of a bounding box technique using a machine learning based Viola-Jones object detection framework with Haar features, scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG) features, deep learning based region proposal, single shot RefineNet for object detection, or single shot MultiBox Detector.

11. The method of claim 1, wherein, Generating the tri-map for each of the one or more objects in the image includes generating the tri-map using a morphological operation, and wherein a kernel size of the morphological operation is determined based on a ratio of a size of the image to a size of each of the one or more objects in the image.

Images