Interactive Image Matting Using Neural Networks

Abstract

Embodiments of the present disclosure relate to interactive image matting using a neural network. A technique for interactive image matting based on a deep neural network (DNN) is disclosed. A method for implementing the technique according to the embodiment includes: generating, by the DNN, an alpha mask associated with an image based on a foreground region position specified by a user in the image. The method also includes applying a first DNN subnet to the image, the first subnet being trained to generate a binary mask based on user input, the binary mask designating pixels of the image as background or foreground. The method also includes applying a second DNN subnet to the generated binary mask, the second subnet being trained to generate a tripartite map based on user input, the tripartite map designating pixels of the image as background, foreground, or an uncertain state. The method also includes applying a third DNN subnet to the generated tripartite map, the tripartite map being trained to generate an alpha mask based on user input.

Description

Technical Field

[0001] The present disclosure relates to image processing. In particular, the present disclosure relates to a neural network-based interactive image matting technique. Background Art

[0002] The selection and masking of objects in an image (a process also known as image matting) is a common and important task in many photographic workflows. For example, it is often desirable to be able to select and extract a person or object from the foreground of an image, and then place the selection against a new background and / or perform other image processing tasks on the selection. Unfortunately, existing photo manipulation tools are cumbersome to use and require a great deal of skill to select objects from an image, but these tools can still produce inaccurate or unsatisfactory results. This is especially true when the background image is complex and the boundaries of the selected object include fine details such as hair or fur, which can be a common situation since many photographs feature people and animals as their subjects. Summary of the invention

[0003] According to some embodiments, a method for image matting is disclosed, the method comprising: generating an alpha mask associated with an image by a processor-based deep neural network (DNN), the generation of the mask being based on a foreground area position specified by a user in the image, wherein the generation comprises: generating a binary mask associated with the image based on the foreground area position specified by the user, the image comprising image pixels, the binary mask designating the image pixels as at least one of a background and a foreground; generating a tripartite map associated with the image based on the generated binary mask and the foreground area position specified by the user, the tripartite map designating the image pixels as at least one of a background, a foreground or an uncertain state; and generating the alpha mask based on the generated tripartite map and the foreground area position specified by the user.

[0004] According to some embodiments, a system for image matting is disclosed, the system comprising: one or more processors, the one or more processors being used to control and / or execute a deep neural network (DNN), the DNN being configured to generate an alpha mask associated with an image, the image comprising image pixels, the generation being based on a foreground area position specified by a user in the image, wherein the DNN comprises: a first subnet, the first subnet being configured to generate a binary mask associated with the image based on the foreground area position specified by the user, the binary mask designating the image pixels as at least one of a background and a foreground; a second subnet, the second subnet being configured to generate a tripartite map associated with the image based on the binary mask and the foreground area position specified by the user, the tripartite map designating the image pixels as at least one of a background, a foreground, or an uncertain state; and a third subnet, the third subnet being configured to generate the alpha mask based on the tripartite map and the foreground area position specified by the user.

[0005] According to some embodiments, a computer program product is disclosed, comprising one or more non-transitory machine-readable media encoded with instructions, which, when executed by one or more processors, cause a process to be performed for generating a tripartite map of an image, the process comprising: generating a tripartite map associated with an image by a deep neural network (DNN), the image comprising image pixels, the DNN being configured to generate the tripartite map based on a binary mask associated with the image and a user-specified foreground region location in the image, the tripartite map designating the image pixels as at least one of background, foreground, or an uncertain state, wherein training of the DNN comprises minimization of a loss function, the minimization of the loss function being based on a comparison of: (1) tripartite map pixels generated from a training image and an associated training binary mask, and (2) ground truth tripartite map pixels associated with the training image. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a top-level block diagram of an image matting system configured according to an embodiment of the present disclosure.

[0007] Figure 2 The conversion of an input image to a binary mask according to an embodiment of the present disclosure is illustrated.

[0008] Figure 3 The conversion of a binary mask to a trimap according to an embodiment of the present disclosure is illustrated.

[0009] Figure 4 The conversion of a tripartite map to an alpha matte according to an embodiment of the present disclosure is illustrated.

[0010] Figure 5 Image synthesis using alpha masks according to an embodiment of the present disclosure is illustrated.

[0011] Figure 6 Illustrated is user input for foreground / background labeling of an image according to an embodiment of the present disclosure.

[0012] Figure 7 is a block diagram of a tripartite graph generating deep neural network (DNN) configured according to an embodiment of the present disclosure.

[0013] Figure 8 is a block diagram illustrating the training of a trigraph generation DNN configured according to an embodiment of the present disclosure.

[0014] Fig. 9 is a block diagram illustrating the training of an end-to-end alpha mask generation DNN configured according to an embodiment of the present disclosure.

[0015] Fig.10 is a block diagram of a high-resolution image matting system configured in accordance with an embodiment of the present disclosure.

[0016] Fig.11 is a flowchart illustrating a method for image matting according to an embodiment of the present disclosure.

[0017] Fig.12 is a block diagram schematically illustrating a computing platform that is configured to perform any of the techniques variously described in this disclosure, as configured according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] As mentioned earlier, image matting is an important task in the photography workflow. Unfortunately, existing photo manipulation tools require a lot of skill to perform these matting tasks and are very time-consuming. In addition, these techniques are often ineffective because they may produce inaccurate or unsatisfactory results, especially when the background image is complex and the boundaries of the selected object include fine details such as hair or fur. In more detail, image matting problems, such as the desire to obtain a high-quality mask of an object, are made more difficult by the fact that it is not enough to simply mark each pixel in the image as foreground or background. This is because the color of many pixels involves a combination of colors from both the foreground and the background. This occurs especially at the edges of objects, where the light received by the camera sensor element arrives from both the object and the background behind the object. This often occurs on objects such as hair and fur, semi-transparent objects, and blurred objects.

[0019] To this end, it would be desirable to have an interactive image matting system that is capable of generating an accurate mask (also referred to as an alpha mask) of a selected object based on only a few (e.g., one to three) user-provided selection indicators (such as mouse clicks or sliding gestures over the object of interest). In addition, it would be desirable for such a system to operate efficiently on larger and relatively high-resolution images. Accordingly, the present disclosure provides a system and method based on a deep neural network (DNN) for generating an accurate alpha mask from a given image with improved computational efficiency. In some embodiments, the alpha mask includes pixels corresponding to pixels of the input image, but provides an indication of the percentage of foreground objects for each image pixel.

[0020] According to an example of such an embodiment, an alpha mask is generated for an input image based on one or more user-specified foreground area positions in the image by a DNN-based method. In more detail, the method for generating the alpha mask includes applying a first subnet of the DNN to the input image. The first subnet is trained to generate a binary mask associated with the image based on the foreground area position specified by the user. The binary mask specifies the image pixels as background or foreground. The method also includes applying a second subnet of the DNN to the generated binary mask. The second subnet is trained to generate a tripartite map associated with the image based on the binary mask and the foreground area position specified by the user. The tripartite map specifies the image pixels as background, foreground or uncertain state. The method also includes applying a third subnet of the DNN to the generated tripartite map. The third subnet is trained to generate an alpha mask based on the tripartite map and the foreground area position specified by the user. The use of the tripartite map results in improved matting performance compared to a system that directly transitions from a binary mask to an alpha mask. In some embodiments, the three subnets can be combined and trained as a single end-to-end network. In addition, according to an embodiment, a technique for improving computational efficiency when processing large and / or high-resolution images is disclosed, as will be explained in more detail below. In another embodiment, a technique for purifying foreground and background colors to minimize or prevent halo effects when compositing a selected mask object into a new background is disclosed.

[0021] Thus, the foregoing framework provides a DNN-based tool for generating alpha masks for image matting and other photo manipulation applications. The intermediate components (binary mask and triplicate map) can also be made available as input to the final product or other photo processing tools, or otherwise provided to the user. The disclosed technology provides an efficient method for generating alpha masks with improved accuracy from relatively little user input through the use of an intermediate triplicate map between a binary mask and an alpha mask. These techniques have significant advantages over existing methods that require extensive technical skills and image manipulation, but often fail to produce acceptable results. In light of this disclosure, many configurations and variations will be apparent.

[0022] System Architecture

[0023] Figure 1is a top-level block diagram of an image matting system 100 configured according to an embodiment of the present disclosure. It can be seen that the image matting system 100 includes a binary mask generation DNN subnet 120, which is configured to generate a binary mask 130 from an input image 110 based on a foreground / background label 105 input by a user. The system also includes a tripartite map generation DNN subnet 140, which is configured to generate a tripartite map 150 from the binary mask 130. The system also includes an alpha mask generation DNN subnet 160, which is configured to generate an alpha mask 170 from the tripartite map 150. In some embodiments, the alpha mask generation DNN subnet 160 can also be configured to generate a foreground / background (FG / BG) color decontamination map 180. The operation of these DNN subnets and the nature of the user input will be described in more detail below. The results of each processing stage (e.g., binary mask 130, triplicate map 150, alpha mask 170, and FG / BG color cleanup map 180) can be provided in their own right as useful end products, however, they can also be provided to image processing tools 190 as a starting point for further photographic manipulation and processing. For example, alpha mask 170 can be used to facilitate image reconstruction, where a selected foreground object is placed on a new background.

[0024] Figure 2 The conversion of an input image 110 to a binary mask 130 according to an embodiment of the present disclosure is illustrated. The input image 110 shows a dog against an out-of-focus background. The binary mask 130 shows a calculated background region 210 and a foreground region 220. In some embodiments, the foreground region pixels may be represented or labeled by a first value (e.g., 255), and the background region pixels may be represented or labeled by a second value (e.g., 0). In some other embodiments, along the edge between the foreground and the background, some pixels of the generated binary mask may take intermediate values ​​between the foreground value and the background value, however this is different from the configuration of the tri-map described below. It can be seen that the binary mask 130 does not capture fine details at the boundary between the background region and the foreground region, so any processing based solely on the binary mask may produce an unrealistic image.

[0025] Figure 3 1 illustrates the conversion of a binary mask 130 to a tri-map 150 according to an embodiment of the present disclosure. The tri-map 150 includes a new background 310 and foreground 320, as well as an area 330 designated as being in an uncertain state between the background and foreground. The uncertain state area 330 allows for more accurate generation of the alpha mask, as will be described in more detail below. In some embodiments, the uncertain state area 330 may include a range of values ​​rather than a single value. For the uncertain state area, this will be displayed as multiple shades of gray rather than a single shade, such as Figure 3A simpler example is shown in the figure.

[0026] Figure 4 The conversion of the tripartite map 150 to the alpha mask 170 according to an embodiment of the present disclosure is illustrated. As can be seen, the resulting alpha mask 170 includes subtle details of the dog's fur and whiskers, which will allow significantly improved results for downstream image processing. For example, Figure 5 An image synthesis 500 using an alpha mask 170 according to an embodiment of the present disclosure is illustrated. The dog in the foreground is placed against a new background with a realistic composition.

[0027] Figure 6 The user input for foreground / background marking of an image according to an embodiment of the present disclosure is illustrated. As described below, the interactive image matting process operates on the input provided by the user to specify the foreground (or optionally, background) area of ​​the input image 110. In this example, three mouse clicks 610 (shown as cross marks) are sufficient to indicate that the dog is a foreground object of interest. In this case, one of the mouse clicks is used to specify the dog's tongue as part of the foreground image. In other cases, only one or two mouse clicks may be sufficient to enable the disclosed technology to generate an alpha mask. Alternatively, a sliding gesture 620 indicated by an arrow (e.g., made with a mouse or a user's finger or stylus) can be used alone or in combination with one or more mouse clicks or finger taps to specify the foreground area of ​​the image. It will be understood that any other convenient method of user input can also be used to quickly and effectively specify the foreground area of ​​the image, such as tapping, flicking, pressing and holding and sliding on a touch screen.

[0028] Figure 7 According to the embodiment of the present disclosure, Figure 1 The block diagram of the trigraph generation DNN subnet 140 is shown. The trigraph generation DNN subnet 140 is shown as including a multi-layer encoder convolutional network 710, which is configured to convert the binary mask 130 (e.g., Figure 3 ) into a set of image feature vectors 720. The trigraph generation DNN subnet 140 is also shown to include a decoder deconvolution network 730, which is configured to transform the feature vector 720 into a trigraph 150 (an example of which is also shown in Figure 3 ). In some embodiments, in view of the present disclosure, the trigraph generation DNN subnet is implemented as a ResNet neural network or a VGG16 neural network, although other such commercially available neural networks may be used. Although the encoder convolution network 710 and the decoder deconvolution network 730 may adopt a standard NN architecture (which includes, for example, a convolution layer, a maximum pooling layer, an activation function, a deconvolution layer, and a depooling layer), the networks 710 and 730 are configured as follows: Figure 8Describes a training procedure to generate triplicate maps for novel use in alpha mask generation.

[0029] As previously described, in some embodiments, the uncertain state region 330 of the tripartite map may include a range of values, such as between 0 and 255, where lower values ​​indicate a greater likelihood of background and higher values ​​indicate a greater likelihood of foreground. Although the range of values ​​indicates that the region is unknown (e.g., uncertain whether it is foreground or background), these values ​​can provide an indication to the subsequent alpha mask generation DNN 160 that the region is more likely to be foreground or more likely to be background. This additional information can improve the performance of the alpha mask generation DNN 160.

[0030] Figure 8 800 is a block diagram illustrating the training of a tripartite map generating DNN subnet 140 configured according to an embodiment of the present disclosure. The DNN training module 830 is configured to perform an iterative training process on the tripartite map generating DNN subnet 140 using a training data body including training images 810. For each training image 810, a corresponding training binary mask 820 and a corresponding target output tripartite map 850 (also referred to as a tripartite map ground truth) are provided. The training binary mask 820 and the target output tripartite map 850 can be generated using any suitable method, including sophisticated hand-drawing techniques. For each training iteration, the training image 810 and the training binary mask 820 are provided to the tripartite map generating DNN subnet being trained, and the network output is compared to the target output tripartite map 850 (e.g., on a pixel basis).

[0031] The loss calculation module 840 is configured to minimize the loss function based on the comparison. In some embodiments, the loss function can be based on the sum of absolute differences or mean square error measurement between the compared pixels. The loss function represents the error in the network trigraph output and is used to update the network weights for the iteration so that the network learns to improve. The training process continues with multiple such iterations and multiple sets of training data (e.g., images 810, binary masks 820, and target outputs 850) until the network performance reaches the desired level of accuracy, at which point a trained trigraph generation DNN is obtained.

[0032] In some embodiments, the binary mask generation DNN subnet 120 and the alpha mask generation DNN subnet 160 can be similarly implemented as multi-layer encoder and decoder convolution / deconvolution networks using ResNet, VGG16 or other suitable neural networks. In view of the present disclosure, these subnets 120 and 160 can also be trained using training data and corresponding real data according to known techniques.

[0033] When alpha mask 170 is used to generate a new image composition (e.g., Figure 5 , where a foreground object is extracted from an existing background and placed on a new background), can sometimes produce undesirable halo or streaking effects, especially if the new background color is different from the old background color. To address this issue, in some embodiments, the alpha mask generation DNN subnet 160 is also configured (e.g., trained) to generate a foreground and / or background (FG / BG) color purification map 180. The foreground color purification map provides a foreground color channel (e.g., a red, green, blue, or RGB channel) associated with the pixels of the alpha mask, and the background color purification map provides a background color channel associated with the pixels of the alpha mask. The color purification map helps reduce or prevent haloing. For example, if a yellow object is adjacent to a blue object, the colors of the edge pixels covering the yellow and blue areas will blend together, resulting in a green shade. If a yellow object is moved onto a white background, the resulting green streaks around the yellow object are undesirable. The color purification map enables image processing tools to determine that for those pixels, the foreground color is actually yellow, so the green streaks can be removed. In other words, while the alpha mask provides information that for example a pixel is 30% foreground, it is also useful to know what color the foreground was, and this can be provided by the FG color cleanup map. The same principle applies to the BG color cleanup map.

[0034] Fig. 9 is a block diagram illustrating the training 900 of an end-to-end alpha mask generation DNN 920 configured according to an embodiment of the present disclosure. In some embodiments, the three DNN subnets (binary mask generation 120, triplicate map generation 140, and alpha mask generation 160) can be treated together or lumped together as a single end-to-end alpha mask generation DNN 920. As such, module 930 can be configured to perform an iterative training process on the end-to-end alpha mask generation DNN 920 based on the training images 810 and the associated target alpha mask 910 (also referred to as the alpha mask ground truth). The loss calculation module 940 is configured to minimize the loss function based on the comparison of the alpha mask ground truth data 910 with the output of the trained end-to-end alpha mask generation DNN (e.g., on a pixel basis) in each iteration. In some embodiments, the comparison can be measured using the sum of absolute differences or mean squared error.

[0035] In some embodiments, the loss calculation module 940 is configured to minimize a loss function based on a comparison of gradients (e.g., edges) between the outputs (at each training iteration) of the end-to-end alpha mask generating DNN being trained, where the gradients are in the background region of the training image 810. The motivation for this approach is that if an edge in the alpha mask generated in a training iteration aligns with an edge in the background region of the training image, it is more likely to be an error. For example, first consider the whiskers of the dog in the input image 110, which are correctly represented in the generated alpha mask 170. However, if the input image 110 is used as a training image and the alpha mask 170 is generated during the training iteration, then if there are similar edge features in the background region of the image that align with what is believed to be a whisker, then the whisker (i.e., edge / gradient feature) may indicate an error. In other words, a loss function based on gradient comparison can help reduce the chance of, for example, a tuft of hair or fur being left behind against a blue sky background.

[0036] In some embodiments, the end-to-end alpha mask generation DNN is implemented as a ResNet neural network or a VGG16 neural network, although other suitable networks may be used.

[0037] Fig.10 1 is a block diagram of a high-resolution image matting system 1000 configured according to an embodiment of the present disclosure. The high-resolution image matting system 1000 is shown to include a downsampling module 1015, a binary mask generation DNN subnet 1030, a tripartite map generation DNN subnet 140, an alpha mask coefficient generation DNN subnet 1050, a coefficient upsampling module 1060, and a high-resolution alpha mask generation module 1070. The high-resolution image matting system 1000 is configured to reduce computational workload and improve performance efficiency by operating on a downsampled, reduced-resolution version of an image and then reconstructing a high-resolution version of the resulting alpha mask, as described below.

[0038] The downsampling module 1015 is configured to downsample the input image 1010 from a first relatively high resolution to a second lower resolution image 1020. The binary mask generation DNN subnet 120 is configured to generate a low resolution binary mask 1030 from the lower resolution input image 1020 based on the foreground / background label 105 input by the user.

[0039] The tripartite map generation DNN subnet 140 is configured to generate a low-resolution tripartite map 1040 from a lower-resolution binary mask 1030 .

[0040] The alpha mask coefficient generation DNN subnet 1050 is configured to generate coefficients representing an alpha mask associated with the lower resolution image 1020. The coefficients are affine parameters that can be used to linearly combine or map the colors of the image to generate an alpha mask. The affine parameters have the following properties: in a transformation based on a linear combination, using these parameters as coefficients, points, lines, and planes are preserved.

[0041] The coefficient upsampling module 1060 is configured to upsample the alpha mask coefficients 1055 from the second lower resolution back to the first higher resolution.

[0042] The high-resolution alpha mask generation module 1070 is configured to use the affine parameter coefficients of the upsampled alpha mask coefficients 1065 to generate a high-resolution alpha mask 1075 as a linear combination of the colors of the high-resolution input image 1010. In some embodiments, the linear combination can be represented by the following equation:

[0043] alpha(i)=R(i)a1(i)+G(i)a2(i)+B(i)a3(i)+a4(i)

[0044] Where alpha(i) is the calculated value of the high-resolution alpha mask at pixel i, R(i) is the value of the red channel of the high-resolution input image 1010 at pixel i, G(i) is the value of the green channel of the high-resolution input image at pixel i, B(i) is the value of the blue channel of the high-resolution input image at pixel i, and a1(i)-a4(i) are the affine parameter coefficients associated with pixel i.

[0045] In some embodiments, more complex linear combinations may be used, such as, for example:

[0046] alpha(i)=R(i) 2 a1(i)+G(i) 2 a2(i)+B(i) 2 a3(i)+R(i)G(i)a4(i)+B(i)G(i)a5(i)+a6(i)

[0047] Where a1(i)-a6(i) are the affine parameter coefficients associated with pixel i. In other embodiments, the affine parameters may be represented as a 3×3 matrix to perform spatial convolution (e.g., to selectively activate selected edge structures in a high-resolution image based on appearance).

[0048] method

[0049] Fig.111100 is a flowchart illustrating a method for interactive image matting based on DNN according to an embodiment of the present disclosure. It can be seen that the method is based on Figure 1 The configuration of the binary mask generation DNN 120, the trigraph generation DNN 120, and the alpha mask generation DNN 160 is described. However, in view of the present disclosure, it will be understood that any number of module configurations can be used to implement the method. It is also noted that the various functions depicted in the method do not need to be assigned to the specific example modules shown. To this end, the depicted example method is provided to give an example embodiment and is not intended to limit the method to any particular physical or structural configuration.

[0050] The method begins at operation 1110 by receiving an image to be processed. The image may be provided from a camera on the platform, a database, or any desired source. In addition, as previously described, one or more foreground area locations within the image are provided by the user, for example, by a mouse click or a sliding gesture.

[0051] At operation 1120, the method continues by applying a first subnet of a DNN to the image, the first subnet being trained to generate a binary mask associated with the image based on the user-specified foreground region locations. The binary mask specifies pixels of the image as belonging to the background or the foreground.

[0052] At operation 1130, a second subnet of the DNN is applied to the generated binary mask, the second subnet being trained to generate a tripartite map associated with the image based on the user-specified foreground region locations. The tripartite map designates image pixels as one of three categories: background, foreground, or an uncertain state.

[0053] At operation 1140, a third subnet of the DNN is applied to the generated tripartite map. The third subnet is trained to generate an alpha mask based on the foreground region locations specified by the user. The alpha mask includes pixels corresponding to pixels of the image, but provides an indication of the foreground percentage for each image pixel.

[0054] In some embodiments, additional operations are performed. For example, in one embodiment, to improve efficiency, the image can be downsampled to a reduced resolution, and the DNN subnet can be trained to generate alpha mask affine parameter coefficients at the reduced resolution. The coefficients can then be used to calculate a linear combination of the colors of the original higher resolution image to generate a higher resolution alpha mask. In addition, in some embodiments, the DNN is further trained to generate a foreground color cleanup map and / or a background color cleanup map. The foreground color cleanup map provides a foreground color channel associated with pixels of the alpha mask, while the background color cleanup map provides a background color channel associated with pixels of the alpha mask.

[0055] Example Platform

[0056] Fig.12 is a block diagram schematically illustrating a computing platform 1200 that is configured to perform any of the techniques as variously described in the present disclosure, configured according to embodiments of the present disclosure. For example, in some embodiments, Figure 1 Image cutout system 100, Figure 7 The three-part graph generation DNN 140, Figure 8 DNN training module 830, Fig. 9 End-to-end image matting system 900, Fig.10 The high-resolution image matting system 1000 or any part thereof, and Fig.11 The method or any part thereof is implemented in a computing platform 1200. In some embodiments, the computing platform 1200 is a computer system, such as a workstation, a desktop computer, a server, a laptop, a handheld computer, a tablet computer (e.g., an iPad tablet computer), a mobile computing or communication device (e.g., an iPhone mobile communication device, an Android mobile communication device, etc.), or other forms of computing or telecommunications devices that are capable of communicating and have sufficient processor power and memory capacity to perform the operations described in the present disclosure. In some embodiments, a distributed computing system is provided that includes a plurality of such computing devices.

[0057] The computing platform 1200 includes one or more storage devices 1290 and / or non-transitory computer-readable media 1230, on which one or more computer-executable instructions or software have been encoded for implementing the various described techniques in the present disclosure. In some embodiments, the storage device 1290 includes a computer system memory or random access memory, such as a persistent disk storage device (e.g., any suitable optical or magnetic persistent storage device, including RAM, ROM, flash memory, USB drive or other semiconductor-based storage medium), a hard drive, a CD-ROM or other computer-readable medium, for storing data and computer-readable instructions and / or software for implementing the various embodiments taught in the present disclosure. In some embodiments, the storage device 1290 also includes other types of memory or a combination thereof. In one embodiment, the storage device 1290 is provided on the computing platform 1200. In another embodiment, the storage device 1290 is provided separately from the computing platform 1200 or remotely. The non-transitory computer-readable medium 1230 includes, but is not limited to, one or more types of hardware memory, non-transitory tangible media (e.g., one or more magnetic storage disks, one or more optical disks, one or more USB flash drives), etc. In some embodiments, a non-transitory computer-readable medium 1230 included in the computing platform 1200 stores computer-readable and computer-executable instructions or software for implementing various embodiments. In one embodiment, the computer-readable medium 1230 is provided on the computing platform 1200. In another embodiment, the computer-readable medium 1230 is provided separately from or remotely from the computing platform 1200.

[0058] The computing platform 1200 also includes at least one processor 1210 for executing computer-readable and computer-executable instructions or software stored in the storage device 1290 and / or the non-transitory computer-readable medium 1230 and other programs for controlling the system hardware. In some embodiments, virtualization is employed in the computing platform 1200 to dynamically share infrastructure and resources in the computing platform 1200. For example, a virtual machine is provided to handle a process running on multiple processors so that the process appears to be using only one computing resource instead of using multiple computing resources. In some embodiments, multiple virtual machines are used with one processor.

[0059] It can also be seen that a bus or interconnect 1205 is also provided to allow communication between the various components listed above and / or other components not shown. The computing platform 1200 can be coupled to a network 1250 (e.g., a local area network or wide area network such as the Internet) through network interface circuitry 1240 to allow communication with other computing devices, platforms, resources, clients, and Internet of Things (IoT) devices.

[0060] In some embodiments, a user interacts with the computing platform 1200 through an input / output system 1260, which interfaces with devices such as a keyboard and mouse 1270, a camera 1275, and / or a display element (screen / monitor) 1280. The keyboard and mouse can be configured to provide a user interface to accept user input and guidance, such as to indicate the foreground and background areas of an image and to otherwise control an image processing application / tool. The camera can be configured to, for example, provide an image for processing using the disclosed techniques. The display element can be configured to, for example, display an image before, during, and after processing using the disclosed techniques. In some embodiments, the computing platform 1200 includes other I / O devices (not shown) for receiving input from the user, such as a pointing device or touchpad, etc., or any suitable user interface. In some embodiments, the computing platform 1200 includes other suitable conventional I / O peripherals. The computing platform 1200 can include and / or be operably coupled to various suitable devices for performing one or more aspects as described in various aspects of the present disclosure.

[0061] In some embodiments, computing platform 1200 runs an operating system (OS) 1220, such as any version of the Microsoft Windows operating system, different versions of the Unix and Linux operating systems, any version of MacOS for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, any operating system for mobile computing devices, or any other operating system capable of running on computing platform 1200 and performing the operations described in the present disclosure. In one embodiment, the operating system runs on one or more cloud machine instances.

[0062] As will be understood in light of this disclosure, Figure 1 and Figure 7-10As shown, the various modules and components of the system can be implemented in software, such as a set of instructions (e.g., HTML, XML, C, C++, object-oriented C, JavaScript, Java, BASIC, etc.) encoded on any computer-readable medium or computer program product (e.g., a hard drive, server, disk, or other suitable non-transient memory or a set of memories), which, when executed by one or more processors, causes the various methods provided in the present disclosure to be performed. It will be understood that in some embodiments, as described in the present disclosure, the various functions and data transformations performed by the computing system can be performed by similar processors with different configurations and arrangements, and the depicted embodiments are not intended to be limiting. The various components of the example embodiment, including the computing platform 1200, can be integrated into, for example, one or more desktop or laptop computers, workstations, tablet computers, smart phones, game consoles, set-top boxes, or other such computing devices. Other components and modules typical of computing systems, such as coprocessors, processing cores, graphics processing units, touch pads, touch screens, etc., are not shown but will become apparent.

[0063] In other embodiments, the functional components / modules are implemented with hardware such as gate-level logic (e.g., FPGA) or application-specific semiconductors (e.g., ASIC). Other embodiments are implemented with a microcontroller having multiple input / output ports for receiving and outputting data and multiple embedded routines for performing the functionality described in the present disclosure. In a more general sense, it will be apparent that any suitable combination of hardware, software, and firmware may be used.

[0064] Other Example Embodiments

[0065] Many example embodiments will be apparent, and the features described herein may be combined in any number of configurations.

[0066] Example 1 includes a method for image matting, the method comprising: generating an alpha mask associated with an image by a processor-based deep neural network (DNN), the generation of the mask being based on a foreground area position specified by a user in the image, wherein the generation comprises: generating a binary mask associated with the image based on the foreground area position specified by the user, the image comprising image pixels, the binary mask designating the image pixels as at least one of a background and a foreground; generating a tripartite map associated with the image based on the generated binary mask and the foreground area position specified by the user, the tripartite map designating the image pixels as at least one of a background, a foreground, or an uncertain state; and generating the alpha mask based on the generated tripartite map and the foreground area position specified by the user.

[0067] Example 2 includes the subject matter of Example 1, wherein the alpha mask comprises alpha mask pixels that correspond to image pixels and provide an indication of a foreground percentage for the corresponding image pixels.

[0068] Example 3 includes the subject matter of Example 1 or 2, wherein the training of the DNN comprises: minimization of a loss function based on a comparison of alpha mask pixels generated from a training image with ground truth alpha mask pixels associated with the training image.

[0069] Example 4 includes the subject matter of any of Examples 1-3, wherein the training of the DNN comprises: minimization of a loss function, wherein the minimization of the loss function is based on a comparison of a gradient of an alpha mask generated from a training image with a gradient in a background region of the training image.

[0070] Example 5 includes the subject matter of any of Examples 1-4, wherein the alpha mask is a first alpha mask, and the method further includes: before generating the binary mask, performing downsampling of the image from a first resolution to a second resolution, wherein the first resolution is higher than the second resolution; performing upsampling of the first alpha mask from the second resolution to the first resolution, wherein the first alpha mask includes affine parameter coefficients associated with the image pixels; and generating a second alpha mask as a linear combination of the colors of the image at the first resolution, the linear combination using the affine parameter coefficients of the first alpha mask.

[0071] Example 6 includes the subject matter of any of Examples 1-5, wherein the DNN is further configured to generate a foreground color cleanup map and / or a background color cleanup map, the foreground color cleanup map providing a foreground color channel associated with the alpha mask pixels, and the background color cleanup map providing a background color channel associated with the alpha mask pixels.

[0072] Example 7 includes the subject matter of any of Examples 1-6, wherein the user-specified foreground area position is specified by a mouse-based input or a touch screen-based input.

[0073] Example 8 includes the subject matter of any of Examples 1-7, wherein the DNN is implemented as a ResNet neural network or a VGG16 neural network.

[0074] Example 9 includes a system for image matting, the system comprising: one or more processors, the one or more processors being used to control and / or execute a deep neural network (DNN), the DNN being configured to generate an alpha mask associated with an image, the image including image pixels, the generation being based on a user-specified foreground region position in the image, wherein the DNN comprises: a first subnet, the first subnet being configured to generate a binary mask associated with the image based on the user-specified foreground region position, the binary mask designating the image pixels as at least one of background and foreground; a second subnet, the second subnet being configured to generate a tripartite map associated with the image based on the binary mask and the user-specified foreground region position, the tripartite map designating the image pixels as at least one of background, foreground, or an uncertain state; and a third subnet, the third subnet being configured to generate the alpha mask based on the tripartite map and the user-specified foreground region position.

[0075] Example 10 includes the subject matter of Example 9, wherein the alpha mask comprises alpha mask pixels that correspond to image pixels and provide an indication of a foreground percentage for the corresponding image pixels.

[0076] Example 11 includes the subject matter of Example 9 or 10, wherein the training of the DNN comprises: minimization of a loss function based on a comparison of alpha mask pixels generated from a training image with ground truth alpha mask pixels associated with the training image.

[0077] Example 12 includes the subject matter of any of Examples 9-11, wherein the training of the DNN comprises: minimization of a loss function, wherein the minimization of the loss function is based on a comparison of a gradient of an alpha mask generated from a training image with a gradient in a background region of the training image.

[0078] Example 13 includes the subject matter of any of Examples 9-12, wherein the alpha mask is a first alpha mask, and the DNN further includes: one or more processors, further configured to control and / or execute a downsampling module to downsample the image from a first resolution to a second resolution prior to operation of the first subnet, wherein the first resolution is higher than the second resolution; one or more processors, further configured to control and / or execute a coefficient upsampling module to upsample the first alpha mask from the second resolution to the first resolution, wherein the first alpha mask includes affine parameter coefficients associated with the image pixels; and one or more processors, further configured to control and / or execute a high-resolution alpha mask generation module to generate a second alpha mask as a linear combination of the colors of the image at the first resolution, the linear combination adopting the affine parameter coefficients of the first alpha mask.

[0079] Example 14 includes the subject matter of any of Examples 9-13, wherein the DNN is further configured to generate a foreground color cleanup map and / or a background color cleanup map, the foreground color cleanup map providing a foreground color channel associated with the alpha mask pixels, and the background color cleanup map providing a background color channel associated with the alpha mask pixels.

[0080] Example 15 includes the subject matter of any of Examples 9-14, wherein the user-specified foreground area location is specified by a mouse-based input or a touch-screen-based input.

[0081] Example 16 includes the subject matter of any of Examples 9-15, wherein the DNN is implemented as a ResNet neural network or a VGG16 neural network.

[0082] Example 17 includes a computer program product comprising one or more non-transitory machine-readable media encoded with instructions that, when executed by one or more processors, cause a process to be performed for generating a tripartite map of an image, the process comprising: generating, by a deep neural network (DNN), a tripartite map associated with an image, the image comprising image pixels, the DNN being configured to generate the tripartite map based on a binary mask associated with the image and a user-specified foreground region location in the image, the tripartite map designating the image pixels as at least one of background, foreground, or an uncertain state, wherein training of the DNN comprises minimization of a loss function, the minimization of the loss function being based on a comparison of: (1) tripartite map pixels generated from a training image and an associated training binary mask, and (2) ground truth tripartite map pixels associated with the training image.

[0083] Example 18 includes the subject matter of Example 17, wherein the DNN comprises: an encoder convolutional network and a decoder deconvolutional network, wherein the encoder convolutional network is used to transform a binary mask into a set of image features; and the decoder deconvolutional network is used to transform a set of image features into a tripartite map.

[0084] Example 19 includes the subject matter of Example 17 or 18, wherein the DNN is implemented as a ResNet neural network or a VGG16 neural network.

[0085] Example 20 includes the subject matter of any of Examples 17-19, wherein the user-specified foreground area position is specified by a mouse-based input or a touch-screen-based input.

[0086] The foregoing description of example embodiments of the present disclosure has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. In view of the present disclosure, many modifications and variations are possible. It is intended that the scope of the present disclosure is not limited by this detailed description, but by the appended claims.

Claims

1. A method for image matting, the method comprising: Generating, by a processor-based deep neural network (DNN), an alpha mask associated with an image, the generating being based on a user-specified foreground region location in the image, wherein the generating comprises: generating a binary mask associated with the image based on the user-specified foreground region location, the image comprising image pixels, the binary mask designating the image pixels as at least one of background and foreground; generating a tripartite map associated with the image based on the generated binary mask and the user-specified foreground region locations, the tripartite map designating the image pixels as at least one of background, foreground, or an indeterminate state; and generating the alpha mask based on the generated tri-map and the foreground area position specified by the user; and The processor-based deep neural network DNN includes: an encoder convolutional network and a decoder deconvolutional network, wherein the encoder convolutional network is used to transform the binary mask into a set of image features, and the decoder deconvolutional network is used to transform the set of image features into the tripartite map.

2. The method of claim 1, wherein the alpha mask comprises alpha mask pixels that correspond to the image pixels and provide an indication of a foreground percentage for the corresponding image pixels.

3. The method according to claim 1, wherein the training of the DNN comprises: Minimization of a loss function based on a comparison of alpha mask pixels generated from a training image with ground truth alpha mask pixels associated with the training image.

4. The method according to claim 1, wherein the training of the DNN comprises: Minimization of a loss function based on a comparison of gradients of an alpha mask generated from a training image with gradients in a background region of the training image.

5. The method of claim 1, wherein the alpha mask is a first alpha mask, the method further comprising: Prior to generating the binary mask, performing downsampling of the image from a first resolution to a second resolution, wherein the first resolution is higher than the second resolution; performing upsampling of the first alpha mask from the second resolution to the first resolution, wherein the first alpha mask comprises affine parameter coefficients associated with the image pixels; as well as A second alpha mask is generated as a linear combination of colors of the image at the first resolution, the linear combination using the affine parameter coefficients of the first alpha mask.

6. The method of claim 1 , wherein the DNN is further configured to generate a foreground color cleanup map and / or a background color cleanup map, wherein the foreground color cleanup map provides a foreground color channel associated with pixels of the alpha mask, and the background color cleanup map provides a background color channel associated with pixels of the alpha mask.

7. The method of claim 1, wherein the user-specified foreground area position is specified by a mouse-based input or a touch screen-based input.

8. The method of claim 1, wherein the DNN is implemented as a ResNet neural network or a VGG16 neural network.

9. A system for image matting, the system comprising: One or more processors, the one or more processors being configured to control and / or execute a deep neural network (DNN), the DNN being configured to generate an alpha mask associated with an image, the image comprising image pixels, the generation being based on a user-specified foreground region position in the image, wherein the DNN comprises: a first subnet configured to generate a binary mask associated with the image based on the user-specified foreground region locations, the binary mask designating pixels of the image as at least one of background and foreground; a second subnet configured to generate a tripartite map associated with the image based on the binary mask and the user-specified foreground region locations, the tripartite map designating the image pixels as at least one of background, foreground, or an indeterminate state; and A third subnet is configured to generate the alpha mask based on the tripartite map and the foreground region position specified by the user; and The DNN is further configured to generate a foreground color cleanup map and / or a background color cleanup map, wherein the foreground color cleanup map provides a foreground color channel associated with the pixels of the alpha mask, and the background color cleanup map provides a background color channel associated with the pixels of the alpha mask.

10. The system of claim 9, wherein the alpha mask comprises alpha mask pixels that correspond to the image pixels and provide an indication of a foreground percentage for the corresponding image pixels.

11. The system of claim 9, wherein the training of the DNN comprises: Minimization of a loss function based on a comparison of alpha mask pixels generated from a training image with ground truth alpha mask pixels associated with the training image.

12. The system of claim 9, wherein the training of the DNN comprises: Minimization of a loss function based on a comparison of gradients of an alpha mask generated from a training image with gradients in a background region of the training image.

13. The system of claim 9, wherein the alpha mask is a first alpha mask, and the DNN further comprises: The one or more processors are further configured to, prior to the operation of the first subnet, control and / or execute a downsampling module to downsample the image from a first resolution to a second resolution, wherein the first resolution is higher than the second resolution; The one or more processors are further configured to control and / or execute a coefficient upsampling module to upsample the first alpha mask from the second resolution to the first resolution, wherein the first alpha mask comprises affine parameter coefficients associated with the image pixels; as well as The one or more processors are further configured to control and / or execute a high-resolution alpha mask generation module to generate a second alpha mask as a linear combination of colors of the image at the first resolution, the linear combination using the affine parameter coefficients of the first alpha mask.

14. The system of claim 9, wherein the user-specified foreground area position is specified by a mouse-based input or a touch screen-based input.

15. The system of claim 9, wherein the DNN is implemented as a ResNet neural network or a VGG16 neural network.

16. A computer program product comprising one or more non-transitory machine-readable media encoded with instructions that, when executed by one or more processors, cause a process to be performed for generating a tripartite map of an image, the process comprising: generating, by a deep neural network (DNN), a tripartite map associated with an image, the image comprising image pixels, the DNN being configured to generate the tripartite map based on a binary mask associated with the image and a user-specified foreground region location in the image, the tripartite map designating the image pixels as at least one of background, foreground, or an uncertain state, wherein training of the DNN comprises minimization of a loss function based on a comparison of: (1) triplicated pixels generated from a training image and an associated training binary mask, and (2) ground truth triplicated pixels associated with the training image, and The DNN comprises: an encoder convolutional network and a decoder deconvolutional network, wherein the encoder convolutional network is used to transform the binary mask into a set of image features, and the decoder deconvolutional network is used to transform the set of image features into the tripartite map.

17. The computer program product of claim 16, wherein the DNN is implemented as a ResNet neural network or a VGG16 neural network.

18. The computer program product of claim 16, wherein the user-specified foreground area position is specified by a mouse-based input or a touch screen-based input.

Images