Closed form method and system for matting a foreground object in an image having a background

Abstract

In a method and system for matting a foreground object F having an opacity α constrained by associating a characteristic with selected pixels in an image having a background B, weights are determined for all edges of neighboring pixels for the image and used to build a Laplacian matrix L. The equation α is solved where α=arg min αT Lα s.t.αi=si, ∀i ∈ S, S is the group of selected pixels, and si is the value indicated by the associated characteristic. The equation Ii=αiFi+(1−αi)Bi is solved for F and B with additional smoothness assumptions on F and B; after which the foreground object F may be composited on a selected background B′ that may be the original background B or may be a different background, thus allowing foreground features to be extracted from the original image and copied to a different background.

Description

RELATED APPLICATONS [0001] This application claims benefit of provisional applications Ser. Nos. 60 / 699,503 filed Jul. 15, 2005 and 60 / 714,265 filed Sep. 7, 2005 whose contents are included herein by reference.FIELD OF THE INVENTION [0002] This invention relates to image matting. REFERENCES [0003] The description refers to the following prior art references, whose contents are incorporated herein by reference. [0004] [1] N. E. Apostoloff and A. W. Fitzgibbon. “Bayesian video matting using learnt image priors” In Proc. CVPR, 2004. [0005] [2] U.S. Pat. No. 6,134,345 A. Berman, P. Vlahos, and A. Dadourian. “Comprehensive method for removing from an image the background surrounding a selected object”, 2000 [0006] [3] Y. Boykov and M. P. Jolly “Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images” In Proc. ICCV, 2001. [0007] [4] Y. Chuang, A. Agarwala, B. Curless, D. Salesin, and R. Szeliski “Video matting of complex scenes”, ACM Trans. Graph., 21(3)...

AI Technical Summary

Benefits of technology

[0047] Furthermore, the closed-form formulation as provided in accordance with the invention enables one to understand and predict the properties of the solution by examining the eigenvectors of a sparse matrix, closely related to matrices used in spectral image segmentation algorithms. In addition to providing a solid theoretical basis for our approach, such analysis can provide useful hints to the user regarding where in the image scribbles should be placed.

[0049] Thus, rather than specifying a trimap, in accordance with the invention, the user scribbles constraints on the opacity of certain pixels. The constraints can be of the form “these pixels are foreground”, “these pixels are background” or the user can give direct constraints on the mixed pixels. The algorithm then propagates these constraints to the full image sequence based on a simple cost function—that nearby pixels in space-time with similar colors should have a similar opacity. The invention shows how to minimize the cost using simple numerical linear algebra, and display high quality mattes from natural images and image sequences.

Problems solved by technology

From a computer vision perspective, this task is extremely challenging because it is massively ill-posed—at each pixel we must estimate the foreground and the background colors, as well as the foreground opacity (“alpha matte”) from a single color measurement.

While mixed pixels may represent a small fraction of the image, human observers are remarkably sensitive to their appearance, and even small artifacts could cause the composite to look fake.

Obviously, this is a severely under-constrained problem, and user interaction is required to extract a good matte.

As a consequence, trimap-based approaches typically experience difficulty handling images with a significant portion of mixed pixels or when the foreground object has many holes [15].

Another problem with the trimap interface is that the user cannot directly influence the matte in the most important part of the image: the mixed pixels.

The requirement of a hand-drawn segmentation becomes far more limiting when one considers image sequences.

While good results have been obtained by intelligent use of optical flow [4], the amount of interaction obviously grows quite rapidly with the number of frames.

Another problem with the trimap interface is that the user cannot directly influence the matte in the most important part of the image: the mixed pixels.

As demonstrated in FIG. 2(b) a sparse set of constraints could lead to a completely erroneous matte.

Although Rother [11] do perform border matting by fitting a parametric alpha profile in a narrow strip around the hard boundary, this is more akin to feathering than to full alpha matting, since wide fuzzy regions cannot be handled in this manner.

While this approach has produced some impressive results, it has the disadvantage of employing an expensive iterative non-linear optimization process, which might converge to different local minima.

Finally, the cost of this method is quite high: 15-20 minutes for a 640 by 480 image.

Images