TECHNIQUES FOR FAST STEREO RECONSTRUCTION FROM IMAGES

Abstract

Computer-implemented procedure comprising the following steps: Performing a stereo alignment on a pair of images; Straightening (204) the image pair so that epipolar lines become either horizontal or vertical; Applying (206) a stereo alignment to the straightened image pair to establish an initial correspondence between pixels from the image pair; Generating a homography matrix transformation; Defining photo consistency between a template window in a base image of the image pair and a warp window of a translated image of the image pair as an implicit function of the homography matrix transformation; Improve the photo consistency of the warp window; Generating a translated pixel from a basic pixel, wherein the generation comprises repeatedly applying (208) the homography matrix transformation to the basic pixel until the absolute value of an increment step is less than a predetermined value; Triangulating (210) correspondence points to create a three-dimensional scene; and Providing the three-dimensional scene for display.

Description

Area

[0001] The subject revealed here generally refers to stereo reconstruction using images from multiple viewpoints. State of the art

[0002] The application of graphics and visual computing is growing in areas such as three-dimensional (3D) games, virtual worlds, mirrored worlds (e.g., Google Earth), and immersive user interfaces. Stereo reconstruction aims to reconstruct compact and / or densely packed 3D scenes from images taken by two or more separately positioned cameras, or, conversely, from images taken by the same camera but from different viewpoints. In stereo reconstruction, the camera's (internal and external) parameters are known through camera calibration. Traditional stereo reconstruction methods are severely limited by either accuracy or processing speed. Consequently, there is a growing demand for fast and accurate stereo reconstruction.

[0003] State of the art is described, for example, in the documents by Pollefeys, M et al., VIDEO-TO-3D, ISPRS Archives, 2002, Vol. XXXIV Part 3 A, pp. 252-257, available on the internet at the URL http: / / www.isprs.org / proceedings / XXXIV / part3 / , Baker, S. et al., Equivalence and Efficiency of Image Alignment Algorithms, Proc. of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, pp. 1-1090 - 1-1097 and Micusik, B. et al., Multi-view superpixel stereo in man-made environments, Technical Report GMU-CS-TR-2008-1, George Mason University, USA, 2008, available on the Internet at the URL http: / / cs.gmu.edu / ~kosecka / Publications / Micusik-Kosecka-GMUTechRep08.pdf.

[0004] The present invention is defined in the attached claims. Brief description of the drawings

[0005] Embodiments of the present invention are shown by way of example and without limitation in the drawings, in which the same reference numerals refer to similar elements. Fig. Figure 1 represents a patch projection onto a stereo image pair. Fig. 2 describes a method for the stereo reconstruction of a stereo image pair in accordance with one embodiment. Fig. Figure 3 describes a method for inverse compositional image alignment (ICIA) in accordance with one embodiment. Fig. Section 4 describes a method that can be used for multi-view stereo reconstruction in accordance with one embodiment. Fig. 5A and Fig. Figure 5B shows a comparison of the results of a stereo reconstruction. Fig.Section 6 describes an embodiment of a computer system in accordance with an embodiment. Detailed description

[0006] References in this description to "(exactly) one embodiment" or "an embodiment" mean that a specific feature, structure, or property described in connection with the embodiment is included in at least one embodiment of the present invention. Therefore, occurrences of the phrase "in (exactly) one embodiment" or "an embodiment" at various points in this description do not necessarily all refer to the same embodiment. Furthermore, the specific features, structures, or properties may be combined in one or more embodiments.

[0007] Two known existing solutions for stereo reconstruction include (1) stereo reconstruction using stereo matching (e.g., normalized cross-correlation (NCC)) and (2) stereo reconstruction based on color or photo consistency optimization. An example of solution (2) is described in Y. Furukawa and J. Ponce, “Accurate, Dense, and Robust Multi-View Stereopsis”, CVPR (2007) (hereinafter “Furukawa’s method”).

[0008] Photoconsistency measures the similarity of correspondence points in a stereo image pair. Photoconsistency can be defined via regions near corresponding points as: ∑x∈Af(IL(x)−IR(x)), where A is a region centered at a corresponding point, I L and I R refer to a left (original) and right (translated) image, f() is a metric function that f(x) = ||x|| 2 may be x is a pixel position and I(x) refers to the gray value at point x.

[0009] Fig. Figure 1 represents a patch projection onto a stereo image pair. The following is a description of the terms from Fig. 1 listed: C1 and C2: Cameras O1 and O2: Camera center point of two cameras (the intersection of the axes) b: Baseline between two cameras I1 and I2: Images captured by C1 and C2 respectively. P: 3D point P1, P2: Projections of 3D point P onto the image plane E1, E2: Epipole line passing through P1 and P2 π: Tangential plane of the 3D model surface at point P n: the normal vector (direction) at P of the tangent plane H: Homography transformation induced by the plane π x=P1 T(x): Template window in the basic image W(x,p): Homography warp window in the translated image

[0010] Furukawa's method projects a patch around a 3D point P into a stereo image pair and computes a photoconsistency metric of corresponding projection points p1 and p2. However, the position P is not precise, and Furukawa's method assumes that the position can be varied along a direction or within a cube near position P. For each position, a photoconsistency dimension exists, and the position with the largest photoconsistency dimension can be the optimized result. The computational speed used by Furukawa's method can be unacceptable due to backprojection of 3D patches into image planes, and its accuracy suffers at the sampling rate near the original position.

[0011] Several implementations provide photoconsistency optimization based on a traditional NCC-based stereo matching method to improve the accuracy of stereo matching and stereo reconstruction. Various implementations employing homography matching convert the photoconsistency measure into an implicit function of pixel coordinates and derive analytical gradients. The optimization is based on homography matching between image planes, which has an analytical gradient and can be solved efficiently.

[0012] In comparison to the NCC-alignment-based method, a photoconsistency optimization from various embodiments that uses homography matching can provide much more accurate stereo reconstruction results through correspondence optimization.

[0013] Various color or photoconsistency optimization methods (e.g., Furukawa's method) use backprojection of 3D patches onto an image plane. However, backprojection of 3D patches onto an image plane is computationally intensive and time-consuming. Accordingly, several other approaches offer reduced computation time, at least compared to Furukawa's method.

[0014] Fig.Section 2 describes a method for the stereo reconstruction of stereo image pairs according to one embodiment. Block 202 includes receiving an input stereo image pair. The stereo image pair can be provided to a computer system by one or more digital cameras or a stereo camera via an input / output interface. The input stereo image pair has the same format as a general image. Each image is, for example, a 2D rectangle that uses (x, y) as coordinates to index a gray value or color of pixels.

[0015] Block 204 features a straightening of the input stereo image pair, so that its epipolar lines become horizontal or vertical.

[0016] Block 206 involves the application of stereo alignment procedures to the straightened input stereo image pair. For example, Block 206 may involve the application of normalized cross-correlation (NCC) to establish the initial correspondences between pixels from this stereo image pair. A graphics processing unit (GPU) of a computer system can perform the image pair straightening and stereo alignment.

[0017] Block 208 features the use of a homograph fit for each pixel in the base image to optimize the photoconsistency between a pixel in the base image and the corresponding pixel in the translated image. The base image is, for example, the left image in Fig. 1, where the translated image is the right image in Fig. 1 is. A central processing unit of a computer system can perform homograph fitting.

[0018] Block 210 features triangulated optimized correspondence points to obtain a 3D scene from a single perspective. Suitable techniques for performing blocks 202, 204, 206, and 210 are described, for example, in H. Hirschmfuller et al., "Real-Time Correlation-Based Stereo Vision with Reduced Border Errors," pages 229–246, Volume 47, IJCV 2002. A computer system's GPU can perform the triangulation of correspondence points to obtain a three-dimensional view from a single perspective.

[0019] The computational procedure for stereo reconstruction in the method according to Fig.Step 2 can be performed on a central processing unit (CPU) or a graphics processing unit (GPU). Generally, the most computationally complex part of the stereo matching process exhibits massive data-level parallelism, which can be accelerated by using the vector processing or multi-threaded processing capabilities of a GPU. Homography matching is a subsequent update procedure that can be performed on a CPU.

[0020] The following describes an exemplary procedure performed in Block 208. Homography is the mapping between two sets of points that correspond to the same 3D point sets but are viewed from different cameras. Homography is used as a transformation to relate two corresponding point sets in two image planes. Given a point P1 in a basic image plane, the following is performed with reference to Fig.1. A homography H, induced by a tangent plane π (n being the normal to the tangent plane), is used to find the corresponding point P2 in the translated image plane. In other words, the homography H translates P1 to P2 via the tangent plane π. The plane π is a tangent plane to the object surface at the corresponding 3D point P.

[0021] Various embodiments apply homography matching of two regions with respect to homography parameters in the optimization of photoconsistency. Referring to Fig.For each pixel P1 in the base image, an m × m window T(x) is centered on pixel P1, where x denotes the image coordinates of the pixels in the base window T(x). The corresponding pixel of x in the translated image is denoted as W(x; p) = H(p)·x, where H(p) is the homography transformation. In various embodiments, H(p) is a 3×3 matrix given by internal and external parameters of a stereo camera, and p is a 3-element parameter relating to the depth and normal of the corresponding 3D point. Other types of homography transformations can be used.

[0022] The homography transformation can be represented as: H(p)=K1⋅(R−tpT)Kr−1, where K l and K r the intrinsic matrices of a corresponding left and right camera are, R is the rotation matrix between the two cameras. t is the translation vector between the two cameras, p=n / d, where n is the normal vector of the plane π and d is the distance between the center of the left camera and the plane π.

[0023] The photo consistency between a pixel x in the base image and the warp window, W(x; p) of the translated image, I, is defined as: ∑x∈A[T(x)−I(W(x;p))]2, where T(x) is a template window in the basic image, I is a translated image and W(x; p) is a homography warp window in the translated image.

[0024] Photoconsistency is an implicit and nonlinear function of the homography parameter p. In various embodiments, an inverse composition image alignment (ICIA) method is used to improve photoconsistency. The ICIA method is modified to provide an optimized homography matrix H(p) and an optimized corresponding pixel in the translated image. A suitable ICIA method is described, for example, in S. Baker, I. Matthews, Lucas-Kanade: “20 Years On: A Unifying Framework”, IJCV (2004). An exemplary ICIA method for homography matching is described with reference to Method 300 from Fig. 3 described.

[0025] Block 302 features the reception of a pixel point x = (u, v, 1) as well as parameters from camera pairs and an initial value of a homography parameter, p. The parameters can include an intrinsic matrix K of a camera and extrinsic elements (e.g., rotation matrix R and translation vector t) and an initial value of a homography parameter, p (which will be defined later). The parameter p can be initialized by a traditional NCC-based procedure in accordance with its definition p = n / d.

[0026] Block 304 involves determining the Hessian matrix for the translated image. Block 304 may, for example, include: (1) evaluating the gradient ∇T of the base window T(x); (2) evaluating the Jacobian determinant for the right-hand image W with respect to p at (x; p0), where the Jacobian determinant is expressed as ∇J = ∂W / ∂p; and (3) determining the Hessian matrix of W with respect to p. The Hessian matrix may be expressed as H = Σ x [∇T∇J]T [∇T∇J] can be expressed. The Hesse matrix corresponds to an improved homography matrix.

[0027] Block 306 involves determining a pixel in the translated image W(x; p). Block 306 can, for example, include: (1) determining I(W(x; p)) by warping the translated image I with W(x; p); (2) determining the error pattern I(W(x; p)) - T(x); (3) calculating the increment step Δp = H -1 ∑ r [∇7∇J] T [I(W(x;p))-T(x)]; and (4) Updating the warp W(x; p) by determining W(x; p) = W(x;p) ·W(x; Ap) -1 The elements (1) to (4) of block 306 are repeated until an absolute value of the increment step, |∇p|, is less than a limit. The limit might be approximately 0.00001, for example. Procedure 300 determines an optimized warp window (W(x,p)) that can be used to generate an optimized homography matrix H = ∑ x [∇T∇J] Tto determine [∇T∇J]. The optimized homography matrix H(p) can be used to find a corresponding pixel in the translated image W(x;p) · [u, v,I] T to determine.

[0028] The ones relating to Fig. The two described stereo homograph matching techniques can be extended for use with multi-view stereo reconstruction. Multi-view stereo can be considered the combination of results from several stereo pairs. Fig. Section 4 describes a method that can be used in accordance with one embodiment for multi-view stereo reconstruction. Block 402 receives several three-dimensional scenes from different stereo views. According to an example, three views are available, namely views C1, C2, and C3. Binocular stereo pairs can be of views C1-C2, C2-C3, and C1-C3.

[0029] Block 404 performs stereo homology and multi-view alignment and integration for each stereo pair. Stereo homology is applied to each stereo pair with reference to... Fig. 2 and Fig. 3. The correspondence optimized by homography fitting can be defined as (x1, x2 = w(x1)). After obtaining the optimized pair (x1, x2 - w(x1)), the pair can be triangulated with the known camera parameters according to the procedure described, for example, in R. Hartley and A. Zisserman: "Multiple View Geometry in Computer Vision", Chapter 12, Cambridge Press, 2nd edition (2003). Triangulation can be used to triangulate a three-dimensional point X. 12 generate from the correspondence of each stereo pair, X 12 =triangulate(x1, x2).

[0030] Assume X 12 is the triangulated result of C1-C2, X 23 is the result of C2-C3 and X 13is the result of C1-C3 and X 12 , X 13 and X 23 correspond to the same 3D point. A 3D point, X, is a function of X. 12 , X 13 and X 23 , namely X=f(X 12 , X 13 , X 23 ), where f() is a multi-view alignment and integration function and X 12 , X 13 and X 23 Each of these are three-dimensional points in world coordinates. In some embodiments, the multi-view alignment and integration technique can be an average function or a best-fit function, although other functions can be used.

[0031] For example, the 3D point is given by: X = (X 12 +X 13 +X 23 ) / 3, if f() is defined as the average function.

[0032] If f() is defined as the best-fit function, the 3D point is chosen where (1) the normalized vector n ialmost perpendicular to the camera plane or (2) n ij exhibits a smallest angle to the direction PO. Thus, f() is considered a selection function: X=Xi is defined, where i=arg mini Angle(POi,ni).

[0033] Block 406 provides a three-dimensional scene based on multiple perspectives. The 3D points defined as X in Block 404 form a three-dimensional scene.

[0034] Fig. 5A reproduces the results of a stereo reconstruction using the standard Middlebury stereo evaluation set, which is described, for example, at http: / / vision.middlebury.edu / . Fig.Figure 5B shows the results of techniques using homograph-matching-based photo consistency optimization. It is evident that these techniques significantly outperform the traditional NCC-matching-based method, at least with regard to image clarity.

[0035] Techniques using homograph-matching-based photoconsistency optimization yield similar results to Furukawa's method, but are faster. Table 1 shows the execution time of the two methods for the stereo reconstruction task. Table 1. Comparison of execution time performance Proceedings Number of optimized corresponding patches Time Techniques that use homograph-matching-based photo consistency optimization ~958K 2s Furukawa's method ~972K 158s

[0036] Fig.Figure 6 shows a block diagram of a computer system 600 in accordance with an embodiment of the present invention. The computer system 600 can comprise a host system 602, a bus 616, and a network interface 620. The computer system 600 can be implemented in a handheld personal computer, a mobile phone, a set-top box, or any computing device. The host system 602 can comprise a chipset 605, a processor 610, a host memory 612, a storage memory 614, and a graphics subsystem 615. The chipset 605 can provide intermediary communication between the processor 610, the host memory 612, the storage memory 614, the graphics subsystem 615, and the bus 616. The chipset 605 can, for example, include a storage adapter (not shown) capable of providing intermediary communication with the storage memory 614.The storage adapter may, for example, be able to communicate with the 614 storage device in compliance with one of the following protocols: Small Computer Systems Interface (SCSI), Fibre Channel (FC) and / or Serial Advanced Technology Attachment (S-ATA).

[0037] In some embodiments, the chipset 605 may have data movement logic capable of performing transfers of information within the host memory 612, or between the network interface 620 and the host memory 612, or generally between any set of components in the computer system 600.

[0038] The 610 processor can be implemented as a Complex Instruction Set Computer (CISC) or a Reduced Instruction Set Computer (RISC) processor, a multi-core or any other microprocessor or central processing unit.

[0039] Host memory 612 can be implemented as a volatile storage device, such as, but not limited to, Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), or Static RAM (SRAM). Storage memory 614 can be implemented as a non-volatile storage device, such as, but not limited to, a magnetic disk drive, an optical disk drive, a tape drive, an internal storage device, an attached storage device, flash memory, battery-backed SDRAM (Synchronous DRAM), and / or a network-accessible storage device.

[0040] The Graphics Subsystem 615 can process images for display, such as still images or video. The Graphics Subsystem 615 can, for example, perform video encoding or decoding. The Graphics Subsystem 615 can, for example, perform activities of a graphics processing unit that have been described with respect to any activity related to Fig. 2, Fig. 3 to Fig. As described in section 4, an analog or digital interface can be used to communicate between the 615 graphics subsystem and a 622 display. The interface can be, for example, any of the following: High-Definition Multimedia Interface, DisplayPort, wireless HDMI, and / or wireless HD-compliant technologies. The 615 graphics subsystem can be integrated into the 610 processor or the 605 chipset. Alternatively, the 615 graphics subsystem can be a standalone card communicatively coupled to the 605 chipset.

[0041] Bus 616 can provide intermediary communication between at least the host system 602 and the network interface 620, as well as other peripheral devices (not shown). Bus 616 can support serial or parallel communication. Bus 616 can support node-to-node or node-to-multi-node communication. Bus 616 can be used with at least Peripheral Component Interconnect (PCI), as described, for example, in the Peripheral Component Interconnect (PCI) Local Bus Specification, Rev. 3.0, February 2, 2004, available from the PCI Special Interest Group, Portland, Oregon, USA (and revisions thereof); PCI Express, as described in the PCI Express Basic Specification of the PCI Special Interest Group, Revision 1.0a (and revisions thereof); and PCI-x, as described in the PCI-X Specification, Rev. 1.1, February 28, 2004.March 2005, described, which is available from the aforementioned PCI Special Interest Group, Portland, Oregon, USA (and revisions thereof); and / or be compatible with Universal Serial Bus (USB) (and related standards) as well as other intermediate interconnection standards.

[0042] The network interface 620 can provide intermediary communication between the host system 602 and a network compliant with any applicable protocol, such as wired or wireless technologies. For example, the network interface can conform to any variety of IEEE 802.3, 802.11, or 802.16. The network interface 620 can communicate with the host system 602 using the bus 616. In one embodiment, the network interface 620 can be integrated into the chipset 605.

[0043] The graphics and / or video processing techniques described here can be implemented in various hardware architectures. For example, the graphics and / or video functionality can be integrated within a chipset. Alternatively, a separate graphics and / or video processor can be used. In a further embodiment, the graphics and / or video functions can be implemented by a general-purpose processor that has a multi-core processor. In yet another embodiment, the functions can be implemented in an electronic end-user device.

[0044] Embodiments of the present invention may, for example, be provided as a computer program product which may comprise one or more machine-readable media containing machine-executable instructions stored thereon which, when executed by one or more machines, such as a computer, a network of computers or other electronic devices, may cause the one or more machines to perform operations in accordance with embodiments of the present invention.A machine-readable medium may include, but is not limited to, floppy disks, optical disks, CD-ROMs (Compact Disc-Read Only Memories) and magneto-optical disks, ROMs (Read Only Memories), RAMs (Random Access Memories), EPROMs (Erasable Programmable Read Only Memories), EEPROMs (Electrically Erasable Programmable Read Only Memories), magnetic or optical cards, flash memory, or other types of media or machine-readable medium suitable for storing machine-executable instructions.

[0045] The drawings and the preceding description provided examples of the present invention. Although they were represented as a number of quite different functional objects, those skilled in the art know that one or more of such elements can also be combined into individual functional elements. Alternatively, certain elements can be divided into several functional elements. Elements from one embodiment can be added to another embodiment. For example, the sequence of the methods described herein can be changed and is not limited to the manner described here. Furthermore, actions in any flowchart need not be implemented in the sequence shown, nor do all activities necessarily have to be performed. Moreover, those activities that are independent of other activities can be performed in parallel with the other activities.The scope of the present invention is by no means limited by these specific examples. Numerous variations are possible, whether explicitly stated in the description or not, such as differences in the structure, dimensions, and use of a material. The scope of the invention is at least as broad as defined by the following claims.

Images