System and method for image stitching with robust camera pose estimation - Patents.com

Abstract

To provide an image forming system and an image forming method for constructing a whole large image of an object.SOLUTION: An image stitching system 100 representing an image forming system acquires sequential images of partial areas of an object obtained by a camera. Two adjacent images of the partial areas include overlap portions, wherein the acquired sequential images correspond to a three-dimensional (3D) surface of the object. An image stitching module 210 configuring the image stitching system 100 comprises a robust camera pose estimation section 220 for estimating camera poses with respect to the acquired sequential images by a robust camera pose estimation method which imposes low-rankness and joint sparsity of relative pose matrices, wherein the acquired sequential images are projected into a 3D surface based on the camera pose estimation, a two-dimensional (2D) image of the object is constructed by interpolating the projected surface image, and the constructed 2D image is outputted.SELECTED DRAWING: Figure 1B

Description

Technical Field

[0001] Technical Field The present invention generally relates to an apparatus and method for stitching images of large scenes or surfaces captured by a camera in an unrecognized and distorted pose.

Background Art

[0002] Background and Prior Art Image stitching technology aims to fuse a collection of overlapping images to achieve a wide view of an object or scene of interest with sufficient resolution in applications such as Google Earth mapping, video stabilization, panoramic image construction, drone image stitching, and 3D structure reconstruction. To achieve the high stitching accuracy required by various applications, it is necessary to compensate for the geometric distortion of the collected images. When compensating for such distortion, the estimation of the camera pose for each image is a very important task. This is because even a slight error in camera pose estimation can lead to obvious distortion and visual inconsistency in the stitched images, which may limit subsequent applications. Therefore, when images are collected, accurately estimating the camera pose is very important and a difficult problem.

[0003] Over the past few decades, pairwise image alignment and stitching methods have been extensively studied as a so-called PnP (perspective-n-point) problem using matching feature points. In particular, for each pair of images to be stitched together, the associated camera pose is estimated using the PnP method, utilizing the matching points based on SIFT features of the image pair so that the images can be stitched together one by one. Pairwise image stitching generally works well for a small number of images where the matching points are clear. However, if the number of images is large or the feature points do not match well, pairwise stitching will not work. This is because any erroneous matching point will cause abnormal camera pose errors, and the impact of these errors will affect the subsequent camera pose estimation used in stitching, ultimately degrading the image stitching performance. To solve this problem, global alignment methods to improve the robustness of stitching, such as bundle adjustment, have been studied. In the bundle adjustment method, instead of considering only one reference image, the camera pose of the target image is estimated by considering matching feature points between the target image and a bundle of superimposed reference images. Therefore, the overall robustness is improved as the number of matching points increases. However, erroneous matching points still inevitably introduce camera pose errors, so the performance, especially for high-precision stitching, remains insufficient. Therefore, there is a need to develop novel image stitching systems and methods that use robust camera pose estimation to improve performance. [Overview of the project]

[0004] Within our proposed framework, we propose a robust camera pose estimation method that can accurately estimate camera pose even with anomalous pairwise camera pose errors. In particular, given a set of indexed images, we first construct all relative camera pose matrices for each image pair. Each entry corresponds to a pair of target and reference images indexed by rows and columns, respectively. If the image pairs overlap, the corresponding entry is realized by solving a PnP (perspective-n-point) problem using matching feature points in the overlapping region. Otherwise, the entry is unobservable. For a 6DoF (6 degrees of freedom) camera pose, we construct six matrices. Each matrix corresponds to one of the six degrees of freedom. As a global alignment method, our proposed method aims to estimate camera pose based on a partially observed relative pose matrix by effectively utilizing the underlying rank 2 relative pose matrix and the combined sparsity of camera pose errors during estimation. Numerical simulations have demonstrated that our proposed method is superior to existing pairwise alignment baselines and global alignment baselines when stitching images.

[0005] In recent years, low-rank and sparsity have been widely studied in applications such as image restoration, signal processing, video processing, machine learning, and Euclidean distance matrix reconstruction, where the low-dimensional structure of signals is effectively utilized. Robust principal component analysis (RPCA) is a well-known method for solving this type of problem. In RPCA, a low-rank matrix and a sparse error matrix are reconstructed from their superimposed observations. Here, inspired by RPCA, but in a different way, we implicitly utilize low-rank properties rather than using explicit low-rank constraints such as matrix kernel norms. Furthermore, if a partially observed matrix is ​​available, we encourage the combined sparsity of the relative pose error.

[0006] Therefore, some embodiments of this disclosure can contribute as summarized below.

[0007] 1. This disclosure proposes a robust image stitching method that effectively utilizes the underlying rank 2 relative pose matrix and the coupled sparsity of camera pose errors. The proposed method exhibits robust performance even under abnormal camera poses.

[0008] 2. This disclosure implicitly explains low-rank and join sparsity, given the partially observed matrices. The proposed method may be applicable to a variety of scenarios involving estimations of a series of linear relationships.

[0009] 3. This disclosure provides numerical experiments using composite images captured by a camera in an unrecognized, distorted posture. Our experiments demonstrated a significant improvement in performance compared to other baseline methods.

[0010] Furthermore, some embodiments of the present invention are based on the understanding that an image forming system is provided for constructing an entire image of an object. This image forming system may include an interface configured to successively acquire images of a sub-region of an object captured by a camera, wherein two adjacent images of the sub-region include an overlapping portion, the successive images correspond to the 3D surface of the object, geometric information (geometric parameters) of the object is provided, and the initial pose of the camera is provided. The image forming system further includes a memory for storing a computer-executable program including a pose estimation method and a stitching method, and an image stitching processor, the image stitching processor including the steps of estimating the successive images by the pose estimation method, projecting the successive images onto the 3D surface based on the camera pose estimation by solving PnP (perspective-n-point), constructing a 2D image of the object based on the estimated camera pose, and outputting the constructed 2D image via the interface.

[0011] Furthermore, another embodiment may provide an image forming method for constructing an entire image of an object. In this case, the method includes the step of successively acquiring images of a portion of an object captured by a camera via an interface, wherein two adjacent images of the portion of the object include an overlapping portion, the successive images correspond to the 3D surface of the object, geometric information of the object is provided, and the initial pose of the camera is provided. The image forming method further includes the step of providing a computer-executable program from memory, which includes a pose estimation method and a stitching method. The image forming method further includes the step of estimating the pose of the successive images by the pose estimation method, projecting the successive images onto the 3D surface based on the camera pose estimation by solving a PnP (perspective-n-point) problem, constructing a 2D image of the object based on the estimated camera pose, and outputting the constructed 2D image via an interface, all of which are performed using an image stitching processor.

[0012] The accompanying drawings, provided to enhance understanding of the present invention, illustrate embodiments of the present invention along with descriptions illustrating the principles of the present invention. [Brief explanation of the drawing]

[0013] [Figure 1A] An example of an image acquisition process according to an embodiment of the present invention is shown. [Figure 1B] A block diagram of an image processing system according to an embodiment of the present invention is shown. [Figure 2] A block diagram showing an image stitching step for generating a large surface image according to an embodiment of the present invention is shown. [Figure 3] This figure shows an algorithm describing robust camera pose estimation according to an embodiment of the present invention. [Figure 4A]This figure compares an exemplary relative camera pose matrix partially observed using PnP according to an embodiment of the present invention with ground truth, a relative pose matrix reconstructed using the least squares method, and a relative pose matrix reconstructed using robust camera pose estimation. [Figure 4B] This figure compares an exemplary relative camera pose matrix partially observed using PnP according to an embodiment of the present invention with ground truth, a relative pose matrix reconstructed using the least squares method, and a relative pose matrix reconstructed using robust camera pose estimation. [Figure 4C] This figure compares an exemplary relative camera pose matrix partially observed using PnP according to an embodiment of the present invention with ground truth, a relative pose matrix reconstructed using the least squares method, and a relative pose matrix reconstructed using robust camera pose estimation. [Figure 4D] This figure compares an exemplary relative camera pose matrix partially observed using PnP according to an embodiment of the present invention with ground truth, a relative pose matrix reconstructed using the least squares method, and a relative pose matrix reconstructed using robust camera pose estimation. [Figure 5A] This figure shows a comparison of relative camera pose error matrices according to an embodiment of the present invention. [Figure 5B] This figure shows a comparison of relative camera pose error matrices according to an embodiment of the present invention. [Figure 5C] This figure shows a comparison of relative camera pose error matrices according to an embodiment of the present invention. [Figure 6] This figure shows the camera pose error when the number of images is increased in a different method according to an embodiment of the present invention. [Figure 7] This figure shows the relative error of camera orientation and the PSNR of the stitched images in different methods according to embodiments of the present invention. [Figure 8A]A diagram comparing the large surface image of ground truth, the image stitched using the least squares method, and the image stitched using robust camera pose estimation according to an embodiment of the present invention. [Figure 8B] A diagram comparing the large surface image of ground truth, the image stitched using the least squares method, and the image stitched using robust camera pose estimation according to an embodiment of the present invention. [Figure 8C] A diagram comparing the large surface image of ground truth, the image stitched using the least squares method, and the image stitched using robust camera pose estimation according to an embodiment of the present invention. [Figure 9A] A diagram comparing the enlarged area image of ground truth, the image stitched using the least squares method, and the image stitched using robust camera pose estimation according to an embodiment of the present invention. [Figure 9B] A diagram comparing the enlarged area image of ground truth, the image stitched using the least squares method, and the image stitched using robust camera pose estimation according to an embodiment of the present invention. [Figure 9C] A diagram comparing the enlarged area image of ground truth, the image stitched using the least squares method, and the image stitched using robust camera pose estimation according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0014] Detailed Description of Preferred Embodiments Hereinafter, various embodiments of the present invention will be described with reference to the drawings. Note that the drawings are not drawn to scale, and elements having the same structure or function are denoted by the same reference numeral throughout the drawings. Also, note that the drawings are merely for facilitating the explanation of specific embodiments of the present invention. These are not intended to explain the invention exhaustively and do not limit the scope of the present invention. In addition, aspects described in conjunction with a given embodiment of the present invention are not necessarily limited to that embodiment and can be implemented in other embodiments of the present invention.

[0015] FIG. 1A is an example showing image acquisition processing according to an embodiment of the present invention. In the image acquisition processing, images are taken one by one. When two images overlap, the relative camera pose between two corresponding camera poses is calculated. When they do not overlap, the corresponding relative pose is set to unknown (uncertain). This figure is an example showing image acquisition processing according to an embodiment of the present invention. Each camera snapshot covers a small area of a large surface (painting) in a state where camera pose disturbance cannot be recognized. The camera moves in a zigzag scan pattern to cover the entire surface in a state where the image snapshots overlap.

[0016] FIG. 1B is a diagram showing an example for explaining image acquisition processing according to an embodiment of the present invention. This figure is a schematic diagram showing an image stitching system 100 for generating a large image of a target surface 140 according to an embodiment of the present disclosure.

[0017] The image stitching system 100 may include a network interface controller (interface) 110 configured to receive an image from a camera / sensor 141 and display (142) the image. The camera / sensor 141 is configured to take an overlapping image of the target surface 140.

[0018] Furthermore, the image stitching system 100 may include a memory / CPU unit 120 for storing a computer-executable program in a storage device 200. The computer-executable program / algorithm may include superimposed images 205 and an image stitching module 210. The image stitching module 210 includes a robust camera pose estimation unit 220. The computer-executable program is configured to connect to the memory / CPU unit 120, which accesses the storage device 200 and loads the computer-executable program.

[0019] Furthermore, the memory / CPU unit 120 is configured to receive images (data) from the camera / sensor 151 or image data server 152 via the network 150 and to perform displacement measurement using the image stitching system 100 described above.

[0020] Furthermore, the image stitching system 100 may include at least one camera positioned to capture an image of the target surface 140, and the at least one camera may transmit the captured image to the display device 142 via an interface.

[0021] In some cases, at least one of these cameras may be a high-resolution digital optical camera. In this case, images of successively acquired subregions of an object are captured by the camera, the high-resolution digital optical camera.

[0022]

number

[0023]

number

[0024]

number

[0025]

number

[0026]

number

[0027]

number

[0028]

number

[0029]

number

[0030]

number

[0031]

number

[0032]

number

[0033]

number

[0034] Figure 3 shows algorithms describing robust camera pose estimation according to several embodiments of the present invention. The overall solution is summarized as Algorithm 1 in Figure 3.

[0035]

number

[0036]

number

[0037]

number

[0038]

number

[0039] To estimate the camera pose, we consider four different methods for comparison: (1) pairwise method, (2) bundle adjustment method, (3) least squares method, and (4) our proposed method. In the case of the pairwise method, we use a SIFT feature-based PnP (perspective-n-point) method for matching points of the image pairs being joined, using prior knowledge of the 3D scene geometry.

[0040]

number

[0041]

number

[0042]

number

[0043]

number

[0044]

number

[0045]

number

[0046] Figure 6 shows camera pose errors when the number of images is increased using different methods according to several embodiments of the present invention. In this figure, the relative errors for the N images shown are plotted. In both the pairwise method and the bundle method, it can be seen that the camera pose error tends to accumulate as the number of images increases. While the least squares method maintains a low level of relative error, our proposed method further reduces the error even in cases of abnormal camera pose errors.

[0047]

number

[0048]

number

[0049] We propose a robust camera pose estimation method for stitching together a set of large images of 3D surfaces with known shapes. To address the issue of camera pose errors accumulating when using conventional PnP methods, we construct a partially observed relative pose matrix for each camera pose parameter and decompose this relative pose matrix into a rank 2 matrix of relative camera poses and a sparse matrix of camera pose errors by effectively utilizing the combined sparsity of camera pose errors when estimating the camera pose. Numerical experiments using images captured by a camera at random poses demonstrate that even when abnormal errors occur in camera pose estimation using conventional PnP methods, our proposed method can detect and suppress estimation errors, resulting in robust camera pose estimation and significantly superior stitching results.

[0050] The embodiments of the present invention described above can be implemented in numerous ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code can be executed on any suitable processor or set of processors, whether located on a single computer or distributed across multiple computers. Such a processor may be implemented as an integrated circuit having one or more processors in an integrated circuit component. However, the processor may also be implemented using circuitry of any suitable format.

[0051] Furthermore, embodiments of the present invention may also be implemented as methods, and examples of such methods have been described. The methods and the actions performed as part of them may be appropriately sequenced. Therefore, even if the actions are illustrated as a series of actions in the exemplary embodiments, the embodiments may be configured to perform the actions in an order different from the illustrated order, and may include performing several actions simultaneously.

[0052] The use of ordinal numbers such as "First," "Second," etc., does not in itself mean that one claim element takes precedence over another, precedes or precedes the other, nor does it mean the temporal order in which the actions included in the method are performed. Rather, it is merely used as a label to distinguish one claim element of a certain name from another claim element of the same name (but in the use of ordinal numbers), thus distinguishing the claim elements.

[0053] While the present invention has been described as an example of a preferred embodiment, it should be understood that various other modifications and changes may be made within the spirit and scope of this disclosure.

[0054] Therefore, the attached claims are intended to include all such modifications and variations within the spirit and scope of the present invention.

[0055] The embodiments of the present invention described above can be realized using hardware, software, or a combination of hardware and software.

[0056] Furthermore, embodiments of the present invention may also be implemented as methods, and examples of such methods have been described. The methods and the actions performed as part of them may be appropriately sequenced. Therefore, even if the actions are illustrated as a series of actions in the exemplary embodiments, the embodiments may be configured to perform the actions in an order different from the illustrated order, and may include performing several actions simultaneously.

[0057] The use of ordinal numbers such as "First," "Second," etc., does not in itself mean that one claim element takes precedence over another, precedes or precedes the other, nor does it mean the temporal order in which the actions included in the method are performed. Rather, it is merely used as a label to distinguish one claim element of a certain name from another claim element of the same name (but in the use of ordinal numbers), thus distinguishing the claim elements.

Claims

1. 1. An imaging system for constructing an entire image of an object, comprising: an interface configured to successively acquire images of a sub-region of the object captured by a camera, wherein two images of the sub-region of the surface of the object include overlapping portions, the successively acquired images correspond to a 3D (three-dimensional) surface of the object, geometric information of the object is provided, and an initial pose of the camera is provided; the imaging system further comprises: a memory for storing a computer executable program including a pose estimation method and a stitching method; and an image stitching processor configured to perform the steps of: estimating the relative camera pose of each image pair by solving perspective-n-point (PnP) problems if the image pairs are overlapping; estimating a pose for the successively acquired images based on relative camera poses; projecting the successively acquired images onto a 3D surface of the object based on the estimated pose for the images; constructing a 2D (two-dimensional) image by interpolating the projected surface images; and outputting the constructed 2D image via the interface.

2. The imaging system of claim 1 , wherein the camera is a high-resolution digital optical camera.

3. The imaging system of claim 1 , wherein the image of the subregion is captured by the camera modeled as a 6DoF (six degrees of freedom) pinhole camera.

4. The imaging system of claim 1 , wherein each of the estimated image poses is determined by a three-dimensional Euler angle vector and a three-dimensional translation vector.

5. The imaging system of claim 1 , wherein the estimated image pose is estimated for all of the consecutively acquired images.

6. 2. The imaging system of claim 1, wherein the estimation of the relative camera pose is performed by solving a perspective-n-point (PnP) problem.

7. 7. The image forming system according to claim 6, wherein solving the perspective-n-point (PnP) problem uses match points based on scale-invariant-feature transform (SIFT) features.

8. The imaging system of claim 1 , wherein the camera pose of at least a first image among the successively acquired images is known.

9. The imaging system of claim 1 , wherein the estimated image pose is updated by a robust camera pose estimation algorithm that applies low-rank and joint sparsity.

10. 1. An image formation method for constructing an entire image of an object, comprising: The method includes a step of successively acquiring images of a partial region of the object captured by a camera via an interface, wherein two images of the partial region include an overlapping portion, the successively acquired images correspond to a 3D (three-dimensional) surface of the object, geometric information of the object is provided, and an initial pose of the camera is provided, and the image formation method further includes: providing, from a memory, a computer executable program including a pose estimation method and a stitching method, the image formation method further comprising: estimating the relative camera pose of each image pair by solving perspective-n-point (PnP) problems if the image pairs are overlapping; estimating a pose for the successively acquired images based on the estimated relative camera pose; projecting the successively acquired images onto a 3D surface based on the estimated pose for the images; constructing a 2D (two-dimensional) image of the object by interpolating the projected surface images; and outputting the constructed 2D image via the interface, using an image stitching processor.

11. The imaging method of claim 10, wherein the camera is a high-resolution digital optical camera.

12. The imaging method of claim 10 , wherein the image of the subregion is captured by the camera modeled as a 6DoF (six degrees of freedom) pinhole camera.

13. The imaging method of claim 10 , wherein each of the estimated image poses is determined by a three-dimensional Euler angle vector and a three-dimensional translation vector.

14. The image formation method of claim 10 , wherein the estimated image pose is estimated for all of the consecutively acquired images.

15. 11. The image forming method according to claim 10, wherein the estimation of the relative camera pose is performed by solving a PnP (perspective-n-point) problem.

16. 16. The image forming method according to claim 15, wherein solving the perspective-n-point (PnP) problem uses coincidence points based on scale-invariant-feature transform (SIFT) features.

17. The image forming method according to claim 10 , wherein the camera pose of at least a first image among the successively acquired images is known.

18. The image formation method of claim 10 , wherein the estimated image pose is updated by a robust camera pose estimation algorithm that applies low-rank and joint sparsity.

Images