Method of 3D panoramic mosaicing of a scene

Abstract

A method of 3D mosaicing of a scene as a 3D mosaic, wherein at least one 3D reconstruction having been obtained without a priori information about the scene, in the course of the following steps of:acquisition of successive images by a panoramic sensor moving along an unconstrained 3D reconstruction trajectory, such that the image of at least one point of the scene is in at least 3 successive 2D panoramic images obtained according to various panoramic system-point of the scene directions,rectification of these images on different rectification planes covering different directions, and matching of the rectified images,3D reconstruction on reconstruction planes on the basis of the rectified matched images,the last 2D panoramic acquired image being called current 2D panoramic image,it further comprises the following steps:A) On the basis of the obtained 3D reconstruction and of the current 2D panoramic image, choosing one or more projection surfaces on which the mosaic will be constructed,B) Selecting sectors of the current 2D panoramic image, and selecting textures in the selected sectors by selecting visible parts that is to say that are not hidden by a surface of the scene, with the aid of the 3D reconstruction,C) Projecting the selected textures onto each projection surface and fusing the textures in each projection surface so as to thus obtain a conformal mosaic on each projection surface.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]This application is a continuation of U.S. application Ser. No. 14 / 646,712, filed on May 21, 2015, which is a National Stage of International patent application PCT / EP2013 / 074277, filed Nov. 20, 2013 which claims priority to foreign French patent application No. FR 1203126, filed on Nov. 21, 2012, the disclosures of which are incorporated by reference in their entirety.FIELD OF THE INVENTION[0002]The field of the invention is that of the 3D mosaicing of a scene on the basis of successive panoramic images of this scene.BACKGROUND[0003]The 3D reconstruction of a scene consists in obtaining, on the basis of successive 2D images of this scene taken from different viewpoints, a so-called 3D reconstructed image such that with each pixel of the reconstructed image, that is to say at any point where the reconstruction declares that there is a scene element, are associated the coordinates of the point of the corresponding scene, defined in a frame...

AI Technical Summary

Benefits of technology

[0037]The panoramic system comprises one or more sensors whose images do not necessarily exhibit any mutual overlap, and makes it possible to cover the whole of the scene to be reconstructed instantaneously (with holes if the sensors are not overlapping) or in the course of the motion.

[0038]This solution also makes it possible to produce in parallel a mosaic with very large spatial and angular extent representing this scene according to all the viewpoints of the panoramic system in the course of its displacement.

[0039]This therefore produces a 3D mosaicing which is a generalization of 2D mosaicing in the sense that the projection can be done on an arbitrary 3D surface, which may itself consist of several plane or non-plane surfaces exhibiting discontinuities. This 3D mosaicing consists, on the basis of successive 2D images of a scene (taken from different viewpoints) and of the 3D reconstruction of the scene in the above sense, in projecting and assembling the various 2D images on the geometric modeling of the 3D reconstruction, thus making it possible to restore the whole of the scene in the form of a textured mosaic overlaid on the various 3D elements of this scene. It makes it possible to restore in a conformal manner an assemblage of images on an arbitrary scene exhibiting relief or 3D elements. The reconstructed 3D mosaic is therefore a textured 3D reconstruction of the scene.

[0040]These methods make it possible to carry out a 3D mosaicing over the widest possible spatial and angular extent.

[0046]A simple, accurate method is thus obtained making it possible for example to produce textured maps on which measurements can be performed, to reconstruct the scene over a large spatial (possibly up to as much as 180°×360°) and angular extent and in real time, without constraints on the trajectory, on an arbitrary scene (without any plane-scene assumptions and without the aid of any prior scene model for example).

[0048]makes it possible to afford a compact solution to the problem of 3D mosaicing of the scene by requiring only a single panoramic system, whilst those of the prior art require several independent sensors and are more complex to implement,

Problems solved by technology

This solution exhibits several drawbacks:the cameras are difficult to calibrate (problems of vibration),an inaccuracy in restitution of the 3D reconstruction on account of a stereo base limited by the spacing between the cameras,low-field and low-extent restitution on account of the limited optical field of the cameras.

Moreover, the finalized 3D reconstruction is not obvious, since it is constructed by assembling local 3D reconstructions (resulting from the method of stereoscopic restitution of 2, often small-field, images) which may be very noisy on account of the limited number of images which made it possible to construct it, of the limited field of the cameras and of the fact that the reconstruction planes dependent on the respective attitudes of the cameras have a geometry that is difficult to measure accurately (the relative position and relative geometry of the cameras serving to do the 3D reconstruction is often inaccurate in practice when dealing with cameras which are 1 or 2 meters apart and liable to vibrate with respect to one another: this is still more evident when these cameras are motorized).

Lastly, the stereoscopic system restores poorly planes which are almost perpendicular to one of the 2 cameras (this is the problem of the restitution of pitched roofs in aerial or satellite stereoscopic imaging).

A moving low-field or mean-field camera, but the 3D reconstruction is limited by the path and the orientation of the camera and is therefore not omnidirectional; moreover, the reconstruction may exhibit holes on account of unchecked motions of the camera or non-overlaps of the latter in the course of its motion.

The algorithms used for 3D reconstruction impose a reconstruction in a frame tied or close to the focal plane of the camera, thereby limiting the possibilities of reconstruction (a single principal reconstruction plane and very limited reconstruction when the camera changes orientation).

The result of the reconstruction is also very noisy and may exhibit numerous errors on account of the small overlap between images, of a constant plane of reconstruction of the reconstructed scene (and of a camera that could deviate from this plane) and of the use of algorithms which for the 3D reconstruction utilize only two images separated by a relatively small distance.

The mosaicing obtained by the ground overlaying of the successive images is inoperative and is not conformal when the scene is not flat and / or comprises 3D elements.

Active sensors, that is to say with telemetry, but here again the 3D reconstruction is not omnidirectional and is not necessarily segmented, the measurements being obtained in the form of scatters of points that are difficult to utilize in an automatic manner.

Moreover, the mesh obtained by these active sensors exhibits the drawback of being angularly non-dense (typically fewer than 4 points per m2 for airborne applications at 1 km height).

The technique is not at the moment suitable for being able to produce a textured image of the scene and must almost always be corrected manually.

All the previous solutions are unsuitable for obtaining a 3D mosaicing for a 3D scene of large dimension, that is to say greater than 500 m×500 m. The 3D instantaneous mosaics obtained exhibit deformations and are limited in angular extent (typically <30°) or spatial extent.

The assembling of the mosaics is complex when the terrain is 3D and the final result does not conform to the geometry of the scene.

Images