Method and system of real-time 3D reconstruction of a scene

Abstract

Performing real-time 3D geometric reconstruction of a scene, comprises receiving (101) data comprising 2D positions of features tracked (e.g. using optical flow) in a video from a mobile system camera

Description

The present invention relates to a method and a system of real-time 3D reconstruction of a scene, adapted for automotive-related image processing applications. Structure-from-Motion is the process of estimating the tridimensional structure of a scene from a set of bidimensional images. In essence, SfM involves three main stages: extraction of features in images (e.g., points of interest, lines, objects, structures, as well as points, lines, and planes of objects and / or structures etc.) and matching these features between images; camera motion estimation (e.g., using relative pairwise camera positions estimated from the extracted features); and reconstruction of the 3D structure using the estimated motion and features (e.g., by optimizing the cost function called reprojection error). Several publications related to this topic is listed below: [1] Ozyesil, 0.; Voroninski, V.; Bash, R.; Singer, A. „A Survey of Structure from Motion", published in 9 May 2017, arXiv:1701.08493v2, [2] Texas Instruments, ..Vision Apps User Guide: Structure From Motion Application", https: / / software-dl.ti.com / jacinto7 / esd / processor-sdk-rtos-iacinto7 / latest / exports / do .cs / yision...gp.ps / docs^ [3] NVIDIA CORPORATION &AFFILIATES, ..Optical Flow Accelerator (0FA)“, October 2022, https: / / deveioper.nvidia.com / docs / drive / drive-os / latest / iinux / sdk / common / topics / nv [4] Wu. C.: "Towards Linear-Time Incremental Structure from Motion." published in 2013 International Conference on 3D Vision - 3DV 2013, Seattle. WA. USA. 2013, pp. 127-134, [5] Snavely, N.; Seitz, S. M.; Szeliski R. „Photo tourism: exploring photo collections in 3D“, July 2006, published in ACM Transactions on Graphics, Volume 25, Issue 3, pp. 835-846. A comprehensive overview of the Structure-from-Motion (SfM) approaches and difficulties is presented in [1], As seen there, in terms of structure estimation, most of the research effort is focused on bundle adjustment (techniques which aim to optimize the total reprojection error), with its direct and indirect subtypes. In practice, for embedded systems, dense optical flow accelerators implemented in hardware are often available (such as those described by [2] or [3], and others). These embedded accelerators have much higher performance than their software counterparts. This moves the processing bottleneck to the reconstruction stage, which makes bundle adjustment methods very expensive by comparison. As of runtime, improvements are often framed in terms of camera poses, where almost linear-time reconstruction through incremental SfM is possible, as disclosed in [4] and [5], However, when using camera poses as a measure of algorithmic complexity and bundle adjustment is still performed, the impact of the number of features is hidden. In the dense optical flow case, this cost is significant, even if the bundle adjustment is restricted to individual features. Comparatively, the solution proposed by [2] addresses the problem of a large number of reconstructed features; nevertheless, through the use of an 8-point algorithm, which handles only pairs of images, the accuracy gains from multi-view, long-baseline reconstruction are lost. Patent literature comprises also documents on inventions underlying structure-from-motion (SfM) techniques. US 10657389 B2 [6] discloses an automotive structure-from-motion (SfM) system which stores a number of image frames acquired from a vehicle-mounted camera in a frame stack according to a frame stack update logic. The SfM system detects feature points, generates flow tracks, and computes depth values based on the image frames, the depth values to aid control of the vehicle. The frame stack update logic selects a frame to discard from the stack when a new frame is added to the stack, and is changed from a „first in, first out" (FIFO) logic to „last in, first out" (LIFO) logic upon a determination that the vehicle is stationary. An optical flow tracks logic is modified based on that determination. The determination is made based on a dual threshold comparison to insure robust SfM system performance. EP 3655299 A1 [7] describes a method for ascertaining an optical flow based on an image sequence recorded by a camera of a vehicle. In the process at least one input signal, which represents vehicle surroundings and / or a driving situation of the vehicle identified using at least one sensor of the vehicle and / or a result of a previous ascertainment of the optical flow, and an image signal representing the image sequence are received. A time difference value is determined using the input signal. At least two individual images of the image sequence which are offset from one another by the time difference value are selected, using the image signal. Coinciding pixels in the individual images are detected to ascertain the optical flow using the coinciding further pixels. To resume, whenever a fast source of optical flow is available (such as in the case of hardware accelerator), optimization is, by comparison, the most computationally expensive operation (up to accuracy tradeoffs). Usual solution presented by prior art was to track fewer features or to use a method with much lower accuracy to fit the resource constraints. Therefore, the technical problem to be solved, in this context, is how to exploit slightly lower accuracy requirements for a considerable reduction in computational cost. It is an object of the present invention to solve the prior art deficiencies and to provide a computationally efficient method of 3D reconstruction of a scene from image data. It is another object of the present invention to provide a 3D reconstruction system able to balance the accuracy requirements with resource constrains. The above-mentioned and other objects are realized by means of the technical features and characteristics mentioned in the independent claims. Further advantageous embodiments are the subject-matter of the dependent claims. By implementing the invention, a lower runtime is obtained in comparison with bundle adjustment running on the same components; clear and lean design; position estimation can be run independently from accumulation. Further aspects of the invention are described below. It will be apparent to those skilled in the art that the above features of embodiments of the inventive method and system of 3D reconstruction of a scene from image data may also be used in conjunction with the following aspects of the invention and vice versa. Figures Figure 1 illustrates an embodiment of a mobile system configured to perform a real-time 3D geometric reconstruction of a scene, according to invention, Fig. 2 shows a flow chart of a real-time 3D geometric reconstruction of a scene method, according to invention. Detailed description For a better understanding of the principles of the present invention, embodiments of the invention will be explained in more detail below with reference to the figures. Like reference numerals are used in the figures for the same or equivalent elements and are not necessarily described again for each figure. Various embodiments described herein are generally directed to image processing techniques related to automotive applications. In image processing or computer vision field, the term “visual features” (simplified “features”) means a piece of information about the content of an image; in the context of this description, features are points, lines and planes of objects or structures present in images captured by a camera mounted on a mobile system, such objects and structures defining a scene in the surroundings of the mobile system. Referring now to Fig. 1, there is shown a diagram of a mobile system 10 configured to perform a real-time 3D geometric reconstruction of a scene, according to one embodiment. The mobile system 10 comprises at least one camera 1 for capturing, in real-time, a streaming sequence of image frames; a plurality of visual features is extracted and tracked from multiple image frames. Extracting and tracking the features over multiple image frames is made in a known manner (for example, by means of optical flow, either sparse or dense), and is not part of this invention. As a result, a set of the feature positions is provided, for example, expressed as a vector P with coordinates in camera image coordinate system: p_img = (x,y)T. Camera 1 is a monocular camera or an event camera (able to detect motion in a scene due to brightness change). At least one processor 2 is configured to receive the features data. A non-transitory computer readable storage medium 3 (i.e., a memory) is provided for storing data and data processing instructions. A 3D geometric perception estimator 4 is configured as a shared feature vector executable on the processor, the shared feature vector containing accumulation matrices of estimates, and a measure of inaccuracy of estimate, as defined. In other words, the accumulation matrix holds coefficients of the equation whose solution is the estimated 3D position of the tracked feature, namely Jacobian of 3D position for each tracked feature. The input data of one feature tracked in more than one image frame from the streaming sequence include 2D position of the feature in multiple image frames. The output of the 3D geometric perception estimator is a point cloud of reconstructed 3D features, expressed at least by their 3D positions. In another embodiment, in addition to the features data, the processor 2 is configured to receive also motion data measured by at least one motion sensor 5, and to add motion data of the features tracked in the image frames. For example, motion sensor 5 is an accelerometer, a gyroscope or a wheel speed sensor. In this case, the output of the 3D geometric perception estimator is a point cloud of reconstructed 3D features, expressed by their 3D positions and orientations. Tu sum up, once all the available data has been input to the estimator, the position, orientation, and additional quantities related to the features tracked in the image frames are determined and stored in the storage 3 for comparison with new estimate for tracking the features. Fig. 2 shows a flow chart of a real-time 3D geometric reconstruction of a scene method, according to invention. The method comprises the following steps: S101: receiving, from a camera of a mobile system, features data comprising 2D positions of features tracked in a streaming sequence of image frames, wherein the camera position is known. S102: providing, based on the received 2D positions of the tracked features, a measure of inaccuracy of the 3D position of each tracked feature. The minimum measure of inaccuracy corresponds to the most probable 3D position of the tracked feature. S103: minimizing the measure of inaccuracy by solving, in real time, an optimization problem with an accumulated linearized equation depending of the 2D position of each tracked feature, the camera position and a non-linear term linking the camera and each tracked feature 2D position; and S104: providing a current estimate of 3D position of each tracked feature, based on the received data and the computed measure of inaccuracy. Features data are not raw sensor data, but are extracted by means of an optical flow component, and provided as feature tracks. In some embodiments, such feature data are either sparse optical flow or dense map. The measure of inaccuracy is modelled by non-linear, probabilistic models under visual and geometric structure constraints, for example by a Gaussian model with a Gaussian probability distribution of the features identified in the streaming sequence of image frames. Further on, the minimization of the measure of inaccuracy employs linearized approximations of the weighted camera ray. For example, a linear approximation is applied by ignoring that some measurements have been made from a closer position and therefore are more accurate. The non-linear term linking the camera position and each tracked feature 3D position depends on the known camera position, provided by the camera, and a weighted camera ray including the distance between the camera position and the 3D position of each feature tracked in the image frame. The inputs to probabilistic model are described as such: point to reconstruct: P, with position vector (in camera image coordinate system) p = (x,y,zY; camera coordinate systems CCi: coordinate system rotation Ri, coordinate system origin <5i; and measured camera ray vector in CCi: pi. Equation below describes the optimization problem: N i=l wherein the terms are: p is 3D position of the feature (to optimize for), <5i is camera position, wt is weighted camera ray, including the distance term in the denominator. Minimization of inaccuracy (step S103) is performed by the following sub-steps: S1031: providing a shared feature vector containing an accumulation matrix for each tracked feature, the accumulation matrix holding coefficients of the equation whose solution is the estimated 3D position of the tracked feature, namely Jacobian of 3D position for each tracked feature, S1032: for each tracked feature, updating the accumulation matrix with the Jacobian of a current 3D position determined in the previous sub-step for a latest image frame, wherein non-linear terms in the Jacobian of the current 3D position are linearized and approximated with a constant value taken from a previous estimate. Or, in other words, the Jacobian is updated by keeping the position in the denominator constant with the value taken from the previous estimate. Sub-step 1031: solving the optimization problem, using the Jacobian term of 3D position for each tracked feature. In the following computations and approximations are described. Camera ray vector for P in camera coordinate system CCi: f = = R-i P 1 \p~ <5i| Under the explicit assumptions of uniform probability distribution for point location and some probability distribution for pt given rt, it is easier computationally to have a Gaussian distribution: P(Aln)~exp Here Oi is approximately pushforward / pullback of one pixel (which would be the inaccuracy of the ray from camera quantization). Possible variants: (a) P (Pi I n)~ exp (b) / (f; X Of P (A I exp -- Cross product is chosen for the most convenient form later on. In order to maximize the cost function nr=i computations and results are the following: By applying a Bayes filter: P(Pi\n) = P(Pi\n)P(rQ P(Pd And assuming an uniform probability for point location, which means constant P(fi) and P(pt). Hence, ]^[max (P^Ia) - ]^[max (P(aIt)) i=l i=l This yields a minimization of the optimization problem N S(p) = i=l HO)2) 9^-2 Cross product versions: Then, 10 N __( 2 hi Ip-Sil Requiring minimization (computations forx only, symmetrical in y, z): 0 = — Approximation in cross product: -> 2 Assuming |p|» |<5i|, it is possible to approximate |p - <5i| with |p|2, such that: N s(p) = i=l -> 2 ((P - $) x wj Ipl2 i=l Let note = xw^, then N vCpx^-uJ — i=l 1 N S(p) N ~ y (ywu - zwiiY - ui>x )2 + (zwiiX - xwiiZ - uiiY )2 + (xwkY - ywiiX - uiiZ )2 Z-i x2 + y2+z2 i=l Again, minimization with respect to x: 0 = „ / (CzwiiX - xwiiZ - + (xwiiY - ywix - uiiZ)wiiYXx2 + y2+z2) - \ I 2 2 2\ I 1=1 V ((ywu - zwi,Y - ui:Xy + (zwiiX - xwiiZ - uiiY)2 + (xwiiY - ywi x - / A first approximation: a less accurate cross product variant is obtained by ignoring that some measurements have been made from a closer position and therefore are more accurate. N i=l By applying minimization with respect to x: 0 = Yi=^Xzwi:X - xwiiZ - uiiYX-wiiZ) + (xwiiY - ywi x - uiiZ)wiiY) 0 = -zwi:X - + (xwiiY - ywix - 0 = x (iXi ((wi,zWi,z + wiiYwiiY ) - z^=1wiiXwiiZ)-y^=1wiiXwiiY) + (£i=1(uiiYwiiZ - uiiZwiiY Equation symmetric in y and z, it is solvable through Gauss-Jordan elimination. A second approximation: to recover some of the lost accuracy, it is possible to scale the vector through the estimate of |p - <5;|-2, which is the denominator in the original equation. This yields the same equation as above, but with a different definition of N S'W=^((p-<5,)xw;)2 i=l Sub-step 1032: for each tracked feature, the accumulation matrix is updated with the Jacobian term determined above for the latest image frame, keeping the position in the denominator constant with the value taken from the previous estimate. This provides the linearity (and thus additivity) of the method, hence the runtime speedup. By employing this method, there is no need to re-estimate the position of a tracked feature (by solving the linear equation of the Jacobian) at every image frame. Since the Jacobian is positive-definite, it is even possible to solve the equation through 5 Gaussian elimination (even without pivoting). As a result, the described method is capable of achieving an optimized runtime of 20ms per camera on an automotive processor (for example, TDA4) with 40k features on a microprocessor core (for example, ARM Cortex-A72) in comparison 10 with 30ms with 10k features on the same hardware. The invention is designed to be used in all applications requiring a large point cloud with limited computing power constraints. This includes autonomous mobile robots (AMR), autonomous unmanned aerial vehicles (UAV) or automatic guided vehicles 15 (AGV), as well as surveys where real-time I low-latency results are more important than accuracy. Bibliographic references Non-Patent Literature [1] Ozyesil, 0.; Voroninski, V.; Basri, R.; Singer, A. „A Survey of Structure from Motion", published in 9 May 2017, arXiv:1701.08493v2, DOI: 10.48550 / arXiv. 1701.08493 [2] Texas Instruments, ..Vision Apps User Guide: Structure From Motion Application", https; / / sgftware"dlj.,com / i^ cs / vision apps / docs / user quide / qroup apps cv demos app sfm.html, [3] NVIDIA CORPORATION &AFFILIATES, ..Optical Flow Accelerator (OFA)“, published in October 2022, https: / / devetoper.nvidia.com / docs / drive / drive-os / iatest / linux / sdk / common / topics / nv !P.gdia..undQ.rs^^ [4] Wu. C.: "Towards Linear-Time Incremental Structure from Motion." published in 2013 International Conference on 3D Vision - 3DV 2013, Seattle. WA. USA. 2013, pp. 127-134, DOI: 10.1109 / 3DV.2013.25 [5] Snavely, N.; Seitz, S. M.; Szeliski R. „Photo tourism: exploring photo collections in 3D“, July 2006, published in ACM Transactions on Graphics, Volume 25, Issue 3, pp. 835-846, DOI: 10.1145 / 1141911.1141964 Patent Literature [6] US 10657389 B2 [7] EP 3655299 A1 10 07 25

Claims

1. A method of performing real-time 3D geometric reconstruction of a scene by means of a processor, the method comprising:5 (101) receiving, from a camera of a mobile system, data comprising 2D positionsof features tracked in a streaming sequence of image frames, wherein the camera position is known,(102) providing, based on the received 2D positions of the tracked features, a measure of inaccuracy of the 3D position of each tracked feature, wherein the 10 minimum measure of inaccuracy corresponds to the most probable 3D position of the tracked feature,(103) minimizing the measure of inaccuracy by solving, in real time, an optimization problem with an accumulated linearized equation depending of the 2D position of each tracked feature, the camera position and a non-linear term linking 15 the camera and each tracked feature 2D position, wherein the minimizing (103)the accumulated linear equation is performed by the following sub-steps: (1031) providing a shared feature vector containing an accumulation matrix for each tracked feature, the accumulation matrix holding coefficients of the equation whose solution is the estimated 3D position of the tracked feature, namely20 Jacobian of 3D position for each tracked feature,(1032) for each tracked feature, updating the accumulation matrix with the Jacobian of a current 3D position for each image frame, wherein non-linear terms in the Jacobian of the current 3D position are linearized and approximated with a constant value taken from a previous estimate, and25 (104) providing a current estimate of 3D position of each tracked feature, based onthe received data and the computed measure of inaccuracy.

2. The method of preceding claim, characterized in that non-linear term linking the camera position and each tracked feature 2D position depends on 30 the known camera position, provided by the camera, and a weighted camera ray including the distance between the camera position and the 2D position of each feature tracked in the image frames.10 07 253. The method of claim ^characterized in that the measure of inaccuracy is modelled by non-linear, probabilistic models under visual and geometric structure constraints.5 4. The method of claims 1 and 3, c h a r a c t e r i z e d in t h a t the measure ofinaccuracy is modelled by a Gaussian model with a Gaussian probability distribution of the features identified in the streaming sequence of image frames.

5. The method of claim 1, c h a ra ct e r i z e d in t h a t the minimization (103) 10 of the measure of inaccuracy employs linearized approximations of the weighted camera ray.

6. A computer-readable medium in which instructions are embodied which when executed by a processor-based computing device perform the method of any of 15 claims 1-5.

7. A mobile system of performing real-time 3D geometric reconstruction of a scene by means of a processor, the system comprising:at least one camera (1) for capturing a streaming sequence of image frames and20 providing data of features tracked in the image frames;a processor (2) configured to receive the data comprising 2D positions of the features tracked in a streaming sequence of image frames, a non-transitory computer readable storage medium (3) for storing data and instructions; and25 a 3D geometric perception estimator (4) executable on the processor for processing the received data, wherein a tracked feature vector of the estimator contains accumulation matrices of estimates for providing a current estimate of 3D position of each tracked feature, characterized in that provision of 3D position of the tracked features is made according to the method of claim 1.

308. The mobile system according to claim 7, characterized in that the image sensor is a monocular camera, or an event camera.

9. The mobile system according to claims 7-8, c h a r a ct e r i z e d in that the mobile system is an automotive system, such as an autonomous mobile robot (AMR), autonomous unmanned aerial vehicle (UAV) or automatic guided vehicle (AGV).10 07 25

Images