Method and system for merging recorded measurements

Abstract

Method for merging recorded measurements, comprising the following steps: Acquiring depth measurements (120) from a scene (105) as a sequence of frames with a depth sensor (110); Acquiring intensity measurements (125) of the scene (105) as a sequence of images with a camera (115), wherein the resolution of the depth sensor (110) is lower than the resolution of the camera (115), and further comprising the following computer-assisted steps; Search (130) for similar locations in several temporally adjacent frames of depth measurements (120); Grouping (130) the similar locations into blocks using the intensity measurements (125); Increasing (140) the resolution of blocks using rank penalty to obtain blocks of higher resolution; Creating (145) a depth image (150) with a resolution higher than the sensor's resolution from the higher resolution blocks; and Repeat (160) the computer-assisted steps until a termination condition is met; where the creation (145) minimizes a cost function that combines a quadratic data fidelity term and a regularizer that controls an error between the depth measurements (120) and the depth image (150); and where the regularizer penalizes one rank of depth image (150).

Description

Technical area

[0001] The present invention relates to sensing systems and methods, and more particularly to merging acquired measurements output by low-resolution depth sensors and high-resolution optical cameras. Background to the state of the art

[0002] Document SCHWARZ, Sebastian; SJÖSTRÖM, Märten; OLSSON, Roger: Temporal consistent depth map upscaling for 3DTV. In: Proceedings / three-dimensional image processing, measurement (3DIPM), and applications 2014, 5 February 2014, Vol. 9013, Art.-Nr. 901302, 8 pp. - ISBN 978-0-8194-9930-1 describes a method for increasing the resolution of depth maps.

[0003] One of the major challenges in computer vision applications is acquiring high-resolution depth maps of scenes. A number of common tasks, such as target reconstruction, robot navigation, and automotive driver assistance, can be significantly improved by supplementing intensity data from optical cameras with high-resolution depth maps. However, with recent sensor technology, directly acquiring high-resolution depth maps is very costly.

[0004] The cost and limited availability of such sensors limit the capabilities of computer vision systems and hamper the implementation of methods based on high-resolution depth mapping. Thus, a number of methods provide numerous alternatives for increasing the spatial resolution of the measured depth data.

[0005] One of the best-known and widely used classes of techniques for improving the spatial resolution of depth data is guided depth super-resolution. Such techniques typically acquire depth maps from the scene using a high-resolution depth sensor and optical images using a high-resolution optical camera. The data acquired by the camera is then used to super-resolve a low-resolution depth map. These techniques exploit the property that both media share common features, such as edges and common structural changes. Therefore, such features in the optical camera data provide information and guidance that significantly improve the super-resolution depth map.

[0006] In the past, most such methods were applied to a single optical image and low-resolution depth map. However, for most practical applications of such methods and systems, a video from the optical camera and a sequence of snapshots of the depth maps are usually acquired.

[0007] One approach models the co-occurrence of edges in depth and intensity using Markov Random Fields (MRFs). An alternative approach is based on joint bilateral filtering, where intensity is used to determine filter weights. Bilateral filtering can be refined by incorporating local depth statistics. Another approach uses geodesic distances to determine the weights. This approach can be extended to dynamic sequences to compensate for different data rates in the depth and intensity sensors. A guided image filtering approach can improve edge preservation.

[0008] Recently, rarity-promoting regularization, a key component of compressed sensing, has delivered even more dramatic improvements in depth super-resolution. For example, improvements have been demonstrated by combining dictionary learning and sparse coding techniques. Another technique relies on weighted generalized total variation (TGV) regularization to impose a piecewise polynomial structure on the depth.

[0009] The conventional MRF approach can be combined with an additional term that supports transformation-domain depth rarity in a form of analysis. One method leverages the MRF model to jointly segment objects and recover higher-quality depth. Depth super-resolution can be achieved by taking multiple snapshots of a static scene from slightly offset views and merging the measurements using the rarity of the weighted gradient depth.

[0010] Many natural images contain repetitions of similar patterns and structures. State-of-the-art image denoising methods, such as nonlocal means (NLM) and block matching and 3D filtering (BM3D), take advantage of this redundancy by processing the image as a structured collection of locations. The NLM formulation can be extended to even more generalized inverse problems using specific NLM regularizers. Similarly, a variational approach can be used for general BM3D-based image reconstruction. In the context of guided depth super-resolution, NLMs are used to reduce the noise component in the estimated depth. Another method combines a block matching procedure with low-rank constraints to improve the resolution of a single depth image. Summary of the invention

[0011] This object is achieved by subject matter having the features according to the independent claims. Advantageous embodiments of the invention are the subject of the figures, the description, and the dependent claims.

[0012] The spatial resolution of depth sensors is often significantly lower than that of conventional optical cameras. Embodiments of the invention improve the resolution of depth images by using higher-resolution optical intensity images as side information. In particular, the embodiments fuse measurements output by low-resolution depth sensors and high-resolution optical cameras.

[0013] Incorporating temporal information into videos can significantly improve results. In particular, the embodiments improve depth resolution by exploiting spatiotemporal redundancy features in depth and intensity images using motion-adaptive low-rank regularization. Results confirm that the embodiments can significantly improve the quality of the estimated high-resolution depth image. This approach can be a first component in methods and systems that employ image processing techniques based on high-resolution depth information.

[0014] A key insight of the invention is based on the realization that information about a particular frame of depth measurements is replicated in some form in temporally adjacent frames. Therefore, frames can be exploited over time to super-resolve the depth map. A challenge lies in determining this information in the presence of scene, camera, and object motion between multiple temporally adjacent frames.

[0015] Another challenge with incorporating time into depth estimation is that depths can vary significantly between frames. This results in abrupt variations in depth values ​​across the temporal dimension and can lead to significant degradation of the result quality. Therefore, it is important to compensate for motion during estimation.

[0016] To this end, the embodiments exploit space-time similarities in the measurements using motion-adaptive regularization. Specifically, the method searches for similar depth locations, which are grouped into blocks, and the blocks are super-resolved and regularized using a rank penalty as a priority constraint. Brief description of the drawings Fig. 1 is a block diagram of a method and system for fusing acquired measurements according to embodiments of the invention. Fig. 2 is a schematic representation of block search and grouping within a time-space search area according to embodiments of the invention. Fig. 3 shows the v-shrinkage operator according to embodiments of the invention. Description of embodiments

[0017] As in Fig. 1, embodiments of the present invention fuse measurements output by low-resolution depth sensors and high-resolution optical cameras to improve the resolution of a depth image.

[0018] The embodiments create a high-resolution depth image by minimizing a cost function that combines a data fidelity term and a regularizer. Specifically, a quadratic data fidelity term is imposed that governs the error between the measured and estimated depth values. The regularizer groups similar depth locations from multiple frames and penalizes the rank of the resulting depth image.

[0019] Because locations from multiple frames are used, the present method is implicitly motion-adaptive. Thus, depth estimates are improved by effectively combining measurements from multiple views of the scene. Overview of the procedure and system

[0020] Depth measurements 120 of a scene 105 are acquired using a depth sensor 110; for example, the depth sensor can be a light radar (LIDAR) sensor. Furthermore, intensity measurements 125 of the scene 105 are acquired using a camera 115; for example, the camera can be a high-resolution optical video camera. A resolution of the depth sensor is lower than a resolution of the camera sensor. The sensor and the camera are calibrated.

[0021] In computer-assisted steps, the depth measurements are searched for similar locations, which are then grouped into blocks using the intensity measurements. A search for similar locations is performed in several temporally adjacent frames of the depth measurements. The details of the search and grouping are described below with reference to Fig. 2 explained.

[0022] The resolution of the blocks is increased using priority constraints 140 to create higher-resolution blocks. Then, the higher-resolution blocks are used to create a high-resolution image 150 145. That is, the resolution of the depth image is higher than the resolution of the depth sensor. The computer-assisted steps are repeated until a termination condition is met, e.g., a predetermined number of times, convergence at the resolution, or the end of the measurements. Problem formulation

[0023] The present method and system acquire the depth measurements {ψ t} t∈ [1, ..., T] of the scene. Each measurement is represented as a subsampled version of a higher resolution depth image ϕ t ∈ ℝ N using a subsampling operator H t The ultimate goal is to obtain the high-resolution depth image Φ tfor all t, where t are temporal indices for the frames of depth measurements.

[0024] In this description of the embodiments, N is used to indicate the number of pixels in each frame, T is used to indicate the number of temporal frames, and M is used to indicate the total number of depth measurements. Furthermore, ψ ∈ ℝ M a vector of all measurements, ϕ ∈ ℝ NT denotes the complete sequence of high-resolution depth images, and H ∈ ℝ M×NT denotes the complete subsampling operator.

[0025] In addition, the intensity measurements 125 are stored as a sequence of high-resolution intensity images X ∈ ℝ NT acquired using the camera.

[0026] Using the depth measurements and intensity measurement, a forward model for the depth recovery problem ψ=Hϕ+e, where e ∈ ℝM Measurement noise. Therefore, the present task is to recover and create the high-resolution depth image 150 based on the measurements ψ and X, and the sampling operator H.

[0027] The depth estimation task is considered as an optimization problem ϕ^=arg minϕ∈ℝNT{12‖ψ−Hϕ‖l22+∑p=1PR(Bpϕ)}, formulated, whereby 12‖ψ−Hϕ‖l22 enforces data fidelity and ∑p=1PR(Bpϕ) is a regularization term that imposes prior constraints on the depth measurements.

[0028] The regularization term is formed by creating groups of locations from each frame in the depth measurements. In particular, a first operator B p , for each group of locations p ∈ [1, ..., P], where P is the number of such created locations. The operator extracts L locations of size B pixels from the frames in the depth measurements ϕ.

[0029] Fig. Figure 2 shows the block search and grouping within a space-time search area 210. The figure shows frames 201, 202, and 202, respectively, at times t-1, t, and t+1. The frames contain various features 205.

[0030] The search area in the current frame t is centered at a reference point 220. The search is performed in identical window positions in several temporally adjacent frames. Similar locations are grouped to form a block β p = B p ϕ. Each block β p = B p ϕ ∈ ℝ B×L is obtained by first selecting the reference location and then finding L - 1 similar locations within the current frame as well as neighboring time frames.

[0031] To determine similarity and group similar locations, intensity measurements in the optical image are used as a guide. To reduce the computational complexity of the search, the search is restricted to a fixed-size space-time window surrounding the reference location 220. The same block search and grouping is performed for all space-time frames by moving the reference location and considering overlapping locations in each frame. Therefore, each pixel in the measurements ϕ can contribute to multiple blocks.

[0032] The adjuncts BpT by B p corresponds to replacing locations in the block at their original positions in the depth measurements ϕ. The adjoint satisfies the following property ∑p=1PBpTBp=R, where R = diag(r1, ..., r N ) ∈ ℝ NT×NT and r n the total number of references for the n-th pixel by the matrices {Bp} p = 1,...P Therefore, the depth measurements ϕ can be calculated with a view to over-complete reproduction using the blocks ϕ=R−1∑p=1PBpTBpϕ be expressed. Rank regularization

[0033] Each block, represented as a matrix, contains several similar locations, i.e., similar columns. Therefore, the matrix is ​​expected to have a low rank, making the rank a natural regularizer for the problem R(β)=rank(β),(β∈ℝB×L) is made.

[0034] By finding a low-rank solution to (2), the similarity of the corresponding blocks is exploited to achieve super-resolution while enforcing consistency with the acquired intensity measurements. However, the rank regularizer (5) is of little practical interest because its direct optimization is intractable. One approach is to convexify the rank by using the nuclear norm: R(β)=λ‖β‖∗≜λ∑k=1min(B,L)σk(β), is replaced, where σ k (β) is the k-th largest singular value of β, and λ > 0 is a parameter controlling the amount of regularization.

[0035] In addition to its convexity, the nuclear norm is an attractive penalty for optimization because the nuclear norm has a closed-form proximal operator: proxλ‖⋅‖∗(ψ)≜arg minβ∈ℝB×L{12‖ψ−β‖F2+λ‖β‖∗}=uηλ(σ(ψ))vT, where ψ = uσv T is a singular value decomposition (SVD) of ψ, and ηλ is a soft threshold function applied to the diagonal matrix σ.

[0036] It is known that non-convex regularizers consistently outperform nuclear norms by providing better denoising capability without loss of important signal components. The following non-convex generalization is proposed for the nuclear norm R(β)=λGλ,v(β)≜λ∑k=1min(B,L)gλ,v(σk(β)), applied, see, for example, R. Chartrand, “Nonconvex splitting for regularized Low-Rank + Sparse decomposition. IEEE Trans. Signal Process., 60(11):5810-5819, November 2012.

[0037] The scalar function g λ,v fulfilled minx∈ℝ{12|x−y|2+λgλ,v(x)}=hλ,v(x), where h λ,v the v-Huber function hλ,v(x)≜(|x|22λif|x|<λ1 / (2−v)|x|vv−δif|x|≥λ1 / (2−v),with δ ≜ (1 / v−1 / 2)λv / (2−v).

[0038] Although g λ,vis non-convex and has no closed-form formula, its proximal operator does not allow a closed-form expression proxλGλ,v(ψ)≜arg minβ∈ℝB×L{12‖ψ−β‖F2+λGλ,v(β)}=uTλ,v(σ(ψ))VT, where Tλ,v is a pointwise v-shrinkage operator, defined as Tλ,v(x)≜max(0,|x|−λ|x|v−1)x|x|.

[0039] Fig. 3 is a graph of the v-shrinkage operator Tv,λ for a fixed λ = 1 at two values ​​of V. For V = 1, the V-shrinkage (12) is equivalent to conventional soft thresholding. As V → 0, the operator approaches hard thresholding, which is similar to principal component analysis (PCA) in the sense that the operator retains only the significant principal components.

[0040] Therefore, the regularizer (8) is a computationally solvable alternative to rank penalty. Although the regularizer is non-convex, it can still be efficiently optimized due to the closed form of its proximal operator. Note that due to the non-convexity of the regularizer, for η < 1, it is difficult to theoretically guarantee global convergence. However, it has been empirically observed that the present methods converge reliably over a wide range of examples. Iterative optimization

[0041] To solve the optimization problem (2) using the rank regularizer (8), the present notation is extended by defining an operator B≜(B1,…,BP) and a vector β≜Bϕ=(β1,…,βP) simplified.

[0042] The minimization is performed using an Augmented Lagrange (AL) method. In particular, critical points of the following AL L(ϕ,β,s)≜12‖ψ−Hϕ‖l22+∑p=1PR(βp)+ρ2‖β−Bϕ‖l22+sT(β−Bϕ) =12‖ψ−Hϕ‖l22+∑p=1PR(βp) +ρ2‖β−Bϕ+s / ρ‖l22−12ρ‖s‖l22, where ρ > 0 is the quadratic parameter, and S is the dual variable that imposes the constraint β = Bϕ.

[0043] Conventionally, the AL method solves (2) by alternating between a joint optimization step and an update step as (ϕk,βk)←arg minϕ∈ℝNT,β∈ℝP×B×L{L(ϕ,β,sk−1)} sk←sk−1+ρ(βk−Bϕk).

[0044] However, the joint minimization in (15) is typically computationally intensive. To reduce the complexity, (15) is separated into a sequence of simpler steps using the alternating direction multiplier method (ADMM).

[0045] These steps are as follows ϕk←arg minϕ∈ℝNT{L(ϕ,βk−1,sk−1)}, βk←arg minβ∈ℝP×B×L{L(ϕk,β,sk−1)}, sk←sk−1+ρ(βk−Bϕk).

[0046] By ignoring the terms that do not depend on the depth measurements ϕ, (17) results in solving a quadratic equation ϕk←arg minϕ∈ℝNT{12‖ψ−Hϕ‖l22+ρ2‖Bϕ−zk−1‖l22} ←(HTH+ρBTB)−1(Hψ+ρBTzk−1), where zk−1≜βk−1+sk−1 / ρ. Solving this quadratic equation is efficient because the inversion is performed on a diagonal matrix. Similarly, (18) is solved by βk←arg minβ∈ℝP×B×L{ρ2‖β−yk‖l22+∑p=1PR(βp)},mityk≜Bϕk−sk−1 / ρ.

[0047] This step can be performed by applying a proximal operator as a block βpk←prox(λ / ρ)Gλ,v(Bpϕk−spk−1 / ρ), for all p ∈ [1, ..., P]. Simplified procedure

[0048] The above iterative optimization procedure can be significantly simplified by decoupling the enforcement of data fidelity from the enforcement of rank-based regularization. This simplified procedure reduces computational complexity while unifying the estimation across the entire space-time depth measurements.

[0049] Due to the inhomogeneous distribution of pixel references generated by matching across the image using a single regularization parameter penalty, pixels with a large number of references are heavily penalized. The resulting non-uniform regularization makes the method potentially oversensitive to the choice of the parameter λ. Instead, the simplified method βpk←arg minβp∈ℝB×L{12‖βp−Bpϕk−1‖F2+R(βp)} ϕk←arg minϕ∈ℝNT{12‖ψ−Hϕ‖l22+ρ2‖ϕ−ϕ˜k‖l22} where ϕ˜k≜R−1BTβk and λ > 0 is the regularization and ρ > 0 are the quadratic parameters.

[0050] To solve (23) the proximal operator βpk←proxλGλ,v(Bpϕk−1), for all p ∈ [1, ..., P]. Then (24) reduces to a linear step ϕk←(HTH+ρl)−1(HTψ+ρϕ˜k).

[0051] There are significant similarities between iterative optimization and simplified methods. The main differences are that the dual variable S and the simplified quadratic subproblem are eliminated (20). Effect of the invention

[0052] The embodiments provide a novel motion-adaptive method and system for generating super-resolution depth images. The method searches for similar locations from multiple frames, which are grouped into blocks, which are then super-resolved using a rank regularizer. This approach can generate high-resolution depth images from low-resolution depth measurements. Compared to conventional techniques, the method preserves temporal edges in the solution and effectively mitigates noise in practical configurations.

[0053] While the present method exhibits higher computational complexity than conventional approaches that process each frame individually, it allows for the incorporation of a highly effective regularization to stabilize the inverse problem associated with super-resolution. The method enables efficient computation and straightforward implementation by reducing the problem to a sequence of straightforward operations. Results demonstrate the significant benefits of incorporating time and motion adaptivity into inverse problems for depth estimation.

[0054] Key contributions include the provision of a new formulation for guided depth resolution incorporating temporal information. In this formulation, the high resolution depth is determined by solving an inverse problem that minimizes cost. This cost includes a quadratic data fidelity term and a motion-adaptive regularizer based on a low-rank penalty for groups of similar locations.

[0055] Two optimization strategies are presented to solve the estimation problem at hand. The first approach is based on exact cost optimization using the alternating direction multiplier method (ADMM). The second approach uses a simplified procedure that alternates between enforcing data consistency and low-order penalty.

Claims

[1] A method for merging acquired measurements, comprising the steps of: Acquiring depth measurements (120) from a scene (105) as a sequence of frames with a depth sensor (110); Acquiring intensity measurements (125) of the scene (105) as a sequence of images with a camera (115), wherein a resolution of the depth sensor (110) is lower than a resolution of the camera (115), and further comprising the following computer-aided steps: Searching (130) for similar locations in several temporally adjacent frames of the depth measurements (120); grouping (130) the similar locations into blocks using the intensity measurements (125); increasing (140) a resolution of the blocks using rank penalty to obtain higher resolution blocks; Creating (145) a depth image (150) with a resolution higher than the resolution of the sensor from the higher resolution blocks; and Repeating (160) the computer-assisted steps until a termination condition is reached; wherein the creating (145) minimizes a cost function that combines a quadratic data fidelity term and a regularizer that regulates an error between the depth measurements (120) and the depth image (150); and where the regularizer penalizes one rank of the depth image (150). [2] The method of claim 1, wherein the depth sensor (110) and the camera (115) are calibrated. [3] The method of claim 1, wherein the depth sensor (110) is a light radar (LIDAR) sensor. [4] The method of claim 1, wherein the camera (115) is a video camera. [5] The method of claim 1, wherein the depth image (150) is motion adaptive by combining measurements from multiple views of the scene (105). [6] The method of claim 1, wherein the searching and grouping (130) uses a search area centered at a reference location and considers overlapping locations in each frame. [7] A method for merging acquired measurements, comprising the steps of: Acquiring depth measurements (120) from a scene (105) as a sequence of frames with a depth sensor (110); Acquiring intensity measurements (125) of the scene (105) as a sequence of images with a camera (115), wherein a resolution of the depth sensor (110) is lower than a resolution of the camera (115), and further comprising the following computer-aided steps: Searching (130) for similar locations in several temporally adjacent frames of the depth measurements (120); grouping (130) the similar locations into blocks using the intensity measurements (125); increasing (140) a resolution of the blocks using rank penalty to obtain higher resolution blocks; Creating (145) a depth image (150) with a resolution higher than the resolution of the sensor from the higher resolution blocks; and Repeating (160) the computer-assisted steps until a termination condition is reached; wherein the creating (145) minimizes a cost function that combines a quadratic data fidelity term and a regularizer that regulates an error between the depth measurements (120) and the depth image (150); and where the regularizer uses a v-shrinkage operator. [8] The method of claim 1 or 7, wherein the cost function is minimized using an augmented Lagrange method. [9] The method of claim 8, wherein the augmented Lagrange method uses an alternating direction method of multipliers. [10] System for merging acquired measurements, comprising: a depth sensor (110) configured to acquire depth measurements (120) from a scene (105) as a sequence of frames; a camera (115) configured to acquire intensity measurements (125) from the scene (105) as a sequence of images, wherein a resolution of the depth sensor (110) is lower than a resolution of the camera (115); and a processor configured to search (130) for similar locations in a plurality of temporally adjacent frames of the depth measurements (120), group (130) the similar locations into blocks using the intensity measurements (125), increase (140) a resolution of the blocks using rank penalty to obtain higher resolution blocks, such that a depth image (150) having a resolution higher than the resolution of the sensor is created from the higher resolution blocks (145); wherein the processor is configured to minimize a cost function that combines a quadratic data fidelity term and a regularizer that regulates an error between the depth measurements (120) and the depth image (150); and where the regularizer penalizes one rank of the depth image (150). [11] System for merging acquired measurements, comprising: a depth sensor (110) configured to acquire depth measurements (120) from a scene (105) as a sequence of frames; a camera (115) configured to acquire intensity measurements (125) from the scene (105) as a sequence of images, wherein a resolution of the depth sensor (110) is lower than a resolution of the camera (115); and a processor configured to search (130) for similar locations in a plurality of temporally adjacent frames of the depth measurements (120), group (130) the similar locations into blocks using the intensity measurements (125), increase (140) a resolution of the blocks using rank penalty to obtain higher resolution blocks, such that a depth image (150) having a resolution higher than the resolution of the sensor is created from the higher resolution blocks (145); wherein the processor is configured to minimize a cost function that combines a quadratic data fidelity term and a regularizer that regulates an error between the depth measurements (120) and the depth image (150); and where the regularizer uses a v-shrinkage operator.

Images