Methods for determining the distance between objects and systems

Abstract

One aspect of the invention relates to a method for determining the distance of objects (10) in a scene (12) using at least one optical detection device (14) and a processor, in which several 2D images of the scene (12) are captured from different viewpoints with the at least one optical detection device (14), wherein a neural network is trained to model the radiation properties and the density of the scene (12) as a function of two spatial coordinates (X, Y) and a viewpoint (ϕ), wherein captured stripes (A) from the captured 2D images and the associated camera positions and orientations are fed into the neural network, the neural network is trained by comparing stripes (B) predicted by volume rendering with the captured stripes (A) and adjusting the parameters of the neural network so that the difference is minimized.Depth information for each pixel from the predicted strips (B) is determined by shooting a beam through each pixel of the strip to calculate the density values ​​at different points and selecting the position with the highest density as the depth information of the corresponding pixel; and a depth map for the strip is generated, representing the distances from the optical detection device (14) to the surfaces of the objects (10) in the scene, from which a distance (r) to the objects (10) is determined. The invention further relates to a system.

Description

[0001] The invention relates to a method for determining the distance between objects according to the preamble of claim 1. Furthermore, the invention relates to a system.

[0002] Determining distances to surrounding obstacles or other dynamic objects is an essential component of modern vehicles. Various sensor modalities are used for this purpose; for example, radar and lidar sensors are primarily used during driving, while ultrasonic sensors are used during parking maneuvers.

[0003] There are various methods or techniques that can estimate depth information from camera images. For example, multiple cameras with different poses or positions can be used (e.g., stereo cameras). Alternatively, images from several consecutive time steps can be processed to estimate depth (Depth from Motion).

[0004] In recent times, however, one method has proven particularly promising in computer vision for obtaining 3D information from images: Neural Radiance Fields (NeRFs).

[0005] The publication “MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving” by WU, Zirui, et al., arXiv preprint arXiv:2307.15058v1, 2023, reveals a photorealistic simulator for autonomous driving based on Neural Radiation Fields (NeRFs). This approach involves modeling foreground objects, such as other vehicles, and the background of a scene separately using their own neural networks. The goal is to create a photorealistic simulation environment for training and testing algorithms. However, this method relies on the full complexity of conventional NeRF models and does not address the problem of high computational intensity for simple distance determination in real-time applications.

[0006] The publication “Uc-nerf: Neural radiance field for under-calibrated multi-view cameras in autonomous driving” by Cheng, Kai, et al., arXiv preprint arXiv:2311.16945v2, 2023, addresses the application of neural radiance fields (NeRFs) to under-calibrated multi-camera systems, such as those found in autonomous vehicles. The focus is on improving robustness and image quality in inadequately calibrated cameras. Solutions are proposed for problems such as color inconsistencies between cameras and inaccurate camera poses that can be caused by vibrations. These include, for example, layer-based color correction, spatially and temporally limited pose refinement, and the generation of additional synthetic training views (virtual warping). These measures aim to improve reconstruction quality under non-ideal input conditions.Simplifying the NeRF model itself to reduce the computational load is not proposed.

[0007] To address the problem of requiring many input images, the publication "Dense Depth Priors for Neural Radiance Fields from Sparse Input Views" by Roessle, Barbara, et al. In: 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022. pp. 12882-12891 proposes a method for improving the quality of NeRF reconstructions with only a small number of input images. This involves converting sparse depth points, obtained from a structure-from-motion process within a camera pose estimation, into dense depth maps using a neural network (a so-called depth completion network). These depth maps, including an estimated uncertainty, are then used to control the NeRF optimization through an additional loss function and targeted sampling.Additional geometric information is added to the NeRF model, which should increase the data efficiency and accuracy of the reconstruction.

[0008] The object of the invention is to further develop a method for determining distance so that a simplified and fast determination of the distance is possible without requiring high processor power.

[0009] This problem is solved by means of a method with the features of claim 1 and by means of a system according to the invention. Advantageous embodiments and further developments can be found in the dependent claims and in the description.

[0010] One aspect of the invention relates to a method for determining the distance of objects in a scene by means of at least one optical detection device, e.g., at least one camera, preferably by means of a plurality of optical detection devices or cameras and at least one processor or electronic computing device coupled to the cameras, which is, for example, configured to perform at least partial Neural Radiance Fields (NeRFs).

[0011] Partial Neural Radiance Fields (NeRFs) is a fully networked neural network capable of generating novel views of complex 3D scenes based on a subset of 2D input images. It is trained to generate views of a scene by interpolating between input images to render a complete scene. A NeRF network is therefore trained to map directly from the viewing direction and spatial position (5D input) to the opacity and color (4D output), using volume rendering to create new views. NeRF is a computationally intensive algorithm, and processing complex scenes can take hours or even days.

[0012] According to the invention, it is provided that several 2D images of a scene are captured from different viewpoints with the at least one optical detection device or camera, wherein the position and orientation of the camera are known for each image.

[0013] In order to solve the problem of the invention and to enable a simplified and fast determination of the distance without requiring high processor power, it is planned to at least partially reduce the complexity of the Neural Radiance Field.

[0014] For this purpose, the inventive method provides for the training of a neural network to model the radiation properties (radiance) and the density of the scene as a function of two spatial coordinates (X, Y) and a viewing angle (phi). Captured strips from the acquired 2D images and the associated camera positions and orientations are fed into the neural network.

[0015] Subsequently, the neural network is trained by comparing stripes predicted by volume rendering with the captured stripes and adjusting the parameters of the neural network to minimize the difference between these stripes.

[0016] Accordingly, depth information for each pixel is determined from the predicted stripes by shooting a beam through each pixel of the stripe to calculate the density values ​​at different points and to choose the position with the highest density as the depth information of the corresponding pixel.

[0017] Subsequently, a depth map is generated for the strip, representing the distances from the camera to the surfaces of the objects in the scene, from which a distance to the objects can in turn be determined.

[0018] The main difference from previously known NeRF networks lies in the reduction of complexity. Conventional NeRF networks are trained to transform five dimensions—namely, the 3D positions x, y, z and the viewing angles theta, phi—into rays with colors R, G, B and a density sigma. It therefore operates with a five-dimensional input space.

[0019] At the same time, conventional NeRF always generates an entire 2D image subsequently using volume rendering.

[0020] Accordingly, it is proposed to use a NeRF that only allows the transformation from two of the 3D positions x, y and from one of the viewing angles phi to the ray with colors R, G, B and density sigma. This restricts the input space to only 3 dimensions. The notations used here apply to a ray in the Fig. 1. Coordinate system shown.

[0021] To enable this transformation, it is intended that only cameras located at a similar height (z) are used, e.g., a height difference could be less than 10 cm.

[0022] Furthermore, images of a scene are taken at this height using one or more cameras.

[0023] The mounting position (intrinsic + extrinsic) of the camera(s) is therefore known. Consequently, the horizontal image line (strip) that runs parallel to the ground plane can be extracted from each image.

[0024] It should be noted that when using a fisheye camera, this effect must be taken into account, and a horizontal image line parallel to the ground plane must be approximated in a preprocessing step. In the simplest case, the camera is positioned exactly parallel to the ground plane. In this case, the middle line of the image can be extracted.

[0025] In a further embodiment of the invention, filtering is performed. Instead of extracting only one image line (strip), namely the one that is exactly parallel to the ground plane, N adjacent image lines (strips) are also extracted. These are then averaged column by column. This means that a single image line (strip) still results, but the values ​​of each pixel are now averaged over several columns. This leads to stabilization, particularly when a larger height difference between the cameras is permitted.

[0026] The result is therefore many image lines (stripes) that are captured at a similar height.

[0027] Accordingly, the plan is to train a modified version of NeRF: From the point / position where an image was captured, rays are emitted in the relevant direction, and different points in time are sampled. Each sampled point is then simply a combination of x, y, and phi.

[0028] It is therefore intended that z is omitted. z is used here to represent the vertical direction / height. z is omitted because every point of the emitted ray lies on the same horizontal plane. This was made possible by extracting only image lines (strips) that are parallel to the ground plane.

[0029] Furthermore, theta is omitted. Here, theta represents the polar angle between a point O of the emitted beam and another point P from which the beams were emitted. This was made possible by extracting only image lines (strips) parallel to the ground plane. Therefore, each point can only be illuminated from exactly one polar angle.

[0030] As with conventional NeRF, for each sampled point, now only x, z, phi, a color R, G, B and a density or opacity are predicted.

[0031] Volume rendering can then be used to derive a single-line image.

[0032] The generated single-line image (image line (strip)) is compared to the original image using an error function, e.g., L2 loss. The volume rendering function is derivable, meaning that during NeRF training, the error (result of the error function) between the generated single-line images and the true single-line image can be used as a training signal for the NeRF network.

[0033] The following steps can now be taken during the application: - An arbitrary novel point (= object for distance measurement) is selected within a scene. It is important to note that the desired point must be located at the same height (z) as the optical detection devices or cameras (a single plane). Rays parallel to the ground plane (theta = constant) are emitted from the novel point in the relevant direction, and different time points are recorded. Each recorded point is then simply a combination of x, y, and phi. - The trained NeRF is used to predict a color and density or opacity for each sampled point. The density or opacity can be used to estimate the distance to an obstacle: For example, the density or opacity of a beam can be integrated over time. Integrating from 0 to a specific time can be interpreted as a probability, indicating how likely an object / obstacle is to already be present in the corresponding beam. This is therefore a learned distance sensor and thus a method for determining the distance to objects, made possible by the inventive method. The probability can also be converted into a binary indicator via a threshold value.

[0034] Further advantages, features, and details of the invention will become apparent from the following description of a preferred embodiment and from the drawings. The features and combinations of features mentioned above in the description, as well as those mentioned below in the figure description and / or shown in the figures alone, can be used not only in the combinations specified, but also in other combinations or individually, without departing from the scope of the invention.

[0035] This shows: Fig. 1 Spherical coordinates in a sphere to illustrate a method for determining distance; Fig. 2. Spherical coordinates within the sphere for a more precise illustration of the method for determining distances, particularly with respect to a plane; and Fig. 3 a top view of a traffic scene 22, which is intended to illustrate a typical application of the procedure,

[0036] In the figures, identical or functionally equivalent elements are provided with the same reference symbols.

[0037] Fig. Figure 1 shows spherical coordinates in a sphere to illustrate a method for determining distances, which is particularly applicable for orientation in describing the advantages of the present invention.

[0038] Here, spherical coordinates are represented with the 3D positions x, y, z and the viewing angles ϑ (theta) and ϕ (phi). Furthermore, the origin O and the point P are shown, between which the distance r is depicted, representing the radius of the sphere shown. Additionally, a height d to the point relative to a plane spanned by x and y is shown.

[0039] Fig. 2 shows the from Fig. 1 known spherical coordinates in the sphere for a more precise illustration of the method for determining distance.

[0040] This diagram depicts two spherical coordinates with the 3D positions x and y and a viewing angle ϕ (phi). It also shows the origin O and the point P, between which the distance r is shown, representing the radius of the plane depicted here between x and y.

[0041] The method for determining the distance of objects 10 at point P in a scene 12 is carried out using at least one optical detection device 14 camera at point O and a processor for at least partial implementation of Neural Radiance Fields (NeRFs).

[0042] The plan is to capture several 2D images of scene 12 from different viewpoints using at least one optical detection device 14 or camera. This would mean that multiple spherical coordinates taken with different cameras and having different origins O would share a common point P of the object. The position and orientation of each camera are known for each image.

[0043] It is intended that a neural network will be trained to model the radiation properties (radiance) and the density of scene 12 as a function of the two spatial coordinates X, Y and the one viewing angle ϕ, whereby captured strips A from the captured 2D images and the associated camera positions and orientations are fed into the neural network.

[0044] The neural network is trained by processing predicted volumetric rendering data. Fig. Two unseen, virtual stripes B are compared with the captured stripes A, and the parameters of the neural network are adjusted so that the difference between stripes A and B is minimized.

[0045] Subsequently, depth information for each pixel from the predicted stripe B is determined by shooting a beam through each pixel of stripe B to calculate the density values ​​at different points (in this case object 10, point P) and choosing the position with the highest density as the depth information of the corresponding pixel.

[0046] Accordingly, a depth map is generated for strip B, which represents the distances from the camera to the surfaces of the objects (distance r) in the scene, from which a distance r to the objects 10 can be determined.

[0047] Fig.Figure 3 shows a top view of a traffic scene 22, which is intended to illustrate a typical application of the procedure, which can be implemented, for example, by motor vehicles 16, 18.

[0048] Several cameras 20a, 20b at the same height h, e.g., mirror cameras on the left (with a first height h1 and on the right with a second height h2), or the same camera (e.g., front camera), can capture single-line images of a traffic scene 22. Furthermore, a third, external camera 20c, linked to the processor 26 executing the process (e.g., also external to the vehicle), with its own third height h3 at the same height h as the other cameras 20a, 20b, can be used to capture additional images of the traffic scene 22. Thus, for example, h1 = h2 = h3 can be the case. This camera can be installed, for example, on another vehicle 16 or elsewhere. In particular, there should be no height difference of more than 10 cm between all cameras.

[0049] The images from all cameras 20a, 20b, 20c can be used to train a NeRF.

[0050] For example, rays can be emitted for a novel point (front of the hood) and the distance r to an obstacle or object 24 can be estimated using NeRF and volume rendering.

[0051] The special feature here is that the dimensionality of the NeRF optimization has been immensely reduced because the input space is now only 3 dimensional, and the number of rays to be emitted when generating novel scenes is also immensely reduced as a result.

[0052] The plan is to reduce the complexity of a Neural Radiance Field (NeRF) by restricting the optimization to lines parallel to the ground plane. This reduces the input space from 5 dimensions to 3. The NeRF is trained for a given scenario (e.g., parking) and then used as an efficient distance sensor.

[0053] The advantages of this approach are manifold; for example, NeRFs delivers results that are virtually unsurpassed in the generation of 3D information from images. The main weakness of NeRFs, namely their computational intensity, is significantly reduced. This enables flexible use in motor vehicles and beyond. Since cameras are already integrated, no additional sensors are required. This leads to cost savings because additional sensors, such as ultrasound, may be unnecessary. Otherwise, redundancy is provided.

Claims

[1] Method for determining the distance between objects (10) in a scene (12) using at least one optical detection device (14) and a processor, wherein: - several 2D images of the scene (12) from different viewpoints are captured with at least the optical detection device (14); characterized by , that - a neural network is trained to model the radiation properties and the density of the scene (12) as a function of two spatial coordinates (X, Y) and a viewing angle (ϕ), wherein - captured strips (A) from the captured 2D images and the associated camera positions and orientations are fed into the neural network; - the neural network is trained by comparing volume-rendered stripes (B) predicted by volume rendering with the captured stripes (A) and adjusting the parameters of the neural network to minimize the difference; - Depth information for each pixel from the predicted stripes (B) is determined by shooting a beam through each pixel of the stripe to calculate the density values ​​at different points and choosing the position with the highest density as the depth information of the corresponding pixel; and - a depth map for the strip, representing the distances from the optical detection device (14) to the surfaces of the objects (10) in the scene, is generated, from which a distance (r) to the objects (10) is determined. [2] Method according to claim 1, characterized by , that the generated images are filtered according to predefined parameters before being fed into the neural network. [3] Method according to claim 1 or 2, characterized by that the predicted stripes are averaged column by column. [4] Method according to any one of the preceding claims, characterized by, that a probability of the measured distance is calculated. [5] Method according to claim 4, characterized by , that the probability above a threshold is also converted into a binary indicator. [6] Method according to any one of the preceding claims, characterized by that the processor is external to the vehicle. [7] Method according to any one of the preceding claims, characterized by , that another external optical recording device is used to capture further images from a different perspective. [8] Method according to claim 7, characterized by , that when using an external optical detection device, a height survey is carried out first. [9] System for carrying out a method according to any one of the preceding claims 1 to 8, comprising all components required to carry out all method steps.

Images