Localization using neural radiance fields

Abstract

A method includes estimating, for a captured image and depth map, a preliminary pose of a setting the preliminary pose as an estimated pose, and repeatedly: generating a generated image and depth map using the estimated pose and a NeRF; calculating a loss function based on a difference between the captured image and the generated image and between the captured depth map and the generated depth map; determining the first estimated pose such that the loss function is reduced. The method further includes terminating the repeatedly generating etc. if the loss function is less than a threshold, and outputting the estimated pose as the pose of the camera.

Description

[0001]Localization using Neural Radiance Fields Technical Field The present disclosure relates to localization. Abbreviations 3GPP 3rdGeneration Partnership Project 5G / 6G / 7G 5th, 6th, 7thGeneration AR Augmented Reality BARF Bundle-Adjusting Neural Radiance Fields BID-NeRF Bundle Adjusted Depth-Supervised Inverted Neural Radiance Field CMY Cyan Magenta Yellow DS-NeRF Depth-Supervised Neural Radiance Field HAIP High Accuracy Indoor Positioning iNeRF Inverted Neural Radiance Field NeRF Neural Radiance Field ORB Oriented Fast and Rotated Brief RGB Red Green Blue RGBD Red Green Blue Depth SLAM Simultaneous Localization and Mapping SE Special Euclidean SfM Structure from Motion SO Special Orthogonal Background Camera localization, or image pose estimation, is a fundamental problem in robotics and computer vision. It refers to the process of determining camera pose (i.e. location of the camera and the viewing direction of the camera) from the visual scene representation (i.e. from an image captured by the camera). Camera localization is essential for many applications such as navigation of autonomous vehicles, structure from motion (SfM), augmented reality (AR) and simultaneous localization and mapping (SLAM). Localization on cm-level is requested in industrial automation, for example. Camera-based localization may provide the means to achieve cm-level accuracy, but the robustness suffers from several drawbacks, like sensitivity to the lighting conditions, dynamic objects, repetitive environments and so on. A Neural Radiance Field (NeRF) is a deep learning based method that can generate novel views of complex 3D scenes, based on a partial set of 2D images. Neural Radiance Fields (NeRFs) may be used to render images or videos of a physical space from arbitrary poses, based on a number of images taken from (a small) number of known poses. NeRFs can be used to reconstruct static or dynamic spaces based on just a few images in a view-dependent mode. NeRFs are trained to represent an implicit mapping between a view-dependent 3D query point (i.e., a spatial location and a viewing direction from that spatial location) and its associated RGB color and volumetric density in the captured image. In detail, for each pixel of the captured image, sample points on the ray through the volume to be imaged are determined. For example, the sample points may be equidistant on the ray. Each sample point is determined by 5 coordinates: its position (x, y, z) and the direction of the ray (Θ, Φ). In the Neural Radiance Field, a color (RGB) and a density (σ) are associated to each sample point. In this model, the color of the pixel in the captured image may be calculated by volume rendering on the ray, i.e., by a weighted addition of the colors of the sample points, wherein the weighting is based on the densities (which may be converted into transmittances). The neural network is trained such that the color of a pixel generated by the volume rendering of the respective ray in the Neural Radiance Field is substantially the same as the color of the pixel in the captured image. Typically, training of the NeRF requires only a few images with known rigid transformations between them. Through an algorithm of volumetric imaging, a new image of arbitrary position and orientation can be synthesized with such a mapping. Due to the differentiability of volumetric imaging and of the NeRF model, this mapping can be reversed. For that purpose, one may assume a pose from which an image was taken and render an image based on the trained NeRF model from that assumed pose. The rendered (generated) image is compared with the actual image. Then the pose is iteratively adapted such that a difference between the generated image and the actually captured image is minimized. Thus, the pose of the camera when actually capturing the image may be estimated. Summary It is an object to improve the prior art. According to a first aspect, there is provided an apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: estimating, for a first captured image and a first captured depth map related to the first captured image, a first preliminarily estimated pose of a first camera for capturing the first captured image setting the first preliminarily estimated pose as a first estimated pose; repeatedly: • generating a first generated image and a first generated depth map using the first estimated pose and a neural radiance field modelling a volume; • calculating a loss function based on a difference between the first captured image and the first generated image and a difference between the first captured depth map and the first generated depth map; • checking whether the loss function is less than a threshold; and • determining the first estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting the first estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the first camera for capturing the first captured image, wherein the first estimated pose of the first camera is defined with respect to a coordinate system of the neural radiance field; each of the first captured image and the first generated image comprises a plurality of pixels, for each of the pixels of the first captured image, the respective pixel comprises a respective captured first color value; for each of the pixels of the first generated image, the respective pixel comprises a respective generated first color value; for each of the pixels of the first generated image, the respective generated first color value is generated by a weighted addition of color values of sample points of a first set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; the color values and densities of the sample points are obtained from the neural radiance field; for at least a portion of the pixels of the first captured image, the first captured depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a first surface within the volume; for at least the portion of the pixels of the generated image, the first generated depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a first virtual surface within the volume. The first preliminarily estimated pose of the first camera may be additionally for capturing the first captured depth map. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: deriving, for a second captured image different from the first captured image and a second captured depth map related to the second captured image, a second preliminarily estimated pose of a second camera for capturing the second captured image and the second captured depth map based on the first preliminarily estimated pose and a known relative pose between an actual first pose for taking the first captured image and an actual second pose for taking the second captured image and the second captured depth map; setting the second preliminarily estimated pose as a second estimated pose; repeatedly: • generating the first generated image and the first generated depth map using the first estimated pose and the neural radiance field and the second generated image and the second generated depth map using the second estimated pose and the neural radiance field; • calculating the loss function based on the difference between the first captured image and the first generated image and a difference between the second captured image and the second generated image and a difference between the first captured depth map and the first generated depth map and a difference between the second captured image and the second generated depth map; • checking whether the loss function is less than the threshold; and • determining the first estimated pose and the second estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold and such that a relative pose between the first estimated pose and the second estimated pose is equal to the known relative pose; wherein the second estimated pose is defined with respect to the coordinate system of the neural radiance field; each of the second captured image and the second generated image comprises a plurality of pixels, for each of the pixels of the second captured image, the respective pixel comprises a respective captured second color value; for each of the pixels of the second generated image, the respective pixel comprises a respective generated second color value; for each of the pixels of the second generated image, the respective generated second color value is generated by the weighted addition of color values of sample points of a second set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for at least a portion of the pixels of the second captured image, the second captured depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a second surface within the volume; for at least the portion of the pixels of the second generated image, the second generated depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a second virtual surface within the volume. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: outputting the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the second camera for capturing the second captured image. For at least one of the rays, at least one of the sample points of the first set of sample points may be outside the volume. The instructions, when executed by the one or more processors, may further cause the apparatus to perform for the at least one of the rays: dividing the ray into equidistant first bins; determining the sample points of the first set of sample points by selecting one point from each of the equidistant first bins. For the at least one of the rays, the sample points of the first set of sample points may be equidistant on the ray. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: inhibiting to generate the first generated image by the weighted addition of color values of sample points of a first finer set of sample points in addition to the weighted addition of the color values of the sample points of the first set of sample points, wherein an average distance between neighbored sample points of the first finer set of sample points is smaller than the average distance between neighbored sample points of the first set of sample points. For at least one of the rays, at least one of the sample points of the second set of sample points may be outside the volume. The instructions, when executed by the one or more processors, may further cause the apparatus to perform for the at least one of the rays: dividing the ray into equidistant second bins; determining the sample points of the second set of sample points by selecting one point from each of the equidistant second bins. For the at least one of the rays, the sample points of the second set of sample points may be equidistant on the ray. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: inhibiting to generate the second generated image by the weighted addition of color values of sample points of a second finer set of sample points in addition to the weighted addition of the color values of the sample points of the second set of sample points, wherein an average distance between neighbored sample points of the second finer set of sample points is smaller than the average distance between neighbored sample points of the second set of sample points. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume. According to a second aspect, there is provided an apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: estimating, for a first captured image, a first preliminarily estimated pose of a first camera for capturing the first captured image; setting the first preliminarily estimated pose as a first estimated pose; deriving, for a second captured image different from the first captured image, a second preliminarily estimated pose of a second camera for capturing the second captured image based on the first preliminarily estimated pose and a known relative pose between an actual first pose for taking the first captured image and an actual second pose for taking the second captured image; setting the second preliminarily estimated pose as a second estimated pose; repeatedly: • generating a first generated image using the first estimated pose and a neural radiance field and a second generated image using the second estimated pose and the neural radiance field; • calculating a loss function based on a difference between the first captured image and the first generated image and a difference between the second captured image and the second generated image; • checking whether the loss function is less than a threshold; and • determining the first estimated pose and the second estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold and such that a relative pose between the first estimated pose and the second estimated pose is equal to the known relative pose; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting one of the first estimated pose and the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the first camera for capturing the first captured image and the pose of the second camera for capturing the second captured image, respectively, wherein the neural radiance field models a volume; the first estimated pose and the second estimated pose are defined with respect to a coordinate system of the neural radiance field; each of the first captured image, the second captured image, the first generated image, and the second generated image comprises a plurality of pixels, for each of the pixels of the first captured image, the respective pixel comprises a respective captured first color value; for each of the pixels of the second captured image, the respective pixel comprises a respective captured second color value; for each of the pixels of the first generated image, the respective pixel comprises a respective generated first color value; for each of the pixels of the second generated image, the respective pixel comprises a respective generated second color value; for each of the pixels of the first generated image, the respective generated first color value is generated by a weighted addition of color values of sample points of a first set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for each of the pixels of the second generated image, the respective generated second color value is generated by the weighted addition of color values of sample points of a second set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; the color values and densities of the sample points are obtained from the neural radiance field. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: outputting the other one of the first estimated pose and the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the other one of the first camera for capturing the first captured image and the second camera for capturing the second captured image. For at least one of the rays, at least one of the sample points of the first set of sample points may be outside the volume. The instructions, when executed by the one or more processors, may further cause the apparatus to perform for the at least one ray: dividing the ray into equidistant first bins; determining the sample points of the first set of sample points by selecting one point from each of the equidistant first bins. For the at least one of the rays, the sample points of the first set of sample points may bee equidistant on the ray. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: inhibiting to generate the first generated image by the weighted addition of color values of sample points of a first finer set of sample points in addition to the weighted addition of the color values of the sample points of the first set of sample points, wherein an average distance between neighbored sample points of the first finer set of sample points is smaller than the average distance between neighbored sample points of the first set of sample points. For at least one of the rays, at least one of the sample points of the second set of sample points may be outside the volume. The instructions, when executed by the one or more processors, may further cause the apparatus to perform for the at least one ray: dividing the ray into equidistant second bins; determining the sample points of the second set of sample points by selecting one point from each of the equidistant second bins. For the at least one of the rays, the sample points of the second set of sample points may be equidistant on the ray. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: inhibiting to generate the second generated image by the weighted addition of color values of sample points of a second finer set of sample points in addition to the weighted addition of the color values of the sample points of the second set of sample points, wherein an average distance between neighbored sample points of the second finer set of sample points is smaller than the average distance between neighbored sample points of the second set of sample points. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume. According to a third aspect, there is provided an apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: estimating, for a captured image, a preliminarily estimated pose of a camera for capturing the captured image; setting the preliminarily estimated pose as an estimated pose; repeatedly: • generating a generated image using the estimated pose and a neural radiance field modelling a volume; • calculating a loss function based on a difference between the captured image and the generated image; • checking whether the loss function is less than a threshold; and • determining the estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting the estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the camera for capturing the captured image, wherein the estimated pose of the camera is defined with respect to a coordinate system of the neural radiance field; each of the captured image and the generated image comprises a plurality of pixels, for each of the pixels of the captured image, the respective pixel comprises a respective captured color value; for each of the pixels of the generated image, the respective pixel comprises a respective generated color value; for each of the pixels of the generated image, the respective generated color value is generated by a weighted addition of color values of sample points of a set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for at least one of the rays, at least one of the sample points of the set of sample points is outside the volume; the color values and densities of the sample points are obtained from the neural radiance field. The instructions, when executed by the one or more processors, may further cause the apparatus to perform for the at least one ray: dividing the ray into equidistant bins; determining the sample points of the set of sample points by selecting one point from each of the equidistant bins. For the at least one of the rays, the sample points of the set of sample points may be equidistant on the ray. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: inhibiting to generate the generated image by the weighted addition of color values of sample points of a finer set of sample points in addition to the weighted addition of the color values of the sample points of the set of sample points, wherein an average distance between neighbored sample points of the finer set of sample points is smaller than the average distance between neighbored sample points of the set of sample points. The instructions, when executed by the one or more processors, may further cause the apparatus to perform: generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume. According to a fourth aspect, there is provided a method comprising: estimating, for a first captured image and a first captured depth map related to the first captured image, a first preliminarily estimated pose of a first camera for capturing the first captured image setting the first preliminarily estimated pose as a first estimated pose; repeatedly: • generating a first generated image and a first generated depth map using the first estimated pose and a neural radiance field modelling a volume; • calculating a loss function based on a difference between the first captured image and the first generated image and a difference between the first captured depth map and the first generated depth map; • checking whether the loss function is less than a threshold; and • determining the first estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting the first estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the first camera for capturing the first captured image, wherein the first estimated pose of the first camera is defined with respect to a coordinate system of the neural radiance field; each of the first captured image and the first generated image comprises a plurality of pixels, for each of the pixels of the first captured image, the respective pixel comprises a respective captured first color value; for each of the pixels of the first generated image, the respective pixel comprises a respective generated first color value; for each of the pixels of the first generated image, the respective generated first color value is generated by a weighted addition of color values of sample points of a first set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; the color values and densities of the sample points are obtained from the neural radiance field; for at least a portion of the pixels of the first captured image, the first captured depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a first surface within the volume; for at least the portion of the pixels of the generated image, the first generated depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a first virtual surface within the volume. The first preliminarily estimated pose of the first camera may be additionally for capturing the first captured depth map. The method may further comprise: deriving, for a second captured image different from the first captured image and a second captured depth map related to the second captured image, a second preliminarily estimated pose of a second camera for capturing the second captured image and the second captured depth map based on the first preliminarily estimated pose and a known relative pose between an actual first pose for taking the first captured image and an actual second pose for taking the second captured image and the second captured depth map; setting the second preliminarily estimated pose as a second estimated pose; repeatedly: • generating the first generated image and the first generated depth map using the first estimated pose and the neural radiance field and the second generated image and the second generated depth map using the second estimated pose and the neural radiance field; • calculating the loss function based on the difference between the first captured image and the first generated image and a difference between the second captured image and the second generated image and a difference between the first captured depth map and the first generated depth map and a difference between the second captured image and the second generated depth map; • checking whether the loss function is less than the threshold; and • determining the first estimated pose and the second estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold and such that a relative pose between the first estimated pose and the second estimated pose is equal to the known relative pose; wherein the second estimated pose is defined with respect to the coordinate system of the neural radiance field; each of the second captured image and the second generated image comprises a plurality of pixels, for each of the pixels of the second captured image, the respective pixel comprises a respective captured second color value; for each of the pixels of the second generated image, the respective pixel comprises a respective generated second color value; for each of the pixels of the second generated image, the respective generated second color value is generated by the weighted addition of color values of sample points of a second set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for at least a portion of the pixels of the second captured image, the second captured depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a second surface within the volume; for at least the portion of the pixels of the second generated image, the second generated depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a second virtual surface within the volume. The method may further comprise: outputting the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the second camera for capturing the second captured image. For at least one of the rays, at least one of the sample points of the first set of sample points may be outside the volume. The method may further comprise, for the at least one of the rays: dividing the ray into equidistant first bins; determining the sample points of the first set of sample points by selecting one point from each of the equidistant first bins. For the at least one of the rays, the sample points of the first set of sample points may be equidistant on the ray. The method may further comprise: inhibiting to generate the first generated image by the weighted addition of color values of sample points of a first finer set of sample points in addition to the weighted addition of the color values of the sample points of the first set of sample points, wherein an average distance between neighbored sample points of the first finer set of sample points is smaller than the average distance between neighbored sample points of the first set of sample points. For at least one of the rays, at least one of the sample points of the second set of sample points may be outside the volume. The method may further comprise for the at least one of the rays: dividing the ray into equidistant second bins; determining the sample points of the second set of sample points by selecting one point from each of the equidistant second bins. For the at least one of the rays, the sample points of the second set of sample points may be equidistant on the ray. The method may further comprise: inhibiting to generate the second generated image by the weighted addition of color values of sample points of a second finer set of sample points in addition to the weighted addition of the color values of the sample points of the second set of sample points, wherein an average distance between neighbored sample points of the second finer set of sample points is smaller than the average distance between neighbored sample points of the second set of sample points. The method may further comprise: generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume. According to a fifth aspect, there is provided a method comprising: estimating, for a first captured image, a first preliminarily estimated pose of a first camera for capturing the first captured image; setting the first preliminarily estimated pose as a first estimated pose; deriving, for a second captured image different from the first captured image, a second preliminarily estimated pose of a second camera for capturing the second captured image based on the first preliminarily estimated pose and a known relative pose between an actual first pose for taking the first captured image and an actual second pose for taking the second captured image; setting the second preliminarily estimated pose as a second estimated pose; repeatedly: • generating a first generated image using the first estimated pose and a neural radiance field and a second generated image using the second estimated pose and the neural radiance field; • calculating a loss function based on a difference between the first captured image and the first generated image and a difference between the second captured image and the second generated image; • checking whether the loss function is less than a threshold; and • determining the first estimated pose and the second estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold and such that a relative pose between the first estimated pose and the second estimated pose is equal to the known relative pose; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting one of the first estimated pose and the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the first camera for capturing the first captured image and the pose of the second camera for capturing the second captured image, respectively, wherein the neural radiance field models a volume; the first estimated pose and the second estimated pose are defined with respect to a coordinate system of the neural radiance field; each of the first captured image, the second captured image, the first generated image, and the second generated image comprises a plurality of pixels, for each of the pixels of the first captured image, the respective pixel comprises a respective captured first color value; for each of the pixels of the second captured image, the respective pixel comprises a respective captured second color value; for each of the pixels of the first generated image, the respective pixel comprises a respective generated first color value; for each of the pixels of the second generated image, the respective pixel comprises a respective generated second color value; for each of the pixels of the first generated image, the respective generated first color value is generated by a weighted addition of color values of sample points of a first set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for each of the pixels of the second generated image, the respective generated second color value is generated by the weighted addition of color values of sample points of a second set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; the color values and densities of the sample points are obtained from the neural radiance field. The method may further comprise: outputting the other one of the first estimated pose and the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the other one of the first camera for capturing the first captured image and the second camera for capturing the second captured image. For at least one of the rays, at least one of the sample points of the first set of sample points may be outside the volume. The method may further comprise for the at least one ray: dividing the ray into equidistant first bins; determining the sample points of the first set of sample points by selecting one point from each of the equidistant first bins. For the at least one of the rays, the sample points of the first set of sample points may be equidistant on the ray. The method may further comprise: inhibiting to generate the first generated image by the weighted addition of color values of sample points of a first finer set of sample points in addition to the weighted addition of the color values of the sample points of the first set of sample points, wherein an average distance between neighbored sample points of the first finer set of sample points is smaller than the average distance between neighbored sample points of the first set of sample points. For at least one of the rays, at least one of the sample points of the second set of sample points may be outside the volume. The method may further comprise for the at least one ray: dividing the ray into equidistant second bins; determining the sample points of the second set of sample points by selecting one point from each of the equidistant second bins. For the at least one of the rays, the sample points of the second set of sample points may be equidistant on the ray. The method may further comprise: inhibiting to generate the second generated image by the weighted addition of color values of sample points of a second finer set of sample points in addition to the weighted addition of the color values of the sample points of the second set of sample points, wherein an average distance between neighbored sample points of the second finer set of sample points is smaller than the average distance between neighbored sample points of the second set of sample points. The method may further comprise: generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume. According to a sixth aspect, there is provided a method comprising: estimating, for a captured image, a preliminarily estimated pose of a camera for capturing the captured image; setting the preliminarily estimated pose as an estimated pose; repeatedly: • generating a generated image using the estimated pose and a neural radiance field modelling a volume; • calculating a loss function based on a difference between the captured image and the generated image; • checking whether the loss function is less than a threshold; and • determining the estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting the estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the camera for capturing the captured image, wherein the estimated pose of the camera is defined with respect to a coordinate system of the neural radiance field; each of the captured image and the generated image comprises a plurality of pixels, for each of the pixels of the captured image, the respective pixel comprises a respective captured color value; for each of the pixels of the generated image, the respective pixel comprises a respective generated color value; for each of the pixels of the generated image, the respective generated color value is generated by a weighted addition of color values of sample points of a set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for at least one of the rays, at least one of the sample points of the set of sample points is outside the volume; the color values and densities of the sample points are obtained from the neural radiance field. The method may further comprise for the at least one ray: dividing the ray into equidistant bins; determining the sample points of the set of sample points by selecting one point from each of the equidistant bins. For the at least one of the rays, the sample points of the set of sample points may be equidistant on the ray. The method may further comprise: inhibiting to generate the generated image by the weighted addition of color values of sample points of a finer set of sample points in addition to the weighted addition of the color values of the sample points of the set of sample points, wherein an average distance between neighbored sample points of the finer set of sample points is smaller than the average distance between neighbored sample points of the set of sample points. The method may further comprise: generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume. Each of the methods of the fourth to sixth aspects may be a method of positioning. According to a seventh aspect, there is provided a computer program product comprising a set of instructions which, when executed on an apparatus, is configured to cause the apparatus to carry out the method according to any of the fourth to sixth aspects. The computer program product may be embodied as a computer-readable medium or directly loadable into a computer. According to some example embodiments, at least one of the following advantages may be achieved: • A highly precise localization may be achieved; • The localization is fast; • The localization needs moderate processing power; • The localization is robust; • The basin of convergence of the optimization is extended. Brief description of the drawings Further details, features, objects, and advantages are apparent from the following detailed description of the preferred example embodiments which is to be taken in conjunction with the appended drawings, wherein: Fig.1 shows some components of a system according to an example embodiment; Fig.2 shows simulation results according to several example embodiments; Fig. 3 shows an algorithm of the pose estimation method according to some example embodiments; Fig.4 shows an apparatus according to an example embodiment; Fig.5 shows a method according to an example embodiment; Fig.6 shows an apparatus according to an example embodiment; Fig.7 shows a method according to an example embodiment; Fig.8 shows an apparatus according to an example embodiment; Fig.9 shows a method according to an example embodiment; and Fig.10 shows an apparatus according to an example embodiment. Detailed description of certain example embodiments Herein below, certain example embodiments are described in detail with reference to the accompanying drawings, wherein the features of the example embodiments can be freely combined with each other unless otherwise described. However, it is to be expressly understood that the description of certain example embodiments is given by way of example only, and that it is by no way intended to be understood as limiting the disclosure to the disclosed details. Moreover, it is to be understood that the apparatus is configured to perform the corresponding method, although in some cases only the apparatus or only the method are described. According to some example embodiments, there are provided NeRF based localization methods that achieve good results in terms of iterations required for convergence and robustness. According to an example embodiment (a first fundamental example embodiment), the method has as inputs a NeRF and a sequence of plural captured RGB images with known relative poses (i.e. known transformations between the locations and viewing directions of the images). A pose should be determined between the coordinate system of any of the images and the coordinate system of the NeRF. This pose is initially not known. By inverting Neural Radiance Fields (iNeRF) algorithms, the Neural Radiance Fields (NeRF) is used for estimation of the pose of the camera with respect to the NeRF coordinate system (`camera pose’). Then, the camera poses of the cameras having captured the other images may be calculated based on the known relative poses and the pose of the at least one image. A pose indicates the transformation between two coordinate system. In the present case, the camera pose indicates the transformation between the coordinate system of the camera and the coordinate system of the NeRF. According to another example embodiment (a second fundamental example embodiment), the method has as input a NeRF, a captured RGB image, and a depth map related to the RGB image. The depth map indicates, in the coordinate system of the camera, for at least some of the pixels in the captured image, the position on the corresponding ray of a surface in the volume modelled by the NeRF. Typically, an object is located at that position, or the surface may be the surface of a wall of a room modelled by NeRF. Again, an inverse NeRF algorithm may be used to estimate, in the coordinate system of the NeRF, the pose of the camera taking the captured image. The depth map may be obtained together with the RGB image by a RGBD camera. As another option, the depth map may be obtained by another sensor (such as LIDAR) than the RGB camera, if the relative pose between the RGB camera and the other sensor is known. Then, the depth for at least some of the pixels of the RGB images may be obtained in the coordinate system of the RGB camera by coordinate transformation based on the known relative pose. In some example embodiments, some pixels of the depth map may be obtained by a RGBD camera, and other pixels of the depth map may be obtained by another sensor. A NeRF models a volume. For the generating of the NeRF, sample points inside the volume are used, but sample points outside the volume are not used. Conventionally, localization uses only NeRF within the volume. However, in some example embodiments (a third fundamental example embodiment), sample points outside the certain volume may be additionally taken into account. The additional sample points may increase the convergence basin. Conventionally, a NeRF may be generated by two iterations, a coarse sampling to determine an approximate position of an object intersecting the ray, and a subsequent fine sampling around the surface of the object. In the localization process according to some example embodiments (a fourth fundamental example embodiment), only the coarse sampling is used in a single iteration. It turns out that the convergence is improved if the fine sampling is omitted. According to some example embodiments, the features described hereinabove for two or more of the above fundamental example embodiments may be combined. Fig.1 shows components of a system where all the described fundamental example embodiments are combined. Namely, the system comprises, apart from the NeRF, a series of n captured RGBD images (n ≥ 2) with known relative poses in between. If at least one pose is estimated between a captured image’s coordinate system and the NeRF’s model coordinate system, the poses of the other captured images with respect to the NeRF model’s coordinate system may be calculated based on the known relative poses. Since the first estimation of a pose may comprise an error, the calculated poses may comprise errors, too. Based on the captured RGBD images, the relative poses, and the first estimation of the pose in the NeRF system as inputs, a method according to some example embodiments (called “BID-NeRF algorithm”, for example) outputs the pose of at least one of the captured pictures. The actions of the BID-NeRF algorithm are now explained at greater detail for an example embodiment combining the above discussed fundamental example embodiments. If another example embodiment does not comprise the features of one or more of the fundamental example embodiments, the corresponding actions of the BID-NERF algorithm may be omitted. In the 1staction, one of the captured images is selected as a reference image, and based on the known relative poses between the pairs of the captured images, for each of the captured images the relative pose with respect to the reference image is calculated. Each of the captured images belongs to at least one of the pairs of the captured images such that the relative poses with respect to the reference image may be calculated. The iterative first-order approximation algorithm may be described as follows: In the 2ndaction, for each input image (i.e., the captured images) an evenly distributed subset of the pixels may be determined. This action may be omitted and all the pixels of the captured images may be used. In another interpretation, the image comprising only the subset of the pixels may be considered as the captured image. This interpretation is used hereinafter. In the 3rdaction, it is assumed that virtual images taken by virtual cameras have the same pixel arrangements as the captured images. For each of the pixels, the parameters defining the ray corresponding to the pixel are determined in the coordinate system of the virtual camera. For example, the virtual camera may be assumed to be a pinhole for which the correspondence of pixel (in the image plane) and the ray from the pixel through the pin hole is known. More sophisticated virtual camera models may be used to correspond better to the camera(s) used for capturing the captured images. In the 4thaction, the parameters of the rays (which are given in the coordinate system of the virtual cameras in the 3rdaction) are transformed with the relative reference pose into a virtual reference coordinate system of a virtual reference camera. In the 5thaction, a pose of the virtual reference camera with respect to the coordinate system of the NeRF is estimated. In some cases, this preliminary estimation might be quite different from the actual pose because not much information about the actual pose is known for the 5thaction. Based on this estimation, the parameters of the rays are transformed into the coordinate system of the NeRF. Typically, the estimations and the resulting transformations comprise an error which is minimized throughout the following actions. The 1stto 5thactions explain one option how to obtain, for each of the pixels of the captured images, corresponding rays, the parameters of which are expressed in the coordinate system of the NeRF. There are alternative options. For example, based on the known relative poses and the estimated pose of the reference image with respect to the coordinate system of the NeRF, one may calculate for some or all of the captured images the transformation from the coordinate system of the respective camera into the coordinate system of the NeRF. In the 6thaction, on each of the rays of the virtual images, sample points are determined by stratified stochastic sampling. For stratified stochastic sampling, the ray is partitioned into evenly spaced bins, and then one sample is drawn at random from within each of the bins. For example, the sample points may be equidistant on the ray. For example, the sample points may be drawn uniformly at random from within each of the bins, such that the probability to be drawn is the same for each of the sample points within a bin. As another option, one may draw the sample point at a certain position (e.g. in the middle) of each bin. In the 7thaction, image rendering is performed for each of the poses (location and viewing direction) estimated for the captured images. That is, RGB and depth values for the pixels are generated with volumetric rendering on the respective rays based on the NeRF. In the 8thaction, the differences between the generated RGB and depth values (the generated images and generated depths) and the captured RGB and depth values (the captured images and captured depths), respectively, are evaluated for the images. For example, an average square error may be determined as a loss function. As another option, which may potentially achieve a more robust optimization in some cases, a different error function, like the Huber error function, can be utilized as the loss function. The 9thand 10thactions are performed if the loss function has a value larger than a predefined threshold. Otherwise, the 11thaction is performed In the 9thaction, the gradient of the error (loss function) is propagated back to the local tangent space of the estimated pose of the reference image using a chain rule. In the 10thaction, the estimation of the pose of the reference camera is updated such that, according to the gradient of the loss function with respect to the local tangent space of the estimated pose, the loss function will be reduced. The poses of the other cameras having taken the captured images are updated correspondingly such that the known relative poses are maintained. The process comprising the 6thto 10thactions is repeated until discrepancies are reduced under a certain limit (i.e., the loss function is smaller than a predefined threshold). In some example embodiments, the process may be terminated also if a certain number of iterations are achieved. If the loss function calculated in the 8thaction has a value smaller than the predefined threshold, the process comprising the 6thto 10thactions is terminated. The process outputs, as an 11thaction, the estimated pose of the reference camera as the actual pose of the reference camera. In some example embodiments, instead of or in addition to the pose of the reference camera, the process may output the pose of another camera having taken any of the captured images. Fig.2 compares the BID-NeRF algorithm (with the features of some or all of the fundamental example embodiments) according to some example embodiments with conventional iNeRF. The insert means: • iNeRF: conventional • iNerf + depth: conventional + depth control (second fundamental example embodiment) • iNeRF + depth + coarse: conventional + depth control + coarse sampling (fourth fundamental example embodiment) • iNerf + depth + coarse + 8img: conventional + depth control + coarse sampling + plural (8) captured images (first fundamental example embodiment) • BiD-NeRF: conventional + depth control + coarse sampling + plural (8) captured images + extended sampling (third fundamental example embodiment) Fig.2 shows how a proportion of localization attempts with a rotational error < 5° (left diagram) and a translational error < 5 cm (right diagram) increases with the number of iterations. As can be seen from Fig. 2 the convergence is improved by each of the example embodiments compared to a conventional example (iNeRF) where none of the features of the fundamental example embodiments is used. BID-NeRF achieved 95% convergence in both error types after 1000 steps, while this percentage is only at 38% for translational and 42% for rotational errors using the iNeRF algorithm. An ablation study shows that the features of each of the fundamental embodiments yielded significant improvements. A detailed technical description of an example embodiment follows: To use NeRFs for image pose estimation (in real-time applications, for example), the performance and robustness of the algorithm should be improved while storage requirements and inference time should be reduced. To achieve this, a method according to some example embodiments introduces the following modifications compared to previous research: (i.) the NeRF models are trained with depth supervision and a depth-based regularization term to the objective loss function is added, (ii.) reference pixels from multiple view directions are sampled based on a moving window of previous frames and their relative transformation estimations, (iii.) a stratified stochastic sampling approach is used where the volume is partitioned into evenly-spaced bins and then one sample is uniformly at random drawn from within each bin. Hierarchical sampling (two subsequent samplings: a coarse sampling with large average distances between neighbored sampling points on the ray, followed by a fine sampling in a small specific range of the ray with smaller average distances between the neighbored sampling points) is omitted during volumetric rendering, meaning only the coarse model is used for pose estimation. With this step the computational complexity of the algorithm is significantly reduced. • Basic equations Transformations between coordinate systems are defined as elements of the Special Euclidean group SE(3) as follows: (1) that is composed of a rotation matrix belonging to the Special Orthogonal group SO(3) and a translation vector . In the 1ststep, for every image a transformation to the coordinate system of the reference image is defined based on the relative transformation as follows: where is the relative transformation between image coordinate systems and . To evaluate the NeRF model the rays are transformed into the NeRF model coordinate system. denotes the estimated transformation between the reference coordinate system of the image sequence and the NeRF model coordinate system. As a result, the transformation from the i'th image coordinate system to the coordinate system of the NeRF model is defined as In the beginning, this transformation is not correct, a goal is to find the optimal transformation. • NeRF basic equations The 3rd–5thsteps of the algorithm are formalized with the following set of equations. Based on the pixel coordinate of the i'th image and the camera intrinsic parameters and the focal lengths, and denoting the principal point coordinates of the camera, a ray is defined in the NeRF model’s coordinate system in the following way: where is the starting point of the radius and is its direction in the camera is the rotational . (In this example without loss of generality the pinhole camera model is used, however, different models might be used in other embodiments.) Stochastic sampling and gradient smoothing with the coarse model In the 6thstep discussed hereinabove, a stratified stochastic sampling approach is used where the ray is partitioned into evenly spaced bins and then one sample is drawn uniformly at random from within each bin. Hierarchical sampling for volume rendering works substantially as follows: Two separate neural networks: a coarse and a fine model were used for rendering. The fine model achieves high accuracy if rays are sampled close to the object surfaces. The purpose of the coarse model is to obtain information about the location of the surfaces in the space, resulting in a more efficient volume sampling method for the fine model. This approach improves the reconstruction capabilities of NeRFs, enabling them to represent finer details with higher frequencies. Finer details lead to higher image gradients in RGB predictions. This effect is undesirable if NeRF is used for pose estimation. Furthermore, eliminating the fine model, a substantial decrease in computational complexity can be achieved. Therefore, in some example embodiments, hierarchical sampling may be omitted completely and only the coarse model may be used for pose estimation. By using the coarse model for rough estimation, the smoother gradients of the RGB and volumetric density values will increase the convergence rate. Furthermore, the σ predictions of the coarse model are also present in areas away from the surface of the object, while the fine model produces much sharper changes. The gradients of the coarse model’s predictions are smoother, making them favorable for first-order optimization algorithms. Using only the coarse model helps the convergence and reduces storage requirements, as only one neural network must be stored as a localization map. • Volume rendering In the 7thstep, after the stratified stochastic sampling of the rays, the sampled points are evaluated by the trained NeRF model. For every sampled point the NeRF model is evaluted and it will output a volumetric density value and a color vector . Volumetric rendering is used to compute the expected color for the ray based on the following equations: , By using the quadrature rule, a numerical estimation of can be formulated as follows: , Eq. (6) distance to the first can be as follows: , where denotes the distance of the i-th sample point from the camera. In the 7thstep of the algorithm, and are computed for every ray. • Localization as a model-based optimization The iNeRF’s original problem formulation can be seen in the following equations , where is the rendered color for ray based on Eq. (6), and is the corresponding color of the reference RGB image pixel. Each ray is transformed to the reference coordinate system from the camera’s local coordinate system with the actual prediction of the camera’s pose. While several first-order optimization algorithms could be potentially used, some example embodiments may use the ADAM optimizer. The updates to the image poses are calculated in the local tangent space of SE(3). The ⊕ operator adds a vector from a Lie algebra to an element from its corresponding Lie group by projecting the vector to a new group element and adding them together through the group’s composition operation. The above equations show the optimal solution to the formulated problem, where is the globally optimal transformation and is the inverse of the initial error represented in the local tangent space. If this inverse vector was known, one may simply add it to the initial pose estimate to get the optimal solution. Instead of directly searching for the optimal vector though, according to some example embodiments, one may iteratively update the pose estimate by the algorithm shown in Fig.3. • Localization with depth supervision As described for the 7thand 8thstep, a depth-based loss term is added to the localization optimization problem. This loss works particularly well if the NeRF model has a good 3D structure representation. It was shown that, by applying depth supervision during training, the model structure is significantly more accurate compared to the training method without depth supervision. This way the learned space representation will contain fewer structural noise or artifacts caused by the errors of volumetric density predictions. The appearance of such artifacts can be best described as the result of the shape-radiance ambiguity in NeRFs. This effect can be reduced by explicitly introducing more constraints in the learning process. Depth supervision can be used as such a constraint, improving the precision of the predicted depth maps. Depth information can also be utilized during pose estimation if depth information is available. Depth maps are free of textures and lighting differences. Therefore, they have smoother image-space gradients compared to RGB images. The following equation shows how depth predictions can be added to the original RGB loss: where is the estimated depth for ray based on Eq. (8), is the depth value of the corresponding ray and is a hyperparameter. The added loss term is the mean-squared error between the estimated depth and the ground-truth depth values , that is similar to the intensity loss term as shown in Eq. (9). The depth predictions do not increase the computational complexity of the volumetric rendering. If dense depth predictions for every RGB pixel cannot be obtained, unlike with RGB- D cameras, other methods can be used as well. The depth supervision during pose estimation also works with sparse depth information. These can be obtained by off-the-shelf structure from motion algorithms such as COLMAP. Another approach would be to use lidar sensors and render separate virtual rays using volumetric rendering to match the sensor’s field of view. • Localization with bundle-adjusted image sequence The relative pose estimates for each input image are obtained by a visual(-inertial) odometry algorithm. Errors in state estimation solutions usually come from an accumulated drift, which is ideally infinitesimal for only a few steps. By applying bundle adjustment, the errors in the pose estimates can be further reduced. With this assumption, a moving window of sequential images can be localized simultaneously. The following equation defines the multi-image loss term: Eq. It averages the loss function introduced in Eq. (10) for the single image use-case, with the only difference that each ray is transformed from the i-th image’s local coordinate system to the reference image’s coordinate system with transformations defined in Eq. (2). is the initial estimate of the absolute pose for the reference K. The use of an image sequence instead of a single image means that rays can be sampled from more viewing angles, with more reference pixel values. This increase in sampling space improves the performance of the inherent optimization. Also, this increase in information is free from any extra calculations, so it does not reduce execution speed. It was already shown, that rendering more rays improves the performance of the optimization. However, evaluating too many rays is often not feasible on embedded systems. Therefore, in some example embodiments, the same total number of rays may be used independently from the number of images. I.e., the rays may be distributed equally between the images such that the number of rays per image depends on the number of images. Thus, comparable results may be achieved, without further increase in computational complexity. Fig. 4 shows an apparatus according to an example embodiment. The apparatus may be a pose estimator or an element thereof. Fig. 5 shows a method according to an example embodiment. The apparatus according to Fig.4 may perform the method of Fig.5 but is not limited to this method. The method of Fig.5 may be performed by the apparatus of Fig.4 but is not limited to being performed by this apparatus. The apparatus comprises means for estimating 110, means for setting 120, means for generating 130, means for calculating 140, means for checking 150, means for determining 160, means for terminating 170, and means for outputting 180. The means for estimating 110, means for setting 120, means for generating 130, means for calculating 140, means for checking 150, means for determining 160, means for terminating 170, and means for outputting 180 may be a estimating means, setting means, generating means, calculating means, checking means, determining means, terminating means, and outputting means, respectively. The means for estimating 110, means for setting 120, means for generating 130, means for calculating 140, means for checking 150, means for determining 160, means for terminating 170, and means for outputting 180 may be an estimator, setter, generator, calculator, checker, determiner, terminator, and outputter, respectively. The means for estimating 110, means for setting 120, means for generating 130, means for calculating 140, means for checking 150, means for determining 160, means for terminating 170, and means for outputting 180 may be a estimating processor, setting processor, generating processor, calculating processor, checking processor, determining processor, terminating processor, and outputting processor, respectively. The means for estimating 110 estimates, for a captured image and a captured depth map related to the captured image, a preliminarily estimated pose of a camera for capturing the captured image (S110). The estimated pose of the camera is defied with respect to a coordinate system of a neural radiance field modelling a volume. For at least a portion of the pixels of the captured image, the captured depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a surface within the volume. The means for setting 120 sets the preliminarily estimated pose as an estimated pose (S120). The following actions of generating (S130), calculating (S140), checking (S150), and determining (S160) may be performed repeatedly, until the condition in S150 is fulfilled. In detail: The means for generating 130 generates a generated image and a generated depth map using the estimated pose and the neural radiance field (S130). Each of the captured image and the generated image comprises a plurality of pixels. For each of the pixels of the captured image, the respective pixel comprises a respective captured color value. For each of the pixels of the generated image, the respective pixel comprises a respective generated color value. For each of the pixels of the generated image, the respective generated color value is generated by a weighted addition of color values of sample points of a set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points. The color values and densities of the sample points are obtained from the neural radiance field. For at least the portion of the pixels of the generated image, the generated depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a virtual surface within the volume. The means for calculating 140 calculates a loss function based on a difference between the captured image and the generated image and a difference between the captured depth map and the generated depth map (S140). The means for checking 150 checks whether the loss function is less than a threshold (S150). In response to checking that the loss function is not less than the threshold (S150 = no), the means for determining 160 determines the estimated pose such that the loss function is reduced (e.g. based on a gradient of the loss function). Then, the generating (S130), calculating (S140), checking (S150), and potentially the determining (S160) may be repeated with the estimated pose determined in S160. On the other hand, in response to checking that the loss function is less than the threshold (S150 = yes), the means for terminating 170 terminates the repeatedly generating, calculating, checking and determining. The means for outputting 180 outputs the estimated pose (i.e., the pose determined when the repeatedly generating, calculating, checking and determining is terminated) as the pose of the camera for capturing the captured image (S180). Fig. 6 shows an apparatus according to an example embodiment. The apparatus may be a pose estimator or an element thereof. Fig. 7 shows a method according to an example embodiment. The apparatus according to Fig.6 may perform the method of Fig.7 but is not limited to this method. The method of Fig.7 may be performed by the apparatus of Fig.6 but is not limited to being performed by this apparatus. The apparatus comprises means for estimating 210, first means for setting 220, means for deriving 222, second means for setting 224, means for generating 230, means for calculating 240, means for checking 250, means for determining 260, means for terminating 270, and means for outputting 280. The means for estimating 210, first means for setting 220, means for deriving 222, second means for setting 224, means for generating 230, means for calculating 240, means for checking 250, means for determining 260, means for terminating 270, and means for outputting 280 may be a estimating means, first setting means, deriving means, second setting means, generating means, calculating means, checking means, determining means, terminating means, and outputting means, respectively. The means for estimating 210, first means for setting 220, means for deriving 222, second means for setting 224, means for generating 230, means for calculating 240, means for checking 250, means for determining 260, means for terminating 270, and means for outputting 280 may be an estimator, first setter, deriver, second setter, generator, calculator, checker, determiner, terminator, and outputter, respectively. The means for estimating 210, first means for setting 220, means for deriving 222, second means for setting 224, means for generating 230, means for calculating 240, means for checking 250, means for determining 260, means for terminating 270, and means for outputting 280 may be a estimating processor, first setting processor, deriving processor, second setting processor, generating processor, calculating processor, checking processor, determining processor, terminating processor, and outputting processor, respectively. The means for estimating 210 estimates, for a first captured image, a first preliminarily estimated pose of a first camera for capturing the first captured image (S210). The first preliminary estimated pose of the first camera is defined with respect to a coordinate system of a neural radiance field modelling a volume. The first means for setting 220 sets the first preliminarily estimated pose as a first estimated pose (S220). The means for deriving 222 derives, for a second captured image different from the first captured image, a second preliminarily estimated pose of a second camera for capturing the second captured image based on the first preliminarily estimated pose and a known relative pose between an actual first pose for taking the first captured image and an actual second pose for taking the second captured image (S222). The estimated pose of the second camera is defined with respect to the coordinate system of the neural radiance field. The second means for setting 224 sets the second preliminarily estimated pose as a second estimated pose (S224) The following actions of generating (S230), calculating (S240), checking (S250), and determining (S260) may be performed repeatedly, until the condition in S250 is fulfilled. In detail: The means for generating 230 generates a first generated image using the first estimated pose and the neural radiance field and a second generated image using the second estimated pose and the neural radiance field (S230). Each of the first and second captured images and the first and second generated images comprises a plurality of pixels. For each of the pixels of the first and second captured images, the respective pixel comprises a respective first captured color value and a respective second captured color value, respectively. For each of the pixels of the first and second generated images, the respective pixel comprises a respective first generated color value and a respective second generated color value, respectively. For each of the pixels of the first and second generated images, the respective generated color value is generated by a weighted addition of color values of sample points of a respective set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points. The color values and densities of the sample points are obtained from the neural radiance field. The means for calculating 240 calculates a loss function based on a difference between the first captured image and the first generated image and a difference between the second captured image and the second generated image (S240). The means for checking 250 checks whether the loss function is less than a threshold (S250). In response to checking that the loss function is not less than the threshold (S250 = no), the means for determining 260 determines the first and second estimated poses such that the loss function is reduced (e.g. based on a gradient of the loss function). Then, the generating (S230), calculating (S240), checking (S250), and potentially the determining (S260) may be repeated with the first and second estimated poses determined in S260. On the other hand, in response to checking that the loss function is less than the threshold (S250 = yes), the means for terminating 270 terminates the repeatedly generating, calculating, checking and determining. The means for outputting 280 outputs at least one of the first and second estimated poses (i.e., the pose determined when the repeatedly generating, calculating, checking and determining is terminated) as the pose of the respective camera for capturing the respective captured image (S280). Fig. 8 shows an apparatus according to an example embodiment. The apparatus may be a pose estimator or an element thereof. Fig. 9 shows a method according to an example embodiment. The apparatus according to Fig.8 may perform the method of Fig.9 but is not limited to this method. The method of Fig.9 may be performed by the apparatus of Fig.8 but is not limited to being performed by this apparatus. The apparatus comprises means for estimating 310, means for setting 320, means for generating 330, means for calculating 340, means for checking 350, means for determining 360, means for terminating 370, and means for outputting 380. The means for estimating 310, means for setting 320, means for generating 330, means for calculating 340, means for checking 350, means for determining 360, means for terminating 370, and means for outputting 380 may be a estimating means, setting means, generating means, calculating means, checking means, determining means, terminating means, and outputting means, respectively. The means for estimating 310, means for setting 320, means for generating 330, means for calculating 340, means for checking 350, means for determining 360, means for terminating 370, and means for outputting 380 may be an estimator, setter, generator, calculator, checker, determiner, terminator, and outputter, respectively. The means for estimating 310, means for setting 320, means for generating 330, means for calculating 340, means for checking 350, means for determining 360, means for terminating 370, and means for outputting 380 may be a estimating processor, setting processor, generating processor, calculating processor, checking processor, determining processor, terminating processor, and outputting processor, respectively. The means for estimating 310 estimates, for a captured image, a preliminarily estimated pose of a camera for capturing the captured image (S310). The estimated pose of the camera is defined with respect to a coordinate system of a neural radiance field modelling a volume. The means for setting 320 sets the preliminarily estimated pose as an estimated pose (S320). The following actions of generating (S330), calculating (S340), checking (S350), and determining (S360) may be performed repeatedly, until the condition in S350 is fulfilled. In detail: The means for generating 330 generates a generated image using the estimated pose and the neural radiance field (S330). Each of the captured image and the generated image comprises a plurality of pixels. For each of the pixels of the captured image, the respective pixel comprises a respective captured color value. For each of the pixels of the generated image, the respective pixel comprises a respective generated color value. For each of the pixels of the generated image, the respective generated color value is generated by a weighted addition of color values of sample points of a set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points. For at least one of the rays, at least one of the sample points of the set of sample points is outside the volume. The color values and densities of the sample points are obtained from the neural radiance field. The means for calculating 340 calculates a loss function based on a difference between the captured image and the generated image (S340). The means for checking 350 checks whether the loss function is less than a threshold (S350). In response to checking that the loss function is not less than the threshold (S350 = no), the means for determining 360 determines the estimated pose such that the loss function is reduced (e.g. based on a gradient of the loss function). Then, the generating (S330), calculating (S340), checking (S350), and potentially the determining (S360) may be repeated with the estimated pose determined in S360. On the other hand, in response to checking that the loss function is less than the threshold (S350 = yes), the means for terminating 370 terminates the repeatedly generating, calculating, checking and determining. The means for outputting 380 outputs the estimated pose (i.e., the pose determined when the repeatedly generating, calculating, checking and determining is terminated) as the pose of the camera for capturing the captured image (S380). Fig.10 shows an apparatus according to an example embodiment. The apparatus comprises at least one processor 810, at least one memory 820 storing instructions that, when executed by the at least one processor 810, cause the apparatus at least to perform the method according to at least one of the following figures and related description: Fig.5, or Fig.7, or Fig.9. Some example embodiments are described with one or more captured RGB images as input. However, in some example embodiments, the colors may be represented by other values. For example, the image may be a CMY image. One piece of information may be transmitted in one or plural messages from one entity to another entity. Each of these messages may comprise further (different) pieces of information. Names of network elements, network functions, protocols, and methods are based on current standards. In other versions or other technologies, the names of these network elements and / or network functions and / or protocols and / or methods may be different, as long as they provide a corresponding functionality. The same applies correspondingly to the terminal. If not otherwise stated or otherwise made clear from the context, the statement that two entities are different means that they perform different functions. It does not necessarily mean that they are based on different hardware. That is, each of the entities described in the present description may be based on a different hardware, or some or all of the entities may be based on the same hardware. It does not necessarily mean that they are based on different software. That is, each of the entities described in the present description may be based on different software, or some or all of the entities may be based on the same software. Each of the entities described in the present description may be deployed in the cloud. According to the above description, it should thus be apparent that example embodiments provide, for example, a localization device, or an element thereof, an apparatus embodying the same, a method for controlling and / or operating the same, and computer program(s) controlling and / or operating the same as well as mediums carrying such computer program(s) and forming computer program product(s). Implementations of any of the above described blocks, apparatuses, systems, techniques or methods include, as non-limiting examples, implementations as hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof. Each of the entities described in the present description may be embodied in the cloud. It is to be understood that what is described above is what is presently considered the preferred example embodiments. However, it should be noted that the description of the preferred example embodiments is given by way of example only and that various modifications may be made without departing from the scope of the disclosure as defined by the appended claims. The terms “first X” and “second X” include the options that “first X” is the same as “second X” and that “first X” is different from “second X”, unless otherwise specified. As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

Claims

Claims:

1. Apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: estimating, for a first captured image and a first captured depth map related to the first captured image, a first preliminarily estimated pose of a first camera for capturing the first captured image setting the first preliminarily estimated pose as a first estimated pose; repeatedly: • generating a first generated image and a first generated depth map using the first estimated pose and a neural radiance field modelling a volume; • calculating a loss function based on a difference between the first captured image and the first generated image and a difference between the first captured depth map and the first generated depth map; • checking whether the loss function is less than a threshold; and • determining the first estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting the first estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the first camera for capturing the first captured image, wherein the first estimated pose of the first camera is defined with respect to a coordinate system of the neural radiance field; each of the first captured image and the first generated image comprises a plurality of pixels, for each of the pixels of the first captured image, the respective pixel comprises a respective captured first color value; for each of the pixels of the first generated image, the respective pixel comprises a respective generated first color value; for each of the pixels of the first generated image, the respective generated first color value is generated by a weighted addition of color values of sample points of a first set ofsample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; the color values and densities of the sample points are obtained from the neural radiance field; for at least a portion of the pixels of the first captured image, the first captured depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a first surface within the volume; for at least the portion of the pixels of the generated image, the first generated depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a first virtual surface within the volume.

2. The apparatus according to claim 1, wherein the first preliminarily estimated pose of the first camera is additionally for capturing the first captured depth map.

3. The apparatus according to any of claims 1 and 2, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform: deriving, for a second captured image different from the first captured image and a second captured depth map related to the second captured image, a second preliminarily estimated pose of a second camera for capturing the second captured image and the second captured depth map based on the first preliminarily estimated pose and a known relative pose between an actual first pose for taking the first captured image and an actual second pose for taking the second captured image and the second captured depth map; setting the second preliminarily estimated pose as a second estimated pose; repeatedly: • generating the first generated image and the first generated depth map using the first estimated pose and the neural radiance field and the second generated image and the second generated depth map using the second estimated pose and the neural radiance field; • calculating the loss function based on the difference between the first captured image and the first generated image and a difference between the second captured image and the second generated image and a difference between the first captured depth map and the first generated depth map and a difference between the second captured image and the second generated depth map; • checking whether the loss function is less than the threshold; and• determining the first estimated pose and the second estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold and such that a relative pose between the first estimated pose and the second estimated pose is equal to the known relative pose; wherein the second estimated pose is defined with respect to the coordinate system of the neural radiance field; each of the second captured image and the second generated image comprises a plurality of pixels, for each of the pixels of the second captured image, the respective pixel comprises a respective captured second color value; for each of the pixels of the second generated image, the respective pixel comprises a respective generated second color value; for each of the pixels of the second generated image, the respective generated second color value is generated by the weighted addition of color values of sample points of a second set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for at least a portion of the pixels of the second captured image, the second captured depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a second surface within the volume; for at least the portion of the pixels of the second generated image, the second generated depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a second virtual surface within the volume.

4. The apparatus according to claim 3, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform outputting the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the second camera for capturing the second captured image.

5. The apparatus according to any of claims 1 to 4, wherein for at least one of the rays, at least one of the sample points of the first set of sample points is outside the volume.

6. The apparatus according to claim 5, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform for the at least one of the rays: dividing the ray into equidistant first bins; determining the sample points of the first set of sample points by selecting one point from each of the equidistant first bins.

7. The apparatus according to claim 6, wherein, for the at least one of the rays, the sample points of the first set of sample points are equidistant on the ray.

8. The apparatus according to any of claims 5 to 7, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform inhibiting to generate the first generated image by the weighted addition of color values of sample points of a first finer set of sample points in addition to the weighted addition of the color values of the sample points of the first set of sample points, wherein an average distance between neighbored sample points of the first finer set of sample points is smaller than the average distance between neighbored sample points of the first set of sample points.

9. The apparatus according to any of claims 3 and 4 or any of claims 5 to 8 dependent on claim 3 or claim 4, wherein for at least one of the rays, at least one of the sample points of the second set of sample points is outside the volume.

10. The apparatus according to claim 9, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform for the at least one of the rays: dividing the ray into equidistant second bins; determining the sample points of the second set of sample points by selecting one point from each of the equidistant second bins.

11. The apparatus according to claim 10, wherein, for the at least one of the rays, the sample points of the second set of sample points are equidistant on the ray.

12. The apparatus according to any of claims 9 to 11, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform inhibiting to generate the second generated image by the weighted addition of color values of sample points of a second finer set of sample points in addition to the weightedaddition of the color values of the sample points of the second set of sample points, wherein an average distance between neighbored sample points of the second finer set of sample points is smaller than the average distance between neighbored sample points of the second set of sample points.

13. The apparatus according to any of claims 1 to 12, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume.

14. Apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: estimating, for a first captured image, a first preliminarily estimated pose of a first camera for capturing the first captured image; setting the first preliminarily estimated pose as a first estimated pose; deriving, for a second captured image different from the first captured image, a second preliminarily estimated pose of a second camera for capturing the second captured image based on the first preliminarily estimated pose and a known relative pose between an actual first pose for taking the first captured image and an actual second pose for taking the second captured image; setting the second preliminarily estimated pose as a second estimated pose; repeatedly: • generating a first generated image using the first estimated pose and a neural radiance field and a second generated image using the second estimated pose and the neural radiance field; • calculating a loss function based on a difference between the first captured image and the first generated image and a difference between the second captured image and the second generated image; • checking whether the loss function is less than a threshold; and • determining the first estimated pose and the second estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold and such that a relative posebetween the first estimated pose and the second estimated pose is equal to the known relative pose; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting one of the first estimated pose and the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the first camera for capturing the first captured image and the pose of the second camera for capturing the second captured image, respectively, wherein the neural radiance field models a volume; the first estimated pose and the second estimated pose are defined with respect to a coordinate system of the neural radiance field; each of the first captured image, the second captured image, the first generated image, and the second generated image comprises a plurality of pixels, for each of the pixels of the first captured image, the respective pixel comprises a respective captured first color value; for each of the pixels of the second captured image, the respective pixel comprises a respective captured second color value; for each of the pixels of the first generated image, the respective pixel comprises a respective generated first color value; for each of the pixels of the second generated image, the respective pixel comprises a respective generated second color value; for each of the pixels of the first generated image, the respective generated first color value is generated by a weighted addition of color values of sample points of a first set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for each of the pixels of the second generated image, the respective generated second color value is generated by the weighted addition of color values of sample points of a second set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; the color values and densities of the sample points are obtained from the neural radiance field.

15. The apparatus according to claim 14, wherein the instructions, when executed by the one or more processors, further cause the apparatus to performoutputting the other one of the first estimated pose and the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the other one of the first camera for capturing the first captured image and the second camera for capturing the second captured image.

16. The apparatus according to any of claims 14 to 15, wherein for at least one of the rays, at least one of the sample points of the first set of sample points is outside the volume.

17. The apparatus according to claim 16, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform for the at least one ray: dividing the ray into equidistant first bins; determining the sample points of the first set of sample points by selecting one point from each of the equidistant first bins.

18. The apparatus according to claim 17, wherein, for the at least one of the rays, the sample points of the first set of sample points are equidistant on the ray.

19. The apparatus according to any of claims 16 to 18, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform inhibiting to generate the first generated image by the weighted addition of color values of sample points of a first finer set of sample points in addition to the weighted addition of the color values of the sample points of the first set of sample points, wherein an average distance between neighbored sample points of the first finer set of sample points is smaller than the average distance between neighbored sample points of the first set of sample points.

20. The apparatus according to any of claims 14 and 15 or any of claims 16 to 19 dependent on claim 14 or claim 15, wherein for at least one of the rays, at least one of the sample points of the second set of sample points is outside the volume.

21. The apparatus according to claim 20, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform for the at least one ray: dividing the ray into equidistant second bins;determining the sample points of the second set of sample points by selecting one point from each of the equidistant second bins.

22. The apparatus according to claim 21, wherein, for the at least one of the rays, the sample points of the second set of sample points are equidistant on the ray.

23. The apparatus according to any of claims 20 to 22, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform inhibiting to generate the second generated image by the weighted addition of color values of sample points of a second finer set of sample points in addition to the weighted addition of the color values of the sample points of the second set of sample points, wherein an average distance between neighbored sample points of the second finer set of sample points is smaller than the average distance between neighbored sample points of the second set of sample points.

24. The apparatus according to any of claims 14 to 23, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume.

25. Apparatus comprising: one or more processors, and memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform: estimating, for a captured image, a preliminarily estimated pose of a camera for capturing the captured image; setting the preliminarily estimated pose as an estimated pose; repeatedly: • generating a generated image using the estimated pose and a neural radiance field modelling a volume; • calculating a loss function based on a difference between the captured image and the generated image; • checking whether the loss function is less than a threshold; and • determining the estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold;terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting the estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the camera for capturing the captured image, wherein the estimated pose of the camera is defined with respect to a coordinate system of the neural radiance field; each of the captured image and the generated image comprises a plurality of pixels, for each of the pixels of the captured image, the respective pixel comprises a respective captured color value; for each of the pixels of the generated image, the respective pixel comprises a respective generated color value; for each of the pixels of the generated image, the respective generated color value is generated by a weighted addition of color values of sample points of a set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for at least one of the rays, at least one of the sample points of the set of sample points is outside the volume; the color values and densities of the sample points are obtained from the neural radiance field.

26. The apparatus according to claim 25, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform for the at least one ray: dividing the ray into equidistant bins; determining the sample points of the set of sample points by selecting one point from each of the equidistant bins.

27. The apparatus according to claim 26, wherein, for the at least one of the rays, the sample points of the set of sample points are equidistant on the ray.

28. The apparatus according to any of claims 25 to 27, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform inhibiting to generate the generated image by the weighted addition of color values of sample points of a finer set of sample points in addition to the weighted addition of the color values of the sample points of the set of sample points, wherein an average distance betweenneighbored sample points of the finer set of sample points is smaller than the average distance between neighbored sample points of the set of sample points.

29. The apparatus according to any of claims 25 to 28, wherein the instructions, when executed by the one or more processors, further cause the apparatus to perform generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume.

30. Method comprising: estimating, for a first captured image and a first captured depth map related to the first captured image, a first preliminarily estimated pose of a first camera for capturing the first captured image setting the first preliminarily estimated pose as a first estimated pose; repeatedly: • generating a first generated image and a first generated depth map using the first estimated pose and a neural radiance field modelling a volume; • calculating a loss function based on a difference between the first captured image and the first generated image and a difference between the first captured depth map and the first generated depth map; • checking whether the loss function is less than a threshold; and • determining the first estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting the first estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the first camera for capturing the first captured image, wherein the first estimated pose of the first camera is defined with respect to a coordinate system of the neural radiance field; each of the first captured image and the first generated image comprises a plurality of pixels,for each of the pixels of the first captured image, the respective pixel comprises a respective captured first color value; for each of the pixels of the first generated image, the respective pixel comprises a respective generated first color value; for each of the pixels of the first generated image, the respective generated first color value is generated by a weighted addition of color values of sample points of a first set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; the color values and densities of the sample points are obtained from the neural radiance field; for at least a portion of the pixels of the first captured image, the first captured depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a first surface within the volume; for at least the portion of the pixels of the generated image, the first generated depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a first virtual surface within the volume.

31. The method according to claim 30, wherein the first preliminarily estimated pose of the first camera is additionally for capturing the first captured depth map.

32. The method according to any of claims 30 and 31, further comprising: deriving, for a second captured image different from the first captured image and a second captured depth map related to the second captured image, a second preliminarily estimated pose of a second camera for capturing the second captured image and the second captured depth map based on the first preliminarily estimated pose and a known relative pose between an actual first pose for taking the first captured image and an actual second pose for taking the second captured image and the second captured depth map; setting the second preliminarily estimated pose as a second estimated pose; repeatedly: • generating the first generated image and the first generated depth map using the first estimated pose and the neural radiance field and the second generated image and the second generated depth map using the second estimated pose and the neural radiance field; • calculating the loss function based on the difference between the first captured image and the first generated image and a difference betweenthe second captured image and the second generated image and a difference between the first captured depth map and the first generated depth map and a difference between the second captured image and the second generated depth map; • checking whether the loss function is less than the threshold; and • determining the first estimated pose and the second estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold and such that a relative pose between the first estimated pose and the second estimated pose is equal to the known relative pose; wherein the second estimated pose is defined with respect to the coordinate system of the neural radiance field; each of the second captured image and the second generated image comprises a plurality of pixels, for each of the pixels of the second captured image, the respective pixel comprises a respective captured second color value; for each of the pixels of the second generated image, the respective pixel comprises a respective generated second color value; for each of the pixels of the second generated image, the respective generated second color value is generated by the weighted addition of color values of sample points of a second set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for at least a portion of the pixels of the second captured image, the second captured depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a second surface within the volume; for at least the portion of the pixels of the second generated image, the second generated depth map comprises an information of a respective location on a ray corresponding to the respective pixel indicating a position of a second virtual surface within the volume.

33. The method according to claim 32, further comprising outputting the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the second camera for capturing the second captured image.

34. The method according to any of claims 30 to 33, wherein for at least one of the rays, at least one of the sample points of the first set of sample points is outside the volume.

35. The method according to claim 34, further comprising, for the at least one of the rays: dividing the ray into equidistant first bins; determining the sample points of the first set of sample points by selecting one point from each of the equidistant first bins.

36. The method according to claim 35, wherein, for the at least one of the rays, the sample points of the first set of sample points are equidistant on the ray.

37. The method according to any of claims 34 to 36, further comprising inhibiting to generate the first generated image by the weighted addition of color values of sample points of a first finer set of sample points in addition to the weighted addition of the color values of the sample points of the first set of sample points, wherein an average distance between neighbored sample points of the first finer set of sample points is smaller than the average distance between neighbored sample points of the first set of sample points.

38. The method according to any of claims 32 and 33 or any of claims 34 to 37 dependent on claim 32 or claim 33, wherein for at least one of the rays, at least one of the sample points of the second set of sample points is outside the volume.

39. The method according to claim 38, further comprising for the at least one of the rays: dividing the ray into equidistant second bins; determining the sample points of the second set of sample points by selecting one point from each of the equidistant second bins.

40. The method according to claim 39, wherein, for the at least one of the rays, the sample points of the second set of sample points are equidistant on the ray.

41. The method according to any of claims 38 to 40, further comprising inhibiting to generate the second generated image by the weighted addition of color values of sample points of a second finer set of sample points in addition to the weightedaddition of the color values of the sample points of the second set of sample points, wherein an average distance between neighbored sample points of the second finer set of sample points is smaller than the average distance between neighbored sample points of the second set of sample points.

42. The method according to any of claims 30 to 41, further comprising generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume.

43. Method comprising: estimating, for a first captured image, a first preliminarily estimated pose of a first camera for capturing the first captured image; setting the first preliminarily estimated pose as a first estimated pose; deriving, for a second captured image different from the first captured image, a second preliminarily estimated pose of a second camera for capturing the second captured image based on the first preliminarily estimated pose and a known relative pose between an actual first pose for taking the first captured image and an actual second pose for taking the second captured image; setting the second preliminarily estimated pose as a second estimated pose; repeatedly: • generating a first generated image using the first estimated pose and a neural radiance field and a second generated image using the second estimated pose and the neural radiance field; • calculating a loss function based on a difference between the first captured image and the first generated image and a difference between the second captured image and the second generated image; • checking whether the loss function is less than a threshold; and • determining the first estimated pose and the second estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold and such that a relative pose between the first estimated pose and the second estimated pose is equal to the known relative pose; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold;outputting one of the first estimated pose and the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the first camera for capturing the first captured image and the pose of the second camera for capturing the second captured image, respectively, wherein the neural radiance field models a volume; the first estimated pose and the second estimated pose are defined with respect to a coordinate system of the neural radiance field; each of the first captured image, the second captured image, the first generated image, and the second generated image comprises a plurality of pixels, for each of the pixels of the first captured image, the respective pixel comprises a respective captured first color value; for each of the pixels of the second captured image, the respective pixel comprises a respective captured second color value; for each of the pixels of the first generated image, the respective pixel comprises a respective generated first color value; for each of the pixels of the second generated image, the respective pixel comprises a respective generated second color value; for each of the pixels of the first generated image, the respective generated first color value is generated by a weighted addition of color values of sample points of a first set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for each of the pixels of the second generated image, the respective generated second color value is generated by the weighted addition of color values of sample points of a second set of sample points on a ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; the color values and densities of the sample points are obtained from the neural radiance field.

44. The method according to claim 43, further comprising outputting the other one of the first estimated pose and the second estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the other one of the first camera for capturing the first captured image and the second camera for capturing the second captured image.

45. The method according to any of claims 43 to 44, whereinfor at least one of the rays, at least one of the sample points of the first set of sample points is outside the volume.

46. The method according to claim 45, further comprising for the at least one ray: dividing the ray into equidistant first bins; determining the sample points of the first set of sample points by selecting one point from each of the equidistant first bins.

47. The method according to claim 46, wherein, for the at least one of the rays, the sample points of the first set of sample points are equidistant on the ray.

48. The method according to any of claims 45 to 47, further comprising inhibiting to generate the first generated image by the weighted addition of color values of sample points of a first finer set of sample points in addition to the weighted addition of the color values of the sample points of the first set of sample points, wherein an average distance between neighbored sample points of the first finer set of sample points is smaller than the average distance between neighbored sample points of the first set of sample points.

49. The method according to any of claims 43 and 44 or any of claims 45 to 48 dependent on claim 43 or claim 44, wherein for at least one of the rays, at least one of the sample points of the second set of sample points is outside the volume.

50. The method according to claim 49, further comprising for the at least one ray: dividing the ray into equidistant second bins; determining the sample points of the second set of sample points by selecting one point from each of the equidistant second bins.

51. The method according to claim 50, wherein, for the at least one of the rays, the sample points of the second set of sample points are equidistant on the ray.

52. The method according to any of claims 49 to 51, further comprising inhibiting to generate the second generated image by the weighted addition of color values of sample points of a second finer set of sample points in addition to the weighted addition of the color values of the sample points of the second set of sample points, whereinan average distance between neighbored sample points of the second finer set of sample points is smaller than the average distance between neighbored sample points of the second set of sample points.

53. The method according to any of claims 43 to 52, further comprising generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume.

54. Method comprising: estimating, for a captured image, a preliminarily estimated pose of a camera for capturing the captured image; setting the preliminarily estimated pose as an estimated pose; repeatedly: • generating a generated image using the estimated pose and a neural radiance field modelling a volume; • calculating a loss function based on a difference between the captured image and the generated image; • checking whether the loss function is less than a threshold; and • determining the estimated pose such that the loss function is reduced in response to checking that the loss function is not less than the threshold; terminating the repeatedly generating, calculating, checking and determining in response to checking that the loss function is less than the threshold; outputting the estimated pose determined when the repeatedly generating, calculating, checking and determining is terminated as the pose of the camera for capturing the captured image, wherein the estimated pose of the camera is defined with respect to a coordinate system of the neural radiance field; each of the captured image and the generated image comprises a plurality of pixels, for each of the pixels of the captured image, the respective pixel comprises a respective captured color value; for each of the pixels of the generated image, the respective pixel comprises a respective generated color value; for each of the pixels of the generated image, the respective generated color value is generated by a weighted addition of color values of sample points of a set of sample points ona ray corresponding to the respective pixel, and the weights of the weighted addition are based on densities of the sample points; for at least one of the rays, at least one of the sample points of the set of sample points is outside the volume; the color values and densities of the sample points are obtained from the neural radiance field.

55. The method according to claim 54, further comprising for the at least one ray: dividing the ray into equidistant bins; determining the sample points of the set of sample points by selecting one point from each of the equidistant bins.

56. The method according to claim 55, wherein, for the at least one of the rays, the sample points of the set of sample points are equidistant on the ray.

57. The method according to any of claims 54 to 56, further comprising inhibiting to generate the generated image by the weighted addition of color values of sample points of a finer set of sample points in addition to the weighted addition of the color values of the sample points of the set of sample points, wherein an average distance between neighbored sample points of the finer set of sample points is smaller than the average distance between neighbored sample points of the set of sample points.

58. The method according to any of claims 54 to 57, further comprising generating the neural radiance field based on sample points within the volume; inhibiting the generating the neural radiance field based on sample points outside the volume.

59. A computer program product comprising a set of instructions which, when executed on an apparatus, is configured to cause the apparatus to carry out the method according to any of claims 30 to 58.

60. The computer program product according to claim 59, embodied as a computer-readable medium or directly loadable into a computer.

Images