NERF-based generation of point cloud

Abstract

There is provided techniques for obtaining a point cloud of a scene from a Neural Radiance Field (NeRF) model of the scene. A method is performed by an image processing device. The method comprises obtaining a NeRF model of the scene. The method comprises determining a depth value and a colour value for one point in the cloud via ray tracing in the NeRF model. This one point corresponds to one pixel in a pixel patch comprises at least three pixels. The method comprises adding this one point, including its colour value and its depth value, to the point cloud only in case the depth value of this one pixel fulfils a distance criterion with respect to depth values of other pixels in the pixel patch.

Description

[0001] NERF-BASED GENERATION OF POINT CLOUD

[0002] TECHNICAL FIELD

[0003] Embodiments presented herein relate to a method, an image processing device, a computer program, and a computer program product for obtaining a point cloud of a scene from a Neural Radiance Field model of the scene.

[0004] BACKGROUND

[0005] Neural volumetric representations, such as Neural Radiance Fields (NeRF), can be used in the process of learning a three-dimensional (3D) scenes representation from sparse input views. In some examples, these sparse input views are defined by, or composed by, a set of input two-dimensional (2D) images and the corresponding camera poses (i.e., one camera pose per 2D image). Using NeRF, the scene can be modeled as a continuous five-dimensional (5D) function that outputs the radiance emitted in each direction (0, ), at each point x, y, z) in 3D space. In other words, NeRF learns a 5D vector-valued function whose input is a 3D location (x,y, z) and a 2D viewing direction (0, ), and whose output is the emitted colour (defined by the radiance) and a volume density (also referred to as structure).

[0006] After a NeRF model is learned, it can be used to render scene views from either user- controlled or some predefined input camera poses. In general terms, the rendering procedure involves a ray casting technique, performed at the pixels of an image that would correspond to the current camera pose. In more detail, for each ray, colour and depth information are extracted from the learned NeRF model via ray tracing. This information (i.e., the extracted colour and depth information) is assigned to the corresponding image pixel. During raytracing, the learned NeRF model is queried in several locations (i.e., points) along the ray, and the outputs from the NeRF model are aggregated into one single value for the corresponding pixel. In the end, if all pixels of an image are processed (rendered), this produces what is often referred to as an RGB-D image (i.e., a red (R), green (G), blue (B) image with depth (D)).

[0007] A (3D) point cloud can be extracted from the NeRF model through the rendering of arbitrary pixels from various camera views. In particular, each sampled pixel, that has assigned colour and depth values as rendered from the NeRF model, directly corresponds to a point in 3D space, coloured with the same colour and located at a known displacement from the camera (in accordance with the camera’s view angle, position, pixel location and the depth value). These points, corresponding to different pixels of different camera poses, can thus be aggregated into a single 3D point cloud.

[0008] The generation of the NeRF models is mainly focused on optimizing the RGB rendering of the scene, while the depth information often comes as a byproduct and is often heuristically estimated. In further detail, even with a well-trained NeRF model that would produce realistic full-image renders of the scene, the geometry of point clouds obtained by the current techniques is often noisy with respect to that points in the 3D space that should be part of an object are displaced in an empty volume or on a surface of another object. The noise often appears where a depth discontinuity would appear in an RGB-D image; for example, near the borders of an object that is close to the camera. An example of this issue is visualized in Fig. 1. In Fig. 1 is schematically illustrated a scene 100 generated from a NeRF model. Fig. 1 shows a scene 100 comprising a first object 120 and a second object 130, both placed on a surface no. A ray 160 as sampled from a camera view, represented by an image 150, is going through the learned scene 100 and passes near the border of the first object 120 and lands on the second object 130. Because of this the depth value may appear anywhere between the ray’s 160 intersection with object 120 and object 130, as represented by black dots along the ray, one of which is marked at reference numeral 140. In other words, when a ray is cast in a way that is passes close to the border of an object, the resulting depth maybe estimated anywhere between this object (object 120 in Fig. 1) and another object (object 130 in fig. 1) that is located behind, including any location in-between. Such noise, especially when in large amounts, often makes it more difficult to (or even prevents from) understanding the correct geometry of the modelled elements. In other words, having a point cloud with a large number of 3D points generated based on incorrect depth values makes the scene 100 noisy and also creates a wrong perception of the scene 100 when viewed in a point cloud viewer, or when the resulting point cloud is processed by some other image processing application.

[0009] Hence, there is still a need for improved generation of point clouds from NeRF models. SUMMARY

[0010] An object of embodiments herein is to enable generation of point clouds from NeRF models where the above issues are resolved, or at least mitigated or reduced.

[0011] A particular object is to enable generation of point clouds from NeRF models, where the noise, as caused by points generated based on incorrect depth values, in the point clouds is reduced.

[0012] A particular object is to enable generation of point clouds from NeRF models, where the number of points generated based on incorrect depth values is reduced.

[0013] According to a first aspect there is presented a method for obtaining a point cloud of a scene from a NeRF model of the scene. The method is performed by an image processing device. The method comprises obtaining a NeRF model of the scene. The method comprises determining a depth value and a colour value for one point in the cloud via ray tracing in the NeRF model. This one point corresponds to one pixel in a pixel patch comprises at least three pixels. The method comprises adding this one point, including its colour value and its depth value, to the point cloud only in case the depth value of this one pixel fulfils a distance criterion with respect to depth values of other pixels in the pixel patch.

[0014] According to a second aspect there is presented an image processing device for obtaining a point cloud of a scene from a NeRF model of the scene. The image processing device comprises processing circuitry. The processing circuitry is configured to cause the image processing device to obtain a NeRF model of the scene. The processing circuitry is configured to cause the image processing device to determine a depth value and a colour value for one point in the cloud via ray tracing in the NeRF model. This one point corresponds to one pixel in a pixel patch comprises at least three pixels. The processing circuitry is configured to cause the image processing device to add this one point, including its colour value and its depth value, to the point cloud only in case the depth value of this one pixel fulfils a distance criterion with respect to depth values of other pixels in the pixel patch.

[0015] According to a third aspect there is presented a computer program for obtaining a point cloud of a scene from a NeRF model of the scene. The computer program comprises computer code which, when run on processing circuitry of an image processing device, causes the image processing device to perform actions. One action comprises the image processing device to obtain a NeRF model of the scene. One action comprises the image processing device to determine a depth value and a colour value for one point in the cloud via ray tracing in the NeRF model. This one point corresponds to one pixel in a pixel patch comprises at least three pixels. One action comprises the image processing device to add this one point, including its colour value and its depth value, to the point cloud only in case the depth value of this one pixel fulfils a distance criterion with respect to depth values of other pixels in the pixel patch.

[0016] According to a fourth aspect there is presented a computer program product comprising a computer program according to the third aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium could be a non-transitory computer readable storage medium.

[0017] Advantageously, these aspects enable generation of point clouds from NeRF models without the above issues.

[0018] Advantageously, these aspects provide point clouds with more correct depth values. In turn, this implies that the sampled geometric noise is reduced.

[0019] Having a point cloud with points that have correct depth values enables a better distinction between small details in the point cloud. Advantageously, these aspects therefore enhance the resolution of the point cloud, thus making it easier to identify intricate details in the point cloud when observed in a point cloud viewer.

[0020] Advantageously, these aspects do not impose any perceivable damage to the point cloud, under the assumption that every point of the point cloud is well -visible from at least one camera angle among the input views.

[0021] Advantageously, these aspects are applicable in the context of different techniques that learn continuous 3D scene representations, such as NeRF, Plenoxels and Gaussian Splatting. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0022] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:

[0025] Fig. 1 schematically illustrates ray tracing in a scene generated from a NeRF model according to an example;

[0026] Fig. 2 is a block diagram of an image processing device according to an embodiment;

[0027] Fig. 3 is a flowchart of methods according to embodiments;

[0028] Figs. 4 and 5 schematically illustrate ray tracing in a scene generated from a NeRF model according to an embodiment;

[0029] Fig. 6 is a flowchart of methods according to embodiments;

[0030] Fig. 7 shows simulation results according to an embodiment;

[0031] Fig. 8 is a schematic diagram showing structural units of an image processing device according to an embodiment; and

[0032] Fig. 9 shows one example of a computer program product comprising computer readable storage medium according to an embodiment. DETAILED DESCRIPTION

[0033] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.

[0034] As noted above, there is still a need for improved generation of point clouds from NeRF models.

[0035] The embodiments disclosed herein therefore relate to techniques for obtaining a point cloud of a scene from a NeRF model of the scene. In order to obtain such techniques there is provided an image processing device, a method performed by the image processing device, a computer program product comprising code, for example in the form of a computer program, that when run on an image processing device, causes the image processing device to perform the method.

[0036] The herein disclosed embodiments are based on providing a denoising technique, according to which a distance criterion with respect to depth values of several pixels in a pixel patch is considered when the depth value of one pixel is to be determined, that helps to significantly reduce the amount of sampled geometric noise.

[0037] In Fig. 2 is provided a block diagram of an image processing device 200 configured for obtaining a point cloud of a scene from a NeRF model of the scene. In general terms, in order to generate a point cloud, 3D points are sampled from the underlying continuous NeRF model by rendering and fusing depth maps. The image processing device 200 receives as input a NeRF model (1, a set of 2D images I, a corresponding set of camera poses C used for rendering with the 2D images, and the desired number of points, N, in the final point cloud. The image processing device 200 provides as output a point cloud P of size N = |P|. The image processing device 200 comprises a number of blocks 210:260. The function of each block will be disclosed below in conjunction with the description of the method in the flowchart of Fig. 3.

[0038] Fig. 3 is a flowchart illustrating embodiments of methods for obtaining a point cloud of a scene 400 from a NeRF model of the scene 400. The methods are performed by the image processing device 200. The methods are advantageously provided as computer programs. In general terms, in order to generate a point cloud, 3D points are sampled from the underlying continuous NeRF model by rendering and fusing depth maps. Parallel reference will be made to Fig. 4. In Fig. 4 is schematically illustrated a scene 400 generated from a NeRF model. The scene 400 is identical to the scene 100 and thus comprises a first object 420 and a second object 430, both placed on a surface 410. However, in contrast to Fig. 1, according to the herein disclosed embodiments, and as will be disclosed below, three rays 460a, 460b, 460c as sampled from pixel patch 470 of a camera view, represented by an image 450, are going through the learned scene 400 and either passes near the border of the first object 420 and lands on the second object 430 (as for rays 460b, 460c) or lands on the first object 420 (as for ray 460a). For each ray 4603:4600, a respective depth value 440a, 440b, 440c is calculated. The depth value of the pixel of interest in the pixel patch 470 is then compared to the depth values of the other pixels in the pixel patch 470 to check whether the depth value of the pixel of interest fulfils a distance criterion with respect to the depth values of the other pixels in the pixel patch 470. The pixel of interest is then only added to the point cloud if this distance criterion is fulfilled.

[0039] S102: The image processing device 200 obtains a NeRF model of the scene 400.

[0040] The herein disclosed embodiments are based on replacing pixel sampling (with its corresponding ray) with sampling of small pixel patches, where the depth value and color value is obtained from one of the pixels in the pixel patch, as in step S106.

[0041] S106: The image processing device 200 determines S106 a depth value, d*, and a colour value for one point 440a in the cloud via ray tracing in the NeRF model. This one point 440a corresponds to one pixel in a pixel patch 470 comprising at least three pixels. In Fig. 2, the ray tracing is implemented by the sample block 220. In Fig. 2, the determination of the depth value is implemented by the depth rendering block 240.

[0042] In the illustrative example of Fig. 4, the pixel patch 470 is composed of nine pixels in total.

[0043] Each sampled point is validated. In case the retrieved point is considered to be an outlier, the validation fails and the point is discarded. In plain words, the validation fails when a distance criterion fails.

[0044] S108: The image processing device 200 adds this one point 440a, including its colour value and its depth value, to the point cloud only in case the depth value d* of this one pixel fulfils a distance criterion with respect to depth values of other pixels in the pixel patch 470. In Fig. 2, the adding of the point to the point cloud is implemented by the point cloud update block 260.

[0045] Different examples of distance criteria will be disclosed below.

[0046] In essence, the procedure defined by steps S102, S106, and S108 thus includes obtaining depth values also for pixels in a neighborhood of the pixel that corresponds to the point of interest, instead of just the pixel for one single pixel, in order to generate the point cloud.

[0047] By means of the distance criterion, points with depth values being outliers among the sampled points can be discarded, thereby improving the overall quality of the resulting point cloud.

[0048] Embodiments relating to further details of obtaining a point cloud of a scene 400 from a NeRF model of the scene 400 as performed by the image processing device 200 will now be disclosed with continued reference to Figs. 2, 3, and 4.

[0049] In some aspects, the image processing device 200 is configured to iteratively construct the point cloud, adding one point after another to the point cloud P. That is, in some embodiments, the point cloud is iteratively constructed by a depth value d* and a colour value being repeatedly determined for a respective point in the point cloud, and this respective point, including its depth value and its colour value, being added to the point cloud. Further, in some aspects, the point cloud is initiated as an empty point cloud; P <- 0. In Fig. 2, this initiation is implemented by the initialize block 210. Further, although the herein disclosed embodiments are described as one point being added after another, this process can have a parallelized implementation in the image processing device 200 via batch-processing (i.e., by the image processing device 200 being configured for sampling pixels in several pixel patches 470 at the same time). However, the process can also have a serialized implementation. Regardless, for simplicity of disclosure, the procedure will hereinafter be described as sequential.

[0050] As disclosed above, the input to the image processing device 200 might in addition to the NeRF mode comprise a set of images and a corresponding set of camera poses. Hence, in some embodiments, the image processing device 200 is configured to perform (optional) step S104.

[0051] S104: The image processing device 200 obtains a set of images from which the NeRF model was generated, and a set of camera poses, with one camera pose per image. The pixel patch 470 is located in one of the images 450, and all the at least three rays 4603:4600 have a common origin 480, 580 (only schematically illustrated in Fig. 4 but in more detail in Fig. 5) as given by the camera pose of this one of the images. In further detail, the camera pose contains the coordinates, 0, of the camera in the 3D space. These coordinates become the origin of the ray for the ray casting.

[0052] As disclosed above, and as illustrated in Fig. 4, there is one respective ray 4603:460 for each of the pixels in the pixel patch 470. That is, in some embodiments, according to the ray tracing, as many rays 4603:4600 as there are pixels in the pixel patch 470, are passed through the NeRF model of the scene 400, with one ray for each respective pixel in the pixel patch 470, for determining one respective depth value for each of the at least three pixels in the pixel patch 470. In Fig. 4, these depth values are illustrated by the location of the points 440a, 440b, 440c along each of the rays 4603:460c.

[0053] In general terms, the depth value for a given point is the distance from the origin 480, 580 to this given point in the scene 400. Parallel reference is here made to Fig. 5 which illustrates two rays 560a, 560b with the same origin 480, 580. The two rays 560a, 560b pass through two different pixels (as marked by “+”) in a pixel patch 570. Each of the two rays 560a, 560b ends at a respective point 540a, 540b (where the respective rays 560a, 560b are assumed to hit some respective surface). The distance for the point 540a is d*. It is further assumed that the distance for the point 540b is the minimum distance of all points for the rays passing through the pixel patch 570. Hence, this distance is denoted dmin. Accordingly, in some embodiments, the point 440a, 540a has a location in the scene 400 where the ray 460a, 560a of this pixel hits a main surface of an object 420 in the scene 400, and the depth value d* of the point 440a, 540a is given by the distance from the origin 480, 580 to this one point 440a, 540a in the scene 400.

[0054] In some aspects, the ray 460a, 560a of the pixel is discretized into a set of points, where each of the points of the ray 460a, 560a is associated with a respective colour vector and density value as given by the NeRF model. In Fig. 2, this discretization is implemented by the discretize block 230. As will be further disclosed next, the location of this point 440a, 540a can then be given by the density values, and the colour value can be given by the colour vectors. In essence, in order to render a pixel at some coordinate (x,y), the NeRF model is queried in a number of points along the ray {0 + uxy• 1 1 1 > O}, resulting is a sequence of values of radiance / color r and density a. This information is aggregated along the ray using, for example, a so-called volume rendering equation. As a result, a single (three- or four-dimensional) color vector rgbx yand a single depth value dx yis obtained for the sampled image pixel. In Fig. 2, the obtaining of the color of each pixel is implemented by the color rendering block 250.

[0055] According to the herein disclosed embodiments, a pixel patch 470, 570, defined by a small grid of pixels around the coordinates (x0,y0), is rendered instead of a single pixel. This results in one depth value dx yfor each pixel in the pixel patch 470, 570, as well as one color value for each such pixel. Among the different color values for the pixels in the pixel patch 470, 570, only the color rgb%0 yoof the pixel of interest is necessary, and the other color values can be discarded. However, the depth values for each of the pixels in the pixel patch 470, 570 are kept.

[0056] In general terms, the pixel patch 470, 570 is of size w x h pixels, where w, h > 1.

[0057] Generally, the pixel patch 470, 570 is composed of a set of adjoining pixels, i.e., where the pixels are neighboring each other. For example, the pixel patch 470, 570 does not have any holes. Further in this respect, although the pixel patch 470, 570 in both Figs. 4 and 5 has been illustrated to be of size 3 x 3 pixels, i.e., where w = h = 3, and where the pixel patch 470, 570 is centered at the pixel of interest (i.e., the pixel for which the depth value is to be determined), the pixel patch 470, 570 can also be of different sizes. For example, when a pixel for which a depth value is to be determined is sampled near the border of the image, a pixel patch of another shape or form may be used. In particular, in some embodiments, the size of the pixel patch 470, 570 is dependent on a location of this one pixel in this one image 450. The size is smaller in case the location of this one pixel is at a border of this one image 450 than in case the location of this one pixel is not at any border of this one image 450. On the other hand, if one and the same size of the pixel patch is to be used for all pixels within the image, the image can be provided with a frame having specified fixed depth values. That is, in some embodiments, in a case a location of this one pixel in this one image 450 is at a border of this one image 450, the depth value of any pixel in the pixel patch 470, 570 falling outside this one image 450 is set to a value larger than a predefined value. Setting the predefined value to a comparative large value, this implies that the depth values of the pixels in the thus added frame may represent a depth values at near-infinite distance.

[0058] As disclosed above, the point 440a, including its colour value and its depth value, is added to the point cloud only in case the depth value d* of this pixel fulfils a distance criterion with respect to the depth values of the other pixels in the pixel patch 470, 570. There can be different examples of such distance criteria.

[0059] In some aspects, the distance criterion is fulfilled only when the depth value d* is not significantly larger than the smallest depth value, above referred to as dmin, of all pixels in the pixel patch 470, 570. Hence, the decision whether or not to add the pixel to the point cloud includes a verification procedure that takes into account the depth values of all the pixels in the pixel patch 470, 570. In particular, in some embodiments, the distance criterion is fulfilled when the depth value d* is not more than a threshold value larger than the smallest depth value of all pixels in the pixel patch 470, 570. If the pixel is accepted (and the depth value is finite), a point at the location 0 + u ■ dXxno>yVno with color value rg °bXxno>yVno is added to the p * oint cloud P. There can be different ways to determine whether or not the depth value d* is not more than a threshold value larger than the smallest depth value of all pixels in the pixel patch 470, 570. Two non-limiting examples will be presented next.

[0060] According to a first example, the distance criterion is fulfilled when for some positive constant E, and where dmindx yis the smallest depth value of all the pixels in the pixel patch 470, 570. In some examples, E < 0.01. In some examples, E < 0.005. In some examples, E < 0.0025. The depth values d* and dmincan potentially coincide if the center of the pixel patch has the smallest depth value.

[0061] The above equation is an example based on the multiplicative distance between the depth values. However, instead, if the scene 400 is given in real-world scale (for example, in metric scale), an additive distance criterion, as in the below second example, can be used.

[0062] According to a second example, the distance criterion is fulfilled when

[0063] [d dmin] for some value a > 0, and where dmin=(min^ dx yis the smallest depth value of all the pixels in the pixel patch 470, 570. In some examples, a > 10cm. In some examples, a > 15 cm. In some examples, a > 20 cm.

[0064] With reference back to Fig. 4, it can be observed that the point 440a is located farther from the origin 480 than the point 440b. Hence, the point 440a does not have the minimum distance dminfor the pixel patch 470 under consideration. Different tunings of the values E and a (depending on which distance criterion is used) can therefore decide whether or not the point 440a is to be included in the point cloud or be regarded as an outlier and thus discarded. The smaller the values of E and a are, the more likely it is that the distance criterion is not fulfilled for the point 440a (and hence that this point 440a is discarded and not included in the point cloud). On particular embodiment for obtaining a point cloud of a scene 400 from a NeRF model of the scene 400 as based on at least some of the above disclosed embodiments, aspects, and examples, and as performed by the image processing device 200 will now be disclosed with reference to the flowchart of Fig. 6.

[0065] 50. The image processing device 200 initializes an empty point cloud; P <- 0.

[0066] 51. The image processing device 200 selects a random image i £ I with a known camera pose from C. The camera pose contains the coordinates, 0, of the camera in the 3D space. These coordinates become the origin of the ray for the ray casting.

[0067] 52. The image processing device 200 selects a random pixel at coordinates within the selected image i, where the pixel is located some distance from the image border.

[0068] For example, x0and y0, might be selected uniformly on the image such that the whole w x h pixel patch with its central pixel placed at is entirely contained within the image bounds. In some examples (as when the pixel patch is of fixed size), this implies that x0and y0are picked at a certain distance from the image border, depending on w and h, respectively. As the result of selecting x0and y0, the unitvector uXo yois obtained, defining the direction of the ray that comes out of the camera center and passes through the pixel at (x0,y0). This direction is calculated from the camera pose (particularly the rotation of the camera) and the intrinsic parameters of the camera such as its focal length. Likewise, a direction uxycan be obtained for every pixel (x, y) of the image patch centered at (x0, yo)-

[0069] 53. The image processing device 200 generates the color values and depth values for all pixels in the w x h patch centered at (x0,y0), using the NeRF model 1.

[0070] 54. The image processing device 200 adds the point 440a, including its colour value and its depth value, to the point cloud only in case the depth value d* of the pixel fulfils any of the above disclosed distance criteria with respect to the depth values of the other pixels in the pixel patch 470. Then, if the point cloud P has the desired size, i.e., in case |P| = N the procedure continues to step S5. Otherwise, i.e., in case |P| < N, the procedure is repeated from step Si for another randomly selected image and / or pixel. S5- The image processing device 200 outputs the generated point cloud P. When the point cloud P achieves the desired size N, the process halts and the point cloud can be provided to some other image processing application.

[0071] Reference is next made to Fig. 7 in which simulation results are shown. Fig. 7 provides a comparison between a first point cloud 700a and a second point cloud 700b. Both the first point cloud 700a and the second point cloud 700b were generated from the same NeRF model, but with the difference that the herein disclosed technique for determining depth values for the points was applied when generating the second point cloud 700b. Two regions in each of the point clouds 700a, 700b for comparison are identified at reference numerals 710a, 710b, 720a, 720b. As can be seen by comparing the region 710a in the first view 700a to the same region 710b in the second point cloud 700b, and by comparing the region 720a in the first point cloud 700a to the same region 720b in the second point cloud 700b, there are much fewer points in the regions 710b and 720b than in the regions 710a and 720a. This implies that herein disclosed technique for determining depth values yield less noise, with less sampled geometric noise that thus yields a cleaner point cloud.

[0072] Fig. 8 schematically illustrates, in terms of a number of structural units, the components of an image processing device 800 according to an embodiment. Processing circuitry 810 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 910 (as in Fig. 9), e.g. in the form of a storage medium 830. The processing circuitry 810 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

[0073] Particularly, the processing circuitry 810 is configured to cause the image processing device 800 to perform a set of operations, or steps, as disclosed above. For example, the storage medium 830 may store the set of operations, and the processing circuitry 810 maybe configured to retrieve the set of operations from the storage medium 830 to cause the image processing device 800 to perform the set of operations. The set of operations maybe provided as a set of executable instructions. Thus the processing circuitry 8io is thereby arranged to execute methods as herein disclosed. The storage medium 830 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory. The image processing device 800 may further comprise a communications (comm.) interface 820 at least configured for communications with other entities, functions, nodes, and devices, e.g., for receiving input to the image processing device 800 from another entity, function, node, or device, and for providing output to another entity, function, node, or device. As such the communications interface 820 may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry 810 controls the general operation of the image processing device 800 e.g. by sending data and control signals to the communications interface 820 and the storage medium 830, by receiving data and reports from the communications interface 820, and by retrieving data and instructions from the storage medium 830. Other components, as well as the related functionality, of the image processing device 800 are omitted in order not to obscure the concepts presented herein.

[0074] The image processing device 800 maybe provided as a standalone device or as a part of at least one further device. Thus, a first portion of the instructions performed by the image processing device 800 maybe executed in a first device, and a second portion of the of the instructions performed by the image processing device 800 may be executed in a second device; the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the image processing device 800 maybe executed. Hence, the methods according to the herein disclosed embodiments are suitable to be performed by an image processing device 800 residing in a cloud computational environment. Therefore, although a single processing circuitry 810 is illustrated in Fig. 8 the processing circuitry 810 maybe distributed among a plurality of devices, or nodes. The same applies to the computer program 920 of Fig. 9.

[0075] Fig. 9 shows one example of a computer program product 910 comprising computer readable storage medium 930. On this computer readable storage medium 930, a computer program 920 can be stored, which computer program 920 can cause the processing circuitry 810 and thereto operatively coupled entities and devices, such as the communications interface 820 and the storage medium 830, to execute methods according to embodiments described herein. The computer program 920 and / or computer program product 910 may thus provide means for performing any steps as herein disclosed.

[0076] In the example of Fig. 9, the computer program product 910 is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product 910 could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 920 is here schematically shown as a track on the depicted optical disk, the computer program 920 can be stored in any way which is suitable for the computer program product 910. The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.

Claims

CLAIMS1. A method for obtaining a point cloud of a scene (400) from a Neural Radiance Field, NeRF, model of the scene (400), the method being performed by an image processing device (200, 800), the method comprising: obtaining (S102) a NeRF model of the scene (400); determining (S106) a depth value, d*, and a colour value for one point (440a, 540a) in the cloud via ray tracing in the NeRF model, wherein said one point (440a, 540a) corresponds to one pixel in a pixel patch (470, 570) comprising at least three pixels; and adding (S108) said one point (440a, 540a), including its colour value and its depth value, to the point cloud only in case the depth value d* of said one pixel fulfils a distance criterion with respect to depth values of other pixels in the pixel patch (470, 570).

2. The method according to claim 1, wherein, according to the ray tracing, as many rays (4603:4600, 560a, 560b) as there are pixels in the pixel patch (470, 570) are passed through the NeRF model of the scene (400), with one ray for each respective pixel in the pixel patch (470, 570) for determining one respective depth value for each of the at least three pixels in the pixel patch (470, 570).

3. The method according to claim 2, wherein the method further comprises: obtaining (S104) a set of images from which the NeRF model was generated, and a set of camera poses, with one camera pose per image, wherein the pixel patch (470, 570) is located in one of the images (450), and wherein all the at least three rays (4603:4600, 560a, 560b) have a common origin (480, 580) as given by the camera pose of said one of the images.

4. The method according to claim 3, wherein said one point (440a, 540a) has a location in the scene (400) where the ray (460a, 560a) of said one pixel hits a main surface of an object (420) in the scene (400), and wherein the depth value d* of saidone point (440a, 540a) is given by a distance from the origin (480, 580) to said one point (440a, 540a) in the scene (400).

5. The method according to claim 4, wherein the ray (460a, 560a) of said one pixel is discretized into a set of points, where each of the points of the ray (460a, 560a) is associated with a respective colour vector and density value as given by the NeRF model, wherein the location of said one point (440a, 540a) is given by the density values, and wherein the colour value is given by the colour vectors.

6. The method according to any preceding claim, wherein the pixel patch (470, 570) is of size w x h pixels, where w, h > 1.

7. The method according to claim 6, wherein w = h = 3, and wherein the pixel patch (470, 570) is centered at said one pixel.

8. The method according to any of claims 1 to 5, wherein the size of the pixel patch (470, 570) is dependent on a location of said one pixel in said one image (450), and wherein the size is smaller in case the location of said one pixel is at a border of said one image (450) than in case the location of said one pixel is not at any border of said one image (450).

9. The method according to any of claims 1 to 7, wherein in a case a location of said one pixel in said one image (450) is at a border of said one image (450), the depth value of any pixel in the pixel patch (470, 570) falling outside said one image (450) is set to a value larger than a predefined value.

10. The method according to any preceding claim, wherein the distance criterion is fulfilled when the depth value d* is not more than a threshold value larger than a smallest depth value of all pixels in the pixel patch (470, 570).

11. The method according to any preceding claim, wherein the distance criterion is fulfilled whenfor some constant E, and where dmindx yis a smallest depth value of all thepixels in the pixel patch (470, 570).

12. The method according to any of claims 1 to 10, wherein the distance criterion is fulfilled whenfor some value a > 0, and where dmin= dx yisasmallest depth value of all thepixels in the pixel patch (470, 570).

13. The method according to any preceding claim, wherein the point cloud is iteratively constructed by a depth value d* and a colour value being repeatedly determined for a respective point in the point cloud, and said respective point, including its depth value and its colour value, being added to the point cloud.

14. An image processing device (200, 800) for obtaining a point cloud of a scene (400) from a Neural Radiance Field, NeRF, model of the scene (400), the image processing device (200, 800) comprising processing circuitry (810), the processing circuitry being configured to cause the image processing device (200, 800) to: obtain a NeRF model of the scene (400); determine a depth value, d*, and a colour value for one point (440a, 540a) in the cloud via ray tracing in the NeRF model, wherein said one point (440a, 540a) corresponds to one pixel in a pixel patch (470, 570) comprising at least three pixels; and add said one point (440a, 540a), including its colour value and its depth value, to the point cloud only in case the depth value d* of said one pixel fulfils a distance criterion with respect to depth values of other pixels in the pixel patch (470, 570).

15. The image processing device (200, 800) according to claim 14, further being configured to perform the method according to any of claims 2 to 13.

16. A computer program (920) for obtaining a point cloud of a scene (400) from a Neural Radiance Field, NeRF, model of the scene (400), the computer programcomprising computer code which, when run on processing circuitry (810) of an image processing device (200, 800), causes the image processing device (200, 800) to: obtain (S102) a NeRF model of the scene (400); determine (S106) a depth value, d*, and a colour value for one point (440a, 540a) in the cloud via ray tracing in the NeRF model, wherein said one point (440a,540a) corresponds to one pixel in a pixel patch (470, 570) comprising at least three pixels; and add (S108) said one point (440a, 540a), including its colour value and its depth value, to the point cloud only in case the depth value d* of said one pixel fulfils a distance criterion with respect to depth values of other pixels in the pixel patch (470, 570).

17. A computer program product (910) comprising a computer program (920) according to claim 16, and a computer readable storage medium (930) on which the computer program is stored.

Images