3D NERF model in absolute scale

Abstract

There is provided techniques for providing a 3D NeRF model of a scene in absolute scale. A method is performed by an image processing device. The method comprises obtaining a 3D NeRF model of the scene, a set of images depicting the scene and cameras positions based on which the 3D NeRF model was generated, localization of a 3D point in the 3D NeRF model, and an action with respect to the 3D point. The cameras positions are in absolute scale. The method comprises associating the 3D point with absolute scale coordinates in 3D space by constraining the 3D point to a coordinate system defined by at least two of the camera poses. The method comprises performing the action with respect to the 3D point.

Description

[0001] 3D NERF MODEL IN ABSOLUTE SCALE

[0002] TECHNICAL FIELD

[0003] Embodiments presented herein relate to a method, an image processing device, a computer program, and a computer program product for providing a three- dimensional Neural Radiance Fields model of a scene in absolute scale.

[0004] BACKGROUND

[0005] In general terms, neural volumetric representations, such as Neural Radiance Fields (NeRF), can be used for problems of learning a three-dimensional (3D) scene representation fl from sparse input views as provided by a set of input images I and corresponding camera poses C (representing the positions and directions from which the set of images was captured). NeRF can be used to model the scene as a continuous five-dimensional (5D) function that outputs radiance emitted in each direction (0, (p), at each point in 3D space (x,y,z). In other words, NeRF can be used to learn a 5D vector-valued function whose input is a 3D location (x,y,z) and a two- dimensional (2D) viewing direction (0, cp), and whose output is emitted color (defined by the radiance) r and a volume density (structure) o, also referred to as structure. Reference is here made to the block diagram 100 in Fig. 1. According to the block diagram 100, camera poses C corresponding to a set of input images I are estimated by a Structure from Motion (SfM) processing block 110. A continuous 3D representation fl of the scene is, based on the camera poses C and the set of input images I, learned by neural network (NN) training in a NeRF processing block 120. The 3D representation fl can then be exposed to a user, or some other application, by a NeRF viewer block 130. After a NeRF model is learned, it can be used to render scene views from either user-controlled or some predefined input camera poses. In general terms, the rendering procedure involves a ray casting technique, performed at the pixels of an image that would correspond to the current camera pose. In more detail, for each ray, colour and depth information are extracted from the learned NeRF model via ray tracing. This information (i.e. , the extracted colour and depth information) is assigned to the corresponding image pixel. During raytracing, the learned NeRF model is queried in several locations (i.e., points) along the ray, and the outputs from the NeRF model are aggregated into one single value for the corresponding pixel. In the end, if all pixels of an image are processed (rendered), this produces what is often referred to as an RGB-D image (i.e., a red (R), green (G), blue (B) image with depth (D)).

[0006] A (3D) point cloud can be extracted from the NeRF model through the rendering of arbitrary pixels from various camera views. In particular, each sampled pixel, that has assigned colour and depth values as rendered from the NeRF model, directly corresponds to a point in 3D space, coloured with the same colour and located at a known displacement from the camera (in accordance with the camera’s view angle, position, pixel location and the depth value). These points, corresponding to different pixels of different camera poses, can thus be aggregated into a single 3D point cloud. In Fig. 2 is schematically shown a setup 200 illustrating how a continuous 3D representation (hereinafter referred to as a 3D NeRF model) of a scene 210 captured in NN weights (dashed lines) can be presented to a user in terms of a rendered (novel) 2D image 220 depicting the scene 240 (solid lines). The 2D image 220 represents a user-defined 2D view 230 of the 3D NeRF model.

[0007] The rendered (novel) view is just a 2D image, as the input 2D images I used to train NeRF, except that the 2D image associated with the view is rendered from the learned NeRF 3D representation. For simplicity, the rendered (novel) views are denoted Inv, representing the 2D image rendered from a corresponding (novel) camera pose Cnv. Thus, worth noting, is that the 3D representation fl can be exposed not only with respect to views corresponding to the camera poses C but from any view. That is, the (novel) camera poses can be defined by user input as provided to the NeRF viewer block 130. Contrary to the limited set of inputs {I , C}, there could be an infinite number of values {Inv ,Cnv}. Because of this ability for accurate scene modeling and novel-view synthesis, NeRF processing can be used for many applications, such as extended reality applications, digital twins, robotics, medical applications, digital asset creation, generation of Business Information Modelling (BIM) representations, etc.

[0008] The block diagram in Fig. 1 and the scheme in Fig. 2 allow visual inspection of the 3D representation of the scene, which might be sufficient for some applications. However, the processes of digital twin generation and BIM generation require measurements to be performed in the 3D representation (i.e., the 3D NeRF model) of the scene in order for dimensions of the objects depicted in the scene to be captured. These processes also involve associating geolocated labels with different parts of the scene. Also other applications could benefit from measurements in absolute scale and the insertion of geolocated labels to be available in the 3D NeRF model. The block diagram in Fig. 1 and the scheme in Fig. 2 do not enable absolute scale measurements to be made. In this respect, the processes of performing measurements and providing labels in 3D NeRF models in absolute scale are challenging for many reasons.

[0009] One reason is that, in the 3D NeRF model, the continuous representation of the scene is captured in NN weights, not in a static 3D point cloud. Another reason is that the NeRF viewer block 130 allows for rendering 2D images from any (novel) view, which means that there could be an infinite space of 2D views, compared to the fixed set of camera poses for the 2D input images. As a result of at least these reasons, it is challenging to establish 2D-3D relation between an image in which measurements are to be made or a geolocated label is to be inserted and an association of those actions with the 3D NeRF model of the scene.

[0010] Hence, there is a need for technologies for improved interaction with 3D NeRF models, for example with respect to measurements and labeling.

[0011] SUMMARY

[0012] An object of embodiments herein is to provide means for improved interaction with 3D NeRF models, for example with respect to measurements and labeling.

[0013] A particular object is to enable an action, requiring a 3D point in the 3D NeRF model to have coordinates in absolute scale (such as in metric scale), to be performed in the 3D NeRF model of a scene.

[0014] According to a first aspect there is presented a method for providing a 3D NeRF model of a scene in absolute scale. The method is performed by an image processing device. The method comprises obtaining a 3D NeRF model of the scene, a set of images depicting the scene and cameras positions based on which the 3D NeRF model was generated, localization of a 3D point in the 3D NeRF model, and an action with respect to the 3D point. The cameras positions are in absolute scale. The method comprises associating the 3D point with absolute scale coordinates in 3D space by constraining the 3D point to a coordinate system defined by at least two of the camera poses. The method comprises performing the action with respect to the 3D point.

[0015] According to a second aspect there is presented an image processing device for providing a 3D NeRF model of a scene in absolute scale. The image processing device comprises processing circuitry. The processing circuitry is configured to cause the image processing device to obtain a 3D NeRF model of the scene, a set of images depicting the scene and cameras positions based on which the 3D NeRF model was generated, localization of a 3D point in the 3D NeRF model, and an action with respect to the 3D point. The cameras positions are in absolute scale. The processing circuitry is configured to cause the image processing device to associate the 3D point with absolute scale coordinates in 3D space by constraining the 3D point to a coordinate system defined by at least two of the camera poses. The processing circuitry is configured to cause the image processing device to perform the action with respect to the 3D point.

[0016] According to a third aspect there is presented a computer program for providing a 3D NeRF model of a scene in absolute scale. The computer program comprises computer code which, when run on processing circuitry of an image processing device, causes the image processing device to perform actions. One action comprises the image processing device to obtain a 3D NeRF model of the scene, a set of images depicting the scene and cameras positions based on which the 3D NeRF model was generated, localization of a 3D point in the 3D NeRF model, and an action with respect to the 3D point. The cameras positions are in absolute scale. One action comprises the image processing device associate the 3D point with absolute scale coordinates in 3D space by constraining the 3D point to a coordinate system defined by at least two of the camera poses. One action comprises the image processing device perform the action with respect to the 3D point.

[0017] According to a fourth aspect there is presented a computer program product comprising a computer program according to the third aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium could be a non-transitory computer readable storage medium. Advantageously, these aspects provide means for improved interaction with 3D NeRF models, for example with respect to measurements and labeling.

[0018] Advantageously, these aspects enable an action (such as performing a distance measurement or inserting a geolocated label), requiring a 3D point in the 3D NeRF model to have coordinates in absolute scale, to be performed in the 3D NeRF model of a scene.

[0019] Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0020] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:

[0023] Fig. 1 is a block diagram comprising an SfM processing block, a NeRF processing block , and a NeRF viewer block according to an example;

[0024] Fig. 2 schematically illustrates a setup showing how a 3D NeRF model of a scene can be presented to a user according to an example;

[0025] Fig. 3 is a block diagram of an image processing device according to an embodiment;

[0026] Fig. 4 is a flowchart of methods according to embodiments;

[0027] Figs. 5 and 6 are block diagrams of image processing devices according to embodiments; Fig. 7 schematically illustrates a setup showing an object of a scene in a 3D NeRF model according to an embodiment;

[0028] Fig. 8 schematically illustrates a procedure for determining a depth value for a 3D point along a ray according to an embodiment;

[0029] Fig. 9 schematically illustrates an example of a user-defined 2D view with measuring and labeling functionality according to an embodiment;

[0030] Fig. 10 is a schematic diagram showing structural units of an image processing device according to an embodiment; and

[0031] Fig. 11 shows one example of a computer program product comprising computer readable storage medium according to an embodiment.

[0032] DETAILED DESCRIPTION

[0033] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.

[0034] As noted above, there is a need for technologies for improved interaction with 3D NeRF models, for example with respect to measurements and labeling.

[0035] The embodiments disclosed herein therefore relate to techniques for providing a 3D NeRF model of a scene 210 in absolute scale (such as in metric scale, e.g., in metric or imperial units). In order to obtain such techniques there is provided an image processing device, a method performed by the image processing device, a computer program product comprising code, for example in the form of a computer program, that when run on an image processing device, causes the image processing device 300 to perform the method. As an introductory example, the process of performing absolute scale measurements and inserting geolocated labels in a 2D image Inv rendered by a NeRF viewer involves relating pixels in the 2D image with a position in 3D space, thus establishing a 2D-3D correspondence. However, since learned NeRF 3D model (i.e., the representation fl) is not a traditional point cloud, or mesh model, the task of establishing this 2D-3D correspondence is not straightforward. One reason for this is that the 2D image corresponds to a 2D view that is transient and cannot be used for information anchoring to specific pixels of specific images of the set of input images I. In the learned NeRF 3D representation, there are inherently no discrete objects and there are no static discrete 3D point clouds. The herein disclosed embodiments are therefore based on establishing a reference in 3D space based on the coordinate system defined by the set of input cameras (i.e., the camera poses C associated with the set of images I) which already are in a Universal Transverse Mercator (UTM) coordinate system.

[0036] In Fig. 3 is provided a block diagram of an image processing device 300 configured for for providing a 3D NeRF model of a scene 210 in absolute scale. The image processing device 300 is configured to receive as input a learned 3D NeRF model (i.e., a 3D representation fl of the scene 210), a set of input 2D images I depicting the scene, the corresponding input camera poses C and input with respect to interaction with the 3D NeRF model. The image processing device 300 provides as output the absolute scale coordinates for 3D points in the 3D NeRF model, where the 3D points are selected through the obtained input. The image processing device 300 comprises a number of blocks 310:360. The function of each block will be disclosed below in conjunction with the description of the method in the flowchart of Fig. 4.

[0037] Fig. 4 is a flowchart illustrating embodiments of methods for providing a 3D NeRF model of a scene 210 in absolute scale. The methods are performed by the image processing device 300. The methods are advantageously provided as computer programs.

[0038] S102: The image processing device 300 obtains a 3D NeRF model of the scene 210, a set of images depicting the scene 210, and cameras positions based on which the 3D NeRF model was generated, localization of a 3D point in the 3D NeRF model, and an action with respect to the 3D point. In Fig. 3, step S102 is implemented by the obtain block 310. Further aspects of step S102 will be disclosed below. The cameras positions are in absolute scale. The action requires the 3D point to have coordinates in absolute scale.

[0039] The information obtained in step S102 is then used for establishing a reference in 3D space, as in step S104.

[0040] S104: The image processing device 300 associates the 3D point with absolute scale coordinates (e.g., in metric or imperial units) in 3D space by constraining the 3D point to a coordinate system defined by at least two of the camera poses. In Fig. 3, step S104 is implemented by the associate block 320. Further aspects of step S104 will be disclosed below.

[0041] Thus, a reference in 3D space is established based on the coordinate system defined by the set of images and corresponding camera poses Ci:C6 (which are already in absolute scale). The 3D point in absolute scale coordinates can then be utilized in different actions performed by the image processing device 300, as in step S106.

[0042] S106: The image processing device 300 performs the action with respect to the 3D point. In Fig. 3, step S106 is implemented by the action block 330. Further aspects of step S106 will be disclosed below.

[0043] As specified in step S104, the 3D point is constrained to a coordinate system defined by at least two of the camera poses. In this respect, the image processing device 300 can provide as output not only the 3D NeRF model of the scene 210 in absolute scale, but also based on how many camera poses the 3D NeRF model of the scene 210 in absolute scale was found.

[0044] Embodiments relating to further details of providing a 3D NeRF model of a scene 210 in absolute scale as performed by the image processing device 300 will now be disclosed with continued reference to Fig. 4.

[0045] Further aspects of step S102, for exampling relating to generating a 3D representation with accurate scale and geolocation, will be disclosed next.

[0046] As disclosed above, the obtained camera poses are in absolute scale. In further detail, the camera poses C as obtained from SfM processing on a set of 2D images (see, Fig. 1) is always correct up to a scaling factor and an arbitrary coordinate system. Two different embodiments for how the camera poses can be obtained in absolute scale based on SfM processing will be disclosed next with reference to Fig. 5 and Fig. 6, respectively.

[0047] In a first embodiment, the camera poses (viz., camera positions) in absolute scale (i.e., the camera positions in absolute scale) are obtained by the image processing device 300 extracting exchangeable image file format (EXIF) data from the set of images. Reference is here made to the block diagram in Fig. 5. The block diagram 500 can be implemented by the image processing device 300. According to the block diagram 500, camera poses C corresponding to a set of input images I are estimated by an SfM processing block 510. Further, positioning information, such as Global Navigation Satellite System (GNSS) coordinates, are extracted from the EXIF data of each input image. The GNSS coordinates are, in a transform block, converted to UTM coordinates (i.e., coordinates in a UTM coordinate system). Since each image is associated with a respective camera pose, this also transforms the camera poses C to a UTM coordinate system, hence yielding camera poses (viz., camera positions) in absolute scale. In some examples, the Kabsch-Umeyama algorithm is used to align the camera poses obtained from the SfM processing to the set of camera poses obtained from the GNSS coordinates. A continuous 3D representation fl of the scene is, based on the camera poses C in absolute scale and the set of input images I, learned by NN training in a NeRF processing block 520. The 3D representation fl in absolute scale can then be exposed to a user, or some other application, by a NeRF viewer block 530.

[0048] In some cases, reliable GNSS data is not available. Then, the camera poses (viz., camera positions) in absolute scale need to be obtained using some other scheme than in Fig. 5. One alternative is to use fiducial markers. Two such fiducial markers Ti and T2 are illustrated in Fig. 2. In particular, in a second embodiment, the camera poses (viz., camera positions) in absolute scale are obtained by the image processing device 300 detecting, in the set of images, geolocated fiducial markers Ti, T2 as placed in the scene 210 (and captured in the input images), and then extracting absolute scale coordinates of the set of images from the geolocated fiducial markers Ti, T2. AS a non-limiting example, the geolocated fiducial markers could be AprilTags placed in the scene. Reference is here made to the block diagram in Fig. 6. The block diagram 6oo can be implemented by the image processing device 300. According to the block diagram 600, camera poses C corresponding to a set of input images I are estimated by an SfM processing block 610. Either the geolocated fiducial markers contain some absolute scale information (e.g., depicting an object such as a rectangle with known dimensions) or the GNSS coordinates of the geolocated fiducial markers could be estimated by positioning a GNSS receiver over their positions, thus yielding a set of GNSS reference points; one for each geolocated fiducial marker. These GNSS reference points are automatically detected by detecting the geolocated fiducial markers and used to transform and scale the camera poses C to a UTM coordinate system. The associated processing is implemented in a detect fiducial markers and transform block 615. A continuous 3D representation fl of the scene is, based on the camera poses C in absolute scale and the set of input images I, learned by NN training in a NeRF processing block 620. The 3D representation fl in absolute scale can then be exposed to a user, or some other application, by a NeRF viewer block 630.

[0049] In some aspects, the localization of the 3D point X is given by a pixel P in a 2D image of the user-defined 2D view 230 of the 3D NeRF model. That is, in some embodiments, the localization of the 3D point X is given by specification of a pixel P in a user-defined 2D view 230 of the 3D NeRF model. Reference is here again made to Fig. 2 where a pixel P is identified in the user-defined 2D view 220. The pixel P in the user-defined 2D view 230 corresponds to the 3D point X in the 3D NeRF model of the scene 210. In some aspects, the pixel P is identified by user input. That is, in some embodiments, the pixel P is marked in the user-defined 2D view 230 through user input. This user input might be obtained via the NeRF view block 530, 630.

[0050] Further aspects of step S104, for example relating to associating a 3D point in the NeRF representation with the pixel in the user-defined 2D view 230 of the 3D NeRF model, will be disclosed next, with reference to Fig. 7 and Fig. 8.

[0051] In some aspects, the 3D point is found via ray tracing. Therefore, in some embodiments, the image processing device 300 is configured to perform (optional) step 8104a as part of constraining the 3D point X to the coordinate system.

[0052] 8104a: The image processing device 300 performs ray tracing along rays 730, 810, 840-1, 840-2 in the 3D NeRF model from at least two of the camera poses Ci:C6 in the 3D NeRF model and from the user-defined 2D view 230, 820 (defined by NV in Fig. 7)-

[0053] A depth value for the 3D point X can then be determined as the 3D point X along the ray 730, 810 from the user-defined 2D point that has minimum distance to the rays 840-1, 840-2 from the at least two camera poses Ci:C6.

[0054] Reference is here made to Fig. 7 which schematically illustrates a setup 700 showing an object O of a scene in a 3D NeRF model. The scene, as represented by the object O, has thus been captured in a set of images by a camera placed at positions Ci:C6. The scene has a coordinate system 710 in absolute scale as given by the camera poses Ci:C6. The set of images has then been used to train the NeRF processing block 520, 620 and to generate digital a 3D representation (i.e., a 3D NeRF model) of the scene, as can be exposed to a user using the NeRF viewer block 530, 630 in terms of user- defined 2D views. In Fig. 7 is further illustrated that the object O is viewed from a user-defined 2D view at a position denoted NV. As can be seen, the user-defined 2D view is different from the camera poses Ci:C6. The user-defined 2D view can be provided as a 2D image by the NeRF viewer block 530, 630. Each pixel in the 2D image corresponds to a ray 730 through the 3D space. Of interest is to obtain a reliable estimate of the depth from the position NV to the surface of the object O hit by the ray, in the coordinate system 710 obtained from the camera poses Ci:C6. The interval of the ray 730 with maximum density is illustrated as a segment 720 (and not a single point along the ray 730). This is due to the uncertainty of the depth estimation from the position NV. The volume of the segment 720 could be defined as a three-sigma span around its peak, or fixed spread, related to an accepted measurement error, e.g., 1-3 cm, depending on the scale. As will be demonstrated next, this uncertainty can be resolved according to the procedure in Fig. 8.

[0055] In Fig. 8 is provided illustrations of a procedure for determining the depth value for the 3D point X along the ray 810. In Fig. 8(a) is shown a setup 800a with the same ray as in Fig. 7 but from another perspective. In more detail, in Fig. 8 is illustrated a ray 810 originating from the user-defined 2D view 820 at position NV and where the 3D point X is placed somewhere in the segment 830. In view of Fig. 7, the point X is thus the point in the segment 720 along the ray 730 at which the ray 730 hits the surface of the object O. The color and volume density in the 3D NeRF model are accumulated along the ray 8io to find an initial segment 830, where the ray 810 likely hits the surface of the object. In Fig. 8(b) is shown a setup 800b where the segment 830 is intersected with rays 840-1, 840-2 from the camera poses Ci and C2 from which images were captured to train the 3D NeRF model. Camera poses from which rays do not intersect with the segment 830 are not considered; for some camera poses this part of the scene might be out of view. Further, rays for which the maximum density (as integrated along the ray) is outside initial segment are also not considered; for some rays, this part of the scene might be occluded, and / or they might hit another surface before passing the segment 830. The sets of remain camera rays 840-1, 840-2 intersecting the segment 830 and having maximum density inside it are used to calculate the position of the 3D point X along the ray 810. This position can be calculated as the point X in the 3D space that has minimum distance (e.g., in Euclidean distance sense) to the set of rays 840-1, 840-2. In Fig. 8(c) is shown a setup 800c showing the position of the 3D point X along the ray 810, where this 3D point is associated with the pixel in the 2D image of the user-defined 2D view at position NV in Fig. 7. The dashed arrow representing the ray 810 thus has a length corresponding to the estimated depth between the position NV and the surface of the object O in Fig. 7.

[0056] Further aspects of step S106, for exampling relating to different types of actions that can be performed with respect to the 3D point (with absolute scale coordinates), will be disclosed next.

[0057] In some aspects, the action involves either to perform a distance measurement with the 3D point as start point and / or to insert a geolocated label at the 3D point. That is, in some embodiments, the action pertains to at least one of: performing a distance measurement with the 3D point Xi (see, Fig. 9) as a start point, associating the 3D point Xi with a geolocated label. Here, the distance measurement is given in absolute scale, and the geolocated label is given in absolute scale coordinates.

[0058] Further aspects of performing a distance measurement with the 3D point as start point will be disclosed next.

[0059] The distance measurement can be initiated with one 3D point being selected as start point (below referred to as 3D start point) and another 3D point being selected as end point (below referred to as 3D end point) for the distance measurement. Thus, for a distance measurement to be performed, also an end point of the distance measurement is needed.

[0060] The 3D start point and the 3D end point can be selected as respective pixels in one or two rendered (novel) 2D views Inv. These pixels can thus each be selected from one rendered (novel) view (i.e., in one rendered 2D image Inv) or from two different rendered (novel) views (i.e., with the pixels of the start and end points belonging to two different rendered 2D images Inv). That is, in one embodiments, both the first 3D point Xi and the second 3D point X2 (see, Fig. 9) are associated with one same user- defined 2D view 230, 820 of the 3D NeRF model, whereas in another embodiment, the first 3D point Xi is associated with a first user-defined 2D view 230, 820 of the 3D NeRF model and the second 3D point X2 is associated with a second user-defined 2D view 230, 820 of the 3D NeRF model. The (novel) views can be freely selected by means of user input via the NeRF viewer and are generated from the 3D NeRF model of the scene 210.

[0061] The absolute scale coordinates for the 3D end point can be obtained by repeating steps S102 and S104 for a second 3D point in the 3D NeRF model. Thus, in some embodiments, the 3D point X is a first 3D point Xi, and the image processing device 300 is configured to perform (optional) steps S102-2 and S104-2.

[0062] S102-2: The image processing device 300 obtains localization of a second 3D point X2 in the 3D NeRF model.

[0063] S104-2: The image processing device 300 associates the second 3D point X2 with absolute scale coordinates in 3D space by constraining the second 3D point X2 to a coordinate system 710 defined by at least two of the camera poses Ci:C6.

[0064] Here, ray tracing, as in step 8104a but for the second 3D point X2 instead of the first 3D point Xi, could be used as part of constraining the second 3D point X2 to a coordinate system 710 defined by at least two of the camera poses Ci:C6. Hence, in some embodiments, step S104-2 comprises a step SiO4-2a of performing raytracing along rays 730, 810, 840-1, 840-2 in the 3D NeRF model from at least two of the camera poses Ci:C6 in the 3D NeRF model and from the user-defined 2D view 230, 820 for constraining the second 3D point X2 to the coordinate system 710. The first 3D point Xi can then serve as a start point for the distance measurement and the second 3D point X2 can then serve as a stop point for the distance measurement. Hence, in some embodiments, the image processing device 300 is configured to perform (optional) step S106-2 as part of performing the action with respect to the 3D point X.

[0065] S106-2: The image processing device 300 determines a distance Di between the first 3D point Xi and the second 3D point X2 in absolute scale by calculating a difference between the absolute scale coordinates of the first 3D point Xi and the absolute scale coordinates of the second 3D point X2. In Fig. 3, step S106-2 is implemented by the distance block 340.

[0066] An illustrative example showing the distance Di between a 3D start point Xi and a 3D end point X2 is provided in Fig. 9.

[0067] In some examples, the distance measurement is represented by the Euclidean distance between the 3D start point and the 3D end point. That is, in some examples, the distance Di (in absolute scale) is determined as: where Xstartrepresents the absolute scale coordinates of the first 3D point and Xstoprepresents the absolute scale coordinates of the second 3D point.

[0068] Further aspects of associating the 3D point Xi with a geolocated label will be disclosed next.

[0069] Generally, associating the 3D point Xi with a geolocated label involves inserting a geolocated label Li at the absolute scale coordinates of the 3D point Xi. It is here thus assumed that the 3D point Xi is selected (i.e., that a localization of a 3D point in the 3D NeRF model is obtained by the image processing device 300 as in step S102) where the geolocated label is to be inserted. The 3D point can be selected as a pixel in a rendered (novel) 2D view Inv. This pixel is a projection of th33D point which is to be annotated. Hence, in some embodiments, the image processing device 300 is configured to perform (optional) step S106-4 as part of performing the action with respect to the 3D point X. S106-4: The image processing device 300 associates the 3D point Xi with a geolocated label Li having geolocation information. In Fig. 3, step S106-4 is implemented by the label block 350.

[0070] The absolute coordinates of the 3D point can be calculated as in step S104 (possibly including optional step 8104a). Therefore, the label associated with the 3D point is geolocated. Thus, the geolocation information is given by the absolute scale coordinates of the 3D point Xi.

[0071] An illustrative example showing a geolocated label Li inserted at a 3D point Xi is provided in Fig. 9.

[0072] In some aspects the labeling operation is persistent. This implies that the geolocated label is stored. Hence, in some embodiments, the image processing device 300 is configured to perform (optional) step S106-6 as part of performing the action with respect to the 3D point X.

[0073] S106-6: The image processing device 300 stores the geolocated label Li with the geolocation information. In Fig. 3, step S106-6 is implemented by the store block 360.

[0074] In this way, the 3D point will be associated with the geolocated label Li even if a new rendered (novel) view is selected, as well as during further sessions of interactions with the 3D NeRF model of the scene 210. This is despite of the transient nature of the (novel) view used when the geolocated label was inserted. Hence, any other future rendered (novel) view, to which the same 3D point Xi is re-projected, will include the same geolocated label Li.

[0075] Reference is next made to Fig. 9 in which is provided an illustrative example of a user-defined 2D view 900 (as represented by a 2D image) with measuring and labeling functionality. In Fig. 9 is illustrated a scene 910 where a geolocated label Li with geolocation information in absolute scale has been inserted at the 3D point Xi in the 3D NeRF model and where a distance Di in absolute scale has been measured between the 3D point Xi in the 3D NeRF model and the 3D point X2 in the 3D NeRF model. That is, even though the user interacts with one or more 2D image(s), the measurements and the insertion of labels are performed using the 3D NeRF model of the scene. One non-limiting and illustrative example of providing a 3D NeRF model of a scene 210, 700, 800 in absolute scale where the NeRF viewer (as implemented by the NeRF viewer block) receives user input for either measuring the distance Di between the two 3D points Xi and X2 in the scene or for labeling the 3D point Xi in the scene will be disclosed next.

[0076] In a step S201, the NeRF viewer receives user input for the user to explore the scene 910 from different viewing directions. For each viewing direction a (novel) view 2D image, as generated from the 3D NeRF model, is rendered on a screen.

[0077] In step S202, the NeRF viewer receives user input responsive to the user having activated a measuring / labeling mode. For this purpose, the NeRF viewer provides a visual user interface, as displayed on the screen, from which the user is enabled to select the measuring / labeling mode from a menu.

[0078] In step S203, the NeRF viewer receives user input that specifies a pixel in the 2D image as currently shown on the screen. The pixel is either a first measuring point or position for label insertion. The specification of the pixel represents the localization of the localization of a 3D point in the 3D NeRF model in step S102. This triggers the image processing device 300 to perform the herein disclosed method for providing a 3D NeRF model of a scene 910 in absolute scale, including step S102, step S104, and S106.

[0079] In step S204, depending on whether the user input defined a measurement action or a labeling action, the image processing device either computes a distance Di between two 3D points Xi and X2 (and thus step S106-2 is performed) or inserts a geolocated label Li at one 3D point Xi (and thus step S106-4 (and possibly step S106-6) is performed). In case step S106-2 is performed, then also steps S102-2 and S104-2 are performed.

[0080] Step S204 is only completed in case the measurement action or the labeling action can be successfully performed by the image processing device. One non-limiting example is here that the volume with maximum density along the ray from from the user-defined 2D view is crossed by rays from at least two camera poses C corresponding to the 2D input images. Therefore, in case step S204 cannot be completed (e.g., when there are uncertainties in the depth value estimation for one of the 3D points), step S205 is entered. In step S205, the NeRF viewer provides visual feedback to the user that the measurement action or the labeling action cannot be successfully performed for the pixel specified in step S203. The user is thereby prompted to select another pixel in the 2D image as currently shown on the screen, or to first select another (novel) view and then select a pixel in the 2D image of that view.

[0081] Fig. 10 schematically illustrates, in terms of a number of structural units, the components of an image processing device 1000 according to an embodiment. Processing circuitry 1010 is provided using any combination of one or more of a suitable central processing unit (CPU), graphics processing unit (GPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 1110 (as in Fig. 11), e.g. in the form of a storage medium 1030. The processing circuitry 1010 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

[0082] Particularly, the processing circuitry 1010 is configured to cause the image processing device 1000 to perform a set of operations, or steps, as disclosed above. For example, the storage medium 1030 may store the set of operations, and the processing circuitry 1010 may be configured to retrieve the set of operations from the storage medium 1030 to cause the image processing device 1000 to perform the set of operations. The set of operations may be provided as a set of executable instructions.

[0083] Thus the processing circuitry 1010 is thereby arranged to execute methods as herein disclosed. The storage medium 1030 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory. The image processing device 1000 may further comprise a communications (comm.) interface 1020 at least configured for communications with other entities, functions, nodes, and devices, e.g., for receiving input to the image processing device 1000 from another entity, function, node, or device, and for providing output to another entity, function, node, or device.. As such the communications interface 1020 may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry 1010 controls the general operation of the image processing device 1000 e.g. by sending data and control signals to the communications interface 1020 and the storage medium 1030, by receiving data and reports from the communications interface 1020, and by retrieving data and instructions from the storage medium 1030. Other components, as well as the related functionality, of the image processing device 1000 are omitted in order not to obscure the concepts presented herein.

[0084] In general terms, the image processing device 1000 may be provided as a standalone device or as a part of at least one further device. Thus, a first portion of the instructions performed by the image processing device 1000 may be executed in a first device, and a second portion of the of the instructions performed by the image processing device 1000 maybe executed in a second device. For example, user-input and image-rendering aspects of the NeRF viewer (including selection of measurement points, selection of label position, etc.) could be performed on a user device (such as a desktop computer, a laptop computer, a tablet computer, or a smartphone), whilst the aspects of view generation, measurement calculation, label region calculation, etc. could be performed by at least one GPU enabled virtual machine in a localized or distributed cloud computational environment. The prerequisite steps of learning the 3D NeRF model parametrization and scaling to absolute scale could performed by (GPU enabled) virtual machines in a localized or distributed cloud computational environment. Hence, the methods according to the herein disclosed embodiments are suitable to be performed by an image processing device 1000 residing in a cloud computational environment. Therefore, although a single processing circuitry 1010 is illustrated in Fig. 10 the processing circuitry 1010 may be distributed among a plurality of devices, or nodes. The same applies to the computer program 1120 of Fig. 11. However, the herein disclosed embodiments are not limited to any particular number of devices on which the instructions performed by the image processing device 1000 may be executed.

[0085] Fig. 11 shows one example of a computer program product 1110 comprising computer readable storage medium 1130. On this computer readable storage medium 1130, a computer program 1120 can be stored, which computer program 1120 can cause the processing circuitry 1010 and thereto operatively coupled entities and devices, such as the communications interface 1020 and the storage medium 1030, to execute methods according to embodiments described herein. The computer program 1120 and / or computer program product 1110 may thus provide means for performing any steps as herein disclosed.

[0086] In the example of Fig. 11, the computer program product 1110 is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product 1110 could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 1120 is here schematically shown as a track on the depicted optical disk, the computer program 1120 can be stored in any way which is suitable for the computer program product 1110. The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.

Claims

CLAIMS1. A method for providing a 3D NeRF model of a scene (210, 910) in absolute scale, the method being performed by an image processing device (300, 1000), the method comprising: obtaining (S102) a 3D NeRF model of the scene (210, 910), a set of images depicting the scene (210, 910) and cameras positions (Ci:C6) based on which the 3D NeRF model was generated, localization of a 3D point (X, Xi, X2) in the 3D NeRF model, and an action with respect to the 3D point (X, Xi, X2), wherein the cameras positions (Ci:C6) are in absolute scale; associating (S104) the 3D point (X, Xi, X2) with absolute scale coordinates in 3D space by constraining the 3D point (X, Xi, X2) to a coordinate system (710) defined by at least two of the camera poses (Ci:C6); and performing (S106) the action with respect to the 3D point (X, Xi, X2).

2. The method according to claim 1, wherein the camera poses (Ci:C6) in absolute scale are obtained by the image processing device (300, 1000) extracting exchangeable image file format data from the set of images.

3. The method according to claim 1, wherein the camera poses (Ci:C6) in absolute scale are obtained by the image processing device (300, 1000) detecting, in the set of images, geolocated fiducial markers (Ti, T2) as placed in the scene (210, 910), and extracting absolute scale coordinates of the set of images from the geolocated fiducial markers (Ti, T2).

4. The method according to any preceding claim, wherein the localization of the 3D point (X, Xi, X2) is given by specification of a pixel (P) in a user-defined 2D view (230, 820) of the 3D NeRF model.

5. The method according to claim 4, wherein the pixel (P) is marked in the user- defined 2D view (230, 820) through user input.

6. The method according to claim 4 or 5, wherein constraining the 3D point (X, Xi, X2) to the coordinate system (710) comprises:performing (8104a, SiO4-2a) ray tracing along rays (730, 810, 840-1, 840-2) in the 3D NeRF model from at least two of the camera poses (Ci:C6) in the 3D NeRF model and from the user-defined 2D view (230, 820); and wherein a depth value for the 3D point (X, Xi, X2) is determined as the 3D point (X, Xi, X2) along the ray (730, 810) from the user-defined 2D point that has minimum distance to the rays (840-1, 840-2) from the at least two camera poses (Ci:C6).

7. The method according to any preceding claim, wherein the action pertains to at least one of: performing a distance measurement with the 3D point (X, Xi) as a start point, associating the 3D point (X, Xi) with a geolocated label.

8. The method according to any preceding claim, wherein the 3D point (X, Xi, X2) is a first 3D point (X, Xi), and wherein the method further comprises: obtaining (S102-2) localization of a second 3D point (X, X2) in the 3D NeRF model; associating (S104-2) the second 3D point (X, X2) with absolute scale coordinates in 3D space by constraining the second 3D point (X, X2) to a coordinate system (710) defined by at least two of the camera poses (Ci:C6); and wherein performing the action with respect to the 3D point (X, Xi, X2) comprises: determining (S106-2) a distance (Di) between the first 3D point (X, Xi) and the second 3D point (X, X2) in absolute scale by calculating a difference between the absolute scale coordinates of the first 3D point (X, Xi) and the absolute scale coordinates of the second 3D point (X, X2).

9. The method according to claim 8, wherein either both the first 3D point (X, Xi) and the second 3D point (X, X2) are associated with one same user-defined 2D view (230, 820) of the 3D NeRF model or the first 3D point (X, Xi) is associated with a first user-defined 2D view (230, 820) of the 3D NeRF model and the second 3D point (X, X2) is associated with a second user-defined 2D view (230, 820) of the 3D NeRF model.

10. The method according to any preceding claim, wherein performing the action with respect to the 3D point (X, Xi) comprises:associating (S106-4) the 3D point (X, Xi) with a geolocated label (Li) having geolocation information, wherein the geolocation information is given by the absolute scale coordinates of the 3D point (X, Xi).

11. The method according to any claim 10, wherein performing the action with respect to the 3D point (X, Xi) comprises: storing (S106-6) the geolocated label (Li) with the geolocation information.

12. An image processing device (300, 1000) for providing a 3D NeRF model of a scene (210, 910) in absolute scale, the image processing device (300, 1000) comprising processing circuitry (1010), the processing circuitry being configured to cause the image processing device (300, 1000) to: obtain a 3D NeRF model of the scene (210, 910), a set of images depicting the scene (210, 910) and cameras positions (Ci:C6) based on which the 3D NeRF model was generated, localization of a 3D point (X, Xi, X2) in the 3D NeRF model, and an action with respect to the 3D point (X, Xi, X2), wherein the cameras positions (Ci:C6) are in absolute scale; associate the 3D point (X, Xi, X2) with absolute scale coordinates in 3D space by constraining the 3D point (X, Xi, X2) to a coordinate system (710) defined by at least two of the camera poses (Ci:C6); and perform the action with respect to the 3D point (X, Xi, X2).

13. The image processing device (300, 1000) according to claim 12, further being configured to perform the method according to any of claims 2 to 11.

14. A computer program (1120) for providing a 3D NeRF model of a scene (210, 910) in absolute scale, the computer program comprising computer code which, when run on processing circuitry (1010) of an image processing device (300, 1000), causes the image processing device (300, 1000) to: obtain (S102) a 3D NeRF model of the scene (210, 910), a set of images depicting the scene (210, 910) and cameras positions (Ci:C6) based on which the 3D NeRF model was generated, localization of a 3D point (X, Xi, X2) in the 3D NeRFmodel, and an action with respect to the 3D point (X, Xi, X2), wherein the cameras positions (Ci:C6) are in absolute scale; associate (S104) the 3D point (X, Xi, X2) with absolute scale coordinates in 3D space by constraining the 3D point (X, Xi, X2) to a coordinate system (710) defined by at least two of the camera poses (Ci:C6); and perform (S106) the action with respect to the 3D point (X, Xi, X2).

15. A computer program product (1110) comprising a computer program (1120) according to claim 14, and a computer readable storage medium (1130) on which the computer program is stored.

Images