A method and system for predicting a data representation of a 3D scene

Abstract

The invention relates to a method of predicting a data representation of a 3D scene, the method comprising: providing a first dataset of 2D images of the scene including 6D-pose information and a depthmap for each image, training a predefined 3D-free-viewpoint module using the first dataset to predict further images including respective depthmaps of the scene for a plurality of further 6D-poses different to the 6D-poses included in the dataset, and predicting a data representation of the scene by merging the images predicted by the 3D-free-viewpoint module.

Description

A method and system for predicting a data representation of a 3D sceneBACKGROUND[1] Obtaining a 3D (i.e. three dimensional) model, i.e. a digital twin of a real 3D scene, may have various technical purposes. The scene may be e.g. an indoor scene, such as a furnished apartment, or a construction site of a building. For example, it may be desirable to be able to virtually move in the 3D scene. In other words, it may be desirable to freely choose a 6D-pose of a virtual camera which can be freely positioned in the virtual scene.[2] Moreover, in case several 3D models are established at different time points, the evolution of the scene may be monitored. For example, it may thus be checked by comparing 3D models of a scene, whether the state of an apartment or a construction has changed.[3] Conventionally, 2D (i.e. two dimensional) image data are obtained with a camera device, for example a smartphone. The 2D image data may comprise single 2D images or a (video) stream of 2D image frames. Typically, a user of the camera device may walk through the scene and simultaneously take several images or a video of different parts of the scene.[4] However, even though smartphone may comprise acceleration sensors, e.g. an IMU (inertial measurement unit), these 2D images may not comprise precise information re. the 6D-pose of camera when taking the images. Hence, it is difficult to set the single 2D images in relation to each other. Vice versa, since it is challenging to reliably and precisely set the single 2D images in relation to each other, it is difficult to derive a change of 6D-pose from the images. Accordingly, constructinga reliable and precise 3D model based on the 2D images is not a trivial task.[5] According to e.g. US11436812B2, to determine types and locations of objects within an environment, a user captures a walkthrough video of the environment and manually identifies and tags the objects in the walkthrough video. For example, at a construction site, a variety of tasks are performed simultaneously on different parts of a building project, and it is difficult to keep track of progress for each aspect of the building project to determine whether the building project is on trackto being completed on time. A general contractor may wish to monitor progress on the construction site by capturing a walkthrough video of the construction site that documents conditions of the construction site. The general contractor may then visually review the walkthrough video and identify objects installed (e.g., light fixtures, cabinets, windows, drywall, paint, cement) by identifying the objects present in the images. The general contractor may periodically capture new walkthrough videos and determine additional objects that have been installed to track progress of the building project over time. However, manual review of videos to count and measure objects is tedious and time consuming.[6] US11436812B2 proposes a system that automatically identifies objects in an environment based on a walkthrough video and an annotated floorplan of the environment. The annotated floorplan indicates locations and types of objects that are expected to be in the environment. The system receives the walkthrough video and generates a 3D model of the environment. The system applies a machine learning model to the walkthrough video to identify regions within frames where objects are captured. After identifying the regions within frames of the walkthrough video that include objects, the system modifies corresponding regions of the 3D model to include the identified objects. For each of the identified objects, the system determines a likelihood of the identified object being present at a location in the environment based on a comparison of the modified 3D model and the annotated floorplan.[7] However, the system of US11436812B2 requires an annotated floorplan to generate a 3D model of the environment. Establishing the annotated floorplan can be a time consuming task, which may also be prone to further errors which can deteriorate the final 3D model.[8] In view of the above, there is a need to solve such drawbacks and associated disadvantages.SUMMARY OF THE DISCLOSURE[9] A computer-implemented method of predicting a data representation of a 3D scene is provided. The method comprises:providing a first dataset of 2D images of the scene including 6D-pose information and a depthmap for each image, training a predefined 3D-free-viewpoint module using the first dataset to predict further images including respective depthmaps of the scene for a plurality of further 6D-poses different to the 6D-poses included in the dataset, and predicting a data representation of the scene by merging the images predicted by the 3D-free-viewpoint module.

[0010] By providing such a method, a data representation may be predicted which represents a precise and reliable model of a real 3D scene. The data representation may be or may comprise at least one of a 3D model, a point cloud, a textured mesh or any other kind of representation of the scene.

[0011] The dataset may typically comprise only a relatively limited selection of possible 6D-poses within the scene, e.g. in the form of image frames of a walkthrough video of the scene.

[0012] A 6D-pose may be defined by 3D-position information and 3D orientation (i.e. 3D-rotational) information, e.g. of a real camera device or of a virtual camera.

[0013] The images of the dataset may be single images or image frames of a video.

[0014] The method may be at least partially performed using an artificial intelligence (Al) model. In particular, the 3D-free-viewpoint module may use or comprise a machine-learning model or algorithm, e.g. a neural network. However, in some examples, the method, in particular the 3D-free-viewpoint module does not require or have a neural network. For example, the 3D-free-viewpoint module may have or use instead a view-dependent sparse voxel model, e.g. Plenoxel (plenoptic volume element), that can optimize to the same fidelity as e.g. Neural Radiance Fields (NeRFs) without any neural network.

[0015] Hence, instead of a NERF algorithm also an equivalent algorithm may be used that allows to create a 3D freeview point model that can synthesize a view from any point of view.

[0016] The dataset may be used to train the 3D-free-viewpoint module, such that the 3D-free-viewpoint module learns to predict further unseen images of the scene (i.e. not included in the dataset) including respective depthmaps.

[0017] Accordingly, during the training phase, the 3D-free-viewpoint module may take in a set of input images of a scene from various viewpoints and optimize theweights of the model (e.g. neural network parameters) to minimize the difference between the rendered images (e.g. comprising colored images and / or depthmaps) generated by the network and the actual input images. Once the training is complete, the trained neural network can be used to generate novel views of the scene from any arbitrary viewpoint.

[0018] In other words, the trained 3D-free-viewpoint module may be configured to predict depth maps (and more in particular images ncluding the respectives depth maps) from any point of view of a virtual camera which was not present in the initial images.

[0019] The predicted images may be merged to generate the data representation. The thereby obtained data representation may thus be more complete and / or precise, as views from more (predicted) 6D-poses may be taken into account, than those present in the initial dataset.

[0020] Moreover, the 3D-free-viewpoint module may be trained to predict depth maps and image radiance data (color and brightness data) instead of only 2D image radiance data (color and brightness data). Due to the additional depth information the predicted 3D model may represent to the 3D scene more correctly. For example, since the 3D-free-viewpoint module (e.g. NeRF) may predict the opacity in space, the depth is usually defined as the distance above which points wouldn't be visible because there are too much accumulated opacity between the point and the virtual camera.

[0021] The 3D-free-viewpoint module may be a Neural Radiance Fields (NeRF) algorithm (or any other algorithm) adapted to use depth maps as training data and / or to use any geometric or 3D training data.

[0022] The 3D-free-viewpoint module may also be referred to as a 6D-free-view- point module, as (once trained on a scene) is possible to predicted a view in any 6D-pose in the scene.

[0023] The predicted images may comprise depth information and also image radiance information. The depth information may be in the form of a depth map. In other words, each predicted image may include a depth map. Additionally, each image may comprise 6D-pose information, or the 6D-pose information of each image may be known based on the output of the 3D-free-viewpoint module.

[0024] Providing the first dataset may comprise:providing an initial dataset of 2D image of the scene including initial 6D-pose information for each image, predicting refined 6D-pose information for each image by inputting the initial dataset into a predefined bundle adjustment module.The first dataset may then be obtained based on the initial dataset of 2D image and the refined 6D-pose information for each image.

[0025] The initial 6D-pose information may for example be only (e.g. noisy) estimates with poor or medium accuracy. According to the method of the present disclosure, such noisy data may be used to generate highly accurate perfect depthmaps. Accordingly, the initial 6D-pose information may be less precise and thus be less suitable for the method. Such initial 6D-pose information may be e.g. based on acceleration data of a IMU of a smartphone, i.e. during recording the images. The refinement of the initial 6D-pose information can improve both the quality of the depth maps and the performance of the 3D-free-viewpoint module. Alternatively, the initial 6D-pose information may be estimated a posteriori instead of during recording. For example, a Structure-From-Motion or a SLAM may be run a posteriori based on the images to estimate the initial 6D-pose information.

[0026] Providing the first dataset may further comprise: predicting a depthmap for each image by inputting an initial dataset of 2D images (without depth information), optionally including the refined 6D-pose information (for the case that the original 2D images have already refined 6D-pose information), into a predefined dense depth estimation module.

[0027] The first dataset may then be obtained based on the initial dataset of 2D image, optionally the refined 6D-pose information for each image, and the depthmap for each image.

[0028] Accordingly, the method may merely use 2D images and initial 6D-pose information as input, e.g. generated by a conventional camera device, such as a smart phone. The refined 6D-pose information and / or the depthmaps may optionally be derived from the images, as described in more detail below. Alternatively, at least one of these two kinds of information (refined 6D-pose information and depthmaps) may be provided separately to the method.

[0029] The dense depth estimation module may use or comprise a machine-learning model or algorithm, e.g. a neural network.

[0030] The dense depth estimation module may comprise (or may be) a multi-view stereo (MVS) algorithm.

[0031] The dense depth estimation module may be configured to estimate depth information in the images, e.g. based on geometric priors in the images.

[0032] In particular, the dense depth estimation module may estimate depth information of homogenous image areas (e.g. of planes with homogeneous texture). For this purpose, geometric priors in the images, such as planes, may be taken into account by the dense depth estimation module.

[0033] The MVS algorithm may be e.g. configured to generate the depth map assuming that the geometry prior takes over during the reconstruction of the planar regions. For example, the MVS algorithm may output a 3D surface map comprising a surface point cloud (or more in general surface representation) representing planar surfaces in the scene.

[0034] Accordingly, the depthmaps obtained by estimating planar priors can improve the results of the 3D-free-viewpoint module in challenging low contrast areas.

[0035] As a consequence, the method can become robust for homogeneous low contrast regions that are assumed to have a geometric prior. A geometric prior may be understood as a predetermined geometric structure, e.g. a plane. In contrast, the 3D-free-viewpoint module (e.g. Nerf) can improve the depthmap's precision on high contrast areas. A multi-view stereo algorithm tends to generate blurry depth maps.

[0036] The dense depth estimation module may further generate a filtered depthmap for each image keeping (only) the 3D information corresponding to homogenous image areas.

[0037] The filtered depthmaps may be used to train the 3D-free-viewpoint module.

[0038] As already (at least partially) outlined above, the 3D-free-viewpoint module may perform at least one of: train an artificial neural network on a dataset of 2D images, respective depth maps, and 6D-pose data of the scene to learn predicting the radiance at any 3D position and viewing direction within the scene,represent the scene as a continuous function using the trained network, such that the radiance of the scene can be synthesized at any 3D position and viewing direction, even for viewpoints that were not captured in the original set of images, render depth maps of the scene from any viewpoint or camera position using the continuous function.

[0039] The bundle adjustment module may extract 2D features for each image, matches the features among the images in the dataset using a co-visibility estimated from the initial 6D-pose data, and minimizes a reprojection error from the positions of these features to generate a refined 6D-pose position data of the scene for each image in the dataset.

[0040] The bundle adjustment module may also be referred to as a feature detection, extraction or matching module.

[0041] The bundle adjustment module may use or comprise a machine-learning model or algorithm, e.g. a neural network. However, the mentioned feature extraction may also be achieved by the bundle adjustment module using another algorithm than a neural network.

[0042] The bundle adjustment module may comprise a deep feature alignment algorithm (e.g. a Pixel-Perfect Structure-from-Motion (PP-SfM) algorithm), which generates a sparse point cloud with feature points. The feature points may in particular represent at least one of: points related to a geometry variation, e.g. intersections between planes (e.g. walls, ceiling, floor), points related to objects boundaries, and / or points with a texture change, pattern, high contrast, etc.

[0043] The resulting sparse point cloud may be considered as merely an intermediate result. However, the deep feature alignment algorithm may allow (by means of the sparse point cloud) to achieve an alignment of the positions of the features as global problem.

[0044] The bundle adjustment module may perform at least one of: extract features, such as corners or edges, or any other kind of invariant image features, from multiple images of the same object in the scene using a neural network, match the features across images to identify corresponding points in the scene in an initial reconstruction process,estimate a pose describing a position and orientation of a camera device when the image was taken, and the 3D structure describing the position of the features in the scene, perform a Featuremetric Refinement process for improving the accuracy of the initial reconstruction by iteratively refining the camera poses and the 3D structure.

[0045] The refinement process may be optionally performed by minimizing the difference between the predicted feature locations in the images and their actual locations.

[0046] The inital dataset may comprise for each image frame a relative pose information, the relative pose information indicating the pose change of a camera device from a first image frame to a second image frame.

[0047] The inital dataset may comprise on-the-fly visual odometry information. For example, in case the camera device is a smartphone, the odometry information may be provided by a e.g. augmented reality (AR) libraries, such as ARCore for Google Android® devices or ARKit libraries for Apple® devices. The initial 6D-pose information, as described above, may be or may be based on the on-the-fly visual odometry information. The camera device may thus apply a Visual Inertial Odometry (VIO) algorithm. Alternatively, the method, e.g. the bundle adjustment module may comprise a VIO algorithm.

[0048] The VIO may be understood as a computer vision technique used for estimating the 6D-pose and velocity of the moving camera device relative to a local starting position. For example, a camera device may be used that takes independent images without 6D-pose information. These images may then be mapped onto a resulting 3D space by matching the features.

[0049] Since the on-the-fly visual odometry information only comprises relative 6D- pose information (i.e. from image to image), an absolute 6D-pose calculated based on this relative 6D-pose information may become erroneous, in particular, in case of a relative large number of images. In other words, the longer the sequence, the accumulation of error becomes bigger and results in a bigger drift. Hence it can be desirable to "close the loop" to refine the 6D-pose information and correct this drift loop closure. This may be accomplished by predicting refined 6D-pose information, as described above.

[0050] An object recognition module may be used to detect predetermined testpatterns in the images and to calibrate and improve precision of the predicted data representation.

[0051] The method may be at least partially performed using an artificial intelligence (Al) model.

[0052] The present disclosure may further refer to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to any one of the preceding method claims.

[0053] The present disclosure may further refer to a system, comprising: at least one processor, and at least one memory storing computer-executable instructions, the computer-executable instructions when executed by the processor cause the computing device to perform a method according to any one of the preceding claims.

[0054] For example, the system may be configured to receive a first dataset of 2D images of the scene including 6D-pose information and / or a depthmap for each image, train a predefined 3D-free-viewpoint module using the first dataset to predict further images including respective depthmaps of the scene for a plurality of further 6D-poses different to the 6D-poses included in the dataset, and predict a data representation of the scene by merging the images predicted by the 3D- free-viewpoint module.

[0055] The processor (or processing unit) may be a component of electronic devices that may be responsible for carrying out computational tasks. There may be different types of processing units, each designed for specific purposes. A processing unit may be or may comprise a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), and / or Application-Specific Integrated Circuit (ASIC). The CPU may be the primary processing unit in a computer that may be responsible for executing most of the instructions and calculations required by the computer. GPUs may be specialized processing units that may be designed to handle the complex calculations required for rendering graphics and video. DSPs may be specialized processing units that may handle signal processing tasks, while FPGAs may be recon-figurable processing units that may be programmed to perform various computational tasks. ASICs may be customized processing units designed to perform a specific set of tasks, optionally making them highly efficient and effective for their intended purpose. These processing units may be found in a wide range of electronic devices, such as medical electronic devices. Medical electronic devices may include computers, smartphones, and other digital devices, optionally enabling these devices to perform various computational tasks efficiently and accurately. The method according to the present disclosure may also run on a virtual server.

[0056] The system may also comprise or be connectable with a camera device, e.g. a smartphone, as described above.

[0057] The present disclosure may also relate to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to any examples of the present disclosure.

[0058] In case the method operations may comprise any aspects (for example physical) which go beyond a mere data processing (for example an ultrasound signal processing), the computer program may further comprise computer-readable instructions which when executed by a data processing system cause any external elements of a system to carry out these operations.

[0059] The present disclosure may also relate to a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any examples of the present disclosure.

[0060] It is intended that combinations of the above-described elements and those within the specification may be made, except where otherwise contradictory.

[0061] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only, are provided for illustration purposes and are not restrictive of the disclosure, as claimed.

[0062] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples of the disclosure and together with the description, and serve to support and illustrate the principles thereof.BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Fig. 1 shows a schematic drawing of a system 100 according to examples of the present disclosure.

[0064] Fig. 2 schematically shows a method of predicting a data representation of a 3D scene according to examples of the present disclosure.DESCRIPTION OF THE DRAWINGS

[0065] Reference will now be made in detail to examples of the disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0066] The present disclosure relates to methods and systems for predicting a data representation of a 3D scene. In particular, the present disclosure relates to machine learning technologies for processing single 2D images to obtain a 3D model of the scene.

[0067] Fig. 1 shows a schematic drawing of a system 100 according to examples of the present disclosure. The system 100 may be configured to method according to the present disclosure. For example of a system for predicting a data representation of a 3D scene.

[0068] The system 100 may comprise:- at least one processor 11,- at least one memory 12 storing computer-executable instructions, the computerexecutable instructions when executed by the processor cause the computing device to perform a method according to any one of the preceding claims,

[0069] The system may further comprise or be configured to be connectable to:- a camera device 13, for example a smartphone, table or other type of device comprising a digital camera,- one or several acceleration sensors 14, e.g. in the form of an IMU (inertial measurement unit),- a display device 15, and- a data storage device 16.

[0070] In its most basic configuration, the system 100 typically includes at least one processing unit (or processor) and memory. In examples, the processor and memory unit may be incorporated into the system such as depicted in FIG. 1, or may be a computer or computer communicatively linked thereto. Depending on the exact configuration and type of computing device, memory (storing, instructions to evaluate ultrasound data or otherwise perform the methods described herein) may be volatile (such as RAM), non-volatile (such as RAM, flash memory, etc.), or some combination of the two. Further, the system 100 may also include storage devices (removable and / or non-removable) including, but not limited to, magnetic or optical disks or tape. Similarly, the system 100 may also have input device(s) such as keyboard, mouse, pen, voice input, etc. and / or output device(s) such as a display, speakers, printer, etc. Also included in the environment may be one or more communication connections, such as LAN, WAN, point to point, etc. In embodiments, the connections may be operable to facility point-to-point communications, connection-oriented communications, connectionless communications, etc.

[0071] The system 100 typically includes at least some form of computer readable media. Computer readable media can be any available media that can be accessed by processing unit (or processor) or other devices comprising the operating environment. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media does not include communication media.

[0072] Communication media embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, microwave, and other wireless media. Combinations ofthe any of the above should also be included within the scope of computer readable media.

[0073] The system 100 may be a single computer operating in a networked environment using logical connections to one or more remote computers. The remote computer may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above as well as others not so mentioned. The logical connections may include any method supported by available communications media. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

[0074] For example, the system may be configured to receive a first dataset of 2D images of the scene from the camera device 13 including 6D-pose information and / or a depthmap for each image, train a predefined 3D-free-viewpoint module using the first dataset to predict further images including respective depthmaps of the scene for a plurality of further 6D-poses different to the 6D-poses included in the dataset, and predict a data representation of the scene by merging the images predicted by the 3D-free-viewpoint module.

[0075] The acceleration sensor 14 may be part of the camera device 13, or may be external thereto. The acceleration sensor 14 may be configured to obtain the acceleration sensor data, based on which the camera device and / or the system may determine the 6D-pose information. The 6D-pose information may also be initial 6D-pose information, which may be refined in this case by the system, as described in context of fig. 2.

[0076] The data representation may constitute a 3D model of the scene, which may be displayed (or at least different views in the scene) on the display device 15.

[0077] Accordingly a 3D (i.e. three dimensional) model, i.e. a digital twin of a 3D scene, for example an indoor scene, such as a furnished apartment or a construction site of a building, may be obtained and visualized by the system. For example, the system may be configured to allow a system user to virtually move in the 3D scene. In other words, it may be desirable to freely choose a 6D-pose of a virtual camera

[0078] The data representation and / or the respective 3D model of the scene may also be stored on the data storage device 16. It is also possible, that data representations and / or the respective 3D models are generated by the system at different time points and each of these is stored on the data storage device 16. Accordingly, an evolution of the scene may be derived or monitored based on a comparison of these stored data stored. For example, it may thus be checked by comparing 3D models of a scene, whether the state of an apartment or a construction has changed.

[0079] According to particular examples, managing and maintaining properties, constructions, devices, vehicles (planes, motors, engines, boats, big transport vehicles, cars,...), complex systems, renovations, objects, etc., which may be represented as 3D scenes, is a time consuming task that is not always standardized and is time consuming and error prone. With the arrival of new smartphones and tablets that integrate several acquisition sensors, like cameras, LIDAR sensors, positioning sensors, and connectivity to external devices (LIDAR, 360° cameras, Hyperspectral cameras, IPS, ...) the system and method according to the present disclosure may allow to generate a digital twin, a 360 view, or a 2D floor map or projections / (all of them or only the required representation of the space) of the environment, aggregate it with the data provided by external sensors and add annotations manually, semi automatically, or automatically depending on the final application requirement. Doing this on the same device and in the same capture simplifies the capture and annotation, reduces errors, and avoids the manual alignment of the data coming from different sources.

[0080] According to a further example, the system 100 may have an architecture, wherein- The system comprises or is associated with a data acquisition device (e.g. a smartphone or other camera device),- The data (obtained by the data acquisition device) is stored in a network connected device (e.g. a server)- The network connected device triggers another computation device that receives the data and performs all or a part of the required computations of the method of the present disclosure. Each algorithm may e.g. run independently on an optimized device and triggered when the previous device finished the processing of the data).- The results may be stored on the network.- The final 3D representation may be accessed by a different device like a smartphone or computer.

[0081] Fig. 2 schematically shows a method of predicting a data representation of a 3D scene according to examples of the present disclosure.

[0082] The method may for example be carried out by a system 100, as described above.

[0083] In a step SI a first dataset of 2D images of the scene is provided. The dataset further includes (refined) 6D-pose information and a depthmap for each image.

[0084] The 6D-pose information may comprise or consist of 3D-position information and 3D-rotational information.

[0085] The 2D images may be taken by a camera device, as e.g. described in context of fig. 1, such as a smartphone. For example, the 2D images may comprise single 2D images or a (video) stream of 2D image frames. Typically, a user of the camera device may walk through the scene and simultaneously take several images or a video of different parts of the scene.

[0086] The dataset may typically comprise only a relatively limited selection of possible 6D-poses within the scene, e.g. in the form of image frames of a walkthrough video of the scene.

[0087] Step SI may comprise one or several optional substeps, in particular, in case an initial dataset does not comprise refined 6D-pose information or depthmaps.

[0088] In such a case, in an optional substep Sla, an initial dataset of 2D images including initial 6D-pose information is provided.

[0089] For example, the initial 6D-pose information may be less precise and thus be less suitable for the method. Such initial 6D-pose information may be e.g. based on acceleration data of a IMU of a smartphone. The refinement of the initial 6D- pose information can improve both the quality of the depth maps and the performance of the 3D-free-viewpoint module.

[0090] Accordingly, the method may merely use 2D images and initial 6D-pose information as input, e.g. generated by a conventional camera device, such as a smart phone. The refined 6D-pose information and the depthmaps may optionallybe derived from the images, as described in more detail below. Alternatively, at least one of these two kinds of information (refined 6D-pose information and depthmaps) may be provided separately to the method.

[0091] For example, the inital dataset may comprise for each image frame a relative pose information, the relative pose information indicating the pose change of a camera device from a first image frame to a second image frame.

[0092] The inital dataset may comprise on-the-fly visual odometry information. For example, in case the camera device is a smartphone, the odometry information may be provided by a e.g. augmented reality (AR) libraries, such as ARCore for Google Android® devices or ARKit libraries for Apple® devices. The initial 6D-pose information, as described above, may be or may be based on the on-the-fly visual odometry information. The camera device may thus apply a Visual Inertial Odometry (VIO) algorithm. Alternatively, the method, e.g. the bundle adjustment module may comprise a VIO algorithm.

[0093] The VIO may be understood as a computer vision technique used for estimating the 6D-pose and velocity of the moving camera device relative to a local starting position. For example, a camera device may be used that takes independent images without 6D-pose information. These images may then be mapped onto a resulting 3D space by matching the features.

[0094] Since the on-the-fly visual odometry information only comprises relative 6D- pose information (i.e. from image to image), an absolute 6D-pose calculated based on this relative 6D-pose information may become erroneous, in particular, in case of a relative large number of images.

[0095] In an alternative example, the bundle adjustment module may comprise a SLAM (Simultaneous Localization and Mapping) algorithm.

[0096] In an optional substep Sib the initial dataset is inputted into a bundle adjustment module (e.g. deep feature alignment algorithm. The bundle adjustment module may predict refined 6D-pose information based on the initial dataset.

[0097] In particular, the bundle adjustment module may extract 2D features for each image, matches the features among the images in the dataset using a covisibility estimated from the initial 6D-pose data, and minimizes a reprojection error from the positions of these features to generate a refined 6D-pose position data of the scene for each image in the dataset.

[0098] The bundle adjustment module may use or comprise a machine-learning model or algorithm, e.g. a neural network. However, the mentioned feature extraction may also be achieved by the bundle adjustment module using another algorithm than a neural network.

[0099] The bundle adjustment module may also be referred to as a feature recognition module. The bundle adjustment module may comprise a deep feature alignment algorithm (e.g. a Pixel-Perfect Structure-from-Motion (PP-SfM) algorithm), which generates a sparse point cloud with feature points. The feature points may in particular represent at least one of: intersections between planes (e.g. walls, ceiling, floor), and objects boundaries.

[0100] The resulting sparse point cloud may be considered as merely an intermediate result. However, the deep feature alignment algorithm may allow (by means of the sparse point cloud) to achieve an alignment of the positions of the features as global problem.

[0101] In other words, the bundle adjustment module may extract features, such as corners or edges, or any other kind of invariant image features, from multiple images of the same object in the scene using a neural network, match the features across images to identify corresponding points in the scene in an initial reconstruction process, estimate a pose describing a position and orientation of a camera device when the image was taken, and the 3D structure describing the position of the features in the scene, and perform a Featuremetric Refinement process for improving the accuracy of the initial reconstruction by iteratively refining the camera poses and the 3D structure.

[0102] The refinement process may be optionally performed by minimizing the difference between the predicted feature locations in the images and their actual locations.

[0103] The present disclosure thus proposes in one particular example to use data provided by AR libraries as a priori of the camera positions and to perform a computer vision pipeline to chain different ways to improve the data and reconstruct the 3D space. In this way, the initialization can be close to the real values, and the exploration espace can be reduced. For example, Pixel Perfect Structure-from-Mo- tion (PP-SfM) may be used to obtain an image alignment refinement using a fea-turemetric error based on dense features predicted by a neural network. The process begins by extracting features, such as corners or edges, from multiple images of the same object or scene. These features are then matched across images to identify corresponding points in 3D space. Once these correspondences have been established, the PP-SfM algorithm uses them to estimate the camera poses and the 3D structure of the scene. The camera poses describe the position and orientation of each camera when the image was taken, and the 3D structure describes the position of the features in the scene. The Featuremetric Refinement stage improves the accuracy of the initial reconstruction by iteratively refining the camera poses and the 3D structure. This refinement is performed by minimizing the difference between the predicted feature locations in the images and their actual locations.

[0104] In an optional substep Sic the initial dataset is inputted into a dense depth estimation module (e.g. multi-view stereo, MVS, algorithm). The dense depth estimation module may predict a depthmap for each image based on the initial dataset.

[0105] The dense depth estimation module may optionally receive as further input the refined 6D-pose information predicted by the bundle adjustment module. Accordingly, this refined 6D-pose information may replace the initial 6D-pose information.

[0106] The dense depth estimation module may use or comprise a machine-learning model or algorithm, e.g. a neural network.

[0107] For example, dense depth estimation module may comprise (or may be) a multi-view stereo (MVS) algorithm.

[0108] The dense depth estimation module may be configured to estimate depth information in the images, e.g. based on geometric priors in the images.

[0109] In particular, the dense depth estimation module may estimate depth information of homogenous image areas (e.g. of planes with homogeneous texture). For this purpose, geometric priors in the images, such as planes, may be taken into account by the dense depth estimation module.

[0110] The MVS algorithm may be e.g. configured to generate the depth map assuming that the geometry prior takes over during the reconstruction of the planar regions. For example, the MVS algorithm may output a 3D surface map comprising a surface point cloud representing planar surfaces in the scene.

[0111] Accordingly, the depthmaps obtained by estimating planar priors can improve the results of the 3D-free-viewpoint module in challenging low contrast areas.

[0112] As a consequence, the method can become robust for homogeneous low contrast regions that are assumed to have a geometric prior. A geometric prior may be understood as a predetermined geometric structure, e.g. a plane.

[0113] The dense depth estimation module may further generate filtered depthmaps keeping only the 3D information corresponding to the homogenous image areas.

[0114] Accordingly, the present disclosure proposes in one particular example to use the (MVS) algorithm to improve the data quality. Multiview stereo (MVS) algorithm is a computer vision technique used to reconstruct a 3D model of an object or scene from multiple 2D images. The basic idea behind MVS is to use the information available in multiple images to create a more accurate and detailed 3D model than what can be achieved from a single image.

[0115] The MVS algorithm typically involves the following steps:Feature detection and matching: In this step, distinctive features (such as corners or edges) are detected in each image, and then matched across all the images.Depth map computation: Once the features are matched, a depth map is computed for each image. A depth map assigns a depth value to each pixel in the image, indicating the distance between the camera and the corresponding point in 3D space.Surface reconstruction: The depth maps are then used to generate a dense point cloud, which represents the 3D positions of the features in the scene. A surface reconstruction algorithm can then be applied to the data representation to generate a complete 3D model of the scene.

[0116] This last part may be particularly advantageous, because we the surface reconstruction can be used to densify planar regions where the texture has little feature points (i.e. uniform color walls for example).

[0117] In an optional step Sid the first dataset may be provided based on the images of the initial dataset, the refined 6D-pose information predicted by the bundle adjustment module and the depthmaps predicted by the dense depth estimation module.

[0118] In a step S2 a predefined 3D-free-viewpoint module (e.g. Neural Radiance Fields (NeRF) algorithm) is trained based on the first dataset, in particular to predict images including respective depthmaps of the scene for a plurality of further 6D- poses different to the 6D-poses included in the dataset.

[0119] In other words, the 3D-free-viewpoint module may train an artificial neural network on a dataset of 2D images, respective depth maps, and 6D-pose data of the scene to learn predicting the radiance at any 3D position and viewing direction within the scene.

[0120] In one example, the filtered depthmaps may be used to train the 3D-free- viewpoint module.

[0121] The 3D-free-viewpoint module may be a Neural Radiance Fields (NeRF) algorithm adapted to use depth maps as training data. Instead of a NERF algorithm also an equivalent algorithm may be used that allows to create a 3D freeview point model that can synthesize a view from any point of view.

[0122] The 3D-free-viewpoint module may also be referred to as a 6D-free-view- point module, as (once trained on a scene) is possible to predicted a view in any 6D-pose in the scene.

[0123] The 3D-free-viewpoint module (e.g. NeRF) can improve the precision of the depthmaps (predicted by the dense depth estimation module) on high contrast areas.

[0124] In a step S3 the 3D-free-viewpoint module is used to predict further (i.e. new or unseen) images including 6D-pose and depth information.

[0125] The predicted images may comprise depth information and also image radiance information. The depth information may be in the form of a depth map. In other words, each predicted image may include a depth map. Additionally, each image may comprise 6D-pose information, or the 6D-pose information of each image may be known based on the output of the 3D-free-viewpoint module.

[0126] In other words, the trained 3D-free-viewpoint module may represent the scene as a continuous function using the trained network, such that the radiance of the scene can be synthesized at any 3D position and viewing direction, even for viewpoints that were not captured in the original set of images. The trained 3D- free-viewpoint module may additionally render (high-quality) depth maps of the scene from any viewpoint or camera position using the continuous function

[0127] In a step S4 a data representation of the scene is predicted based on the predicted further images (and optionally the first dataset). The data representation may be obtained based on the predicted images, e.g. by merging these images in a 3D model.

[0128] Optionally, an object recognition module (not shown in fig. 2) may be used to detect predetermined test-patterns in the images and to calibrate and improve precision of the predicted data representation.

[0129] Accordingly, the present disclosure proposes in one particular example to use the NeRF algorithm in the method. The Neural Radiance Fields (NeRF) algorithm is a state-of-the-art technique for synthesizing high-quality 3D scenes from a collection of 2D images. The key idea behind NeRF is to use a deep neural network to learn a continuous function that maps from 3D positions and viewing directions to the radiance (color and brightness) of the scene.

[0130] The NeRF algorithm typically involves the following steps:Training: In this step, a deep neural network is trained on a set of 2D images of the scene. The neural network is designed to predict the radiance and opacity at any 3D position and viewing direction within the scene.Representation: Once the neural network is trained, it can be used to represent the 3D scene as a continuous function. This allows the radiance of the scene to be synthesized at any 3D position and viewing direction, even for viewpoints that were not captured in the original set of images.Rendering: The continuous function learned by the neural network can be used to render high-quality images of the scene from any viewpoint or camera position.

[0131] The NeRF algorithm has several advantages over traditional 3D rendering techniques. For example, NeRF can handle complex lighting and shading effects, and can synthesize high-quality images even from a small number of input images. NeRF can also be used to render scenes with moving objects or changing lighting conditions, making it useful for applications such as virtual reality and augmented reality.

[0132] Throughout the description, including the claims, the term "comprising a" should be understood as being synonymous with "comprising at least one" unless otherwise stated. In addition, any range set forth in the description, including the claims should be understood as including its end value(s) unless otherwise stated.Specific values for described elements should be understood to be within accepted manufacturing or industry tolerances known to one of skill in the art, and any use of the terms "substantially" and / or "approximately" and / or "generally" should be understood to mean falling within such accepted tolerances.

[0133] The terms "record" and "receive" may be used synonymously throughout this disclosure unless denoted differently.

[0134] Although the present disclosure herein has been described with reference to particular examples, it is to be understood that these examples are merely illustrative of the principles and applications of the present disclosure.

[0135] It is intended that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims.

[0136] A reference herein to a patent document or any other matter identified as prior art, is not to be taken as an admission that the document or other matter was known or that the information it contains was part of the common general knowledge as at the priority date of any of the claims.

Claims

CLAIMS1. A computer-implemented method of predicting a data representation of a 3D scene, comprising: providing a first dataset of 2D images of the scene including 6D-pose information and a depthmap for each image, training a predefined 3D-free-viewpoint module using the first dataset to predict further images including respective depthmaps of the scene for a plurality of further 6D-poses different to the 6D-poses included in the dataset, and predicting a data representation of the scene by merging the images predicted by the 3D-free-viewpoint module.

2. The method of claim 1, wherein the 3D-free-viewpoint module is a Neural Radiance Fields (NeRF) algorithm adapted to use depth maps as training data, and / or the predicted images comprise depth information and also image radiance information.

3. The method of claim 1 or 2, wherein providing the first dataset comprises: providing an initial dataset of 2D images of the scene including initial 6D-pose information for each image, predicting refined 6D-pose information for each image by inputting the initial dataset into a predefined bundle adjustment module.

4. The method according to any one of the preceding claims, wherein providing the first dataset further comprises: predicting a depthmap for each image by inputting an initial dataset of 2D images, optionally including the refined 6D-pose information, into a predefined dense depth estimation module.

5. The method according to any one of the preceding claims, whereinthe dense depth estimation module comprises a multi-view stereo algorithm, optionally configured to estimate depth information based on geometric priors in the images.

6. The method according the preceding claim, wherein the dense depth estimation module further generates a filtered depthmap keeping the 3D information corresponding to the homogenous image areas, wherein the filtered depthmaps are used to train the 3D-free-viewpoint module.

7. The method according to any one of the preceding claims, wherein the 3D-free-viewpoint module performs at least one of: train an artificial neural network on a dataset of 2D images, respective depth maps, and 6D-pose data of the scene to learn predicting the radiance at any 3D position and viewing direction within the scene, represent the scene as a continuous function using the trained network, such that the radiance of the scene can be synthesized at any 3D position and viewing direction, even for viewpoints that were not captured in the original set of images, render depth maps of the scene from any viewpoint or camera position using the continuous function.

8. The method according to claim 1, wherein the bundle adjustment module extracts 2D features for each image, matches the features among the images in the dataset using a co-visibility estimated from the initial 6D-pose data, and minimizes a reprojection error from the positions of these features to generate a refined 6D-pose data of the scene for each image in the dataset.

9. The method according to any one of the preceding claims, wherein the bundle adjustment module comprises a deep feature alignment algorithm, which optionally generates a sparse point cloud with feature points.

10. The method according to any one of the preceding claims, wherein the bundle adjustment module performs at least one of: extract features, such as corners or edges, or any other kind of invariant image features, from multiple images of the same object in the scene using a neural network, match the features across images to identify corresponding points in the scene in an initial reconstruction process, estimate a pose describing a position and orientation of a camera device when the image was taken, and the 3D structure describing the position of the features in the scene, perform a Featuremetric Refinement process for improving the accuracy of the initial reconstruction by iteratively refining the camera poses and the 3D structure, wherein the refinement process is optionally performed by minimizing the difference between the predicted feature locations in the images and their actual locations.

11. The method according to any one of the preceding claims, wherein the depth map reconstruction module comprises a Multiview stereo (MVS) algorithm, in particular configured to generate the depth map assuming that the geometry prior takes over during the reconstruction of the planar regions.

12. The method according to any one of the preceding claims, wherein the dataset comprises for each image frame a relative pose information, the relative pose information indicating the pose change of a camera device from a first image frame to a second image frame, and / or the dataset comprises on-the-fly visual odometry information.

13. The method according to any one of the preceding claims, wherein an object recognition module is used to detect predetermined test-patterns in the images and to calibrate and improve precision of the predicted data representation.

14. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to any one of the preceding method claims.

15. A system, comprising: at least one processor, and at least one memory storing computer-executable instructions, the computerexecutable instructions when executed by the processor cause the computing device to perform a method according to any one of the preceding claims.

Images