IMAGE SELECTION SYSTEM FOR 3D RECONSTRUCTION

Abstract

A computer system for the three-dimensional reconstruction of a real-world environment is presented. The system comprises a preprocessing module, an image selection module, and a reconstruction module. The preprocessing module is configured to receive a set of images of the real-world environment and determine a pixel metric for each image within the set. The image selection module is configured to select a subset of images from the set based on a diversity of image subsets determined by the pixel metric and graph optimization. The reconstruction module is configured to reconstruct a three-dimensional representation of the real-world environment based on the selected image subset. A helper system, including the computer system, is also presented.

Description

Title of the invention: IMAGE SELECTION SYSTEM FOR 3D RECONSTRUCTION DOMAIN

[0001] The present invention relates to the processing of graphics and, in particular, the three-dimensional reconstruction of a real-world environment. CONTEXT

[0002] Three-dimensional (3D) graphic representations or models of a real-world environment are widely used for various technical purposes in computer graphics, for example, in conjunction with virtual or augmented reality applications. For instance, 3D models of buildings, technical facilities, or locations are used to train, e.g., firefighters or technicians when access to the actual building, facility, or location is not possible (or access is limited). Other examples include assisting users in planning their routes or visits to their destinations, e.g., to or from concerts or in case of an emergency, or presenting real-world environments to gamers.

[0003] As can be seen from the examples, it may be desirable to provide as accurate a representation as possible of real-world environments, such as objects or areas, with limited computing resources, such as processing power or storage. Although some initial approaches to addressing this requirement already exist, further improvements in processes and systems are still needed. SUMMARY

[0004] In this context, a computer system for the three-dimensional reconstruction of a real-world environment is presented. The system comprises a preprocessing module, an image selection module, and a reconstruction module. The preprocessing module is configured to receive a set of images of the real-world environment and determine a pixel metric for an image within the image set. The image selection module is configured to select a subset of images from the image set based on a diversity of image subsets determined by the pixel metric and graph optimization. The reconstruction module is configured to reconstruct a three-dimensional representation of a real-world environment based on the selected image subset.

[0005] In some embodiments, to determine the pixel metric, the preprocessing module is configured to determine multiple segments on an image within a set of images, generate integrations of multiple segments on the image, and combine the integrations to determine a plurality of concepts present on the image, whereby a concept relates to a recurring pattern of pixel color distribution and the concepts present on the image represent the pixel metric of the image. In other embodiments, to select the subset of images, the image selection module is configured to generate a bipartite graph with an image connected to one or more concepts present on the image, and to apply graph optimization to the bipartite graph to select the subset of images, such that a diversity of concepts is maximized for the subset of images.In other embodiments, graph optimization is based on an influence maximization problem. In some embodiments, the segments represent different semantic regions of the real-world environment. In other embodiments, to generate the integrations, the preprocessing module is configured to trim and reduce the number of multiple segments. In other embodiments, to combine the integrations, the preprocessing module is configured to group similar pixel patterns from multiple images in the image set to form a concept.

[0006] In other embodiments, to determine the pixel metric, the preprocessing module is configured to determine a depth map of an image within the image set and to create a plurality of points in a point cloud representing a three-dimensional structure of a real-world environment based on the depth map, whereby the points present in the image represent the pixel metric of the image. In other embodiments, to select the subset of images, the image selection module is configured to generate a visibility graph with an image connected to one or more concepts present in the image, and to apply graph optimization to the visibility graph in order to select the subset of images, such that point diversity is maximized for a reduced number of images.In other embodiments, to create the plurality of points, the preprocessing module is configured to project the depth images into a world space linked to the real-world environment and to merge the pixels that lie within the distance threshold of one voxel per point.

[0007] In some embodiments, the image set includes images of the real-world environment taken from multiple viewpoints; to determine a pixel metric for each image in the image set, the modulus of The preprocessing module is configured to apply one or more machine learning models and / or computer algorithms. In other embodiments, the selected image subset is independent of any model applied by the reconstruction module to reconstruct the three-dimensional representation.

[0008] A user assistance system for presenting a user with a three-dimensional representation of a real-world environment is also presented. The user assistance system includes the computer system as described herein. In some embodiments, the purpose of presenting the user with the three-dimensional representation includes at least one of the following: training the user to control a machine, assisting the user in navigating an environment, and / or improving a virtual three-dimensional representation in computer graphics.

[0009] The preceding paragraphs are provided by way of general introduction and are not intended to limit the scope of the following claims. The embodiments described, as well as other advantages, will be better understood with reference to the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF FIGURES

[0010] The preceding objects, features and advantages and other objects, features and advantages of the present subject will become evident from the following description of exemplary embodiments with reference to the accompanying drawings, in which similar numbers are used to represent similar elements, wherein:

[0011] The [Fig. 1] is an overview of a 3D reconstruction computer system according to the invention.

[0012] Fig. 2 presents processes executed by a preprocessing module in accordance with embodiments related to a concept.

[0013] Fig. 3 presents processes executed by an image selection module in accordance with embodiments related to concepts.

[0014] Fig.4A presents an example of a bipartite graph and Fig.4B presents a pseudocode of an optimization algorithm for greedy graphs according to the embodiments concerned.

[0015] Figure 5 presents processes executed by a preprocessing module in accordance with point cloud embodiments.

[0016] Figure 6 shows processes executed by an image selection module according to embodiments related to point clouds.

[0017] Fig. 7A presents an example of a visibility graph and Fig. 7B presents pseudocode for an optimization algorithm for greedy graphs according to embodiments related to point clouds.

[0018] Fig. 8 is a table comparing different approaches with the solutions presented herein.

[0019] The [Fig.9] is a schematic representation of internal components of a computer system, which may implement all or part of the functionalities described herein. DETAILED DESCRIPTION

[0020] The present invention relates to methods and systems for three-dimensional (3D) reconstruction of a real-world environment. Creating a 3D model of a real-world environment, e.g., an object or a scene, using multiple images from different viewpoints has long been a problem in computer graphics. Before the advent of deep learning techniques, such as deep neural networks and deep learning specifically for 3D reconstruction, traditional methods such as motion-based structure and multi-view stereo (i.e., different capture locations) were widely used. The introduction of neural radiance field (NeRF) models in, e.g., Mildenhall et al., "Nerf: Representing scenes as neural radiance fields for view synthesis," ACM Communications, vol.65, no. 1, pp. 99-106, 2021, revolutionized the rendering of new views by leveraging neural networks to create photorealistic 3D models where color is a function of camera position. This innovation has led to an increase in research on radiation fields, including improved NeRF models as well as alternative techniques such as Gaussian bursting and related work.

[0021] Despite these advances, radiation field-based processes typically require a large number of images (also called video frames, since images can be extracted from videos, i.e., they relate to video frames) to train a high-quality 3D representation. This is partly due to the lack of a systematic approach to capturing optimal video frames, given that the requirements are highly object-dependent, influenced by the object's geometry and color distribution. Researchers have proposed uncertainty estimation as a strategy to address this problem. While uncertainty-based techniques outperform random sampling, they require modifications to the NeRF architecture and training regime, such as continuous learning.

[0022] Other methods use the Tammes problem to predict video frame positions based solely on camera positions. While effective for synthetic data and spherical camera positions, this approach is less applicable to real-world data where video frame selection is also influenced by scene geometry and color distribution. Furthermore, the preceding approaches do not typically incorporate high-level concepts such as parts of objects in a 3D scene that could improve the interpretability of the video frame selection process.

[0023] Given the constraints on computing resources, it is often impractical to use all available video frames for 3D reconstruction, although this would generally lead to the best and most accurate reconstruction possible. Therefore, the challenge lies in selecting the best subset of video frames within a specified limit k (of video frames) that maximizes the quality of the 3D reconstruction. This constraint necessitates an efficient and effective selection strategy to ensure that the chosen video frames capture the essential features and variations of the scene presented herein.

[0024] Before proceeding to the solution presented herein, information on current approaches to addressing the aforementioned challenge is provided. Recent advances in neural radiation fields (NeRF) have focused on improving efficiency through reduced video frame requirements and enhanced computing strategies. Jain et al., “Putting nerf on a diet: Semantically consistent few-shot view synthesis,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 5885–5894, focuses on resolving the overfitting problems that are very common in traditional NeRF implementations when trained with limited data.For example, a loss of semantic consistency is introduced, and it uses a pre-trained image classifier to ensure that the images rendered from different viewpoints are not only photorealistic but also semantically consistent. This approach contrasts with the solution presented here, which focuses on semantic consistency to improve view synthesis rather than directly influencing video frame selection.

[0025] Pan et al., “Activenerf: Learning where to see with uncertainty estimation” (“Activenerf: Learning where to look with uncertainty estimation”, at the European Conference on Computer Vision. Springer, 2022, pp. 230-246, It incorporates an active learning framework to selectively improve the NeRF training process. Therefore, radiation field values ​​are modeled as a Gaussian distribution, and variance is the measure of uncertainty. The most informative training views are chosen by estimating the uncertainty in the model's predictions. While this approach aims to minimize uncertainty in view rendering, the solution presented here maximizes conceptual diversity between video frames, focusing on enriching the model's training dataset from a varied conceptual perspective rather than from an uncertainty reduction perspective.

[0026] Yu et al., “pixelnerf: Neural radiance fields from one or a few images,” in Proceedings of the 1TEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 4578–4587, proposes a variant of NeRF that is trained on one or a few images by conditioning pixel-aligned features extracted from those images using a pre-trained convolutional neural network (CNN). In contrast, the solution presented here does not rely on conditioning specific pixel features but rather selects various video frames based on the pixel metric by evaluating global coverage to improve the robustness and generalization of the reconstruction. Furthermore, this approach cannot select the best video frames for 3D reconstruction that satisfy the limit.

[0027] Goli et al., « Bayes’ rays: Uncertainty quantification for neural radiance fields “Bayes Rays: Uncertainty Quantification for Neural Radiation Fields,” arXiv preprint arXiv:2309.03185, 2023, introduces a post-hoc framework for quantifying uncertainty in pre-trained NeRF models, thereby improving robustness without altering the original training process. Unlike the solution presented here, which inherently incorporates diversity in video frame selection, this approach focuses on evaluating and managing uncertainty after training. Zhan et al., “Activermap: Radiance field for active mapping and planning,” arXiv preprint arXiv:2211.12656, 2022, integrates NeRF with active vision tasks, with an emphasis on dynamic environments.It uses RGB data only for both online 3D reconstruction and active planning, contrasting with the solution presented here, which relies on pre-captured multiview datasets to derive the diversity of concepts. Marza et al., “Autonerf: Training implicit scene representations with autonomous agents” (Autonerf: Representations. (of implicit training scenes with autonomous agents), arXiv preprint arXiv:2304.11241, 2023, develops an autonomous data-gathering framework in which embedded agents collect data for training the NeRF. This approach contrasts with the solution presented here in that it dynamically generates training data through exploration rather than selection from a static set of pre-existing video frames.

[0028] Kopanas and Drettakis, "Improving nerve quality by progressive camera placement for free-viewpoint navigation," 2023, proposes metrics to guide camera placement for NeRFs. It aims to optimize the distribution of camera angles for better reconstruction quality, which differs from the solution presented here, which consists of selecting various images based on their relevance. Cerkezi and Favaro, "Sparse 3D reconstruction via object-centric ray sampling," arXiv preprint arXiv:2309.03008, 2023, reduces view requirements using a novel object-centric sampling technique.This process focuses on the efficiency of sample use, which contrasts with the solution presented here, which focuses on maximizing conceptual diversity in the selected video frames.

[0029] Each of these previous works contributes to the field of 3D reconstruction, and none of them addresses the problem of video frame selection by exploiting image segmentation and aggregation to maximize diversity—which is the focus of the solution presented herein. Current methodologies not only improve the interpretability and relevance of the selected video frames but also remain independent of camera positions, making them highly adaptable to various real-world scenarios.

[0030] The solution presented herein operates on a (potentially large) set of images of the real-world environment that needs to be reconstructed. Using all the images in this image set for reconstruction would be impractical since storage and processing resources are generally limited. Therefore, it is beneficial to determine a smaller number of images forming a subset so that the 3D reconstruction can still be performed at a sufficiently good quality, e.g., with a predefined number of pixels, 3D image clarity, and the like. This result is achieved according to the invention by defining a pixel metric for an image within the image set in order to assess the image's relevance in the reconstruction process. Consequently, the image subset is selected from the image set based on a diversity of image subsets.Diversity of subsets. The image is determined based on pixel metrics and graph optimization. Finally, the three-dimensional representation of the real-world environment is reconstructed based on the selected image subset.

[0031] To select images from the image set, two approaches are described herein: a concept maximization approach that employs image segmentation followed by clustering to identify key concepts in a multiview image set, i.e., images captured in different locations and / or from different angles. A concept is defined herein as a recurring pattern of pixel color distribution across multiple images. A clustering problem (e.g., density-based) with the objective of maximizing the inclusion of various concepts in the selected video frames is then applied, thereby improving the process of selecting video frames for 3D reconstruction. In contrast to the concept maximization approach, a point maximization algorithm for a point cloud can be used.This algorithm is designed for scenarios where depth maps and camera positions are already available, allowing this approach to be used as a standalone process. Both approaches optimize video frame selection for multiview 3D reconstruction by maximizing diversity within a given video frame limit k.

[0032] The main advantages of the solution presented herein are camera position independence, model independence, and diversity-based pixel metric selection. In particular, the solution presented herein requires no information regarding camera position, making it applicable in varied and realistic environments using only RGB images as input. Furthermore, the solution is decoupled from 3D reconstruction models, thus improving its usability for different reconstruction techniques. In addition, high-level pixel metrics for diversity evaluation are used for video frame selection, thereby improving the interpretability and explainability of the process.

[0033] Figure 1 is an overview of a 3D reconstruction computer system 10 according to the invention. The computer system 10 comprises a preprocessing module 11, an image selection module 12, and a reconstruction module 13. The computer system 10 can be implemented by a single computer or processing system (as shown in Figure 1) but can also be implemented in a distributed manner by multiple computers, servers, or in the cloud. In such a distributed environment, the multiple computers can be connected via a network, which can be a wide area network, a global network, the Internet, or a similar network, and can be a public or private network, and can include multiple interconnected networks, as is known in the industry.

[0034] The preprocessing module 11 is configured to receive a set of images 14 of the real-world environment. Such images may be images, such as photographs or video frames, of the real-world environment taken from multiple viewpoints. The real-world environment may relate to several objects, such as technical installations, buildings, or the like, or a scene, such as a view of a street, a cave, or a tunnel structure, or the like.

[0035] The preprocessing module 11 can also be configured to determine a pixel metric 15 for an image in the image set 14. The pixel metric 15 can take several forms and can generally describe the content shown on the image in an abstract way. In some embodiments, the pixel metric 15 of the image can relate to concepts that are present on the image. The concepts relate to a recurring pattern of pixel color distribution over multiple images. This means that a concept can identify the same object, region, etc., on multiple images. Therefore, the pixel metric 15 can in this example provide a list of concepts or a vector representing concepts present on the image. In these embodiments, the preprocessing module 11 can be configured to determine multiple segments, e.g.regions of interest, on an image within the image set, to generate integrations, e.g. vector representations, of multiple segments on the image, and to assemble the integrations to determine the plurality of concepts present on the image.

[0036] In other embodiments, the pixel metric 15 of the image may refer to points in a point cloud that are present on the image. Therefore, a set or vector of points representing a 3D structure of the real-world environment shown on an image may define the pixel metric 15 of the image in this example. In these embodiments, the preprocessing module 11 may be configured to determine (e.g., calculate or extract from the image if the depth map has already been calculated upstream) a depth map of an image in the image set and create a plurality of points in a point cloud present on the image.

[0037] The image selection module 12 is configured to select a subset of images 16 from the image set 14 based on a diversity of the subsets of images based on the pixel metric 15 of the images in the image set 14 and based on a graph optimization. In some embodiments, in which the pixel metric 15 relates to concepts present on the image, the image selection module 12 can be configured to generate a bipartite graph with images connected to one or more concepts, i.e. the concepts present on a respective image according to the pixel metric 15. A graph optimization can then concern an influence maximization problem and can maximize the number of concepts for a fixed limit of k images, i.e., k images forming the image subset 16. The number k, i.e., how many images should be selected, can be defined by a user or automatically according to the available computing or storage resources and / or according to the number of images required by the 3D reconstruction model to be used in the reconstruction module 13.

[0038] In other embodiments, in which the pixel metric 15 relates to points present on the image, the image selection module 12 can be configured to generate a visibility graph with images connected to points, i.e. the points present on a respective image according to the pixel metric 15. A graph optimization can then relate to a maximization problem and can maximize the number of points for a fixed limit of k images, i.e., k images forming the subset of images 16.

[0039] The reconstruction module 13 is configured to reconstruct a three-dimensional representation 17 of a real-world environment based on the selected image subset 16. The selected image subset 16 may be independent of a reconstruction model applied by the reconstruction module 13 to reconstruct the 3D representation. For example, the selected image subset 16 works with neural radiation field (NeRF) models and with any other 3D reconstruction model, preferably machine learning 3D reconstruction models, since the selection objectively improves the diversity on the selected images and, consequently, improves the selection for 3D reconstruction models in general.

[0040] The computer system described herein, in various embodiments, can be used in an assistance system to present a user with a three-dimensional representation of a real-world environment. The purpose of presenting the user with the 3D representation may include at least one of the following: training the user to control a machine, assisting the user in navigating an environment, and / or enhancing a virtual three-dimensional representation in computer graphics. The efficiency of the computer system described herein surpasses systems implementing other algorithms due to the specific technical consideration of the image data structure to maximize diversity in the image subset.

[0041] Figure 2 shows processes executed by the preprocessing module 11 according to embodiments related to a concept. The preprocessing module 11 is configured to determine multiple segments on an image in the set of images 14 as indicated in Box 21. To optimize the selection of images (video frames) for 3D reconstruction, the preprocessing module 11 identifies and focuses on the most informative parts of each image. This result is achieved through the segmentation process in these embodiments. Segmentation isolates distinct regions or objects within the images. Segmentation involves dividing an image into segments, in which each segment represents a different region or semantic object. These segments are typically represented by masks, which are binary to multi-category images that highlight the regions of interest.

[0042] Segmentation techniques for collecting RGB images from multiple viewpoints are known in the industry, such as the Segment-Anything Model (SAM) by Kirillow et al.; “Segment anything” in the IEEE / CVF International Conference on Computer Vision, 2023n pp 4015-4026. These models typically perform semantic segmentation by dividing each image into distinct segments, each associated with a semantic label. By adjusting a probability threshold parameter, these models can be refined to be more or less conservative in their segmentation.

[0043] The output of the segmentation process in the preprocessing module 11 is a set of segmented images, where each segment corresponds to a different object or region on the image, represented by masks. This segmentation ensures that the most informative parts of the images are precisely identified and used for 3D reconstruction.

[0044] The preprocessing module 11 then generates integrations of multiple segments on the image, as shown in Box 22. To facilitate efficient stitching (as shown in Box 23) and improve computing efficiency, the resulting segments can—in certain embodiments—be cropped and their number reduced before generating the integrations. The integrations transform the segmented regions into a format that captures their nuanced features in a compact digital format.

[0045] To generate these integrations, EfficientNet by Tan and Le, "EfficientNet: Rethinking model scaling for convolutional neural networks," during the International Conference on Machine Learning, PMLR, 2019, pp. 6105-6114, can be applied, but other pre-trained vision models can also be used to generate these integrations. In some embodiments, element multiplication between the RGB image and the binary masks can be performed to obtain the segments in Box 21. In these embodiments, all the segments are the same size and can be fed into EfficientNet in a batch for inference. fast, using a graphics processing unit (GPU) for calculation. The output is a set of integrations that capture the essential characteristics of these segments.

[0046] The preprocessing module 11 is further configured to gather the integrations in order to determine the plurality of concepts present in the image, i.e., the pixel metric 15 of the image set 14, as shown in Box 23. As already indicated above, a concept relating to a recurring pattern in the pixel color distribution and the concepts present in an image represent the pixel metric 15 of the image. The integrations generated in Box 22 allow the segments to be grouped into significant clusters based on their visual characteristics.

[0047] Integrations can be clustered using the Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBScan) framework by McInnes et al., “hdbscan: Hierarchical density-based clustering,” in the Journal of Open Source Software, 2017, 2(11), 205. HDBScan is a hierarchical clustering framework that excels in environments with variable cluster density, which is typical of complex image data used in semantic segmentation. HDBScan works with a few key hyperparameters, such as minimum cluster size and minimum samples, which influence the granularity and noise sensitivity of the data. It is noted that other clustering algorithms can also be applied, as is known to the person skilled in the art.

[0048] The cluster groups similar pixel patterns from multiple images into clusters, referred to as concepts in this invention. Each cluster, or concept, represents recurring patterns in the images, facilitating detailed analysis and use in various applications such as 3D reconstruction.

[0049] Figure 3 shows processes executed by an image selection module 12 according to concept-related embodiments. The image selection module 12 is configured, for selecting the subset of images 16, to generate a bipartite graph with an image connected to one or more concepts on the image, as shown in Box 31. An example of a bipartite graph is shown in Figure 4A.

[0050] Figure 4A presents an example of a bipartite (and simplified) graph of embodiments related to the concept. Consider the three images 41, 42, and 43 and the three concepts 44, 45, and 46. Each image is then connected to the concepts present in its respective image. For example, image 41 may indicate all the concepts 44, 45, and 46, but images 42 and 43 each indicate only two concepts. Therefore, if all the concepts must be present for the 3D reconstruction, it is sufficient to select image 41 for the image subset 16. It is noted that The bipartite graph is usually larger, e.g., hundreds or millions of images (or frames, e.g., of videos) and hundreds of concepts depending on the real-world environment to be reconstructed.

[0051] Moving on to Fig. 3, the image selection module 12 is also configured to apply graph optimization to the bipartite graph in order to select the image subset 16, such that a diversity of concepts is maximized for the image subset 16. This graph-based approach helps to maximize the diversity of concepts in the limit k of images, i.e., in a determined number of k images forming the image subset 16 from N images in the image set 14. Due to the combinatorial complexity of f Æ )--iïL- in some embodiments, it may be beneficial to find \ k J ~ (N-kîk ' ' 1 1 Approximate but effective solutions. In this embodiment, the image or video frame selection process can iteratively choose video frames that contribute most to the overall concept coverage. A greedy algorithm can be used, which selects the video frames that contribute most to the overall concept coverage, as explained below. It should be noted, however, that other graph optimization algorithms mentioned above that solve the maximization problem can also be applied.

[0052] Initially, an empty set of selected video frames S is generated. For each image in the image set 14, the set of concepts to which the image is connected in the graph is determined. For each concept associated with an image, the specific pixels belonging to that concept are identified. A utility function U(S, i) is defined that calculates the number of new unique pixels contributed by image i for its associated concepts, excluding those already covered by the current set S. The algorithm then iteratively selects the image from the remaining set that maximizes this utility function, ensuring that each new image added to the selection maximizes the diversity of concepts. This process continues until the desired number of video frames k (e.g., a user-defined value or one automatically generated as described above) is reached or until there are no more images to select.A pseudocode for this greedy algorithm can be found in [Fig.4B]. The greedy approach ensures that the selected video frames collectively maximize the diversity of high-level concepts, thereby improving the efficiency and interpretability of the 3D reconstruction process.

[0053] The image selection module 12 then determines, based on the optimized graph from the image set 14, the image subset 16 which generally has a significantly lower number of images, i.e., k ≤ N. Consequently, less storage resources are required and the 3D reconstruction model requires a smaller subset of images to provide the same 3D image quality, i.e., the 3D reconstruction can be calculated faster.

[0054] Figure 5 shows processes performed by the preprocessing module 11 according to point cloud-related embodiments. The preprocessing module 11 is configured to determine a depth map of an image in the image set 14, as shown in Box 51, in order to determine the pixel metric. "Determine" should be understood broadly and may include the preprocessing module 11 calculating the depth maps of the images in the image set 14, or the depth maps already being present, e.g., pre-calculated and attached (at least partially) to the images in the image set 14. The depth maps represent the distance from the camera to objects in the scene, providing a point cloud representation of the real-world environment captured in 3D.

[0055] The preprocessing module 14 is further configured to create a plurality of points in a point cloud representing a 3D structure of the real-world environment based on the depth map as shown in Box 52. In this example, the points present on the image represent the pixel metric 15 of the image. In other words, the depth maps are projected into a 3D world space to create point clouds. These point clouds represent the 3D structure of the scene. To manage computational complexity and improve efficiency, points that are close to each other can be merged. This can be achieved in some embodiments by reducing the voxel sampling. This process reduces the number of points by grouping them into voxels, which are small cubic volumes, and retaining only one point per voxel.

[0056] Figure 6 shows processes executed by an image selection module 12 according to embodiments related to point clouds. The image selection module 12 is configured, in order to select the subset of images, to generate a visibility graph with an image connected to one or more points present on the image, as shown in Box 61. An example of a visibility graph is shown in Figure 7A.

[0057] Figure 7A presents an example of a (simplified) visibility graph of point cloud embodiments. Consider the three images 71, 72, and 73 and the six points (or voxels) 74, 75, 76, 77, 78, and 79. Each image is then connected to the points present in its respective image. For example, image 71 may only display points 74 and 75, but image 72 displays all the other points 76, 77, 78, and 79. Image 73 displays points 75, 76, and 77, which are also present in the other images. Therefore, if all the points must be present for the 3D reconstruction, it is sufficient to select images 71 and 73 for the image subset 16. It is noted that the visibility graph is usually larger, e.g. hundreds or millions of images (or frames, e.g., of videos) and hundreds of points depending, e.g., on user configurations.

[0058] Moving on to [Fig. 6], the image selection module 12 is also configured to apply graph optimization to the visibility graph to select the subset of images so that point diversity is maximized for a reduced number of images, as shown in Box 62. In some embodiments, it may be beneficial to find approximate but efficient solutions. In these embodiments, this image or video frame selection process may use a greedy algorithm to maximize the number of points (or voxels) covered. This greedy algorithm iteratively selects video frames k that contribute most to the overall coverage of the point cloud, ensuring that the selected video frames capture the maximum detail and variation of the scene. Pseudocode for this greedy algorithm can be found in [Fig. 7B].

[0059] The image selection module 12 then determines, on the basis of the optimized graph from the image set 14, the image subset 16 which generally has a significantly lower number of images, i.e. k « N. Consequently, fewer storage resources are required and the 3D reconstruction model requires a smaller subset of images to provide the same 3D image quality, i.e. the 3D reconstruction can be calculated more quickly.

[0060] Fig. 8 is a table comparing random sampling, farthest view sampling, ActiveNeRF (with Bayesian estimation, -BE, and with continuous learning, -CL) and DietNeRF with the solutions presented herein.

[0061] Random sampling refers to the random selection of k video frames from a dataset of N video frames. FVS starts with an initial video frame chosen randomly and iteratively adds the video frames that are furthest from the already selected set. For each iteration, FVS finds the video frame with the minimum maximum Euclidean distance (minmax) from its camera position to that of the current set of selected video frames. This process continues until the desired number of k video frames is selected. By maximizing the spatial distances between the selected video frames, this method aims to capture a wide range of perspectives, thus ensuring complete scene coverage.

[0062] For the 3D reconstruction in this comparison, the vanilla neuronal radiation field (NeRF) model was used, with the help of the dataset Local light field fusion (LLFF) by Mildenhall et al., "Local light field fusion: Practical view synthesis with prescriptive sampling guidelines," ACM Transactions on Graphics (TOG), vol. 38, no. 4, pp. 1-14, 2019. This dataset provides eight different realistic scenes with two configurations: spherical and non-spherical. Our goal is to select k video frames from N available video frames, where k is the limit and N is the total number of video frames in the scene. PSNR, SSIM, and LPIPS metrics by Zhang et al., "The unreasonable effectiveness of deep features as a perceptual metric," in IEEE Conference Proceedings on Computer Vision and Pattern Recognition, 2018, pp.Images 586-595 were used to evaluate the reconstruction quality. The NeRF model was trained for 50,000 periods using images selected by the respective video frame selection algorithms being compared. The remaining images were used as test images to evaluate the model. For comparison, random sampling and farthest view sampling (FVS) methods were also performed. ActiveNeRF and DietNeRF were included for comparison with results from the literature. These methods were trained for 200,000 periods, and their results are presented for spherical cases with settings of 10 and 20 video frames. ActiveNeRF BE (Bayesian Estimation) does not involve NeRF in the video frame selection loop, while ActiveNeRF CL (Continuous Learning) integrates the model into the loop.DietNeRF modifies the NeRF loss function using image integrations from CLIP.

[0063] In the experiments, the images in the LLFF dataset were downsampled to a resolution of 378x504 pixels. For concept maximization as described herein, image segmentation was performed using the SAM model with a probability threshold set at 0.8. Segmented regions were cropped and their number reduced by a factor of 4, and segmentation masks smaller than the square root of the height-width product were discarded to improve computational efficiency. Integrations were generated using the EfficientNet model. These integrations were then clustered using HDBScan to group similar pixel patterns from multiple images into "concepts," with the minimum cluster size set at N / 4. Outliers were discarded to ensure high-quality clusters.The relationships between the images and their corresponding concepts were modeled in a graph. The greedy video frame selection algorithm was applied to maximize the diversity of concepts, as shown in [Fig. 4B]. For the maximization based on... On the point cloud, COLMAP by Schonberger and Frahm, "Structure-from-motion revisited" in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4104-4113, was used to compute depth maps and create point clouds. A visibility graph was derived to maximize the selection of unique points via a greedy algorithm, detailed in [Fig. 7B].

[0064] The average of the results, as shown in [Fig. 8], obtained from eight scenes from the LLFF dataset, reveals key information about the effectiveness of each method. For all settings with 10, 20, and 25 observations—that is, the number of images in the selected subset or k as described above—the results of the concept maximization method outperformed all other methods for both spherical (same camera distances from the object / scene) and non-spherical (different camera distances from the object / scene) image configurations. This concept-based method achieved the highest metrics, thus indicating its superior ability to maximize concept diversity within a limited number of video frames. Point cloud point maximization also showed good performance compared to the other approaches.With 25 observations, the computer systems implementing the two methods presented herein showed high performance, with concept maximization slightly outperforming the overall quality metric. In general, the results highlight the effectiveness of the concept maximization approach in improving 3D reconstruction by maximizing the diversity of high-level concepts. The point cloud approach also proved to be a robust method, particularly advantageous when depth data is already available for the images in a dataset. These findings demonstrate the potential of the video frame selection approaches for 3D reconstruction described herein.

[0065] Figure 9 is a graphical representation of internal components of a system A computer system 90 implementing the functionality of one or more components, as described herein. The computer system 90 includes at least one processor 91, a user interface 92, a network interface 93, and main memory 96, which communicate with each other via a bus 95. Optionally, the computer system 90 may also include static memory 97 and a hard disk drive (not specified), which communicate with each other via the bus 95. A video display, an alphanumeric input device, and a cursor control device may be provided as examples of the user interface 92. In addition, the computer system 90 may also include one or more graphics processing units (GPUs) 94.

[0066] GPUs 94 may also include a plurality of GPU cores or stream multiprocessors, which comprise many different components, such as at least one register, at least one cash and / or shared memory, and a plurality of ALUs, FPUs, Tensor Processing Units (TPUs) or Tensor Cores, and / or other optional processing units. GPUs can perform multiple computations simultaneously, thereby enabling the distribution of training processes and the acceleration of machine learning operations.

[0067] The main memory 96 may be random access memory (RAM) and / or any other volatile memory. The main memory 96 may store program code 98a and may also store additional program data 98b required to provide all or part of the functionality described herein. In addition, the main memory 96 may also include a cache 99.

[0068] According to another aspect, a computer program comprising instructions may be provided. These instructions, when the program is executed by a computer, cause the computer to perform the processes described herein. The program code embedded in any of the systems described herein is capable of being distributed individually or collectively as a program product in various forms. In particular, the program code may be distributed using a computer-readable storage medium containing computer-readable program instructions to cause a processor to execute aspects of the embodiments described herein.

[0069] Computer-readable storage media, which are inherently non-transient, may include tangible volatile and non-volatile, and removable and non-removable media, implemented in any information storage process or technology, such as computer-readable instructions, data structures, program modules, or other data.Computer-readable storage media may also include random access memory (RAM), read-only memory (ROM), programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other semiconductor memory technology, portable compact disc (CD-ROM) read-only memory or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be read by a computer.

[0070] A computer-readable storage medium should not be interpreted as transient signals in themselves (e.g., radio waves or other propagating electromagnetic waves, electromagnetic waves propagating through a transmission medium such as a waveguide, or electrical signals transmitted via a cable). Computer-readable program instructions can be downloaded to a computer, another type of programmable data processing device, or other device from a computer-readable storage medium or to an external computer or external storage device via a network.

[0071] It should be noted that while particular embodiments and variations have been described herein, other modifications and alternatives will be obvious to those skilled in the art. In particular, examples are provided to illustrate the principles and to supply a number of specific methods and arrangements for implementing these principles.

[0072] In certain embodiments, the functions / actions specified in the flowcharts, sequential diagrams, and / or block diagrams may be reordered, processed serially, and / or simultaneously without departing from the scope of the invention. Furthermore, all flowcharts, sequential diagrams, and / or block diagrams may include more or fewer blocks than those illustrated, in a manner consistent with the embodiments of the invention.

[0073] The terminology used herein is intended to describe particular embodiments only and is not intended to limit the embodiments of the invention. It will be understood that the terms "includes" and / or "comprising," when used in this specification, denote the presence of features, integers, processes, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, processes, operations, elements, components, and / or groups thereof. Furthermore, to the extent that the terms "includes," "having," "has," "with," "composed of," or variations thereof are used either in the detailed description or in the claims, such terms are intended to be inclusive in a manner similar to the term "comprising."

[0074] While a description of the various embodiments has illustrated the process, and while these embodiments have been described in great detail, the applicants do not intend to restrict or limit in any way the scope of the attached claims with regard to such details. Additional advantages and modifications will readily become apparent to those skilled in the art. The invention in its broadest aspects is therefore not limited to the specific details, apparatus, and representative process and illustrates the examples indicated and described. Accordingly, the described embodiments should be understood as being provided by way of example, for the purpose of teaching the general features and principles, but should not be construed as limiting the scope as defined in the attached claims.

Claims

Demands

1. A computer system for three-dimensional reconstruction of a real-world environment, the system comprising: a. a preprocessing module; b. an image selection module and c. a reconstruction module; wherein the preprocessing module is configured to: • receive a set of images of the real-world environment and • determine a pixel metric for an image in the image set; wherein the image selection module is configured to: • select a subset of images in the image set based on a diversity of image subsets based on the pixel metric and a graph optimization; wherein the reconstruction module is configured to: • reconstruct a three-dimensional representation of a real-world environment based on the selected image subset.

2. The system of claim 1, wherein, to determine the pixel metric, the preprocessing module is configured to: • determine multiple segments on an image in the image set; • generate integrations of multiple segments on the image; and • assemble the integrations to determine a plurality of concepts present on the image, where a concept relates to a recurring pattern of pixel color distribution, and where the concepts present on the image represent the pixel metric of the image.

3. The system of claim 2, wherein, for selecting the subset of images, the image selection module is configured to: • generate a bipartite graph with an image connected to one or more concepts present on the image and • apply graph optimization on the bipartite graph to select the subset of images, so that a diversity of concepts is maximized for the subset of images.

4. The system of claim 3, in which graph optimization is based on an influence maximization problem.

5. The system according to any one of claims 2 to 4, wherein the segments represent different semantic regions of the real-world environment.

6. The system according to any one of claims 2 to 5, wherein, to generate the integrations, the preprocessing module is configured to trim and reduce the number of multiple segments.

7. The system according to any one of claims 2 to 6, wherein, to assemble the integrations, the preprocessing module is configured to group similar pixel patterns from multiple images in the image set to form a concept.

8. The system of claim 1, wherein, to determine the pixel metric, the preprocessing module is configured to: • determine a depth map for an image in the image set and • create a plurality of points in a point cloud representing a three-dimensional structure of the real-world environment based on the depth map, wherein the points present on the image represent the pixel metric of the image.

9. The system of claim 8, wherein, for selecting the subset of images, the image selection module is configured to: • generate a visibility graph with an image connected to one or more points present on the image and • apply graph optimization to the visibility graph to select the subset of images, so that a diversity of points is maximized for a reduced number of images.

10. The system of claim 8 or 9, wherein, to create a plurality of points, the preprocessing module is configured to project the depth maps into a world space linked to the real-world environment and to merge the pixels that lie within the distance threshold of one voxel per point.

11. The system according to any one of the preceding claims, wherein the image set comprises images of the real-world environment taken at multiple points of

12. view. The system according to any one of the preceding claims, wherein, to determine a pixel metric for each image in the image set, the preprocessing module is configured to apply one or more machine learning models and / or computer algorithms.

13. The system according to any one of the preceding claims, wherein the selected image subset is independent of a model applied by the reconstruction module to reconstruct the three-dimensional representation.

14. An assistance system for presenting a user with a three-dimensional representation of a real-world environment comprising the computer system of any one of claims 1 to 13.

15. The assistance system of claim 14, wherein the object of the user presentation with the three-dimensional representation comprises at least one of the user training to control a machine, user assistance in navigating an environment, and / or the improvement of a three-dimensional virtual representation in computer graphics.