Incremental parametric modeling based on 3D scan data

Abstract

Incremental parametric modeling based on 3D scan data. The invention relates to a method of generating a parameter set for modeling a real-world object in a scene. The parameter set represents information about a spatial extent and position of a first object in a frame of reference associated with the scene. The method comprises: (a) accessing input data based on a point cloud, the point cloud being recorded from different viewpoints; (b) providing an array representation of the point cloud using scan coordinates as rows and columns; (c) identifying image blocks in the array corresponding to the object; (d) deriving the parameter set based on the identified image blocks and known viewpoints.

Description

Technical Field

[0001] This invention relates to a method for generating parametric models of a set of real-world objects in a scene. The parametric models represent information about the spatial extent, position, and orientation of the respective objects within a reference frame associated with the scene, particularly the relative pose of the set of objects. The invention also relates to a computer program product and a reality capture system configured to perform the method. Background Technology

[0002] Digital twins have revolutionized the planning, design, and maintenance of everything from individual mechanical components to entire city infrastructures and beyond. Essentially, a digital twin is a virtual copy of a physical object or system, bridging the real world and digital data. Using digital twins, organizations can simulate, visualize, and analyze their operations in a risk-free and cost-effective virtual environment. It empowers broad decision-making capabilities by identifying performance gaps, potential failures, or areas requiring optimization or improvement through real-time monitoring and predictive analytics.

[0003] However, the creation of digital twins relies heavily on accurate and detailed 3D modeling. The 3D model gives the digital twin its form and structure. Using representation methods such as Computer-Aided Design (CAD) or Building Information Modeling (BIM), engineers digitally replicate the physical properties of a product or structure, laying the foundation for digital twins.

[0004] State-of-the-art 3D modeling processes are not without challenges. Typically, it's a supervised process performed by expert engineers, requiring significant time and expertise, especially when modeling complex entities. Furthermore, ensuring the reliability of these models is crucial, as even minor discrepancies can lead to major problems in downstream tasks. The field of 3D modeling has made significant progress with the advent of RGBD imaging and point clouds in various applications such as computer vision, robotics, geographic information systems, and 3D modeling. RGBD images combine color (RGB) and depth (D) information. Point clouds represent objects in a 3D Cartesian coordinate system. In the context of this invention, a point cloud is a dense, continuous 3D point cloud. While the required density depends heavily on the specific application, these point clouds contain millions of individual points with corresponding real-world coordinates and potentially further correlated data. For example, for typical construction site surveying, 1000 points per square meter on a surface 10 meters from the scanner and perpendicular to the scanning direction can be considered a dense point cloud.

[0005] One of the main challenges in 3D modeling from RGBD or 3D point cloud data is handling large and noisy datasets. Techniques such as voxel methods, sparse voxel meshes, or SuperPoint representations have been developed to address the computational cost associated with high-resolution 3D data, which has a significant impact on detail preservation.

[0006] Novel deep learning techniques have achieved significant improvements in both image-based and point cloud-based approaches. Particularly in image-based applications, the technology has evolved to the point of visual foundational models. These foundational models are trained on vast amounts of data and accumulated knowledge about different types of objects that can be located, described, and separated from the background. The trained models are able to extract semantic information from images during inference and possess excellent scene understanding capabilities.

[0007] The expectation is to use this modeling approach not only in 2D but also in 3D. However, handling disordered, irregular, and noisy 3D point clouds presents unique challenges. In particular, state-of-the-art models lack robustness, efficiency, reliability, and applicability across different scenarios. Therefore, improving the accuracy and efficiency of the 3D modeling process remains an unresolved challenge.

[0008] Furthermore, the data acquisition phase is typically conducted in field conditions, without access to powerful computing resources. Therefore, executing 3D processing algorithms on "low-end" edge devices inevitably faces several limitations. This is especially true for 3D modeling algorithms, which involve handling complex data structures, computationally intensive arithmetic, and massive amounts of data, while also demanding high reliability of the output. Most state-of-the-art solutions rely on powerful neural networks for 3D segmentation and consistent modeling processes, which, to date, are still primarily performed manually by expert operators. Automation is crucial, both for optimizing the modeling process, reducing costs and delivery time, and for eliminating human bias in the results due to interpretation or unexpected errors.

[0009] Furthermore, bringing the modeling process to the field supports multiple downstream tasks and allows operators to effectively utilize the time available for surveying equipment operation. Typically, such a phase is not effectively utilized by surveyors. Performing field modeling from scratch during a scanning operation is impractical due to its time-consuming nature and the need for specialized hardware unavailable on-site. However, this time window enables the execution of a semi-supervised modeling process. In other words, the user only reviews the system's suggestions and makes only minor adjustments as needed. Moreover, this supervision can be efficiently performed on a companion device (i.e., a tablet). Summary of the Invention

[0010] In view of the above, one object of the present invention is to provide a more robust and less resource-intensive modeling method based on point clouds.

[0011] Another objective of this invention is to reduce the level of expertise required for operators to perform modeling.

[0012] This invention relates to a computer-implemented method for generating a first set of parameters for a parameterized model representing a first object in a scene. The scene contains a set of real-world objects.

[0013] In this invention, objects and scenes represent real-world environments, such as buildings with doors and windows, and objects are part of said environment. An object in this invention can represent a part of a physical entity; in particular, the facade of a building can be considered an object. An object in this invention extends at least in one dimension (1D), and particularly in two dimensions (2D). However, a numerical representation, such as an image portion corresponding to an object, may also be simply referred to as "object" if the context is clear.

[0014] There are no fundamental obstacles to applying the invention to situations where the set of real-world objects contains only one or a few unrelated objects. However, many aspects of the invention are interpreted based on scenarios involving (partial) sets of objects that are spatially and functionally interconnected. While the invention is advantageous in these cases, these examples should not be construed as limiting.

[0015] The first parameter set represents information about the spatial extent and position of a first object within a frame of reference associated with the scene. Specifically, multiple parameter sets associated with various objects represent the spatial relationships between objects in the scene relative to each other, i.e., extent, shape, position, and orientation. A parametric model in the sense of this invention means describing real-world objects with as few necessary parameters as possible. For example, a wall segment can be represented as a rectangle with a corresponding 2D extension and position. For more complex objects, the parameter sets can be appropriately complexified accordingly. Parameter sets can include semantic data about the functional roles of objects or data about relationships between objects. For example, a parameter set can specify that a given object is designated as a "window," which has a certain spatial relationship with another object designated as a "wall." In this case, adjustments to the parameter set of the "wall" object can be automatically applied to the parameter set of the "window" object. Those skilled in the art will understand that the term parameter set does not mean a single, continuous block of data, but rather a collection of data associated with an object.

[0016] The method includes (a) accessing input data representing a scene, the input data being based on (i) a first point cloud associated with a first viewpoint and acquired by a point cloud recording device, and (ii) a second point cloud associated with a second viewpoint and acquired by a point cloud recording device, wherein the first and second viewpoints are different; and (b) based on the input data, providing a set of arrays including a first array associated with the first viewpoint and a second array associated with the second viewpoint, wherein each array includes (i) rows having first image coordinate values ​​associated with first scan coordinates, (ii) columns having second image coordinate values ​​associated with second scan coordinates, and (iii) representations of the corresponding first and second scan coordinates. The steps include: (c) providing a first set of image patches by processing the first array individually using an image processing algorithm, wherein each image patch in the first set is associated with a single real-world object in the scene; (d) providing a second set of image patches by processing the second array individually using an image processing algorithm, wherein each image patch in the second set is associated with a single real-world object in the scene; (e) identifying a first patch in the first set of image patches and a second patch in the second set of image patches by jointly processing the first and second arrays, wherein the first and second patches correspond to a first object; and (f) deriving a first set of parameters based on the identified first and second patches and the first and second viewpoints. Those skilled in the art will understand that the step numbers should not be interpreted as instructions for sequential execution of the steps, but rather as an enumeration for improved readability. The above steps can be performed in any reasonable order.

[0017] Feature coordinates may be associated with different concepts. Therefore, the following conventions apply throughout the specification. The true three-dimensional (3D) coordinates (Cartesian or polar coordinates) of an object are referred to as real-world coordinates. Coordinates related to beam deflection within the scanning device are referred to as scan coordinates. Coordinates representing array rows and columns are referred to as image coordinates to reflect their 2D nature.

[0018] Those skilled in the art will also understand that, for the purposes of this invention, it is implied that the first and second viewpoints provide an overlapping view of the scene, and in particular, the first object must be recorded from both viewpoints. A viewpoint can be understood as a fixed position, specifically a position within the scene itself. However, the invention is equally applicable to scans with “continuous positions,” such as moving scans, where LiDAR and image data streams are provided. For this type of scan, virtual viewpoints can be created.

[0019] The point cloud recording device in the sense of this invention can be any suitable instrument. A non-exclusive list of point cloud recording devices includes laser scanners, such as the laser scanner disclosed in EP 3 825 720 A1; profilometry instruments, including multibeam profilometry instruments, such as the instrument disclosed in EP 3 816 657 A1; lidar devices, including multibeam lidar devices, such as the device disclosed in EP 3 460 519 A1; time-of-flight cameras; laser trackers or total stations. The structural features disclosed in the aforementioned documents, particularly the beam steering mechanism and the generation and format of point clouds, are incorporated herein by reference.

[0020] The point cloud recording device of this invention is configured to provide a dense 3D point cloud from a fixed location. While point cloud recording devices capable of generating full-dome vertex clouds have advantages due to their large data volume, the method of this invention is equally applicable to forward-scanning lidar or profilometers with a limited elevation angle range and similar alternatives.

[0021] Furthermore, those skilled in the art will understand that the method according to the invention is not limited to the case where identical point cloud recording devices are used to acquire the first and second point clouds, provided that these devices can be considered equivalent to each other. In other words, the first and second point clouds must be structurally compatible.

[0022] The array's structure makes it resemble a 2D image. For forward-looking multi-pixel LiDAR, image coordinates might be the actual pixels. For a laser scanner with a rotating mirror, image coordinates can be derived from angular (scan) coordinates. Advantageously, this representation allows the application of image processing algorithms designed for 2D images. Particularly advantageous is that machine learning methods are "unbiased" for distorted views created through this image simulation. In other words, 2D algorithms and neural networks that can be efficiently processed in 3D space but operate in 2D can be applied. This significantly reduces the computational requirements on the device. Due to the rotating scan operation, 2D data (e.g., objects) is distorted in a given representation; however, the method and 2D algorithms of this invention are able to robustly process semantic information.

[0023] Intensity values ​​can be viewed as scalar values, such as distance from the viewpoint or the signal level of a returned pulse. Intensity values ​​can also be viewed as vectors, such as color and distance information. There is no fundamental difference in the applicability of these two cases to image processing methods or machine learning. For transparency, the first case will be discussed in detail below. Specific aspects of the second case can be applied accordingly.

[0024] A concrete example of an image processing algorithm might be the aforementioned 2D vision foundational model. Many state-of-the-art models are known. A non-exclusive list includes "OneFormer: One Transformer to Rule Universal ImageSegmentation" or "Mask2Former: Masked-attention Mask Transformer for Universal Image Segmentation," or custom models based on lightweight backbone networks (such as "EfficientNet" or "MobileViT") with efficient segmentation heads, as proposed in "EfficientPS: Efficient Panoptic Segmentation." The latter approach is particularly advantageous for processing on real-time devices. Those skilled in the art can develop appropriate models based on the listed or suitable alternatives.

[0025] For example, the scene might be a building under construction, and the measurement task might be deriving a floor plan. The model might automatically identify the environment and / or provide user input about the task to the model, such as in the form of drop-down menus. The scene then consists of walls, floors, ceilings, windows, doors as objects, and irrelevant items (e.g., building materials or equipment placed in the scene). The processing of the array, whether individual or joint, should be understood in this context. Joint processing may specifically include combining the first and second blocks and the parameter set to make them consistent / coherent, removing duplicates, or robustly detecting weak classifications with the help of more locations / blocks.

[0026] After the initial point cloud acquisition from the first viewpoint is completed, the model may process the point cloud and identify image patches in a first array. Each image patch corresponds to a single specific object and preferably provides a semantic label regarding a potential role, such as whether a particular image patch belongs to the floor. The model specifically leverages scene understanding gained through training on large amounts of input data (e.g., unstructured areas at elevation coordinates may belong to the ceiling). The same process can then be repeated for the second viewpoint. The model then utilizes the redundant information available between overlapping scans to provide robust and reliable predictions. In particular, it identifies corresponding image patches in both arrays based on the relative arrangement and / or semantic labels of the image patches. Finally, a set of parameters is proposed for the identified objects. Those skilled in the art will understand that this set of parameters may not be final. For example, the full extent of the ceiling may not be determined from the first and second viewpoints.

[0027] In some embodiments, the measurement data includes (a) data related to the distance from the viewpoint, and / or (b) data related to the level of the returned beam signal, and / or (c) data related to the surface color, and / or (d) a surface-related normal vector defined by a plurality of points in the point cloud, particularly a normal vector corresponding to a local tangent plane. Those skilled in the art will understand that the above list is non-exclusive and may alternatively or additionally include further (raw) measurement data, radiometric measurements, or geometric features. In some specific embodiments, the intensity value is based on a vector containing surface color-related data (particularly RGB data) and data related to the distance from the viewpoint.

[0028] In some embodiments, the point cloud recording device is implemented as a laser scanner. The laser scanner includes a first scanning element that defines the first scan coordinates as azimuth coordinates. The first scanning element can be implemented as a support unit providing pivoting capability relative to a base (particularly a tripod base). The laser scanner also includes a second scanning element that defines the second scan coordinates as elevation coordinates. The second scanning element can be implemented as a rotating mirror, particularly a fast-rotating mirror pivotally mounted to the support unit. Those skilled in the art will appreciate that the laser scanner described herein includes corresponding motor components, angle encoders, and control elements to record the azimuth of the scanning element and control its movement. The first image coordinates are linearly proportional to the azimuth coordinates, and the second image coordinates are linearly proportional to the elevation coordinates. In other words, the array can be considered as a quasi-image with linearized polar coordinates. In some specific embodiments, the laser scanner is configured to acquire a full dome scan.

[0029] In some implementations, the input data is provided as a set of point clouds, particularly during scene scanning. Specifically, the point clouds are provided in polar coordinates. Alternatively or additionally, encoder data regarding the scanned elements may also be provided as a substitute for (real-world) polar coordinates. Performing modeling during scanning from another viewpoint improves efficiency.

[0030] In some implementations, the derivation of the first parameter set is based on heuristic methods, particularly a weighted combination of predictor factors. The weights used to combine the predictor factors can be defined, for example, by the distance / range of each block measured by the device from each scan position or viewpoint. Thus, the features or objects predicted for each block are scaled by the inverse of the distance or the inverse of the square of the distance. This results in a lower contribution / weight for features predicted by the neural network for blocks measured from a distance compared to features predicted for blocks computed from a close distance. After computing the combination of predictor factors, each block has a robust prediction. Heuristic modeling is required to parameterize the objects and define external relationships. Complementary heuristics, such as the viewpoint (the angle between the surface normal and the incident direction of the lidar beam), can be used.

[0031] Alternatively or additionally, a “learned combination of predictors” can be used. This approach can be based on using an “attention” mechanism that “learns” how to integrate the predictions of multiple predictors without any heuristics for calculating weights.

[0032] In some implementations, the method also includes manual adjustment. Manual adjustment includes (a) displaying a first parameter set, (b) accessing user input data, and (c) confirming or adjusting the first parameter set based on the user input data. Specifically, the user input data can be provided to the model for retraining.

[0033] In some implementations, manual adjustment also includes (a) displaying at least a portion of the first and / or second point cloud, and (b) displaying a parametric model of a first object referenced to the displayed point cloud. Such implementations are advantageous because a direct visual comparison between the proposed model and the original point cloud allows human operators to identify potential errors more quickly. This is particularly beneficial for users with lower levels of expertise, as the information displayed in this way follows more self-evident principles.

[0034] This manual adjustment enables "human-in-the-loop" supervision and correction. Operators can supervise the predictions made by the computer-implemented method, using these human inputs to influence upcoming predictions. In this way, the set of parameters verified by the operator (true positives), as well as corrections to erroneous predictions (false positives), and especially undetected objects (false negatives), can be utilized in a "feedback mechanism" to enhance predictions in subsequent iterations.

[0035] In some implementations, the parameter set includes confidence predictions of confidence values ​​associated with numerical or semantic information stored in the parameter set. This allows a human operator to supervise the modeling process each time and focus on validating predictions with lower confidence when needed. User feedback regarding such confidence can also be implicitly or explicitly integrated into the method. That is, the confidence or reliability associated with each identified real-world object can be increased or decreased accordingly based on user feedback.

[0036] In some implementations, the set of real-world objects includes a second object. The method further includes (a) identifying a third block in the first set of image patches and a fourth block in the second set of image patches by jointly processing the first and second arrays, the third and fourth blocks corresponding to the second object; and (b) deriving a second set of parameters for a parameterized model of the second object based on the identified third and fourth blocks, the first viewpoint, and the second viewpoint, particularly the first set of parameters. As will immediately be recognized by those skilled in the art, the steps for parameterizing the second object are substantially similar to those for parameterizing the first object. In other words, the same optional features can be applied accordingly.

[0037] A particularly useful implementation of parameterizing second objects is the so-called incremental approach, where the model leverages scene understanding. For example, after identifying and parameterizing a wall segment, a window on that wall can be identified and parameterized, especially by providing constraints between the wall segment object and the window object.

[0038] In some implementations, the set of real-world objects includes a third object, and the input data is further based on a third point cloud associated with a third viewpoint and acquired by a point cloud recording device. Features associated with the second viewpoint and the second point cloud can be applied accordingly to the third viewpoint and the third point cloud. The method also includes (a) providing the set of arrays based on the input data, such that it includes a third array associated with the third viewpoint, which is functionally equivalent to the second array; (b) providing a third set of image patches by processing the third array separately using an image processing algorithm, wherein each image patch in the third set is associated with a single real-world object in the scene; (c) identifying a fifth patch in the first set of image patches and a sixth patch in the third set of image patches by jointly processing the first and third arrays, the fifth and sixth patches corresponding to the third object; and (d) deriving a third set of parameters for a parameterized model of the third object based on the identified fifth and sixth patches, the first viewpoint, the second viewpoint, and the third viewpoint, and in particular the first set of parameters. As will be readily apparent to those skilled in the art, the steps for parameterizing the third object are substantially similar to those for parameterizing the first object. In other words, the same optional features can be applied accordingly.

[0039] A particularly useful implementation of parameterizing a third object is the so-called cumulative approach, where the model utilizes scene understanding. For example, a wall segment can be parameterized to connect to a previously identified ceiling section. Those skilled in the art will understand that knowledge gained from parameterizing the third and / or second object can be used to adjust the first set of parameters corresponding to the first object accordingly.

[0040] In some implementations, each parameter set includes object classifications or semantic labels corresponding to the real-world roles of associated real-world objects. The method also includes (a) deriving a scene graph based on object classifications, where the scene graph represents the structural and functional relationships of real-world objects, and (b) adjusting the parameter sets based on a set of constraints corresponding to the scene graph. Constraints could be, for example, that a chair object cannot be directly connected to a ceiling, or enforce geometric priors such as the perpendicularity of wall seams. This is referred to as "constraint-based optimization," and its purpose is to leverage prior knowledge to eliminate errors in the modeling process. This can be the final step in the modeling process.

[0041] In some implementations, a scene graph represents hierarchical relationships, that is, it references parent objects, such as walls in a building, and related child objects, such as windows on that wall. In other words, a scene graph condenses information relevant to scene understanding.

[0042] Specifically, a scene graph can be constructed where nodes define the semantic and parametric representation of each identified real-world object. This is supplemented by edges in the graph, representing external relationships / intersections between the objects. This alternative not only provides an end-to-end AI solution without any heuristic-based modeling process, but also provides all the necessary information for generating derivatives such as planar graphs or parametric 3D models that rely on knowledge of object relationships for consistent modeling.

[0043] In some specific embodiments, the method further includes (a) assigning a first block and a second block to a first object classification corresponding to a first object, (b) providing a first prediction for a first unassigned image block regarding a second object classification based on image coordinates of the first unassigned image block relative to the first block or the second block and a scene graph, wherein the first unassigned image block corresponds to an unsorted object, (c) providing a second prediction based on the first prediction regarding an additional set of parameters associated with the unsorted real-world object, and (d) identifying a second unassigned image block corresponding to an unsorted real-world object based on the second prediction.

[0044] In some specific embodiments, the scene is a building and / or a construction site or a portion thereof. Scene diagrams correspond to one of floor plans, construction drawings, blueprints, CAD models, and BIM. Those skilled in the art will understand that alternative 2D or 3D models, particularly those used in the construction or manufacturing industries, can be used. Furthermore, the applicability of the invention is not limited to currently available modeling tools, but also includes any reasonable further development of said modeling tools. A set of object classifications includes floors, walls, ceilings, windows, and doors. Alternative object classifications can also be used; for example, in the case of industrial complexes or large mechanical parts, pipes, tanks, and valves can be used to determine the scene diagram. Those skilled in the art can adapt the invention to a given scene based on typical structural and functional models used in typical scenes.

[0045] In some implementations, the method further includes suggesting alternative viewpoints based on the scene graph to obtain additional point clouds. These suggestions may be based on confidence levels associated with a set of parameters. In other words, the computer-implemented method proposes positions for a human operator to cover potential gaps in the 3D scene or maximize confidence, thereby improving the reliability of the parametric model.

[0046] In some implementations, the first set of parameters corresponds to a bounded, smooth surface, particularly a rectangle. Those skilled in the art will understand that rectangular shapes are very common for buildings. However, when the scenario differs from a building or construction site, such as machinery or a factory, the system can also propose other types of schematic geometry.

[0047] The present invention also relates to a computer program product. This computer program is stored on a machine-readable medium or embodied by electromagnetic waves. The computer program includes program code segments. The computer program (or particularly the program code segments) has computer-executable instructions for performing embodiments of a computer-implemented method. Those skilled in the art will understand that the computer program product can be implemented as a "standalone" computer program or as a specific extension or sub-function of a more general computer program.

[0048] The computer program product of the present invention specifically includes code segments for performing the following operations: (a) accessing input data representing a scene, particularly accessing a set of point clouds; (b) based on the input data, providing the set of arrays including a first array associated with a first viewpoint and a second array associated with a second viewpoint; (c) providing a first set of image patches by processing the first array individually; (d) providing a second set of image patches by processing the second array individually; (e) identifying a first block in the first set of image patches and a second block in the second set of image patches by jointly processing the first and second arrays, the first and second blocks corresponding to a first object; and (f) deriving a first set of parameters based on the identified first and second blocks and the first and second viewpoints. Those skilled in the art will understand that the features of the computer program product and the computer-implemented method correspond to each other, and further descriptions of the features of the method can be applied accordingly to the features of the computer program product.

[0049] The present invention also relates to a reality capture system, which includes a point cloud recording device and a computing unit.

[0050] The point cloud recording device includes a scanning beam source configured to emit scanning signals along a scanning direction. The scanning beam source can be configured to emit a periodic scanning pulse sequence. Periodicity in the context of this invention also includes quasi-periodic pulse sequences, wherein the pulse sequence is further modulated by periodic and / or random modulation signals.

[0051] The point cloud recording device also includes a detector configured to detect return signals returning from object points in the scene. Based on the emission of the scan signal and the detection of the return signal, the distance between the point cloud recording device and the object point can be derived. Specifically, the distance between the point cloud recording device and the object point is determined based on the time-of-flight of the scan and return pulses.

[0052] The point cloud recording device also includes a first scanning element configured to scan a scanning direction about an azimuth axis. Specifically, the first scanning element is implemented as a frame or support that provides azimuthal rotation relative to a static base (particularly a tripod) via a motorized device. More specifically, the first scanning element is configured to provide continuous rotation at a first rotational speed. Those skilled in the art will understand that the point cloud recording device is equipped with appropriate encoders and control elements to measure and adjust the state of the first scanning element.

[0053] The point cloud recording device also includes a second scanning element configured to scan the scanning direction about an elevation axis. Specifically, the second scanning element is implemented as a rotating mirror rotatably mounted to a frame or support and provided with an elevation angle rotation relative to the frame by a motorized device. More specifically, the second scanning element is configured to provide continuous rotation at a second rotational speed higher than the first rotational speed. Those skilled in the art will understand that the point cloud recording device is equipped with appropriate encoders and control elements to measure and adjust the state of the second scanning element.

[0054] The computing unit is configured to process data provided by a point cloud recording device as input data. The computing unit is configured to execute embodiments of the computer program product of the present invention. The computing unit specifically includes or accesses non-transitory memory. This non-transitory memory contains the computer program product in an executable manner. The computing unit also includes wired and wireless interfaces providing data transfer from the point cloud recording device and / or to a database storing raw or processed format point cloud data. Needless to say, the computing unit includes a suitable processor and operating memory to execute the computer program product of the present invention.

[0055] The computing unit may be a field computer, particularly a tablet computer, associated with the point cloud recording device. Attached Figure Description

[0056] By way of example only, specific embodiments of the present invention will be described more fully below with reference to the accompanying drawings, wherein:

[0057] Figure 1 The point cloud recording device during scene scanning is illustrated schematically.

[0058] Figure 2 Schematic illustration of the corresponding Figure 1 Scene diagram of the scene;

[0059] Figure 3 The illustration shows the basis Figure 1 An array of point clouds in the scene is presented as a quasi-image;

[0060] Figure 4 The flowchart illustrates some key steps of the implementation of the method of the present invention.

[0061] Figure 5 The flowchart illustrates some key steps in the implementation of user feedback. Detailed Implementation

[0062] Figure 1A point cloud recording device 1, implemented as a laser scanner, is schematically shown during point cloud recording in scene 2. The depicted point cloud recording device 1 includes a first scanning element 112, depicted as a frame rotatable relative to a tripod base 100. The first scanning element 112 is configured to scan a scanning direction 131 about an azimuth axis 110 at a first rotational speed 111 (particularly a constant first rotational speed). The point cloud recording device 1 includes a second scanning element 122, depicted as a rotating mirror. The second scanning element 122 is configured to scan the scanning direction 131 about an elevation axis 120 at a second rotational speed 121 (particularly a constant second rotational speed 121). Typically, the second rotational speed 121 is much faster than the first rotational speed 111. Preferably, a ratio of rotational speeds 111 and 121 is provided to enable the recording of substantially isotropic point clouds. Furthermore, providing a higher rotational speed for the lighter second scanning element 122 is easier to achieve than providing it for the larger first scanning element 112. The point cloud recording device 1 includes a scanning beam source 141 configured to emit scanning signals along the scanning direction 131, a detector 142 configured to detect return signals returned from object points 132 in scene 2, and corresponding signal processing and control elements 143. The intersection of the azimuth axis 110 and the elevation axis 120 can be considered as the (first) viewpoint 301 in the sense of this invention.

[0063] Scene 2 (depicted as the interior of a building under construction) comprises a set of real-world objects 201-203, 205, 221, 252, 291-293. A typical task is to determine the floor plan corresponding to Scene 2, such as checking for discrepancies with architectural drawings. To do this, the layout of some objects in Scene 2 must be defined, such as walls 202, 203, 205, floor 201, door 252, or window 221. Other objects (such as temporarily stored building materials 291, construction markings 292 on wall 201, or railings 293) represent distractions. Human surveyors intuitively understand these principles. However, actually identifying and parameterizing important objects in a noisy point cloud, especially on a low-performance field computer, is a challenging task requiring experience.

[0064] Figure 2 Schematic illustration of the corresponding Figure 1 Scene diagram 20 depicts the scene. The same reference numerals refer to the same... Figure 1The same real-world objects are shown in the figure, while objects not explicitly shown in the figure are depicted with a gray background. Screen Figure 20 illustrates the spatial constraints and functional relationships between real-world objects 201-206, 221, 251, 252, 293. For example, a parameter set 702 containing information about the spatial extent and location of the corresponding wall element 202 is also depicted in Figure 20. Those skilled in the art will understand that similar parameter sets are associated with all real-world objects 201-206, 221, 251, 252, 293. For transparency, the depiction of additional parameter sets has been omitted.

[0065] The depicted scene diagram 20 provides an understanding of the scene and is constructed hierarchically. Floor 201, wall elements 202-205, and ceiling 206 are primary elements. Windows 221, 251, door 252, and railing 293 are secondary elements. Solid lines represent constraints between elements. In other words, a change in the parameter set 702 associated with wall element 202 may result in a corresponding change in the parameter sets associated with floor 201, ceiling 206, adjacent wall elements 203, 205, and window 221. Arrows indicate dependencies or unilateral relationships. In other words, the parameter set of window 221 must be chosen such that window 221 is located on wall element 202.

[0066] The railing 293 is marked with a dashed boundary, which indicates that it is unknown whether the object actually belongs to the scene, or in other words, whether it should be considered in the derivation of the scene diagram.

[0067] This scene diagram 20 can be provided in advance or by the model, taking into account the scanning task. For example, during the derivation of the floor plan of a furnished room, it can be assumed in advance that there is a floor 201, a ceiling 206, and multiple wall segments 202-205 connecting the two. The preset set of objects can be parameterized, and further objects are added in a cumulative and incremental manner.

[0068] Figure 3 It shows the corresponding Figure 1 A visualization of the first array 411 of the scene shown. The first array 411 has rows with first image coordinate values ​​c1 associated with first scan coordinates (i.e., azimuth coordinates of the point cloud recording device). The first array 411 has columns with second image coordinate values ​​c2 associated with second scan coordinates (i.e., elevation coordinates of the point cloud recording device). For example, the image coordinates c1, c2 are linearly proportional to the spherical coordinates. Those skilled in the art will understand that while this representation allows for easy transfer of point cloud data, other representations are possible. In particular, the shown choice emphasizes the areas near the nadir and zenith, while most important objects are located near the central plane of the laser scanner. However, this drawback can also be compensated for with proper training of the neural network.

[0069] The first array 411 has intensity values ​​representing measurement data associated with corresponding first and second scan coordinates. For transparency, the intensity values ​​in the depicted embodiment correspond to the surface reflectivity, which can be derived from the return signal intensity normalized according to the object point distance. As a result, it is easier to distinguish different objects 201-206, 221, 251, 252, 291-293. Furthermore, boundary lines separating the objects are added to aid in differentiation.

[0070] Construction marker 292 produces artifacts in this representation due to its high reflectivity, while windows 221 and 251 are almost non-reflective, thus appearing as holes in the image in the first array 411. Similarly, the white-painted ceiling 206 and wall segments 202-205 are more reflective than the floor 201. Due to the way the point cloud is captured (or the rectangular shape of the building is converted to polar coordinates), the first array 411 appears as a distorted image to a human observer. However, this does not pose a particular problem for artificial intelligence, as it is trained using similar input data. Based on this training, the model can identify the first patch 511 of the top region of the first array 411 as the area associated with the ceiling 206. In this specific example, the boundary 512 associated with the ceiling 206 may also be identified. Similarly, further image patches can be identified, thus providing a first set of image patches associated with the first viewpoint. A second set of image patches can be provided during the processing of the second array associated with the second viewpoint.

[0071] Those skilled in the art will understand that the examples depicted were chosen for pedagogical reasons, not because of their particularly beneficial properties. Rather, providing intensity values ​​as vectors (e.g., incorporating distance values, surface normals, and / or gradients, color information) is advantageous for robustness in identifying image patches and associating them with real-world objects. Since RGB processing involves three separate sets of “intensity” information, there are no fundamental obstacles for image processing algorithms to this vector-based approach.

[0072] Figure 4A flowchart illustrates an implementation of method 4 for deriving the first parameter set. Flow lines / commands are depicted with thick lines, while data lines are depicted with dashed lines. For transparency, some flow lines and / or data lines may not be shown in the schematic flowchart. Furthermore, this and any further flowcharts focus on certain aspects of the invention; that is, information about other aspects can be abstracted. In the first step, access 401 represents input data 400 for the scene. Input data 400 is based on a set of point clouds obtained from known viewpoints 301, 302, specifically implemented as or equivalent to said point clouds. In subsequent steps, arrays 41 and corresponding viewpoints 301, 302 are provided based on input data 400. Subsequently, a first set of image blocks 51 is provided based on a first array 411, and a second set of image blocks 52 is provided based on a second array 412. Each image block in sets 51, 52 is associated with a single real-world object. Preferably, the image blocks represent almost all of the real-world object, but image blocks corresponding to one or more fundamental features of the real-world object are equally applicable. Subsequently, the first block 511 in the first group of image blocks 51 and the second block 521 in the second group of image blocks 52 are identified. The first image block 511 and the second image block 521 correspond to the first object. Finally, the first parameter set 71 is derived based on the identified first block 511 and second block 521, as well as the first viewpoint 301 and the second viewpoint 302.

[0073] Figure 5 The flowchart illustrates an implementation of manual adjustment 5. In the first step, prediction 710 stores numerical and semantic information in the first parameter set 71, such as... Figure 4 As shown. Those skilled in the art will understand that this step may be performed through multiple iterations, and details are omitted for transparency reasons. The prediction 710 of the first parameter set 71 also includes the prediction 720 of the associated confidence value 72. The prediction 720 of the confidence value 72 is depicted as a separate step, again for transparency reasons. In practical implementations, the prediction 710 of numerical and semantic information and the confidence value 72 may be intertwined.

[0074] When the (preliminary) first parameter set 71 is calculated, it is displayed 730 for user verification. Although not displayed, a graphical representation of one or more (partial) point clouds of the objects based on the first parameter set may also be displayed in the same frame of reference. System access 740 User input 74. When the user confirms the first parameter set 71, it is retained. Alternatively, the first parameter set 71 can be adjusted 751 based on user input. Adjustment 751 can be performed in a manner that predicts a new first parameter set 71. User input 74 and the first parameter set 71 before and after adjustment 751 can be provided to the training database.

[0075] Although the invention has been described above with reference to specific embodiments, it must be understood that numerous modifications and combinations of different features of the embodiments can be made. All such modifications are within the scope of the appended claims.

Claims

1. A computer-implemented method (4) for generating (710) a first parameter set (71) of a parameterized model representing a first object in a scene (2), wherein - The scene (2) contains a set of real-world objects (201-206, 221, 251, 252, 291-293). - The first parameter set (71) represents information about the spatial extent and location of the first object in a reference frame associated with the scene (2). The method (4) includes: - Access (401) represents the input data (400) of the scenario (2), the input data (400) being based on - The first point cloud associated with the first viewpoint (301) and acquired by the point cloud recording device (1), and - A second point cloud associated with a second viewpoint (302) and acquired by the point cloud recording device (1), wherein the first viewpoint (301) and the second viewpoint (302) are different. - Based on the input data (400), a set of arrays (41) is provided (410), the set of arrays including a first array (411) associated with the first viewpoint (301) and a second array (412) associated with the second viewpoint (302), wherein each array (411, 412) includes: - A row containing the first image coordinate value (c1) associated with the first scan coordinate. - A column containing the second image coordinate value (c2) associated with the second scan coordinate, and - Represents the intensity value of the measurement data associated with the corresponding first and second scan coordinates. - A first set of image blocks (510) is provided by processing the first array (411) individually using an image processing algorithm, wherein each image block of the first set of image blocks (51) is associated with a single real-world object (201-206, 221, 251, 252, 291-293) in the scene (2). - A second set of image blocks (520) is provided by processing the second array (412) individually using the image processing algorithm, wherein each image block of the second set of image blocks (52) is associated with a single real-world object (201-206, 221, 251, 252, 291-293) in the scene (2). - By jointly processing the first array (411) and the second array (412), the first block (511) in the first group of image blocks (51) and the second block (521) in the second group of image blocks (52) are identified (61), the first block (511) and the second block (521) corresponding to the first object, and - Based on the identified first block (511) and second block (521) and the first viewpoint (301) and the second viewpoint (302), derive (710) the first parameter set (71).

2. The method (4) according to claim 1, further comprising manual adjustment (5), wherein, The manual adjustment (5) includes: - Display (730) the first parameter set (71). - Access (740) user input data (74), and - Confirm the first parameter set (71) based on the user input data (74) or adjust (751) the first parameter set (71) according to the user input data (74). In particular, the first parameter set (71) includes a confidence prediction associated with a confidence value (72), which is associated with numerical or semantic information stored in the first parameter set (71).

3. The method (4) according to claim 2, wherein, The manual adjustment (5) also includes: - Display at least a portion of the first point cloud and / or the second point cloud, and - Displays a parametric model of the first object with reference to the displayed point cloud.

4. The method (4) according to any one of the preceding claims, wherein, The measurement data includes: - Data related to the distance from the viewpoints (301, 302), and / or - Data related to the returned beam signal level, and / or - Data related to surface color, and / or - The normal vector associated with the surface defined by multiple points in the point cloud. In particular, the intensity value is based on a vector containing surface color-related data, particularly RGB data, and data related to the distance from the viewpoint (301, 302).

5. The method (4) according to any one of the preceding claims, wherein - The point cloud recording device (1) is implemented as a laser scanner, which has a first scanning element (112) and a second scanning element (122). The first scanning element defines the first scanning coordinates as azimuth coordinates, and the second scanning element defines the second scanning coordinates as elevation coordinates. The laser scanner is specifically configured to acquire full dome scans. - The first image coordinate (c1) is linearly proportional to the azimuth coordinate, and - The second image coordinate (c2) is linearly proportional to the elevation coordinate.

6. The method according to claim 5, wherein, The input data (400) is provided as a set of point clouds (401), particularly during the scanning of the scene (2).

7. The method (4) according to any one of the preceding claims, wherein, The derivation (710) of the first parameter set (71) is based on a heuristic method, in particular a weighted combination of predictors.

8. The method (4) according to any one of the preceding claims, wherein, The set of real-world objects (201-206, 221, 251, 252, 291-293) includes a second object, and the method (4) further includes: - By jointly processing the first array (411) and the second array (412), a third block in the first group of image blocks (51) and a fourth block in the second group of image blocks (52) are identified, the third block and the fourth block corresponding to the second object, and - Based on the identified third and fourth blocks, the first viewpoint (301) and the second viewpoint (302), and especially the first parameter set (71), a second parameter set for the parameterized model of the second object is derived.

9. The method (4) according to any one of the preceding claims, wherein - The set of real-world objects (201-206, 221, 251, 252, 291-293) includes a third object, and - The input data (400) is further based on a third point cloud associated with the third viewpoint and acquired by the point cloud recording device (1). in, The method (4) further includes: - Based on the input data (400), the set of arrays (41) is provided such that the set of arrays includes a third array associated with the third viewpoint. - A third set of image patches is provided by processing the third array individually using the image processing algorithm, wherein each image patch in the third set of image patches is associated with a single real-world object (201-206, 221, 251, 252, 291-293) in the scene (2). - By jointly processing the first array (411) and the third array, the fifth block in the first group of image blocks (51) and the sixth block in the third group of image blocks are identified, the fifth block and the sixth block corresponding to the third object, and - Based on the identified fifth and sixth blocks, the first viewpoint (301), the second viewpoint (302) and the third viewpoint, and in particular the first parameter set (71), a third parameter set for the parameterized model of the third object is derived.

10. The method (4) according to claim 8 or 9, wherein - Each of the parameter sets (71, 702) includes an object classification corresponding to the real-world role of the associated real-world object (201-206, 221, 251, 252, 291-293). - The method (4) further includes: - Based on the object classification, a scene diagram (20) is derived, wherein the scene diagram (20) represents the structural and functional relationships of the real-world objects (201-206, 221, 251, 252, 291-293), particularly as hierarchical relationships, and - Adjust the parameter set (71,702) based on a set of constraints corresponding to the scene graph (20). In particular, method (4) further includes: - Assign the first block (511) and the second block (521) to the first object category corresponding to the first object. - Based on the image coordinates (c1, c2) of the first unassigned image patch relative to the first block (511) or the second block (521) and the scene graph (20), a first prediction regarding the classification of a second object is provided for the first unassigned image patch, wherein the first unassigned image patch corresponds to an unsorted real-world object. - Based on the first prediction, provide a second prediction regarding an additional set of parameters related to the unsorted real-world object, and - Based on the second prediction, identify the second unassigned image patch corresponding to the unsorted real-world object.

11. The method (4) according to claim 10, wherein - The scenario (2) is a building and / or a construction site, - The scene diagram (20) corresponds to one of the following: floor plan, construction drawing, blueprint, computer-aided design model, and building information model. - A set of object categories includes floor (201), walls (202-205), ceiling (206), windows (221,251) and doors (252).

12. The method (4) according to any one of claims 10 to 11, the method comprising suggesting an additional viewpoint based on the scene graph (20) to obtain an additional point cloud.

13. The method (4) according to any one of the preceding claims, wherein, The first parameter set (71) corresponds to a bounded smooth surface, in particular a rectangle.

14. A computer program product comprising program code, the computer program product being stored on a machine-readable medium or embodied by electromagnetic waves and comprising program code, the program code comprising program code segments and having computer-executable instructions for performing the calculation steps of the method (4) according to any one of claims 1 to 13.

15. A reality capture system, the reality capture system comprising a point cloud recording device (1) and a computing unit, wherein, The point cloud recording device (1) includes: - A scanning beam source (141) configured to emit a scanning signal along the scanning direction (131), particularly emitting a periodic scanning pulse sequence. - A detector (142) configured to detect a return signal from an object point (132) in the scene (2), wherein the distance between the point cloud recording device (1) and the object point (132) can be deduced based on the emission of the scan signal and the detection of the return signal, particularly based on the time of flight of the scan and return pulses. - A first scanning element (112), configured to scan the scanning direction (131) about an azimuth axis (110), wherein, in particular, the first scanning element (112) is configured to provide continuous rotation at a first rotational speed (111), and - A second scanning element (122) configured to scan the scanning direction (131) about an elevation axis (120), and in particular, wherein the second scanning element (122) is configured to provide continuous rotation at a second rotation speed (121) higher than the first rotation speed (111). Wherein, the computing unit - Configured to process data provided by the point cloud recording device (1) as input data (400), and - Configured to execute the computer program product as described in claim 14.

Images