Real-time visualization of the subject during scanning

Abstract

Real-time visualization of a subject during a scan. The invention relates to a method for visualizing the progress of a scanning process of a subject. The method comprises in real-time: receiving data points from a scanning device performing a scanning process of a subject; estimating a surface normal at each data point; adding the data points to a 3D grid, each data point being assigned to a cell of the grid; for each cell, computing an average data point; associating average data point information to an associated node of the grid, the information comprising at least the position of the average data point in the respective cell and an average surface normal; classifying the associated nodes of the grid into internal and external nodes based on the position and the average surface normal; for each internal node, placing a vertex at the position of the average data point associated with the internal node; generating a mesh connecting all average data points, using the internal and / or external nodes to define an inside and / or outside of the mesh; and visualizing the mesh to a user of the scanning device.

Description

Technical Field

[0001] This invention relates to a computer-implemented method that visualizes a preview of an object in real time while it is being scanned (i.e., before the scan is complete and before the final high-precision mesh is created after the scan process is finished). In this way, an initial visualization based on partial data can be provided to help the user complete the scan. Background Technology

[0002] It is known to perform 3D scanning on real-world objects or environments to capture three-dimensional data of their shape in order to generate a 3D model of the object. 3D scanners can be based on many different technologies. These include terrestrial laser scanners (TLS) and handheld scanners, such as Hexagon's Absolute Scanner AS1. Handheld laser scanners can be mounted on an absolute arm and / or tracked by a laser tracker to allow the acquisition of the absolute coordinates of the measured points.

[0003] When scanning real-world objects with this handheld laser scanner, hundreds of thousands of points can be acquired per second as the user moves the scanner across the object's surface. For example, using a line laser, points can be captured as single strips. As the scanner moves, more and more strips are acquired. These points are agglomerated by computer software for later use in geometry and tolerances (GD&T) or for reconstruction purposes.

[0004] When scanning an object, the user must ensure that the entire surface of the object is scanned. However, with existing methods, it is impossible to know exactly which parts of the object still need to be scanned before the mesh is calculated from the scan data. For example, existing methods (such as Reshaper, Polyworks, or Geomagic) first aggregate a large number of capture points before meshing them and merging them into an existing mesh. This solution results in a two-stage workflow, where the user must first scan surface patches and then examine the visualized mesh to find missing parts. Other methods use a so-called truncated symbolic distance function to generate a mesh from the scan points. For this purpose, the distance to the local surface reconstruction is recorded in a voxel grid. The mesh can be extracted from these distance values ​​using, for example, the well-known Marching Cubes method (William Lorensen and Harvey Cline, “Marching cubes: A high resolution 3D surface construction algorithm,” Computer Graphics, Vol. 21, No. 4, July 1987, pp. 163–169). However, due to the computational overhead of local surface approximation and the number of values ​​that must be adjusted for each insertion point, this approach is not suitable for real-time mesh generation that requires tens to hundreds of updates per second. Summary of the Invention

[0005] Therefore, the object of the present invention is to provide an improved method for providing visual feedback to a user of a handheld laser scanner about the scanned portion of an object.

[0006] One particular objective is to provide a method that allows for faster and more real-time, and especially instantaneous, delivery of visual feedback.

[0007] At least one of these objectives is achieved by means of the following methods and systems.

[0008] A first aspect of the present invention relates to a first computer-implemented method for visualizing the progress of a scanning process of an object. The method includes the following steps: real-time...

[0009] - Receive data points from the scanning device that performs the scanning process on the object; - Add the received data points to a 3D raster, which includes nodes and edges; - Assign each data point in the added data points to the nearest node of the raster; - For each nearest node of the grid to which data points are assigned, calculate the average data point, wherein the position of each average data point is calculated based on the positions of all points assigned to the corresponding nearest node; - Generate a grid connecting all average data points; and - Visualize the grid to the user of the scanning device.

[0010] According to some implementations of the first method, mesh generation is performed using a moving cube algorithm.

[0011] According to some embodiments of the first method, receiving data points includes receiving the surface normals of each data point. For example, the surface normals may be estimated surface normals and / or may correspond to an inverted scan direction. According to some embodiments, for each average data point, an average surface normal is calculated. According to some embodiments, the mesh is generated based on the average surface normals.

[0012] According to some implementations of the first method - The scanning process is performed continuously, and the data points are received continuously from the scanning device; - For each nearest node of the grid that has been assigned a new data point, calculate the new average data point; - The grid is continuously updated to connect all average data points, including new average data points; and - The continuously updated grid is visualized in real time.

[0013] According to some implementations of the first method, the grid is visualized to the user within one-tenth of a second after the data points are received.

[0014] According to some embodiments of the first method, the scanning device is a handheld laser scanner, such as a line scanner, and performing the scanning process includes: the user moving the scanning device across the surface of the object. According to some embodiments, receiving data points from the scanning device includes: receiving at least 100,000 data points per second.

[0015] According to some implementations of the first method, the three-dimensional grid is a sparse grid or a fine grid.

[0016] According to some implementations, the first method includes the following steps: after the scanning process is completed, calculating a 3D model of the object based on data points.

[0017] A second aspect of the invention relates to a first scanning system. The scanning system includes: a handheld laser scanner configured to scan an object by emitting a measuring beam toward the surface of the object; and a display device including a display and a computing unit. The computing unit is configured to perform a visualization process for visualizing the progress of the scanning process of the object performed by the scanning device, the visualization process including: real-time...

[0018] - Receive data points from the scanning device; - Add the received data points to a 3D grid, which includes nodes and edges; - Assign each data point in the added data points to the nearest node of the raster; - For each nearest node of the grid to which data points are assigned, calculate the average data point, wherein the position of each average data point is calculated based on the positions of all points assigned to the corresponding nearest node; - Generate a grid connecting all average data points; and - To enable the display to visualize the grid to the user of the system.

[0019] According to some implementations of the first system, the computing unit is part of the display device.

[0020] According to some implementations of the first system, the computing unit is configured to calculate a 3D model of the object based on data points after the scanning process is completed.

[0021] According to some embodiments of the first system, a handheld laser scanner is mounted to an articulated arm. According to some embodiments, a display device is mounted to either the handheld laser scanner or the articulated arm.

[0022] A third aspect of the invention relates to a first computer program product comprising program code for performing a first method. For example, the computer program product may be run by a computing unit of a first system. In some embodiments, the computer program product is stored in non-volatile memory.

[0023] A fourth aspect of the invention relates to a second computer-implemented method for visualizing the progress of a scanning process of an object. The method includes the following steps: real-time...

[0024] - Receive data points from the scanning device that performs the scanning process on the object; - Estimate the surface normal at each data point in the received data points; - Add the received data points to a 3D grid, which includes nodes and edges that form cells, and assign each data point to a cell within the grid; - For each cell having at least two assigned data points, calculate the average data points, wherein the position of each average data point is calculated based on the positions of all points assigned to the corresponding cell, and the average surface normal of each average data point is calculated based on the estimated surface normals of all points assigned to the corresponding cell. - Associate average data point information with at least one associated node, the average data point information including at least the position of the average data point in the corresponding cell and the average surface normal, the associated node being a node of the corresponding cell; - Based on the position of the average data points and the average surface normal, classify one or more associated nodes of the raster as internal nodes and classify one or more associated nodes of the raster as external nodes; - For each internal node that is connected to an external node via an edge of the grid, place the vertex at the location of the average data point associated with that internal node; - Generate a grid connecting all average data points, wherein the inner nodes and / or the outer nodes are used to define the inner and / or outer sides of the grid; and - Visualize the grid to the user of the scanning device.

[0025] According to some embodiments of the first or second method, receiving data points from a scanning device includes: continuously receiving multiple data point strips, each data point strip including multiple data points arranged along a corresponding strip, wherein the scanning device is a line scanner.

[0026] According to some implementations of the first or second method, estimating the surface normal of the data points in the first strip includes: - Perform triangulation among the data points of the first strip, the data points of the previously received strip, and the data points of the subsequently received strip, such that multiple triangles are adjacent to each data point of the first strip, and - Calculate the surface normal of each data point in the first strip as the normal sum of the normals of the triangles adjacent to that data point.

[0027] In some implementations, triangulation includes generating for each data point of the first strip: a triangle formed with the two nearest data points of the previously received strip, a triangle formed with the two nearest data points of the subsequently received strip, at least one triangle formed with adjacent data points on the first strip and data points of the previously received strip, and at least one triangle formed with adjacent data points on the first strip and data points of the subsequently received strip.

[0028] According to some implementations of the second method, generating the mesh includes using a moving cube algorithm.

[0029] According to some implementations of the second method, the average data point information includes at least one of the sum of the data points in the corresponding cell and the weight of each data point in the corresponding cell.

[0030] According to some implementations of the second method, the average data point information is associated with multiple associated nodes. For example, each node of a corresponding cell is an associated node of the average data point information of that cell.

[0031] According to some embodiments of the second method, the scanning process is performed continuously, and data points are continuously received from the scanning device. For each cell of the grid assigned a new data point, a new average data point is calculated. The grid is continuously updated to connect all average data points including the new average data point, and the continuously updated grid is visualized in real time.

[0032] According to some implementations of the second method, the grid is visualized to the user within one-tenth of a second after the data points are received.

[0033] According to some embodiments of the second method, the scanning device is a handheld laser scanner, such as a line scanner, and performing the scanning process includes: the user moving the scanning device across the surface of the object. For example, receiving data points from the scanning device includes: receiving at least 100,000 data points per second.

[0034] According to some implementations of the second method, the three-dimensional grid is a sparse grid or a fine grid.

[0035] According to some implementations, the second method includes the following steps: after the scanning process is completed, calculating a 3D model of the object based on data points.

[0036] A fifth aspect of the invention relates to a second scanning system. The scanning system includes: a handheld laser scanner configured to scan an object by emitting a measuring beam toward the surface of the object; and a display device including a display and a computing unit. The computing unit is configured to perform a visualization process for visualizing the progress of the scanning process of the object performed by the scanning device, the visualization process including: real-time...

[0037] - Receive data points from the scanning device; - Estimate the surface normal at each data point in the received data points; - Add the received data points to a 3D grid, which includes nodes and edges that form cells, and assign each data point to a cell within the grid; - For each cell with at least two data points assigned, calculate the average data points, where the position of each average data point is calculated based on the positions of all points assigned to the corresponding cell, and the average surface normal of each average data point is calculated based on the estimated surface normals of all points assigned to the corresponding cell. - Associate average data point information with at least one associated node, the average data point information including at least the position of the average data point in the corresponding cell and the average surface normal, the associated node being a node of the corresponding cell; - Based on the location of the average data points and the average surface normal, classify one or more associated nodes of the raster as internal nodes and classify one or more associated nodes of the raster as external nodes; - For each internal node connected to an external node via an edge of the grid, place the vertex at the location of the average data point associated with that internal node; - Generate a grid connecting all average data points, wherein the inner nodes and / or the outer nodes are used to define the inner and / or outer sides of the grid; and - To enable the display to visualize the grid to the user of the scanning device.

[0038] According to some implementations of the second system, the computing unit is part of the display device.

[0039] According to some implementations of the second system, the computing unit is configured to calculate the 3D model of the object based on data points, for example, after the scanning process is completed.

[0040] According to some embodiments of the second system, a handheld laser scanner is mounted to an articulated arm. For example, a display device may be mounted to the handheld laser scanner or the articulated arm.

[0041] A sixth aspect of the invention relates to a computer program product comprising program code for performing a second method. For example, the computer program product may be run by a computing unit of a second system. In some embodiments, the computer program product is stored in non-volatile memory.

[0042] The methods and systems described in this paper can also be combined with each other in various ways. For example, steps described only with respect to the first method can be integrated into the second method, and vice versa. Attached Figure Description

[0043] The invention will now be described in detail with reference to exemplary embodiments accompanied by the accompanying drawings, wherein: Figure 1 An exemplary system according to the present invention is shown when scanning an object; Figures 2a to 2b An example of a sparse grid hierarchy is shown. Figures 3a to 3f The steps of a first exemplary embodiment of the method according to the present invention are illustrated; Figure 4 This is a flowchart illustrating a first exemplary embodiment of the method according to the present invention; Figures 5a to 5d An exemplary process for estimating the surface normals of captured data points is illustrated; Figures 6a to 6f The steps of a second exemplary embodiment of the method according to the invention are illustrated; and Figure 7 This is a flowchart illustrating a second exemplary embodiment of the method according to the present invention. Detailed Implementation

[0044] Figure 1An exemplary system according to the invention is illustrated when scanning object 1. The system includes a handheld 3D laser scanner 2 that emits a measuring beam toward the surface of object 1 in the form of a laser fan 3 to measure strips of surface points as the laser fan 3 moves across the surface. Optionally, to allow for the acquisition of the absolute coordinates of the measuring points, the handheld laser scanner 2 may be mounted on an articulated arm and / or tracked by a laser tracker as it is moved by the user. The system also includes a display device 4, which includes a display 5 on which a preview image 6 can be displayed in real time to the user of the handheld laser scanner 2. Although shown here as a separate device, the display device 4 may be integrated into the laser scanner 2 or configured to be mounted on the laser scanner 2 or the articulated arm. The preview image 6 shows the portions of the object that have been scanned, allowing the user to scan the entire object 1 more quickly without gaps. A computing unit (not shown here) of the system is configured to perform the method according to the invention, which allows visualization of the preview image 6 in a manner emphasizing speed and interactivity. For example, the computing unit may be part of the display device 4.

[0045] Figure 2a and Figure 2b An exemplary hierarchy of sparse grids used in an exemplary embodiment of the method according to the invention is illustrated. This method employs an efficient approach to clustering neighboring points within an axis-aligned, regular, sparse grid hierarchy. For example, the sparse grid hierarchy can be constructed in a manner similar to that described in the OpenVDB method. For this purpose, a fine grid spacing is selected, and initial coarse cells are constructed such that the first batch of provided points are located near their centers.

[0046] The initial coarse cell contains a fixed raster and has a finite size. Users may scan points not contained in this initial coarse cell. Multiple coarse cells allow scanning large objects or objects far apart from each other. Although they correspond to cells in a regular raster, they are not stored in the regular raster but are associated with dimension indices. Given an initial coarse cell C[0,0,0] with minimum corner points (xMin, yMin, zMin) and side length hCoarse as defined above, the index of the coarse cell containing point P=(x, y, z) can be calculated as I=[floor((x-xMin) / hCoarse, floor((y-yMin) / hCoarse), floor((z-zMin) / hCoarse)]. At the coarse level, a map is used through which coarse cells can be accessed by providing such indices. Only those coarse cells containing points are created.

[0047] If a point is inserted outside the boundary of the initial coarse cell, an additional coarse cell containing that new point is created, and the algorithm continues with that coarse cell for a given point. The first coarse cell defines a three-dimensional regular structured grid, where the smallest corner point of the initial coarse cell defines the origin of the grid. Cells in this grid can be identified using a three-dimensional index. Coarse cells can be accessed using this three-dimensional index and are created only when needed.

[0048] Figure 2b The diagram shows a 2D representation of three levels of a sparse hierarchical raster, L-1, L-2, and L-3. L-3 is the base raster. Each cell of the base raster can hold a sub-raster of half its side length. This sub-raster is considered to be in the next level, L-2. If a point is inserted into a cell of the base raster L-3, it is forwarded to the sub-raster. If such a sub-raster does not exist, it is created, and the point is forwarded to the new sub-raster.

[0049] Similar to the base raster L-3, each cell in raster L-2 can contain a sub-raster. If a point is inserted into a cell of that raster, it is forwarded to the corresponding sub-raster. Similarly, if such a sub-raster does not exist, it is created, and the point is forwarded to the new sub-raster. The grid is associated with each raster in level L-2 (each corresponding to a cell in the parent level L-3). This grid is then used for rendering. After meshing is performed in the leaf raster of the top level L-1, the individual grids of all sub-rasteres are joined to the associated grid of level L-2. The grid must only be joined again when a new point is inserted into a raster in level L-2.

[0050] Each fine-grained raster cell is grouped into a structured raster containing N×N×N cells (e.g., N=8). In a sparse hierarchical raster structure with L levels, the structured rasters residing at level L-1 are the leaves of the structure. At level L-2, the structured raster is contained within sparse raster nodes of M×M×M cells (e.g., M=16). A single grid is associated with each of these sparse raster nodes at level L-2. When preparing the visualization for the next frame, all those sparse raster nodes at level L-2, containing sub-rasters whose timestamps are more recent than the last grid update, must be accessed.

[0051] Now the grid and each such sparse raster cell can be updated in parallel: - Summes the values ​​collected from the fine-grained raster cells in the associated active corner node, namely, the sum of positions, the sum of estimated normals, and the number of points contained in the cell.

[0052] - For each fine-grained cell containing both empty and non-empty corner nodes, triangulation can now be constructed between the average positions of the non-empty corner nodes. This can be done using the triangle table described in the "Moving Cube" algorithm. For example, non-empty nodes can be treated as internal nodes, and empty nodes as external nodes. Instead of placing vertices on cell edges, they can be placed at the average positions of the associated non-empty nodes stored in the inserted edge.

[0053] - By comparing the normals of the generated triangles with the average estimated normals of the corresponding non-empty nodes, triangles facing the interior of the scanned object can be eliminated quickly on the fly.

[0054] During the described mesh update process, updates performed in individual cells are completely decoupled from updates in other cells. This algorithm is therefore suitable for parallel execution. For all cells in level L-2 where the mesh has just been updated, the associated render buffer on the GPU can now be updated. Due to the locality of the described update process, typically only a small portion of the total vertex data of the visualized mesh needs to be updated. Therefore, the amount of data transferred to the GPU is small, and the update efficiency is high.

[0055] Compared to other raster-based meshing methods for point clouds, the insertion of new points is entirely local and affects only the single cell containing a given point. This allows for a much higher number of points to be processed per second, and ultimately allows for a much higher update rate for visualizations of large point sets. The latter is crucial for smooth and interactive visualizations, which in turn is essential for an efficient scanning process.

[0056] Figures 3a to 3f The steps of a first exemplary embodiment of the method according to the invention are illustrated. Although actual scanning is typically performed in 3D, the raster is shown here in 2D for clarity and simplicity.

[0057] This method allows for the instantaneous generation of a very fast yet detailed mesh-based visualization of the scanned surface while the scan is still in progress. New points can be rapidly inserted into the existing structure and are immediately taken into account in the efficient local updates of the visualized surface mesh. The method allows viewing the preview mesh from any perspective and smoothly adapts the viewing angle to the orientation of a moving laser scanner, even as new points are inserted and the mesh grows. This method allows users of 3D strip-based laser scanners to receive instantaneous feedback on the acquired surface during the scanning process (i.e., within fractions of a second). This allows users to monitor and adjust their scanning process the moment they acquire points of a specific structure of the scanned part.

[0058] Unlike other methods, the proposed method combines point aggregation and mesh generation for visualization into a single stage. Points are interpolated onto a continuously growing mesh in real time. The visual feedback provided to the user is immediate, and allows for tighter control over the scanning process.

[0059] Figure 3a An empty, fine grid 10 is shown, comprising nodes (represented by circles) that form cells and edges (represented by lines). The hollow circles shown here represent "inactive" nodes 11. Figure 3b In this process, seven scan points 20a-g are added to the fine grid 10. For each of the points 20a-g, an estimated surface normal 22a-g (e.g., scan direction) is provided, which is represented here by a short line next to each point.

[0060] To insert this batch of points into the preview visualization, for each point in points 20a-g, find the cell in fine grid 10 that geometrically contains that point. For each such fine grid cell, store the sum of the positions of the points already inserted into that cell, the sum of the estimated normals 22a-g for each inserted point 20a-g (e.g., the reverse scan direction), and the number of inserted point positions.

[0061] exist Figure 3c In this process, for each node in the raster, the average neighboring data points 21a-d and their respective surface normals 23a-d are determined. Initially, all nodes are considered inactive (inactive node 11). If the distance from a node (cell corner) to one of the inserted points 21a-d is less than or equal to half the cell diameter, then the node / corner is subsequently considered active. In each fine cell, a bit pattern can be stored, which generates the active node 12 for that cell. Figure 3b Scan points 20a-g are all close to four nodes in grid 10. Therefore, these nodes are "active" nodes 12 (represented by solid circles), while other nodes without any neighboring points remain inactive nodes 11. For all points closest to the first active node 12a, an average data point 21a and a corresponding average surface normal 23a are determined. The same operation is performed for the second active node 12b (average data point 21b, average surface normal 23b) and the third active node 12c (average data point 21c, average surface normal 23c). Since only one data point 20g is closest to the fourth active node 12d, the average data point 21d (with average surface normal 23d) is equal to that data point. The calculation of average data points 21a-d can be based on the number of previously stored point locations and the sum of the stored point locations. Similarly, the calculation of average surface normals 23a-d can be based on the number of previously stored point locations and the sum of the stored estimated normals.

[0062] Figure 3d and Figure 3e An example illustrates how to apply the moving cube algorithm. Figure 3d In this diagram, active nodes (i.e., nodes with at least one associated data point) are considered "inside". Inactive nodes are considered "outside". This information is sufficient to generate triangles using the moving cube / square rule. For visualization, the positions of the generated vertices 31a-h are visualized at the centers of the intersecting edges.

[0063] exist Figure 3e In the process, the positions of each generated vertex 31a-h are selected as the positions of the associated average points 21a-d of the interior corner points (=active nodes) of the intersecting moving cube edges. Those triangles whose normals point in the opposite direction to at least one of the average normals 23a-c associated with the corner points of the triangles are filtered out. The resulting surface connects the generated average data points 21a-d to create a mesh 35. It must be understood that although the mesh here is represented by 2D polylines, in 3D, it is a mesh of triangles 35. The mesh 35 representing the triangles of the object's surface can then be visualized to the user to indicate the progress of the scan.

[0064] Figure 3f Finally, an example is shown on how adding additional data points alters the triangular mesh, thus updating the visualization presented to the user in real time. Here, an additional measurement point 20n is added along with its estimated normal, thereby changing the position of the average neighbor point (21a), i.e., becoming the new average neighbor point 21n. Furthermore, the normal of this point is adapted (not shown here), and the point counter of node 12a is incremented from 2 to 3. Due to the change in the position of the average neighbor point 21n, the mesh 35 is locally altered; in the example shown, only the left segment is changed.

[0065] Advantageously, using the described method, adding more data points to an existing grid 10 is highly efficient. Each grid node stores the sum of all neighboring data points and the number of neighboring data points. When a new point 20n is added, the nearest node must be found, the point's position is added to the sum of positions at that node, and the neighboring data point counter is incremented by 1.

[0066] To reduce raster artifacts in the reconstructed mesh, data points can be clustered at cells. Before meshing, those clustered values ​​must be considered across all active corner nodes of the modified cells. To do this, in the first step, the values ​​in the corner nodes of each modified cell must be cleared. Then, for each cleared node, the clustered values ​​of all adjacent cells that are active at that node are added to the sum of the values ​​at that node's location. Thus, the nodes effectively average the data points from all adjacent cells. This reduces the impact of outliers and minimizes raster-related artifacts in the extracted surface mesh.

[0067] Therefore, here, the position of the newly inserted point 20n is added to the sum of the positions of the new point assigned to node 12a, the point counter of node 12a is incremented by 1, and the timestamp of the parent cell of the node's cell is updated so that it is known that the parent cell contains data that has not yet been considered in the resulting grid.

[0068] Figure 4 The flowchart illustrates a first exemplary embodiment of the computer-implemented method 100 according to the present invention.

[0069] Multiple data points are captured by scanning the object. This can be performed by a user holding a handheld laser scanner. The captured data points are received in real time (110), and as per [the relevant information]... Figure 3b As illustrated, 120 data points are added to the grid. Each added data point is assigned 130 to the nearest node of the grid. For each node that has been assigned at least two data points, an average data point is calculated 140 using the positions of all data points assigned to that node. This may include summing the values ​​from all data points assigned to the same node, such that the sum of positions, the sum of estimated normals, and the number of points are assigned to that node. A mesh connecting the average data points is generated 150 and, while the user continues scanning the object, this mesh is visualized in real-time as a preview model of the object 160 to the user. Steps 110 to 160 are repeated, with the surface continuously updated as additional data points are captured, until the scan is complete. Finally, when the scan is complete, a “real” 3D model of the scanned object 190 can be calculated.

[0070] The illustrated method can be derived from Figure 1 The system executes the method. For example, a computer program running on the system's computing unit may include program code for executing the method.

[0071] Figures 5a to 5d An exemplary process for estimating the surface normal of captured data points is illustrated. For this process, strips of data points are required, for example, as captured by a line scanner. The strips S of captured data points... nBefore inserting the sparse raster hierarchy, the surface normal can be estimated at various points within the strip. For this purpose, consider the previous strip S. n-1 And the next one with S n+1 The two points, and through (e.g., from left to right) "stitching" the strip S n This generates a triangulation between the three stripes. This allows the strip S to be... n The normal to each point in the triangle is calculated as the normalized sum of the normals of all adjacent triangles.

[0072] Figure 5a This illustrates three consecutive stripes S containing data points to be inserted into a sparse raster hierarchy. n S n-1 S n+1 Lines connect adjacent data points within each individual strip. Figure 5b It shows Figure 5a The three bands S n S n-1 S n+1 Among them, from strip S n-1 and S n Starting from the leftmost point, select strip S. n-1 and S n Choose one of two adjacent data points in S such that the newly introduced connection is shortest. Then add the normal of the corresponding triangle to S. n The estimated normal of the associated data points. Figure 5c Three stripes S are shown n S n-1 S n+1 , where, traversing S to the right n-1 and S n For each adjacent data point in S, the shorter connection is selected each time. For each introduced connection, the normal of the corresponding triangle is added to S. n The data points that are related in the middle. Figure 5d Three stripes S are shown n S n-1 S n+1 ,in, Figures 5a to 5c Steps using strip S n and S n+1 Repeat. The resulting S n The estimated surface normals in the model produce a sufficient approximation of the associated surface normals of the underlying object. The triangulation itself can be discarded later and does not need to be used to generate the final mesh.

[0073] This estimation process can optionally be used for the above regarding... Figures 3a to 3f and Figure 4 The described method and the following about Figures 6a to 6f and Figure 7 The method described.

[0074] Figures 6a to 6f The steps of a second exemplary embodiment of the method according to the invention are illustrated. Again, for clarity and simplicity, the grid is shown here in 2D. The surface normals of individual data points are estimated, particularly before the point is added to the grid. Specifically, as... Figures 5a to 5d The initial estimate is performed as illustrated.

[0075] exist Figure 6a In this process, six scan points 20a-f are added to a fine grid 10. The fine grid 10 includes nodes 11 (represented by circles) that form cells and edges (represented by lines). For each point in the grid 20a-f, estimated surface normals 22a-f are provided, indicated here by short lines next to the corresponding points.

[0076] exist Figure 6b In this approach, the average of the positions 21a-c and the average of the normals 23a-c for each data point are maintained in each cell by storing the sums of the positions and the sums of the normals together with the sums of the weights. Therefore, new data points can be inserted simply by adding their positions, normals, and weights to the corresponding sums. For example, the weights of the data points can be related to confidence values.

[0077] exist Figure 6c In the process of "Extracting the Raster," for each node, the average position 24a-b and the sum of normals 25a-b of the cells adjacent to that node are calculated. For each non-empty node, it can now be determined whether the node is located inside or outside the surface being reconstructed. This is achieved by evaluating the plane equation based on the average position 24a-b and the sum of normals 25a-b, as well as the average position 21a-c and the average normal 23a-c stored in the node. If a node, according to the regular raster pattern, is located in front of the plane, it is considered to be outside. If it is located behind the plane, it is considered to be inside.

[0078] exist Figure 6d In this process, all grid nodes are classified as either "inner" nodes 14 (represented here by squares) or "outer" nodes 15 (represented here by triangles) by comparing the positions of all grid nodes with the plane passing through the associated average position with the average normal. If a cell contains both empty and non-empty nodes, the empty node can be classified as either "inner" node 14 or "outer" node 15 by comparing its regular grid position with the plane stored in the adjacent non-empty nodes.

[0079] exist Figure 6eIn the process, for each cell, the position of unclassified corner nodes is evaluated relative to their position on a plane stored in the cell's other corner points, classifying them as either "inner" nodes 14 (squares) or "outer" nodes 15 (triangles). Average data points 21a-c and average positions 24a-b create a grid 35. It must be understood that while this grid is represented by 2D polylines, in 3D it is a triangular grid 35. This triangular grid 35, representing the surface of an object, can then be visualized to the user to indicate the progress of the scan.

[0080] For each fine-grained cell including both "inner" corner node 14 and "outer" corner node 15, a triangulation between the average positions at the inner corners can be constructed. For this, a "triangulation cube" table as described by Lorensen et al. in "Moving Cubes" can be used. Alternatively, instead of placing the triangle vertices on the edges of the extraction grid as performed in a regular moving cube, the vertices of the generated triangles can be placed at the associated average positions of the associated "inner" nodes on the edges of each cell connecting the "inner" and "outer" nodes. The moving cube rules can then be applied. This approach significantly reduces the number of triangles generated.

[0081] Figure 6f Finally, an example is shown how adding additional data points changes the triangular mesh, thus updating the visualization presented to the user in real time. Here, an additional measurement point 20n with its estimated normal is added, which causes the average data point 21a and the corresponding average position 24a to change position, becoming the new average neighbor point 21n and the new average position 24n. Due to the change in the position of the average neighbor point 21n, the mesh 35 is locally altered; in the example shown, only the left segment is changed.

[0082] Optionally, to improve the smoothness and visual fidelity of the generated mesh with minimal performance overhead, the edges connecting two nodes of the generated triangulation can be refined on-the-fly (e.g., using cubic Hermite spline interpolation) where the angle between the average normals of the two nodes is greater than a threshold. These splits can be performed individually in each cell before inserting the generated triangles into the larger set of triangles at the parent cell.

[0083] Advantageously, using the implementation of the described method, adding more data points to an existing raster 10 is highly efficient. The sum of all neighboring data points and the number of neighboring data points are stored in each raster cell. When a new point 20n is added, the surrounding cell must be found, the position of the point is added to the sum of positions at that cell, and the data point counter is incremented by 1. To update the mesh, the outlined process is then repeated only for those cells that received the new data point and those cells that share raster nodes with the changed cell. Extending this algorithm to arbitrary point clouds is straightforward. While stripe structures allow for efficient normal estimation and local mesh updates, the underlying technique can be used for arbitrary point clouds with estimated normals.

[0084] One possible scenario for this second implementation is its application to structured light scanning. Here, images of the object are captured from multiple different viewpoints. For each image, a point cloud is generated. If no normals are provided for a given point cloud, these normals can be estimated later, for example, using 2D Delaunay triangulation of the projected point set and inferring the normals of the points from the associated triangles. Points with associated estimated normals can then be inserted into a sparse grid structure, and the mesh can be updated on an in real-time, as described above with respect to this implementation.

[0085] In another possible extension of the proposed method, one could consider estimating the dot product between the normal and the scan direction to derive a scalar value representing the confidence or degree of certainty at a given data point. The influence of each data point can be scaled by its confidence or degree of certainty rating when averaging the position and normal over a sparse raster structure. Instead of simply counting the number of points in a cell, the weights of each data point can be summed, and the average can be calculated based on the summed weights. Therefore, points scanned on the surface the scanner is pointing at will receive a higher weight than those accidentally captured when the scanner reaches the edge of an object. These accidentally scanned points typically have a higher noise ratio and thus receive a lower weight.

[0086] Figure 7 The flowchart illustrates a second exemplary embodiment of the computer-implemented method 100 according to the present invention.

[0087] Multiple data points are captured by scanning the object. This can be performed by a user holding a handheld laser scanner. The captured data points are received in real time, and, for example, as per... Figures 5a to 5d As described, the surface normal at each of the 115 received points is estimated. The data points and their estimated surface normals are added to the raster (e.g., as shown in the image). Figure 6a (As shown).

[0088] By storing the sum of positions and the sum of normals together with the sum of weights, the average of the positions and the average of the normals for 125 data points are obtained in each raster cell. This allows new data points to be added based on their positions, their estimated normals, and optional weights.

[0089] For each node adjacent to a cell containing data points (“extracted raster”), 142 calculate the average position and normal sum of the cells adjacent to that node. These average position and normal sums are associated with that node (“associated node”). 144 Other nodes remain empty. For each associated (i.e., non-empty) node, it is now possible to determine whether the node is inside or outside by evaluating the plane equation based on the average position and average normal stored in the node. If the node's position according to the regular raster pattern is in front of the plane, the node is classified as an outside node. 146 If it is behind the plane, the node is classified as an inside node.

[0090] For each internal node connected to an external node via an edge of the grid, the vertex is placed at the position of the average data point associated with that internal node. Therefore, for each cell including both internal and external nodes, a triangulation between the average positions of the interior corner points can be constructed. This is very similar to a traditional moving cube. However, unlike in a moving cube where triangle vertices are placed on the edges of the grid, the vertices of the generated triangles are placed at the associated average position of the associated internal node on each cell edge connecting the internal and external nodes.

[0091] While the user continues scanning the object, a mesh connecting the average data points is generated (150) and visualized in real-time as a preview model of the object (160) for the user. Steps 110 to 160 are repeated, with the surface continuously updated as additional data points are captured, until the scan is complete. Finally, when the scan is complete, a “real” 3D model of the scanned object (190) can be calculated.

[0092] The illustrated method can be derived from Figure 1 The system executes the method. For example, a computer program running on the system's computing unit may include program code for executing the method.

[0093] Although the invention has been illustrated above with reference to some preferred embodiments, it should be understood that many modifications and combinations of different features of the embodiments can be made. All such modifications are within the scope of the appended claims.

Claims

1. A computer-implemented method (100) for visualizing the progress of a scanning process of an object (1), the method comprising the steps of: Real-time - Receive (110) data points (20) from the scanning device (2) that performs the scanning process of the object (1); - Estimate the surface normal (22) at each data point (20) in the data points (20) received by (115); - Add (120) the received data points to a three-dimensional grid (10), the three-dimensional grid (10) including nodes (11, 12) forming cells and edges, each data point being assigned to a cell in the grid; - For each cell having at least two data points assigned, calculate (142) the average data point (21), wherein the position of each average data point (21) is calculated based on the position of all points assigned to the corresponding cell, and the average surface normal (23) of each average data point (21) is calculated based on the estimated surface normal (22) of all points assigned to the corresponding cell. - Associate the average data point information with at least one associated node (144), the average data point information including at least the position of the average data point (21) in the corresponding cell and the average surface normal (23), the associated node being a node of the corresponding cell; - Based on the location of the average data point (21) and the average surface normal (23), one or more associated nodes of the grid are classified (146) as internal nodes (14), and one or more associated nodes of the grid are classified (146) as external nodes (15). - For each internal node (14) connected to an external node (15) by the edge of the grid, the vertex is placed (148) at the location of the average data point (21) associated with that internal node; - Generate (150) a grid (35) connecting all average data points (21), wherein the inner and / or outer sides of the grid are defined using the inner nodes (14) and / or the outer nodes (15); and - Visualize (160) the grid (35) to the user of the scanning device (2).

2. The method (100) according to claim 1, wherein, Receiving (110) data points (20) from the scanning device (2) includes: continuously receiving multiple data point stripes (S n S n+1 S n-1 Each data point strip includes multiple data points (20) arranged along the corresponding strip, and in particular, the scanning device (2) is a line scanner.

3. The method (100) according to claim 2, wherein, Estimate (115) the first band (S) n The surface normal (22) of the data point (20) includes: - In the first band (S n The data points (20) and previously received stripes (S) n-1 ) data points and subsequently received stripes (S n+1 Triangulation is performed between the data points of ) such that multiple triangles are connected to the first strip (S) n The data points (20) are adjacent to each other, and - The first band (S) n The surface normal (22) of each data point (20) in the equation is calculated as the normal sum of the normals of the triangles adjacent to that data point. Specifically, the triangulation includes: the first strip (S) n The data points (20) of each data point are generated: - with the previously received strip (S) n-1 The triangle formed by the two nearest data points; - with the subsequently received strip (S) n+1 The triangle formed by the two nearest data points; - with the first band (S) n Adjacent data points on the ) and the previously received strip (S) n-1 At least one triangle formed by the data points; and - with the first band (S) n Adjacent data points on the ) and the subsequently received strips (S n+1 At least one triangle formed by the data points of ).

4. The method (100) according to any one of the preceding claims, wherein, Generating (150) meshes (35) involves using the moving cube algorithm.

5. The method (100) according to any one of the preceding claims, wherein, - The average data point information includes at least one of the sum of the data points in the corresponding cell and the weight of each data point in the corresponding cell; and / or - The average data point information is associated (144) with multiple associated nodes, wherein each node of the corresponding cell is an associated node of the average data point information of the cell.

6. The method (100) according to any one of the preceding claims, wherein, - The scanning process is performed continuously, and the data points (20) are continuously received (110) from the scanning device (2). - For each cell of the grid (10) that has been assigned a new data point (20n), calculate (145) the new average data point (21n). - The grid (35) is continuously updated to connect all average data points (21) including the new average data point (21n); and - The continuously updated grid (35) is visualized in real time (160).

7. The method (100) according to any one of the preceding claims, wherein, Within one-tenth of a second after receiving (110) the data point (20), the grid (35) is visualized (160) to the user.

8. The method (100) according to any one of the preceding claims, wherein, - The scanning device (2) is a handheld laser scanner, particularly a line scanner, and - Performing the scanning process includes: the user moving the scanning device (2) on the surface of the object (1), Specifically, receiving (110) data points (20) from the scanning device (2) includes receiving at least 100,000 data points (20) per second.

9. The method (100) according to any one of the preceding claims, wherein, The three-dimensional grid (10) is either a sparse grid or a fine grid.

10. The method (100) according to any one of the preceding claims, the method (100) comprising the following steps: After the scanning process is completed, a 3D model of the object (1) is calculated (190) based on the data points (20).

11. A scanning system, the scanning system comprising: - A handheld laser scanner (2) configured to scan the object (1) by emitting a measuring beam (3) toward the surface of the object (1). - Display device (4), the display device (4) including display (5), and - Computing unit, Its features The computing unit is configured to execute a visualization process for visualizing the progress of the scanning process of the object (1) performed by the scanning device (2), the visualization process including: real-time... - Receive (110) data points (20) from the scanning device (2); - Estimate the surface normal (22) at each data point (20) in the data points (20) received by (115); - Add (120) the received data points to a three-dimensional grid (10), the three-dimensional grid (10) including nodes (11, 12) forming cells and edges, each data point being assigned to a cell in the grid; - For each cell having at least two data points assigned, calculate (142) the average data point (21), wherein the position of each average data point (21) is calculated based on the position of all points assigned to the corresponding cell, and the average surface normal (23) of each average data point (21) is calculated based on the estimated surface normal (22) of all points assigned to the corresponding cell. - Associate the average data point information with at least one associated node (144), the average data point information including at least the position of the average data point (21) in the corresponding cell and the average surface normal (23), the associated node being a node of the corresponding cell; - Based on the location of the average data point (21) and the average surface normal (23), one or more associated nodes of the grid are classified (146) as internal nodes (14), and one or more associated nodes of the grid are classified (146) as external nodes (15). - For each internal node (14) connected to an external node (15) by the edge of the grid, the vertex is placed (148) at the location of the average data point (21) associated with that internal node; - Generate (150) a grid (35) connecting all average data points (21), wherein the inner and / or outer sides of the grid are defined using the inner nodes (14) and / or the outer nodes (15); and - To enable the display (5) to visualize (160) the grid (35) to the user of the scanning device (2).

12. The scanning system according to claim 11, wherein, The computing unit is part of the display device (4).

13. The scanning system according to claim 11 or claim 12, wherein, The computing unit is configured to calculate (190) a 3D model of the object (1) based on the data points (20), particularly to calculate (190) a 3D model of the object (1) after the scanning process is completed.

14. The scanning system according to any one of claims 11 to 13, wherein, The handheld laser scanner (2) is mounted on the articulated arm, particularly in which, The display device (4) is mounted on the handheld laser scanner (2) or the articulated arm.

15. A computer program product comprising program code for performing the method (100) according to any one of claims 1 to 10.

16. A computer-implemented method (100) for visualizing the progress of a scanning process of an object (1), the method comprising the steps of: Real-time - Receive (110) data points (20) from the scanning device (2) that performs the scanning process of the object (1); - Add (120) the received data points to a three-dimensional grid (10), the three-dimensional grid (10) including nodes (11, 12) and edges; - Assign (130) each of the added data points to the nearest node (12) of the raster (10); - For each nearest node (12) of the grid (10) to which data points are assigned, calculate (140) the average data point (21), wherein the position of each average data point (21) is calculated based on the positions of all points assigned to the corresponding nearest node (12); - Generate (150) a grid (35) connecting all average data points (21); and - Visualize (160) the grid (35) to the user of the scanning device (2).

17. The method (100) according to claim 16, wherein, The generation of (150) meshes (35) is performed using the moving cube algorithm.

18. The method (100) according to claim 16 or claim 17, wherein, Receiving (110) data points (20) includes receiving the surface normal (22) of each data point, wherein the surface normal (22) is an estimated surface normal and / or corresponds to the reverse scanning direction.

19. The method (100) according to claim 18, wherein, For each average data point (21), calculate the average surface normal (23).

20. The method (100) according to claim 19, wherein, A (150) mesh (35) is generated based on the average surface normal (23).

21. The method (100) according to any one of claims 16 to 20, wherein, - The scanning process is performed continuously, and the data points (20) are continuously received (110) from the scanning device (2). - For each nearest node (12) of the grid (10) that has been assigned a new data point (20n), calculate the new average data point (21n). - The grid (35) is continuously updated to connect all average data points (21) including the new average data point (21n); and - The continuously updated grid (35) is visualized in real time (160).

22. The method (100) according to any one of claims 16 to 21, wherein, Within one-tenth of a second after receiving (110) the data point (20), the grid (35) is visualized (160) to the user.

23. The method (100) according to any one of claims 16 to 22, wherein, - The scanning device (2) is a handheld laser scanner, particularly a line scanner, and - Performing the scanning process includes: the user moving the scanning device (2) on the surface of the object (1), Specifically, receiving (110) data points (20) from the scanning device (2) includes receiving at least 100,000 data points (20) per second.

24. The method (100) according to any one of claims 16 to 23, wherein, The three-dimensional grid (10) is either a sparse grid or a fine grid.

25. The method (100) according to any one of claims 16 to 24, the method comprising the following steps: After the scanning process is completed, a 3D model of the object (1) is calculated (190) based on the data points (20).

26. A scanning system, the scanning system comprising: - A handheld laser scanner (2) configured to scan the object (1) by emitting a measuring beam (3) toward the surface of the object (1). - Display device (4), the display device (4) including display (5), and - Computing unit, Its features are, The computing unit is configured to execute a visualization process for visualizing the progress of a scanning process of the object (1) performed by the scanning device (2), the visualization process including: real-time... - Receive (110) data points (20) from the scanning device (2); - Add (120) the received data points to a three-dimensional grid (10), the three-dimensional grid (10) including nodes (11, 12) and edges; - Assign (130) each of the added data points to the nearest node (12) of the raster (10); - For each nearest node (12) of the grid (10) to which data points are assigned, calculate (140) the average data point (21), wherein the position of each average data point (21) is calculated based on the positions of all points assigned to the corresponding nearest node (12); - Generate (150) a grid (35) connecting all average data points (21); and - The display (5) visualizes (160) the grid (35) to the user of the system.

27. The scanning system according to claim 26, wherein, The computing unit is part of the display device (4).

28. The scanning system according to claim 26 or claim 27, wherein, The computing unit is configured to calculate (190) a 3D model of the object (1) based on the data points (20) after the scanning process is completed.

29. The scanning system according to any one of claims 26 to 28, wherein, The handheld laser scanner (2) is mounted on the articulated arm, particularly in which, The display device (4) is mounted on the handheld laser scanner (2) or the articulated arm.

30. A computer program product comprising program code for performing the method (100) according to any one of claims 16 to 25.

Images