Occlusion culling techniques for point cloud visualization

Abstract

The invention relates to a shielding elimination technology for point cloud visualization. The present invention relates to a method for visualizing a point cloud on a display of a computing device, where the visualizing comprises changing an observation point onto the point cloud and efficiently processing changes of points of the point cloud that are differently masked for different observation points. The method includes an occlusion culling test using a depth buffer for sampling a first depth from a previous observation point associated with a previous frame toward a test point to determine whether the test point is visible or occluded when viewed from the previous observation point. If it is determined that the test point is occluded when viewed from the previous observation point, a heuristic test is performed as part of an occlusion culling test to determine whether a change in the observation point since the previous frame may have revealed that the previous coverage point is visible from the new observation point.

Description

Technical Field

[0001] The present invention relates to a method for visualizing a point cloud on a display of a computing device, wherein the visualization includes changing observed points onto the point cloud and effectively processing changes of points in the point cloud that are differently occluded for different observed points. Background Art

[0002] Unwanted overdraw of points in a point cloud occurs when multiple points positioned substantially along the same viewing axis but at different depths from an observed point are visualized. During visualization, such points (also referred to as on-screen points) fall into the same pixel or adjacent pixels. So-called occlusion culling is the industry standard way to mitigate overdraw, e.g., to prevent unnecessary work of a graphics card from updating the same pixel or pixel region multiple times, e.g., to avoid memory contention that would slow down rendering.

[0003] Occlusion culling works by performing an occlusion test before rendering an object to the screen to determine whether the object will be occluded by other geometries in the scene. Objects that are considered occluded are skipped from rendering. When the observed point is changed, the occlusion state can change, i.e., a previous covered point when viewing the point cloud from a previous observed point becomes visible from the new observed point, and a point that was previously visible becomes covered when the observed point is changed. Changing the observed point is also commonly referred to as changing the frame, where each frame represents the point cloud seen from the corresponding observed point.

[0004] There are many methods for determining whether a point is occluded. As an example, the prior art involves first rendering all objects that were visible in the last frame and generating a hierarchical depth buffer from the resulting depth buffer. A hierarchical depth buffer is a depth texture with multiple levels of progressive texture mapping (mip maps), where the maximum depth of any sub-pixel is used to generate each mip level of the progressive texture mapping, so that the occlusion test is conservative. After generating the hierarchical depth buffer, an occlusion test is performed on all objects against the hierarchical depth buffer, and any objects that are found to be visible but have not yet been rendered are now rendered. To test whether an object is occluded, the hierarchical depth map is sampled at the location of the object at the mip level, where each texel is large enough to cover the entire object.

[0005] The typical occlusion culling techniques as described above have several drawbacks when processing / visualizing point clouds. Since point clouds tend to have many gaps, they create poor occluders and rarely fully occlude objects behind them. Additionally, while triangle mesh geometries typically consist of individual objects that have well-fitting bounding boxes that can be individually occlusion-tested, point clouds are typically provided with a so-called LOD (level-of-detail) structure where points are segmented into predefined volumes. For example, European Patent Application No. EP23216367.5, filed by the same applicant, discloses a method that includes performing level-of-detail selection on multiple index data sets that are not necessarily aligned with each other. Grouping points into predefined volumes will typically include multi-segment disconnected objects where the combined bounding box contains a large amount of empty space, making it less likely that they will be fully occluded. Summary of the Invention

[0006] Accordingly, an object of the present invention is to provide an improved method for visualizing point clouds.

[0007] Another object is to provide a more efficient visualization of point clouds while maintaining a visually correct view of the scene when represented by the current frame of the point cloud.

[0008] The present invention relates to a computer-implemented method for visualizing different frames of a point cloud using a computing device with limited memory, where each of the different frames represents the point cloud as seen from a corresponding observation point. The method includes an occlusion culling test of the points in the point cloud to determine which points of the point cloud should be culled for the visualization frame. Hereinafter, the points in the point cloud that are tested by the occlusion culling test for one of the frames are referred to as test points, and one of the frames is referred to as a test frame.

[0009] The occlusion culling test includes using a depth buffer that provides depth information from a previous frame (the frame before the test frame when the observation point is changed to the point cloud). For example, the depth information provides information about the distance (also referred to as depth) along a line from the observation point of the previous frame to a point in the point cloud, where the distance indicates the location where the geometry is first encountered that blocks the view from the observation point to the corresponding point in the point cloud further along the corresponding line. For a test point, the occlusion culling test includes sampling a first depth from the previous observation point associated with the previous frame towards the test point using the depth buffer to determine whether the test point is visible or occluded when viewed from the previous observation point. If it is determined that the test point is occluded when viewed from the previous observation point, a heuristic test that is part of the occlusion culling test is performed to determine whether the change in the observation point since the previous frame may have revealed a (previously covered) point that is visible from the new (later) observation point.

[0010] The heuristic test includes calculating the position of a first depth point corresponding to a first depth. The first depth point lies on a line passing through the previous observation point and passing through the test point or a position adjacent to the test point, e.g., where the adjacent position is used to mitigate numerical precision issues (see below). For example, the adjacent position is less than the point diameter away from the test point.

[0011] The heuristic test also includes calculating the position of a sampling point on a line from the observation point associated with the test frame (the observation point from which the point cloud represented by the test frame is observed) to the test point, where the distance from the test point to the sampling point is equal to the distance from the test point to the first depth point multiplied by a factor between 0.75 and 1.25. For example, the difference between the distance from the test point to the sampling point and the distance from the test point to the first depth point is chosen to be less than the distance from the test point to the first depth point. In particular, the distance from the test point to the sampling point is equal to the distance from the test point to the first depth point multiplied by a factor between 0.9 and 1.1. More particularly, the distance from the test point to the sampling point is equal to the distance between the test point and the first depth point.

[0012] As an example, the computer-implemented method is configured to provide different settings for the factor to be multiplied by the distance from the test point to the first depth point, e.g., where the method includes settings for the factor. The factor can be freely set within the range of 0.75 to 1.25 and / or a set of selectable fixed factors can be provided for the user to choose from.

[0013] The heuristic test also includes sampling a second depth along a line passing through the sampling point from the previous observation point (e.g., by using a depth buffer), and calculating the position of a second depth point corresponding to the second depth. The second depth point is used to evaluate a culling criterion that depends on the relationship between the distance from the second depth point to the sampling point and the distance from the second depth point to the test point. Based on the evaluation of the culling criterion, the test point is considered to be culled for visualizing the test frame.

[0014] As an example, if the distance from the second depth point to the sample point is shorter than the distance from the second depth point to the test point multiplied by an aggressiveness factor, the culling criterion provides for the test point to be considered culled for visualizing the test frame. The aggressiveness factor defines the aggressiveness of culling the test point. For example, the computer-implemented method includes setting the aggressiveness factor, e.g., to provide a trade-off between computational capacity and a reasonable representation of the point cloud. The larger the aggressiveness factor, the more points are "wrongly" culled (i.e., the more points in the point cloud that are not occluded as seen from the observation point associated with the test frame are culled). A lower value of the aggressiveness factor will make the culling more conservative, up to the limit where the aggressiveness factor is zero (where no points will be culled). The factor can be fixed or user-set, e.g., where the user freely defines a custom value, or where a set of fixed selectable values is provided to the user. In particular, the aggressiveness factor is less than 10. For example, the aggressiveness factor is 1, i.e., such that if the distance from the second depth point to the sample point is shorter than the distance from the second depth point to the test point, the test point is considered culled for visualizing the test frame.

[0015] As an example, occluded points are culled by assigning them positions outside the frustum of the observation point associated with the test frame.

[0016] In one embodiment, the sampling of the first depth and / or the calculation of the first depth point includes biasing (e.g., shifting the position for the purpose of the calculations of the above steps) the test point in depth or (e.g., laterally) in position. For example, such biasing is provided to mitigate numerical precision issues. In particular, a bias less than five point diameters, more particularly less than two point diameters, is used.

[0017] In another embodiment, the occlusion culling test further includes calculating the position of another sample point, where the position of the other sample point is the position where the test point is shifted towards the observation point associated with the test frame by, e.g., one point diameter, and the first depth point lies on the line from the previous observation point to the other sample point.

[0018] In another embodiment, the depth buffer is a hierarchical depth buffer, e.g., where the method includes generating a hierarchical depth buffer at the end of the occlusion culling test for each frame.

[0019] In particular, the occlusion culling test further includes calculating the mipmap levels to be used when sampling the first depth and the second depth such that the hierarchical depth buffer is sampled at the lowest mipmap level where a single texel in the depth texture is large enough to contain the entire test point. For example, the mipmap level is given as the log2 of the screen point size for visualizing the test frame, rounded up to the nearest integer.

[0020] In another embodiment, sampling of the first depth and calculation of the first depth points are performed by using the model-view-projection matrix from the previous frame.

[0021] Specifically, the method includes calculating the screen position and the depth of the calculated test points by using the model-view-projection matrix from the previous frame. The screen position is used to sample the first depth, wherein, based on a criterion for comparing the depth of the first depth point and the calculated test points, it is determined whether the test points are occluded or visible when viewed from the previous observation point. As an example, in order to sample the depth from the previous frame, the so-called screen coordinates of the test points are calculated by using the model-view-projection matrix from the previous frame, where the screen coordinates are 3D coordinates, two of which represent the screen position of the test points and are used to sample the depth texture, while the third component is the depth to be compared in the occlusion culling test.

[0022] In a further embodiment, if it is determined that the test points are visible when viewed from the previous observation point, the heuristic test is skipped.

[0023] In a further embodiment, if it is determined that the test points are visible when viewed from the previous observation point, the test points are visible for visualizing the test frame.

[0024] In another embodiment, the occlusion culling test is performed in the vertex shader. As an example, for instance, compared to performing the occlusion culling test for each part of the point cloud when issuing a draw call, the occlusion culling test is performed for each individual point in the point cloud in the vertex shader.

[0025] In another embodiment, the occlusion culling test is performed for individual points or groups of points in the point cloud in the compute shader, for example by creating a list of points to be rendered by the compute shader, wherein the test points in the point cloud that will be culled for visualizing the test frame are excluded from the list.

[0026] In another embodiment, the method includes visualizing the test frame based on performing the occlusion culling test for each point in the point cloud. As an example, by performing the occlusion test at each point level, it is no longer related to how these points are grouped, and gaps in the occlusion geometry will only prevent individual points from being occluded rather than entire parts.

[0027] In another embodiment, the method includes a rasterization step and a fragment shader step, wherein if it is considered that the test points will be culled for visualizing the test frame, the rasterization step and the fragment shader step are skipped for the test points. Thus, for example, although with the method of the present invention, the cost of evaluating the vertex shader may still occur, for the culled points, the rasterization step and the fragment shader step (where overdraw is relevant) can be skipped.

[0028] The present invention also relates to a computer program product comprising program code having computer-executable instructions for performing the method according to any of the embodiments described above.

[0029] While the above techniques are particularly beneficial for rendering point clouds, the described occlusion culling tests can also be applied to the rendering of triangle meshes. As an example, in this context, the occlusion culling tests will then be performed at the object level rather than in the vertex shader.

[0030] Accordingly, the present invention also relates to a computer-implemented method and a computer program product, which substantially include the above steps, but are adapted to render a mesh model, such as a digital model provided by a triangle mesh rather than a point cloud. As an example, such adaptation is based on a computer-implemented method for visualizing different frames of a mesh model provided by a triangle mesh, for example, using a computing device with limited memory. Each of the different frames represents the mesh model as seen from a corresponding observation point, wherein the method includes an occlusion culling test of the points of the mesh model to determine which points of the mesh model should be culled for the visualization frame. For a point (test point) of the mesh model being tested for visualization in a frame (test frame) by the occlusion culling test, the occlusion culling test includes using a depth buffer providing depth information from a previous frame to sample a first depth from a previous observation point associated with the previous frame towards the test point to determine whether the test point is visible or occluded when viewed from the previous observation point.

[0031] If it is determined that the test point is occluded when viewed from the previous observation point, the occlusion culling test includes performing a heuristic test, which includes: (i) calculating the position of a first depth point corresponding to the first depth; (ii) calculating the position of the sampling point on the line from the observation point associated with the test frame to the test point, wherein the distance from the test point to the sampling point is equal to the distance from the test point to the first depth point multiplied by a factor between 0.75 and 1.25; (iii) sampling a second depth along the line passing through the sampling point from the previous observation point and calculating the position of a second depth point corresponding to the second depth; and (iv) considering the test point to be culled for the visualization test frame based on a culling criterion depending on the relationship between the distance from the second depth point to the sampling point and the distance from the second depth point to the test point. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The methods and computer program products according to different aspects of the present invention will now be described or explained in more detail only by way of example with reference to working examples schematically illustrated in the accompanying drawings. Identical elements are labeled with the same reference numerals in the figures. In particular,

[0033] Figure 1 : Schematically depicts the main steps of an embodiment of a computer-implemented method according to the present invention;

[0034] Figure 2 : Schematically depicts the parameters determined or checked in an embodiment of a heuristic secondary test depicted by Figure 1 Depicted. Detailed implementation

[0035] Figure 1 Schematically depicts the main steps of an embodiment of a computer-implemented method according to the present invention. The method relies on depth information from a previous frame (e.g., the frame immediately preceding when the viewing point is changed) to predict whether points in the point cloud are occluded in the current frame. At the end of each frame, a depth buffer 1 is generated from the scene depth, such as a hierarchical depth buffer. Then the depth buffer 1 is provided to the subsequent frame (e.g., the immediately following next frame) together with a copy of the model-view-projection matrix 2 of the point cloud portion.

[0036] Based on the depth buffer 1 and the model-view-projection matrix 2, a first depth sampling 3 is performed to check whether the test point is visible 4 or occluded 5 when viewed from the previous viewing point. As an example, for each point of the point cloud portion to be rendered, the vertex shader first checks whether the point is visible from the viewpoint (viewing point) of the camera in the previous frame. If the point is visible 4, it is still considered visible 6. If the point is outside the frustum of the previous frame, the point is considered not occluded (i.e., visible). If the point is occluded 5, a heuristic secondary test 7 is performed to determine whether the change in the camera position since the previous frame may have revealed the point such that it can be newly considered visible 6. If not, the point is culled 8.

[0037] Figure 2 Schematically depicts the parameters determined or checked in an embodiment of the heuristic secondary test 7 depicted by Figure 1 Depicted. For example, the heuristic test is initiated by calculating the mipmap level to be used when sampling the depth buffer, e.g., where this is given as the log2 of the screen point size, rounded up to the nearest integer.

[0038] Then the position of the first sampling point 9 is calculated, where the first sampling point 9 is given as the original position of the test point 10 shifted by one diameter (e.g., the camera position) towards the current frame viewing point 11. As an example, this shift (i.e., the first sampling point 9) is used to avoid self-occlusion.

[0039] In the next step, the model-view-projection matrix from the previous frame is used to calculate the screen coordinates and depth of the test point 10 of the previous frame using its previous observation point 12. If the test point lies outside the viewing frustum of the previous frame, the point is considered unoccluded and the following steps are skipped.

[0040] Sample the depth buffer using the last frame screen coordinates. If the sampled depth is higher than the calculated depth, assume the test point is unoccluded and skip the following steps.

[0041] The next steps include calculating the position of the first depth point 13 corresponding to the sampled depth and the distance between the first depth point 13 and the first sample point 9. Calculate the position of the second sample point 14, where the position of the second sample point 14 is given as the original position of the test point 10 offset towards the current frame observation point 11 by an amount equal to the distance between the test point 10 and the first depth point 13. Sample the depth buffer at the position of the second sample point 14 to calculate the position of the second depth point 15 corresponding to the sampled depth.

[0042] If the position of the second depth point 15 calculated from the second depth sample is closer to the position of the second sample point 14 compared to the position of the test point 10, the test point 10 is considered occluded. However, in the example shown, the test point 10 is now visible from the current observation point 11, which is also the result of the heuristic test, since the distance to the second depth point 15 is closer to the test point 10 than the position of the second sample point 14 (e.g., by assuming an aggressiveness factor of 1).

[0043] Although the above part has illustrated the present invention with reference to some preferred embodiments, it must be understood that many modifications and combinations of different features of the embodiments can be made. All such modifications are within the scope of the appended claims.

Claims

1. A computer-implemented method for visualizing different frames of a point cloud using a computing device with limited memory, each of the different frames representing the point cloud as seen from a corresponding observation point, wherein: The method comprises an occlusion culling test of points in the point cloud to determine which points of the point cloud should be culled for visualizing the frame, wherein, for a frame in the point cloud for visualizing the frame, i.e., the test frame, the points tested by the occlusion culling test, i.e., the test points, the occlusion culling test comprises: Using a depth buffer (1) providing depth information from a previous frame, sampling a first depth (3) from a previous observation point (12) associated with the previous frame towards the test point (10) to determine whether the test point (10) is visible (4) or occluded (5) when viewed from the previous observation point (12), Wherein, if it is determined that the test point (10) is occluded when viewed from the previous observation point (12) (5), the occlusion culling test includes performing a heuristic test (7), the heuristic test including calculating a position of a first depth point (13) corresponding to the first depth; calculating the position of a sampling point (14) on a line from the observation point (11) associated with the test frame to the test point (10), wherein the distance from the test point (10) to the sampling point (14) is equal to the distance from the test point (10) to the first depth point (13) multiplied by a factor between 0.75 and 1.25, sampling a second depth along a line from the previous observation point through the sampling point (14), and calculating the position of a second depth point (15) corresponding to the second depth, and Based on a culling criterion depending on the relationship between the distance from the second depth point (15) to the sampling point (14) and the distance from the second depth point (15) to the test point (10), it is considered that the test point (10) is to be culled (8) for visualizing the test frame.

2. The method according to claim 1, wherein: The sampling of the first depth and / or the calculation of the first depth point (13) comprises offsetting the test point (10) in depth or position, in particular by less than five point diameters, more particularly by less than two point diameters.

3. The method according to claim 2, wherein: The occlusion culling test further comprises calculating the position of another sampling point (9), wherein the position of the another sampling point (9) is the position of the test point (10) shifted towards the observation point (11) associated with the test frame, in particular shifted by one point diameter, and the first depth point (13) is located on a line from the previous observation point (12) to the another sampling point (9).

4. A method according to any one of the preceding claims, wherein: The depth buffer (1) is a layered depth buffer, in particular wherein: The method includes generating the layered depth buffer at the end of an occlusion culling test for each frame.

5. The method according to claim 4, wherein: The occlusion culling test further comprises calculating a mipmap level to be used when sampling the first depth and the second depth such that the layered depth buffer (1) is sampled at the lowest mipmap level for which a single texel in the depth texture is large enough to contain the entire test point (10), in particular wherein the mipmap level is given as the log2 of a screen point size used to visualize the test frame, rounded up to the nearest integer.

6. A method according to any one of the preceding claims, wherein: The sampling of the first depth and the calculation of the first depth point (13) are performed by using the model-view-projection matrix (2) from the previous frame.

7. The method according to claim 6, comprising: A screen position and the calculated depth of the test point (10) are calculated using the model-view-projection matrix (2) from the previous frame, and the first depth is sampled using the screen position, wherein a determination is made as to whether the test point (10) is occluded (5) or visible (4) when viewed from the previous observation point (12) based on a criterion comparing the first depth point (13) and the calculated depth of the test point (10).

8. A method according to any one of the preceding claims, wherein: If it is determined that the test point (10) is visible when viewed from the previous observation point (4), the heuristic test (7) is skipped.

9. The method according to any one of the preceding claims, comprising: If it is determined that the test point (10) is visible when viewed from the previous observation point (4), then it is considered that the test point (10) will be visible (6) for visualization of the test frame.

10. A method according to any one of the preceding claims, wherein: The occlusion culling test is performed in the vertex shader.

11. A method according to any one of the preceding claims, wherein: The occlusion culling test is performed in a compute shader on individual points or groups of points of the point cloud, in particular by creating a list of points to be rendered by the compute shader, wherein test points (10) in the point cloud that are to be culled for visualizing the test frame are excluded from the list.

12. A method according to any one of the preceding claims, comprising: The test frame is visualized based on performing the occlusion culling test for each point in the point cloud.

13. A method according to any one of the preceding claims, comprising a rasterization step and a fragment shader step, wherein: If the test point (10) is considered to be culled (8) for visualization of the test frame, the rasterization step and the fragment shader step are skipped for the test point.

14. A method according to any one of the preceding claims, wherein: If the distance from the second depth point (15) to the sampling point (14) is shorter than the distance from the second depth point (15) to the test point (10) multiplied by an aggressiveness factor, in particular wherein the aggressiveness factor is less than 10, more particularly wherein the aggressiveness factor is 1, the rejection criterion provides that the test point (10) is considered to be rejected (8) for visualization of the test frame.

15. A computer program product comprising a program code having computer executable instructions for performing the method according to any one of the preceding claims.

16. A computer-implemented method for visualizing different frames of a mesh model using a computing device with limited memory, each of the different frames representing the mesh model as seen from a corresponding observation point, wherein: The method comprises an occlusion culling test of points in the mesh model to determine which points in the mesh model should be culled for visualizing the frame, wherein, for a frame in the mesh model for visualizing the frame, i.e., a test frame, the points tested by the occlusion culling test, i.e., the test points, the occlusion culling test comprises: Using a depth buffer (1) providing depth information from a previous frame, sampling a first depth from a previous observation point (12) associated with the previous frame towards the test point (10) to determine whether the test point (10) is visible (4) or occluded (5) when viewed from the previous observation point (12), Wherein, if it is determined that the test point (10) is occluded when viewed from the previous observation point (12) (5), the occlusion culling test includes performing a heuristic test (7), the heuristic test including calculating a position of a first depth point (13) corresponding to the first depth; calculating the position of a sampling point (14) on a line from the observation point (11) associated with the test frame to the test point (10), wherein the distance from the test point (10) to the sampling point (14) is equal to the distance from the test point (10) to the first depth point (13) multiplied by a factor between 0.75 and 1.25, sampling a second depth along a line from the previous observation point through the sampling point (14), and calculating the position of a second depth point (15) corresponding to the second depth, and Based on a culling criterion depending on the relationship between the distance from the second depth point (15) to the sampling point (14) and the distance from the second depth point (15) to the test point (10), it is considered that the test point (10) is to be culled (8) for visualizing the test frame.

17. A computer program product comprising a computer program product having program code with computer executable instructions for performing the method according to claim 16.

Images