A Real-time Sonar Point Cloud Visualization and Interaction Method for Dynamic Data Management

Abstract

The present invention provides a real-time sonar point cloud visualization interaction method for dynamic data management, which relates to the field of visualization technology. The purpose of the present invention is to improve the efficiency and effect of real-time visualization interaction of massive point cloud data simultaneously. During the data organization and management process, bricks are designed for physical partitioning, and spatial fields are dynamically allocated to solve the memory consumption problem generated by conventional spatial fields. Virtual panoramic regions are defined using bricks to manage the scanning regions of each continuously incoming frame of sonar data. Memory is dynamically allocated for the collected point cloud data in units of bricks, and the memory of bricks not involved is empty. Combining the characteristics of multi-level brick partitioning organization, the ray casting algorithm is improved for real-time rendering, and the rendering resolution of the algorithm is fixed. Through the calculation of empty voxel jumping and distance-based spatial interpolation for the data points within the bricks, in terms of interaction operations, after panning and zooming the viewing field, the three-dimensional points to be rendered under the new viewing field are determined by calculating the bricks within the viewing field region.

Description

Technical Field

[0001] The present invention relates to the field of visualization technology, and in particular to a dynamic data management real-time sonar point cloud visualization interaction method. Background Art

[0002] With the improvement of hardware level, sonar data acquisition technology has developed rapidly, and long-term large-scale sonar data generates a huge amount of three-dimensional point cloud data. The three-dimensional point cloud data records the three-dimensional intensity information and geometric position information of the space. Using this information, the three-dimensional point cloud can be visually rendered in real time to output a three-dimensional topographic map based on sonar intensity mapping. When visualizing and rendering the terrain with three-dimensional point cloud data, it is necessary not only to efficiently organize and manage the three-dimensional point cloud data, but also to have high real-time performance during user interaction. Most of the existing rendering systems use traditional methods such as pre-caching and hierarchical traversal to store data. Such a processing system often results in a long data organization process, which is not only not conducive to real-time visualization, but also leads to a general rendering quality. In addition, due to the continuous improvement of the accuracy of three-dimensional scanned point clouds, the density of point cloud sampling has increased rapidly, the data storage scale has become larger and larger, and various factors interfere with the point cloud data, resulting in problems such as irregular and uneven distribution of the point cloud data, which significantly increases the burden of the point cloud in visual representation. Therefore, on the premise of ensuring the completeness and authenticity of the point cloud data, it is necessary to find a method to solve the problems of massive data organization and scheduling, and further realize real-time interaction with massive point cloud data, including operations such as translation and zooming, which is particularly important in visualization research.

[0003] The existing public literatures "Wang Chenglei, Lian Yi, Zeng Xiaoming, He Long, Cui Tiejun, Du Peng, Chen Pengfei. Research on 3D Point Cloud Visualization Based on WebGL [J]. Technology Innovation and Application, 2017(35):41+45." and "Yang Fan. Research on the Organization and Scheduling Method of Massive Point Cloud Data in and out of Memory [D]. Beijing University of Civil Engineering and Architecture, 2017." provide methods for organizing three-dimensional point cloud data in a massive space, and introduce methods such as triangular mesh slicing based on edge folding and common quadtrees, KD trees, etc. as spatial indexes for point cloud data organization. In particular, the quadtree organization method mentioned in the literature is easy to implement and has good operability, and is more suitable for organizing massive point cloud data.

[0004] The existing public literature "Chen Yidong. Automatic Fusion and Interaction Method of Panoramic / Point Cloud Data in Network Environment [D]. Southwest Jiaotong University, 2021. DOI: 10.27414 / d.cnki.gxnju.2021.001800." combines the advantages of point clouds with accurate spatial position information, fuses point cloud data with panoramic images for visualization, establishes a matching mechanism based on automatic mapping of panoramic / point cloud data, and realizes cross-modal interaction analysis of multi-source data in the fusion scene. The method proposed in the literature can support the automatic fusion and interaction of point cloud data, improve the display efficiency of a large amount of point cloud data, and provide an idea on how to perform efficient data organization fusion, visualization, and interaction analysis in a network environment.

[0005] Since sonar data is obtained by accumulating several frames scanned over a long time, its most significant feature is layer-by-layer scanning, real-time updating, and a large amount of data. The existing methods are not very effective in organizing and visually expressing such a large amount of point cloud data in a massive space. Most methods deal with point clouds with a small amount of data. If dealing with massive data, it will create an extremely large conventional space field, taking too long to process and being unfavorable for visual scheduling.

[0006] The data organization and management methods proposed in the above literature, although achieving the management and call of point cloud data, cannot perform efficient scheduling and fast calculation for the data of several frames updated in real time in a massive space, and it is difficult for the existing organization and management methods to meet the high requirements of real-time visualization for efficiency in improving the organization efficiency of massive point cloud data. In addition, although the visual interaction method in the literature has the advantage of data adaptability, for the characteristics of irregular data points, uneven distribution, and a large span of data value ranges of massive point cloud data, it cannot ensure that the data accumulated in several frames is completely visualized during the final real-time rendering. In particular, during the interaction process, due to the need to dynamically update massive data, the transformation operation also poses higher requirements for data scheduling and imaging efficiency. Therefore, the literature method can neither ensure the completeness and authenticity of massive data during the organization process nor is it suitable for the visualization and real-time interaction of massive three-dimensional point cloud data. Summary of the Invention

[0007] Aiming at the deficiencies of the existing technology, the present invention provides a dynamic data management real-time sonar point cloud visualization and interaction method.

[0008] A dynamic data management real-time sonar point cloud visualization and interaction method specifically includes the following steps:

[0009] Step 1: Apply for a virtual space area and create a brick space field to organize data;

[0010] The point cloud data information in the image domain obtained by three-dimensional forward sonar scanning is processed and converted to the Cartesian coordinate system, and continuously transmitted into the brick in the form of several frames. The data information includes the spatial geographical location information coordinates X, Y, Z values, and the reflection intensity value Value.

[0011] After the brick allocates the actual occupied space field for the data information, it stores all the imaging parameter information of the incoming point cloud data points into the brick. After interpolating the data points, the brick is updated simultaneously. Each data point has a corresponding nbrick after being divided into multiple levels, and contains all its parameter information and the corresponding nbrickID. The nbrickID is used as the index of the data point, and its calculation formula is as follows:

[0012] nbrickID = nz × nBrickNumInASlice + ny × nWidBrickNum + nx

[0013] Among them, (nx, ny, nz) are the three-dimensional coordinates of the data point, nBrickNumInASlice is the number of brick sub-blocks in one layer, and nWidBrickNum is the number of brick sub-blocks in one row.

[0014] After partitioning the point cloud data into sub-blocks using the brick, through formula calculation, the ID number of the brick sub-block corresponding to the data point is obtained, and finally the point cloud data based on brick multi-level partitioning is obtained. Next, the data will be organized using the nbrickID number as the index.

[0015] Step 2: Use an improved ray casting algorithm based on brick multi-level partitioning for rendering, perform brick calculation processing and visualization on the point cloud data;

[0016] Step 2.1: Empty voxel jump processing and calculation of the brick;

[0017] For the point cloud data based on brick multi-level partitioning, perform an empty voxel judgment on the point cloud data: If the data in the collected brick is empty, there is no need to perform transfer function mapping, skip the current brick block, and process the next data point; If the data in the collected brick is not empty, retain it, and perform lighting calculation and color synthesis on the data in the brick.

[0018] Perform a distance-based spatial interpolation calculation on the point cloud data based on brick multi-level partitioning: Obtain the three-dimensional coordinates and intensity information Valuei of the neighborhood points of the interpolation point through the brick, and then perform weighted averaging with the distance between the interpolation point and other data points in the neighborhood as the weight. The calculated distance-weighted value is used as the intensity value of the current interpolation point and stored in the brick as data information; among them, the junction point data existing between bricks is discarded during the interpolation process.

[0019] The distance calculation uses the Euclidean distance, and the calculation formula for the intensity value is as follows:

[0020]

[0021] Among them, di represents the distance from the i-th data point in the neighborhood to the interpolation point, and Wi is the weight of the i-th point to the interpolation point.

[0022] Value = ∑Wi * Valuei

[0023] Among them, Valuei is the intensity information of the i-th point in the neighborhood, and the obtained Value is the final intensity value of the current interpolation point.

[0024] Step 2.2: Use the improved ray casting algorithm to present the brick field of view map at a fixed resolution;

[0025] Step 2.2.1: Starting from each pixel point on the screen, emit a ray along the line of sight. When this ray passes through the point cloud data, sample at equal distances along the ray direction. In Step 2.1, the data points that have undergone interpolation calculation processing have their optical properties. Gradient estimation aims to determine the gradient of the approximate surface normal for classification and coloring. Using principal component analysis, calculate the normal vector of the data point position, and the normal vector In the spherical coordinate system, it can be represented by θ:

[0026]

[0027] Then, judge the gradient value of this point according to the cos value of the angle between the normal vector and the ray:

[0028]

[0029] Among them, the data point normal vector is The ray direction is

[0030] Step 2.2.2: Synthesize the sampled data points on the ray in the front-to-back order to calculate the color value of the pixel point on the screen corresponding to this ray. The color synthesis operation for each sampling point along the ray has the formula:

[0031]

[0032] Among them, C is the final synthesized color, C i and a i are respectively the color value and opacity of the sampling data point i on the light ray. n is the number of sampling data points on the light ray.

[0033] Step 2.2.3: According to the interpolation, gradient estimation, coloring and synthesis of the data points in Steps 2.1 - 2.2.2, further improvements are made during rendering, that is, the resolution is fixed to draw a view range map with stable imaging.

[0034] Step 3: Implement real - time interaction of the point cloud within the brick to complete the real - time sonar point cloud visualization interaction method;

[0035] When performing real - time interaction operations of selection, translation, rotation and scaling on the point cloud data based on the multi - level partitioning of the brick, within the process of view transformation and scaling, select the bricks inside the view area, perform screen mapping on the point cloud data according to the ID of the brick; regard the point cloud data as being interacted with in units of bricks, judge the nbrick data points that change during the interaction process, select them corresponding to their nbrickID, and finally select the new brick area after interaction for visualization.

[0036] The beneficial effects produced by adopting the above technical solutions are as follows:

[0037] The present invention provides a real - time sonar point cloud visualization interaction method for dynamic data management, which dynamically applies regions based on multi - level partitioning of bricks, realizing efficient organization and scheduling of data information; improves the ray - casting algorithm for drawing, improves the sampling accuracy, and performs real - time visualization of the imaging field of view at a fixed resolution; combines bricks to perform dynamic update processing on data, improving the rendering efficiency during the interaction process. This method meets the requirements of massive point cloud data for real - time visualization interaction at the levels of data organization, visualization and interaction in sequence. Brief Description of the Drawings

[0038] Figure 1 It is a diagram of the real - time sonar point cloud visualization interaction method in the embodiment of the present invention

[0039] Figure 2 It is a schematic diagram of the brick organizational structure of the point cloud data in the embodiment of the present invention

[0040] Figure 3 It is a flow chart of the improved brick empty - voxel jumping method in the embodiment of the present invention Detailed Embodiment

[0041] The following will further describe in detail the specific implementation manners of the present invention in conjunction with the accompanying drawings and embodiments. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0042] A real-time sonar point cloud visualization interaction method for dynamic data management, as Figure 1 shown, specifically includes the following steps:

[0043] Step 1: Apply for a virtual space area and create a brick space field to organize data;

[0044] The meaning of "brick" is to divide into blocks or partitions, and its significance lies in dividing the incoming massive point cloud data into brick sub-blocks for organization within a very large space area in the future. The image domain point cloud data information obtained by three-dimensional forward-looking sonar scanning is processed and converted into the Cartesian coordinate system, and continuously transmitted into the brick in the form of several frames. The data information includes the spatial geographical location information coordinates X, Y, Z values, and reflection intensity value Value.

[0045] Each frame of data will continuously fill the brick space. According to the scanning area of each frame of sonar data, the involved bricks are determined, and data coverage is repeated. Memory is not allocated for the brick areas that have not been collected.

[0046] After the brick allocates the actual occupied space field for the data information, all the imaging parameter information of the incoming point cloud data points is stored in the brick. After interpolation calculation of the data points, the brick is updated simultaneously. Each data point has a corresponding nbrick after multi-level partitioning, and includes all its parameter information and the corresponding nbrickID. The nbrickID is used as the index of the data point, and its calculation formula is as follows:

[0047] nbrickID = nz × nBrickNumInASlice + ny × nWidBrickNum + nx

[0048] Where, (nx, ny, nz) are the three-dimensional coordinates of the data point, nBrickNumInASlice is the number of brick sub-blocks in one layer, and nWidBrickNum is the number of brick sub-blocks in one row.

[0049] After partitioning the point cloud data into sub-blocks by using bricks, through formula calculation, the ID number of the brick sub-block corresponding to the data point is obtained, and finally the point cloud data based on brick multi-level partitioning is obtained. Next, the data will be organized with the nbrickID number as the index.

[0050] For subsequent scheduling during visual interaction implementation. The organizational structure of brick for massive point cloud data is as Figure 2 shown.

[0051] Step 2: Use an improved ray casting algorithm based on brick multi-level partitioning to perform rendering, and conduct brick calculation processing and visualization on the point cloud data;

[0052] During the rendering process, according to the multi-level organizational characteristics of data by brick, the empty voxel skipping method is improved, and at the same time, a distance-based spatial interpolation calculation is performed on the point cloud data to ensure the image quality of data visualization. After the data is calculated and processed, a view range map is output at a fixed resolution, as Figure 3 shown.

[0053] Step 2.1: Empty voxel skipping processing and calculation of brick;

[0054] For the point cloud data based on brick multi-level partitioning, perform empty voxel judgment on the point cloud data: If the data in the collected brick is empty, there is no need to perform transfer function mapping, skip the current brick block, and process the next data point; thus avoiding invalid access, reducing data processing for empty bricks, reducing the rendering time, and the more empty voxels skipped, the more obvious the acceleration effect. The processing process of the improved brick empty voxel skipping method is shown in the figure. If the data in the collected brick is not empty, retain it, and perform lighting calculation and color synthesis on the data in this brick;

[0055] To quickly filter massive data, perform another distance-based spatial interpolation calculation and processing on the point cloud data based on brick multi-level partitioning: Obtain the three-dimensional coordinates and intensity information Valuei of the neighborhood points of the interpolation point through brick, and then perform weighted average with the distance between the interpolation point and other data points in the neighborhood as the weight. The closer the data point is to the interpolation point, the greater the weight assigned, and the calculated distance weighted value is used as the intensity value of the current interpolation point, and is stored as data information in the brick; among them, the data at the junction points between bricks is discarded during the interpolation process.

[0056] The Euclidean distance is used for distance calculation, and the following formula is used for the calculation of the intensity value:

[0057]

[0058] Among them, di represents the distance from the i-th data point in the neighborhood to the interpolation point, and Wi is the weight of the i-th point to the interpolation point.

[0059] Value = ∑Wi * Valuei

[0060] Among them, Valuei is the intensity information of the i-th point in the neighborhood, and the obtained Value is the final intensity value of the current interpolation point.

[0061] Implement fast calculation of the data within the brick, and finally obtain more refined and more operable massive point cloud data.

[0062] Step 2.2: Adopt the improved ray casting algorithm to present the brick field of view map at a fixed resolution;

[0063] Step 2.2.1: Starting from each pixel point on the screen, emit a ray along the line of sight. When this ray passes through the point cloud data, sample at equal distances along the ray direction. In Step 2.1, the data points that have been processed by interpolation calculation have their optical properties, such as color intensity values and opacity, etc.; Gradient estimation aims to determine the gradient of the approximate surface normal for classification and coloring. Using principal component analysis, calculate the normal vector of the data point position, and the normal vector In the spherical coordinate system, it can be represented by θ:

[0064]

[0065] Then, judge the gradient value of this point according to the cos value of the angle between the normal vector and the ray:

[0066]

[0067] Among them, the data point normal vector is The ray direction is

[0068] Step 2.2.2: Synthesize the sampled data points on the ray in the front-to-back order, and calculate the color value of the pixel point on the screen corresponding to this ray. The color synthesis operation for each sampling point along the ray is calculated by the formula:

[0069]

[0070] Among them, C is the final synthesized color, and C i and a i are respectively the color value and opacity of the sampling data point i on the ray. n is the number of sampling data points on the ray.

[0071] Step 2.2.3: Based on the interpolation, gradient estimation, coloring, and synthesis of data points in Steps 2.1 - 2.2.2, further improvements are made during rendering. That is, the resolution is fixed, such as 512 * 512, to draw a field-of-view map with stable imaging. Due to the relationship between the size of the imaging screen and the change of the viewing point, data points within the field of view can be rendered, that is, selectively draw the bricks within the field of view. Therefore, during the rendering process, the resolution is fixed and will not change due to the change of the field-of-view position and operation transformation. In the case of a variable field of view, regardless of the data volume, the bricks where the data points are located will be included in the visualized field-of-view area, ensuring that a field-of-view map with stable imaging is drawn. If the display area is large, the data points are fine; if the display area is small, they are coarser. Thus, ensuring the stable frame rate of real-time visualization.

[0072] Step 3: Implement real-time interaction of the point cloud within the brick to complete the real-time sonar point cloud visualization interaction method; after improving the ray casting algorithm, the real-time visualization effect of the point cloud data is improved. On this basis, it is still necessary to meet the application requirements for efficient interaction (such as selection, movement, and scaling) of massive three-dimensional point cloud data.

[0073] When dealing with interactive operations, traditional methods often choose to traverse all data points, judge according to the change of the three-dimensional coordinates of the data points within the screen, map the new screen coordinates, and continuously visualize the data points after a series of interactive operation transformations. However, in a massive space, as the amount of point cloud data increases and several frames of data are continuously input, the real-time updated data will lead to a greater amount of calculation. The operation of massive data points will cause the screen refresh efficiency to be slow and cannot meet the needs of real-time interaction.

[0074] When performing real-time interactive operations such as selection, translation, rotation, and scaling on the point cloud data based on brick multi-level partitioning, during the field-of-view transformation and scaling process, select the bricks inside the field-of-view area, and perform screen mapping on the point cloud data according to the ID of the brick; regard the point cloud data as being interacted with in units of bricks, judge the nbrick data points that change during the interaction, select them corresponding to their nbrickID, and finally select the new brick area after interaction for visualization. This can not only ensure that the rendering frame rate tends to be stable, but also reduce the huge amount of calculation in the case of frequent interaction, realizing efficient rendering and dynamic update processing of the point cloud.

[0075] The above description is only a preferred embodiment of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) disclosed in the embodiments of the present disclosure that have similar functions.

Claims

1. A real-time sonar point cloud visualization and interaction method for dynamic data management, characterized in that It includes the following steps: Step 1: Apply for a virtual space area and create a brick space field to organize data; Step 2: Use an improved ray casting algorithm based on multi-level brick partitioning to perform rendering, and conduct brick calculation processing and visualization on the point cloud data; Step 2.1: Process and calculate the empty voxels of the brick; For the point cloud data based on multi-level brick partitioning, judge the empty voxels of the point cloud data: If the data in the collected brick is empty, there is no need to perform transfer function mapping, skip the current brick block, and process the next data point; If the data in the collected brick is not empty, retain it, and perform lighting calculation and color synthesis on the data in this brick; Perform another distance-based spatial interpolation calculation processing on the point cloud data based on multi-level brick partitioning: Obtain the three-dimensional coordinates and intensity information Valuei of the neighborhood points of the interpolation point through the brick, and then perform weighted averaging with the distance between the interpolation point and other data points in the neighborhood as the weight. The calculated distance weighted value is used as the intensity value of the current interpolation point and is stored in the brick as data information; among them, the data of the intersection points between bricks are discarded during the interpolation process; Step 2.2: Use the improved ray casting algorithm to present the brick field of view map at a fixed resolution; Step 2.2.1: Starting from each pixel point on the screen, emit a ray along the line of sight. When this ray passes through the point cloud data, sample at equal distances along the ray direction. Use principal component analysis to calculate the normal vector of the data point positions. The normal vector can be represented in the spherical coordinate system by , θ: Then judge the gradient value of this point according to the cos value of the angle between the normal vector and the ray: Among them, the data point normal vector is The light direction is Step 2.2.2: Synthesize the sampled data points on the ray in the front-to-back order to calculate the color value of the pixel point corresponding to this ray on the screen. The color synthesis operation for each sampled point along the ray is calculated by the formula: Among them, C is the final composite color, C i with a i are the color value and opacity of the sampling data point i on the light, respectively; n is the number of sampling data points on the light; Step 2.2.3: According to the interpolation, gradient estimation, coloring, and synthesis of the data points in Steps 2.1 - 2.2.2, further improve during rendering, that is, fix the resolution and draw a stable field of view map; Step 3: Implement real-time interaction of the point cloud within the brick to complete the real-time sonar point cloud visualization interaction method.

2. A real-time sonar point cloud visualization interaction method for dynamic data management according to claim 1, characterized in that, Specifically, in Step 1, the point cloud data information of the image domain obtained by three-dimensional forward sonar scanning is processed and converted to the Cartesian coordinate system, and is continuously transmitted into the brick in the form of several frames. The data information includes the spatial geographical location information coordinates X, Y, Z values, and reflection intensity value Value; After the brick allocates the actual occupied space field for the data information, all the imaging parameter information of the incoming point cloud data points is stored in the brick. After performing interpolation calculation on the data points, the brick is updated simultaneously. Each data point has a corresponding nbrick after multi-level partitioning, and includes all its parameter information and the corresponding nbrickID. The nbrickID is used as the index of the data point; After partitioning the point cloud data into sub-blocks using the brick, through formula calculation, the ID number of the brick sub-block corresponding to the data point is obtained, and finally the point cloud data based on multi-level brick partitioning is obtained. Next, the data will be organized using the nbrickID number as the index.

3. A real-time sonar point cloud visualization and interaction method for dynamic data management according to claim 2, characterized in that, The calculation formula of the nbrickID is as follows: nbrickID = nz × nBrickNumInASlice + ny × nWidBrickNum + nx where (nx, ny, nz) are the three-dimensional coordinates of the data point, nBrickNumInASlice is the number of brick sub-blocks in a layer, and nWidBrickNum is the number of brick sub-blocks in a row.

4. A real-time sonar point cloud visualization interaction method for dynamic data management according to claim 1, characterized in that In the calculation of the intensity value described in Step 2.1, the Euclidean distance is used for distance calculation, and the following formula is used for the calculation of the intensity value: where di represents the distance from the i-th data point in the neighborhood to the interpolation point, and Wi is the weight of the i-th point to the interpolation point; Value = ∑Wi * Valuei where Valuei is the intensity information of the i-th point in the neighborhood, and the obtained Value is the final intensity value of the current interpolation point.

5. A real-time sonar point cloud visualization and interaction method for dynamic data management according to claim 1, characterized in that, Specifically, in Step 3, when performing real-time interactive operations of selection, translation, rotation, and scaling on the point cloud data based on brick multi-level partitioning, during the view transformation and scaling process, select the bricks inside the view area, and perform screen mapping on the point cloud data according to the ID of the bricks; regard the point cloud data as being interacted with in units of bricks, judge the nbrick data points that change during the interaction, select them corresponding to their nbrickID, and finally select the new brick area after interaction for visualization.

Images