A point cloud hole completion system and method for a three-dimensional imaging system

Abstract

The application relates to a three-dimensional imaging system point cloud hole completion system and method, and belongs to the technical field of point cloud hole completion. First, point cloud data, color images and intensity data of the same field of view under different integration times are acquired, and a hole region and a boundary in the point cloud data are identified and marked. Secondly, the spatial distribution of depth data is acquired by using color image features, and visual features in the color images and corresponding spatial positions of the holes in the point cloud data are accurately aligned. Finally, intensity data under different integration times are acquired, the holes are completed in a wide dynamic range fusion mode, and the data quality of the holes is further improved by combining a radial basis function interpolation algorithm. The technical scheme provided by the application can accurately identify the hole region in the point cloud data, fill and optimize the hole information, maintain the spatial continuity and consistency of the point cloud data, and finally improve the spatial accuracy and visual quality of the overall point cloud data.

Description

Technical Field

[0001] This invention relates to a point cloud hole completion system and method for a three-dimensional imaging system, belonging to the field of point cloud hole completion technology, and particularly to a point cloud hole completion method, device and storage medium for a three-dimensional imaging system. Background Technology

[0002] With the rapid development of 3D imaging technology, 3D imaging systems have been widely used in various fields such as industrial inspection, cultural heritage protection, virtual reality, and autonomous driving. 3D imaging systems typically acquire the 3D information of objects using technologies such as laser scanning, structured light, and Time-of-Flight (ToF) to form point cloud data. Point cloud data can accurately describe the shape and spatial position of objects and is the foundation for 3D reconstruction and analysis.

[0003] However, in practical applications, due to various reasons such as limitations in scanning angle, object reflection characteristics, and occlusion, point cloud data often contains gaps, meaning that some areas lack point clouds. This results in incomplete 3D models and loss of spatial information, severely impacting data quality and the effectiveness of subsequent applications. Especially in application scenarios with stringent requirements for high precision and visual effects, such as the 3D reconstruction of cultural and artistic works and sophisticated industrial design, the completeness and accuracy of point cloud data are paramount.

[0004] Currently, point cloud hole completion mainly relies on interpolation algorithms and deep learning techniques. While these methods improve the integrity of point cloud data to some extent, they still face problems such as inaccurate edge recognition, unnatural hole completion results, and high computational resource consumption. Therefore, developing a method that can accurately identify hole regions, efficiently complete and optimize hole information is of great significance for improving the data processing capabilities and application scope of 3D imaging systems. Summary of the Invention

[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a point cloud hole completion system and method for a three-dimensional imaging system. First, point cloud data, color images, and intensity data of the same field of view at different integration times are acquired. An edge detection algorithm and a neighborhood density-based analysis method are used to accurately identify and mark the hole regions and boundaries in the point cloud data. Second, the spatial distribution of depth data is obtained using color image features, accurately aligning the visual features in the color image with the corresponding spatial positions of the holes in the point cloud data. Finally, intensity data at different integration times are acquired, and a wide dynamic range fusion method is used to complete the hole regions. Furthermore, a radial basis function interpolation algorithm is combined to further improve the data quality of the hole regions.

[0006] The technical solution of this invention is:

[0007] A method for filling holes in point clouds in a 3D imaging system includes the following steps:

[0008] S1, acquire point cloud data P1, color image M1, and intensity data D1 at integration time T1 through a three-dimensional imaging system;

[0009] Point cloud data P2, color image M2, and intensity data D2 at integration time T2 are acquired using a 3D imaging system.

[0010] …

[0011] The integration time T is obtained through a three-dimensional imaging system. n Point cloud data P n Color image M n Intensity data D n ;

[0012] S2, Select the point cloud data P with the fewest void areas obtained in step S1. i And use an edge detection algorithm to process the selected point cloud data P i Segmentation and identification of spatially discontinuous regions;

[0013] S3, based on point cloud data P i The point cloud density determination step 2 identifies the boundary density of spatially discontinuous regions, and combines this with a neighborhood density algorithm to further identify and label the point cloud data P. i The boundary of the void area;

[0014] S4, the color image M obtained in step S1 i The feature points and the point cloud data P marked in step S3 i The boundary of the void region in the data is merged to obtain the merged point cloud data P. i Color information and point cloud information of the boundary of the void region X i ;

[0015] S5, based on X obtained in step S4 i Extract intensity data D1, D2, D3…D at different integration times. n The corresponding region is then filled in using a wide dynamic range fusion method;

[0016] S6. The radial basis function interpolation algorithm is used to interpolate the area filled in step S5 to increase the amount of point cloud data in the hole area and ensure that the point cloud density of the hole area is the same as the point cloud density of the original point cloud data of the 3D imaging system, that is, the hole filling of the point cloud of the 3D imaging system is completed.

[0017] In step S2, the method for segmentation and recognition is as follows: a deep learning model is established and the established deep learning model is trained. During the training process, point cloud data of hollow regions with coarse labels and real objects are input, and gradient calculation value G is output. Hollow region boundaries are filtered according to the output gradient calculation value G.

[0018] Let the high threshold of the output gradient calculation value G be G. h The low threshold is G l ;

[0019] If G≥G h Then the point cloud data corresponding to the gradient calculation value G will be directly determined as the boundary of the void region;

[0020] If G l ≤G≤G h The neighborhood density algorithm is used to calculate the density distribution of each point in the cavity region and its neighborhood points. The relatively sparse regions are identified as the boundaries of the cavity region.

[0021] If G≤G l If the model fails to distinguish between real objects and empty areas, it will be unable to train the model further and the high and low thresholds will be adjusted.

[0022] In step S4, a checkerboard calibration board or a circular calibration board is used to calibrate the three-dimensional imaging system during fusion to determine the intrinsic and extrinsic parameters of the three-dimensional imaging system, and the color image is cropped to ensure that the imaging field of view of the color image is consistent with that of the point cloud data.

[0023] In step S5, the method for completing the extracted corresponding region is as follows: calculate the weight of the point cloud data at each integration time, synthesize a point cloud dataset with a wide dynamic range, and calculate the linear region of each pixel (m, n) in the dynamic range of the three-dimensional imaging system.

[0024]

[0025] Where j = 1, 2, 3…n;

[0026] Find the maximum and minimum points in the second derivative, and denote the integration time at each point as T. min and T max , that is, the left and right endpoints of the linear region;

[0027] For linear regions [T] min， T max The internal response function is used for high-dynamic fusion, and the fusion result is:

[0028] D x =aT x +b

[0029] In the formula, Tx D is the integration time; x The data represents intensity; a represents the weight within the linear region; b represents the bias; x = 1, 2, 3…n;

[0030] For [T] min， T max Outside the interval, a nonlinear response function is used for high-dynamic fusion, and the fusion result is as follows:

[0031]

[0032] In the formula, μ is the mean; σ is the standard deviation;

[0033] In step S6, the interpolation method is as follows: select the type of radial basis function according to the point cloud density requirements and the distribution characteristics of the hole region, select the Gaussian function for the outer boundary of the hole region, select the polynomial function for the interior, and perform interpolation to complete the hole region.

[0034] A point cloud hole completion system for a three-dimensional imaging system, the completion system includes an image acquisition module, a preprocessing module, a data fusion module and a hole completion module;

[0035] The image acquisition module is used to acquire point cloud data, color images, and intensity data at different integration times;

[0036] The preprocessing module is used to perform statistical analysis on the acquired point cloud dataset using edge detection algorithms and neighborhood density algorithms to identify hole regions and boundary features in the point cloud data.

[0037] The data fusion module is used to analyze point cloud data, color images and intensity data at different integration times, obtain response functions in different integration intervals, and perform data fusion on the hole region using a wide dynamic range.

[0038] The hole completion module is used to select an appropriate interpolation function to complete the hole regions and boundaries in the point cloud data based on the distribution characteristics of the hole regions and boundaries.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] 1. Improved hole completion accuracy: This method accurately identifies point cloud holes and boundaries through edge detection and neighborhood density algorithms, which significantly improves the accuracy of hole identification compared with traditional methods, providing a high-precision foundation for 3D model reconstruction and achieving more realistic 3D visual effects.

[0041] 2. Optimize visual quality and spatial coherence: By combining wide dynamic range fusion and radial basis function interpolation algorithms, this method not only increases the amount of point cloud data in the hole region, but also ensures that the density after completion is consistent with the original data, significantly improving visual quality and data coherence.

[0042] 3. Enhance the flexibility and robustness of data processing: By aligning point cloud data with color image features at different integration times and adding deep learning edge detection, this method demonstrates high flexibility and stability when dealing with point cloud holes in complex environments.

[0043] 4. A method for filling holes in point clouds in a three-dimensional imaging system, the method comprising the following steps: First, acquiring point cloud data, color images, and intensity data at different integration times within the same field of view; using edge detection algorithms and neighborhood density-based analysis methods to accurately identify and mark hole regions and boundaries in the point cloud data; second, using color image features to obtain the spatial distribution of depth data, accurately aligning visual features in the color image with the corresponding spatial positions of holes in the point cloud data; finally, acquiring intensity data at different integration times, filling hole regions using a wide dynamic range fusion method, and further improving the data quality of hole regions by combining a radial basis function interpolation algorithm. The technical solution provided by this invention can accurately identify hole regions in point cloud data, fill and optimize hole information, maintain the spatial coherence and consistency of point cloud data, and ultimately improve the spatial accuracy and visual quality of the overall point cloud data. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the point cloud completion method of the present invention;

[0045] Figure 2 This is a schematic diagram of the response function under different integration times in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the interpolation algorithm in an embodiment of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0048] To achieve the above objectives, this invention provides a method for completing point cloud holes in a three-dimensional imaging system, such as... Figure 1 As shown, it includes the following steps:

[0049] S1 obtains different integration times T1, T2, T3…T through a three-dimensional imaging system. n Next, point cloud data P1, P2, P3…Pn and color images M1, M2, M3…M n And intensity data D1, D2, D3…D n ;

[0050] S2 selects the point cloud data P with the fewest void areas. i Processing is performed using an edge detection algorithm on the point cloud data P. i Segmentation and recognition of discontinuous regions in mid-space are performed to obtain real object regions and hollow regions;

[0051] S3 based on point cloud data P i The point cloud density is used to determine the boundary density of the hole region, and the neighborhood density algorithm is used to further identify and mark the boundary of the hole region in the point cloud data.

[0052] S4 uses color image M i Feature points and point cloud data P i Spatial distribution, aligning visual features in color images with the corresponding spatial locations of holes in point cloud data;

[0053] S5 based on point cloud data P i Location information of the hollow cavity region, and extraction of intensity data D1, D2, D3…D at different integration times. n In the corresponding region, a wide dynamic range fusion method is used to fill in the void area;

[0054] S6 combines radial basis function interpolation algorithm to interpolate the completed point cloud data, increasing the amount of point cloud data in the hole area and ensuring that the point cloud density in the area is the same as the original point cloud data output by the 3D imaging system.

[0055] Preferably, the edge detection algorithm in step S2 employs a deep learning method. During model training, it takes labeled hole regions and real object point cloud data as input, outputs gradient calculation values ​​G, and defines a high threshold G. h and low threshold G l The boundaries of the void areas are then selected.

[0056] If G≥G h If so, it is directly determined as the boundary of the void region;

[0057] If G l ≤G≤G h Then, combining the neighborhood density algorithm in step S3, the density distribution of each point in the void region and its neighborhood points is calculated, and the relatively sparse regions are identified as the boundaries of the void region.

[0058] If G≤G l If the model fails to distinguish between real objects and empty areas, it will continue to train the model and adjust the high and low thresholds.

[0059] Preferably, in step S4, it is necessary to combine three-dimensional imaging system calibration technology, using a checkerboard calibration board or a circular calibration board, to determine the intrinsic and extrinsic parameters of the three-dimensional imaging system, and to crop the color image to ensure that the imaging field of view of the color image is consistent with that of the point cloud data.

[0060] Preferably, in step S5, the weights of the point cloud data at each integration time are calculated to synthesize a point cloud dataset with a wide dynamic range. The integration times T1, T2, and T3 are obtained. j …T n Intensity data D1, D2, D j …D n Calculate the linear region of each pixel (m, n) in the dynamic range of the three-dimensional imaging system.

[0061]

[0062] Find the maximum and minimum points in the second derivative, and denote the integration time at each point as T. min and T max That is, the left and right endpoints of the linear region. For example... Figure 2 As shown, the linear region [T] min， T max High dynamic fusion is performed using the internal response function.

[0063] D x =aT x +b

[0064] In the formula T x D is the integration time; x denoted as intensity data; a represents the weight within the linear region; b represents the bias.

[0065] [T min， T max Outside the interval, a nonlinear response function is used for high dynamic fusion.

[0066]

[0067] In the formula, μ is the mean and σ is the standard deviation.

[0068] Preferably, in step S6, the type of radial basis function is selected according to the point cloud density requirements and the distribution characteristics of the cavity region. The outer boundary of the cavity region is selected as a Gaussian function, the interior is selected as a polynomial function, and the cavity region is interpolated and completed.

[0069] like Figure 3 As shown, the central circular block represents the eigenvalue to be determined, and the coordinates of the surrounding blocks are (m...). i n i ), (m i+1 ni ), (m i n i+1 ), (m i+1 n i+1 During the process of filling holes, the corresponding function is used to fill the holes based on the known surrounding feature values ​​and the point cloud density requirements.

[0070] This invention also provides a point cloud hole completion system for a three-dimensional imaging system, comprising:

[0071] Image acquisition module: Acquires point cloud data, color images, and intensity data at different integration times;

[0072] Preprocessing module: Performs statistical analysis on the acquired point cloud dataset using edge detection and neighborhood density algorithms to identify hole regions and boundary features in the point cloud data;

[0073] Data fusion module: Analyzes point cloud data, color images and intensity data at different integration times, obtains response functions in different integration intervals, and uses wide dynamic range to perform data fusion on hole regions;

[0074] Hole Completion Module: Based on the distribution characteristics of hole regions and boundaries, a suitable interpolation function is selected to complete the hole regions and boundaries in the point cloud data.

[0075] Preferably, a point cloud hole completion device for a three-dimensional imaging system includes: a processor and a memory;

[0076] The memory stores an application program that can be executed by the processor, which enables the processor to perform the steps of the point cloud hole completion method of the three-dimensional imaging system.

[0077] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the point cloud hole completion method of the three-dimensional imaging system.

[0078] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for filling holes in point clouds in a three-dimensional imaging system, characterized in that... Includes the following steps: S1, acquire point cloud data P1, color image M1, and intensity data D1 at integration time T1 through a three-dimensional imaging system; Point cloud data P2, color image M2, and intensity data D2 at integration time T2 are acquired using a 3D imaging system. … The integration time T is obtained through a three-dimensional imaging system. n Point cloud data P n Color image M n Intensity data D n ; S2, Select the point cloud data P with the fewest void areas obtained in step S1. i And the selected point cloud data P i Segmentation and identification of spatially discontinuous regions; S3, based on point cloud data P i The point cloud density determination step S2 identifies the boundary density of spatially discontinuous regions, and identifies and marks the point cloud data P. i The boundary of the void area; S4, the color image M obtained in step S1 i The feature points and the point cloud data P marked in step S3 i The boundary of the void region in the data is merged to obtain the merged point cloud data P. i Color information and point cloud information of the boundary of the void region X i ; S5, based on X obtained in step S4 i Extract intensity data D1, D2, D3…D at different integration times. n The corresponding region is then filled in, and the extracted corresponding region is completed; S6. Interpolate the area filled in step S5 to complete the point cloud hole filling of the 3D imaging system.

2. The method for completing point cloud holes in a three-dimensional imaging system according to claim 1, characterized in that: In step S2, an edge detection algorithm is used to process the selected point cloud data P. i Segmentation and identification of spatially discontinuous regions.

3. A method for completing point cloud holes in a three-dimensional imaging system according to claim 1 or 2, characterized in that: In step S2, the method for segmentation and recognition is as follows: a deep learning model is established and the established deep learning model is trained. During the training process, point cloud data of hollow regions with coarse labels and real objects are input, and gradient calculation value G is output. Hollow region boundaries are filtered according to the output gradient calculation value G. Let the high threshold of the output gradient calculation value G be G. h The low threshold is G l ; If G≥G h Then the point cloud data corresponding to the gradient calculation value G will be directly determined as the boundary of the void region; If G l ≤G≤G h The neighborhood density algorithm is used to calculate the density distribution of each point in the cavity region and its neighborhood points. The relatively sparse regions are identified as the boundaries of the cavity region. If G≤G l If the model fails to distinguish between real objects and empty areas, it will continue to train the model and adjust the high and low thresholds.

4. The method for completing point cloud holes in a three-dimensional imaging system according to claim 1, characterized in that: In step S3, the point cloud data P is further identified and labeled using the neighborhood density algorithm. i The boundary of the void area in the middle.

5. The method for completing point cloud holes in a three-dimensional imaging system according to claim 1, characterized in that: In step S4, a checkerboard calibration plate or a circular calibration plate is used to calibrate the three-dimensional imaging system during fusion to determine the intrinsic and extrinsic parameters of the three-dimensional imaging system, and the color image is cropped to ensure that the imaging field of view of the color image is consistent with that of the point cloud data.

6. The method for completing point cloud holes in a three-dimensional imaging system according to claim 1, characterized in that: In step S5, the extracted corresponding region is completed using a wide dynamic range fusion method.

7. A method for completing point cloud holes in a three-dimensional imaging system according to claim 1 or 6, characterized in that: In step S5, the method for completing the extracted corresponding region is as follows: calculate the weight of the point cloud data at each integration time, synthesize a point cloud dataset with a wide dynamic range, and calculate the linear region of each pixel (m, n) in the dynamic range of the three-dimensional imaging system. Where j = 1, 2, 3…n; Find the maximum and minimum points in the second derivative, and denote the integration time at each point as T. min and T max , that is, the left and right endpoints of the linear region; For linear regions [T] min， T max The internal response function is used for high-dynamic fusion, and the fusion result is: D x =aT x +b In the formula, T x D is the integration time; x The data represents intensity; a represents the weight within the linear region; b represents the bias; x = 1, 2, 3…n; For [T] min， T max Outside the interval, a nonlinear response function is used for high-dynamic fusion, and the fusion result is as follows: In the formula, μ is the mean and σ is the standard deviation.

8. The method for completing point cloud holes in a three-dimensional imaging system according to claim 1, characterized in that: In step S6, the region completed in step S5 is interpolated using a radial basis function interpolation algorithm.

9. A method for completing point cloud holes in a three-dimensional imaging system according to claim 1 or 8, characterized in that: In step S6, the interpolation method is as follows: select the type of radial basis function according to the point cloud density requirements and the distribution characteristics of the hole region, select the Gaussian function for the outer boundary of the hole region, select the polynomial function for the interior, and perform interpolation to complete the hole region.

10. A point cloud hole completion system for a three-dimensional imaging system, characterized in that: The completion system includes an image acquisition module, a preprocessing module, a data fusion module, and a hole completion module; The image acquisition module is used to acquire point cloud data, color images, and intensity data at different integration times; The preprocessing module is used to perform statistical analysis on the acquired point cloud dataset using edge detection algorithms and neighborhood density algorithms to identify hole regions and boundary features in the point cloud data. The data fusion module is used to analyze point cloud data, color images and intensity data at different integration times, obtain response functions in different integration intervals, and perform data fusion on the hole region using a wide dynamic range. The hole completion module is used to select an appropriate interpolation function to complete the hole regions and boundaries in the point cloud data based on the distribution characteristics of the hole regions and boundaries.

Images