A method for intelligently extracting the point cloud of stacked construction waste particles

Abstract

The present invention relates to the technical field of intelligent detection of building materials, and particularly relates to a method for intelligently extracting the point cloud of stacked construction waste particles. The method includes the following steps: S1. Obtaining the point cloud data of construction waste particles; S2. Converting the stacked point cloud into a depth image; S3. Using the Waste-Net model to extract the segmentation mask of the particles; S4. Extracting the depth image of completely visible particles; S5. Segmenting and extracting the point cloud of completely visible particles. This method can achieve the rapid and accurate segmentation of the point cloud of completely visible stacked construction waste particles, and effectively filter the point cloud data of incompletely visible particles, solving the problem that traditional segmentation methods cannot be automatically segmented in complex stacked environments.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent detection of building materials, and particularly relates to a method for intelligently extracting the point cloud of stacked muck particles. Background Art

[0002] Figure 1 A muck particle feature detection system based on 3D scanning is shown. The muck particles present a complex stacked shape, which is consistent with the working conditions faced in engineering practical applications. Patent CN119360363A discloses a particle gradation detection method for uniformly dispersing and arranging particles. This method uses a clustering algorithm to segment the particle point cloud data. However, this method is obviously not applicable to the complex stacked working conditions involved in this case. Therefore, it is particularly important to develop a fast extraction algorithm for particle point clouds that can adapt to complex stacked working conditions.

[0003] There has been no relevant research on the segmentation of muck particle point clouds under complex stacked working conditions. Common point cloud segmentation methods include clustering-based algorithms, geometric feature-based methods, and deep learning-based methods. The existing technologies face many challenges, especially in the processing of complex stacking and partially occluded particles.

[0004] (1) The DBSCAN algorithm is a representative algorithm based on clustering and is widely used in the segmentation of uniformly dispersed particles. However, in the complex environment of stacked particles, affected by the adhesion and overlapping occlusion of particle point clouds, the DBSCAN algorithm will have over-segmentation or under-segmentation, and even cannot be segmented. Moreover, the computational complexity is relatively high, making it difficult to meet the efficiency requirements of real-time detection.

[0005] (2) Geometric feature-based segmentation methods (such as region growing and edge detection) can be used for particle boundary extraction. However, when there are overlaps or occlusions between particles, it is difficult to accurately distinguish the boundaries of particles. Especially when the particle shapes are complex or the surfaces are irregular, the segmentation accuracy drops significantly, and the computational complexity is also relatively high.

[0006] (3) Deep learning-based segmentation methods (such as PointNet) can effectively segment point cloud data with obvious features, such as segmenting the point clouds of ground, vehicles, or indoor objects such as tables, walls, and beams. Its principle is to extract the geometric features of the object point cloud and use deep learning algorithms for segmentation, which is a method of directly segmenting point clouds. However, these methods rely on a large amount of labeled data for training. The training process requires a large amount of computing power to process complex point cloud data, with high computational resource requirements and poor real-time performance. On the other hand, the complex stacked particles in this case have complex and variable geometric features and no fixed geometric form. Therefore, deep learning-based segmentation methods are not applicable to the point cloud segmentation of such complex stacked particles.

[0007] The complex stacking situation of muck particles targeted by the present invention brings two major challenges to point cloud segmentation and analysis:

[0008] ① The adhesion and occlusion problems between particles are the core difficulties in the segmentation task. In actual engineering scenarios, muck particles often show a disordered stacking state, with blurred contact surface boundaries, making it difficult for traditional geometric feature-based segmentation methods (such as region growing and watershed algorithms) to accurately distinguish adjacent particles. If the defective point clouds of partially occluded particles are misjudged as complete particles, it will significantly affect the accuracy of subsequent grading analysis. Although existing research has tried to improve clustering algorithms, such as variants of the density-based clustering algorithm DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or introduce morphological processing, their adaptability to complex stacking conditions is still limited.

[0009] ② The balance between algorithm efficiency and accuracy is another major challenge. Engineering applications require that the segmentation algorithm has real-time processing capabilities (such as processing millions of point clouds per second), while existing high-precision methods (such as the voxel-based 3D convolutional network PointNet) often have high computational complexity and are difficult to meet the real-time requirements. At the same time, the diversity of particle shapes (such as pebbles, gravel, flakes, etc.) requires the algorithm to have strong generalization ability, but the actual labeled data is scarce, restricting the performance of supervised learning models. Some studies have tried to alleviate data dependence through transfer learning or weakly supervised methods, but their stability in cross-scene applications still needs to be verified.

[0010] In summary, how to quickly and automatically extract and segment completely visible particle point clouds from complexly stacked particle point clouds to provide an algorithm basis for on-line monitoring of muck particle characteristics in engineering is still an urgent problem to be solved in this field. Summary of the Invention

[0011] The present invention provides a method for intelligently extracting stacked muck particle point clouds, aiming to solve the technical problems of muck particle point cloud segmentation under complex stacking conditions.

[0012] The present invention provides a method for intelligently extracting stacked muck particle point clouds, including the following steps:

[0013] S1. Acquisition of muck particle point cloud data: The original point cloud data of muck particles on the loading surface is scanned by a 3D camera;

[0014] S2. Conversion of stacked point clouds into depth images: Taking the loading surface as the reference plane, generating a particle depth image based on the Z-axis coordinate values of the particle point cloud data;

[0015] S3. The Waste-Net model extracts the segmentation mask of particles: Based on the U-Net network, combined with the Laplacian operator, the particle boundary attention mechanism module, and the convolutional block attention module, the Waste-Net model is constructed to process the input particle depth image, and finally output the segmentation mask of completely visible particles;

[0016] S4. Extraction of the depth image of completely visible particles: Based on the segmentation mask generated by the Waste-Net model, the depth image of completely visible particles is extracted;

[0017] S5. Segmentation and extraction of the point cloud of completely visible particles: Based on the depth image of completely visible particles, the three-dimensional point cloud data of completely visible particles is extracted.

[0018] As a further improvement of the present invention, the step S2 includes:

[0019] S21. Read the particle point cloud in PLY format through the Open3D library, extract the Z coordinates of all particle point clouds, and construct a histogram of Z values;

[0020] S22. By calculating the region with the minimum histogram density, determine the height of the reference plane projected by each point in the point cloud data, and calculate the distance from each point to the reference plane. Finally, obtain the depth value of each point;

[0021] S23. Use the binned_statistic_2d function in the SciPy library to perform two-dimensional binning statistics on the points in the point cloud data, calculate the average distance value in each bin, and generate a depth image;

[0022] S24. Normalize the generated depth image, apply the Viridis colormap of Matplotlib for coloring, map the depth value to the specified interval, generate a color depth image, and save it in image format.

[0023] As a further improvement of the present invention, the step S21 specifically includes:

[0024] Use the Open3D library to read the particle point cloud P in PLY format, extract the Z coordinates of all particle point clouds in P, denoted as the set , evenly divide multiple bins in the Z direction, and calculate the histogram of Z values hist ( Z ), as shown in Equation (1):

[0025] (1)

[0026] where B is the number of bins of the histogram, b i is thei The boundary of a bin.

[0027] As a further improvement of the present invention, step S22 specifically includes:

[0028] Next, find the bin index with the minimum point density, as shown in Equation (2); finally, the height of the reference plane for projection can be calculated by Equation (3); for each point , calculate its distance to the reference plane d i , as shown in Equation (4);

[0029] (2)

[0030] (3)

[0031] (4)

[0032] Wherein, x i represents the coordinate of the i th point on the X-axis, y i represents the coordinate of the i th point on the Y-axis, z i represents the coordinate of the i th point on the Z-axis; hist ( Z ) i represents the number of point clouds in the i th bin in the Z direction; Z base represents the height of the projection reference plane on the Z-axis, and are respectively the two boundaries of the bin with the minimum density.

[0033] As a further improvement of the present invention, in step S3, the Waste-Net model processes the input particle depth image and finally outputs a segmentation mask of the fully visible particles, specifically including:

[0034] S31. Conversion from point cloud to depth image: The collected particle point cloud data is preprocessed and converted into a corresponding depth image. The Z coordinate of each point cloud represents the distance from the point cloud to the reference plane, and each pixel value of the depth image represents the distance between the particle and the reference plane;

[0035] S32. Dataset construction: Pair the particle depth image with the annotation mask of the fully visible particle region to form a training dataset;

[0036] S33. Encoder module processing: The encoder gradually extracts the features of the image through multiple convolutional layers, while reducing the size of the image and increasing the number of channels of the feature map through max-pooling operations;

[0037] S34. Laplacian operator processing: The Laplacian operator performs edge enhancement processing on the input image, highlighting the particle boundary information by calculating the second-order derivative of the pixels;

[0038] S35. Particle boundary attention mechanism module processing: Feature fusion is performed on the output feature map of the encoder and the edge feature map processed by the Laplacian operator;

[0039] S36. CBAM module processing: The CBAM module applies channel and spatial attention mechanisms to the feature map processed by the particle boundary attention mechanism module to generate a weighted image of the important feature regions;

[0040] S37. Skip connection and decoder module processing: The feature map processed by the CBAM module is skip-connected with the low-level feature map of the encoder and enters the decoder module. The decoder module restores the spatial resolution of the image through deconvolution operations and generates the final segmentation mask.

[0041] As a further improvement of the present invention, in step S35, the processing process of the particle boundary attention mechanism module specifically includes:

[0042] S35a. Feature map input: Receive the output feature map from the encoder and the edge feature map processed by the Laplacian operator;

[0043] S35b. Extraction of edge-enhanced features: Obtain the particle depth image information extracted by the encoder;

[0044] S35c. Feature fusion: Multiply the particle depth image information directly with the edge feature map processed by the Laplacian operator element by element to obtain an intermediate process quantity of the edge-enhanced features, and then add the intermediate process quantity of the edge-enhanced features to the particle depth image information element by element to obtain a new intermediate quantity of the feature map.

[0045] As a further improvement of the present invention, in step S36, in the CBAM module, the process of generating the weighted image specifically includes:

[0046] S36a. The channel attention mechanism performs two pooling operations on the input feature map: average pooling and max pooling, generating the average value and the maximum value of the feature map respectively, and extracting global information and significant information;

[0047] S36b. The two pooling results are processed by a 1x1 convolution and the ReLU activation function is used. Each channel is weighted by the Sigmoid activation function to generate a weighted channel feature map;

[0048] S36c. The spatial attention mechanism generates a feature map representing spatial importance through average pooling and max pooling operations on the feature map in the spatial dimension, performs a convolution operation on the feature map representing spatial importance, and uses the Sigmoid activation function for normalization to obtain the weighting coefficients for each spatial position;

[0049] S36d. The CBAM module combines channel attention and spatial attention, applies spatial weighting to the channel-weighted feature map, and generates a weighted image.

[0050] As a further improvement of the present invention, the step S4 includes: obtaining the depth image of completely visible particles through the one-to-one pixel correspondence between the depth image and the segmentation mask;

[0051] The process of establishing the one-to-one pixel correspondence between the depth image and the segmentation mask includes:

[0052] S41. Generation of the depth image: Each point in the point cloud is mapped to the pixel position of the depth image through its (x, y) coordinates, and the z value is used as the pixel value of the depth image;

[0053] S42. Generation of the segmentation mask: The segmentation mask generated by the deep learning model corresponds one-to-one with the pixel positions of the depth image, representing the target area in the image;

[0054] S43. Mapping from pixel to point cloud: The digitize function is used to map the (x, y) coordinates of the point cloud to the pixel positions of the depth image to establish the mapping relationship between the pixel and the point cloud;

[0055] S44. Connection between the segmentation mask and the depth image: Through the mapping relationship between the pixel and the point cloud, the segmentation mask is made to correspond to each pixel value in the depth image.

[0056] As a further improvement of the present invention, the step S5 specifically includes:

[0057] S51. Based on the depth image of completely visible particles, using a preset pixel-point cloud mapping function, each selected pixel point in the image is corresponded to the corresponding position in the three-dimensional point cloud space, and the three-dimensional point cloud data of completely visible particles is extracted through reverse indexing.

[0058] As a further improvement of the present invention, the step S51 specifically includes:

[0059] The pixel-point cloud mapping function is achieved by establishing the correspondence between the pixels in the depth image segmentation mask and the point indices in the point cloud; assuming the scanned particle point cloud , each point p i has coordinates ([[]] x i , y i , z i ). The size of the depth image is n×n , where n is the image height and width; each point in the particle point cloud is assigned to an image pixel, and a functional relationship is established between the X and Y coordinates of the point cloud and the X and Y coordinates of the pixel grid of the image:

[0060] Calculate the X and Y indices of each point in the particle point cloud in the image i x and i y , as shown in Eqs. (5) and (6);

[0061] (5)

[0062] (6)

[0063] digitize is to map x i and y i into predefined bins, X bins and Y bins are the bin boundaries of the point cloud in the X and Y directions respectively;

[0064] Calculate the flat index of each point, convert the 2D index to a 1D index, as shown in Eq. (7); use a mapping function or data structure to store all the points corresponding to each pixel, as shown in Eq. (8);

[0065] (7)

[0066] (8)

[0067] where j is an integer from 0 to n 2 -1, representing each pixel in the depth image, and the set contains the indices of the points in the point cloud; index flat,i is a variable used to represent the 1D index of the image in image processing tasks, iy,i represents the y - coordinate of a pixel in the image, i x,i represents the x - coordinate of a pixel in the image; Mapping pixel to point j represents the corresponding point in three - dimensional space to which the pixel position in the image is mapped.

[0068] The beneficial effects of the present invention are as follows: This method is applicable to the precise segmentation of muck particles and the real - time detection of grading characteristics, and can efficiently handle the problems of stacking, overlapping, and occlusion between particles. The present invention can achieve the rapid and accurate segmentation of the completely visible particle point cloud of stacked muck, and at the same time effectively filter the particle point cloud data that is not completely visible, solving the problem that traditional segmentation methods cannot be automatically segmented in a complex stacking environment. This method can also be widely applied to other fields, such as mineral particle screening, building sand and gravel detection, agricultural particle fertilizer grading, etc. Its intelligent segmentation ability can meet the needs of different industries in the segmentation of stacked particle point clouds, especially suitable for large - scale point cloud data processing and real - time monitoring at the engineering site, and has significant engineering application value. Brief Description of the Drawings

[0069] Figure 1 is a schematic diagram of the application background of the method for intelligently extracting the stacked muck particle point cloud of the present invention;

[0070] Figure 2 is a schematic diagram of the automatic segmentation process of the complex stacked particle point cloud in the present invention;

[0071] Figure 3 is a flowchart of the method for intelligently extracting the stacked muck particle point cloud of the present invention;

[0072] Figure 4 is a comparison diagram of the particle point cloud and the depth image of the present invention;

[0073] Figure 5 is a structural diagram of the Waste - Net network model in the present invention;

[0074] Figure 6 is a structural diagram of the particle boundary attention mechanism (PEA) module in the present invention;

[0075] Figure 7 is a comparison diagram of the particle depth image, the Waste - Net segmentation mask, and the depth image after segmentation in the present invention;

[0076] Figure 8 is a comparison diagram of the extraction of the completely visible particle point cloud in the present invention;

[0077] Figure 9 is a schematic diagram of the processing of the feature map by element - by - element addition in the present invention. Detailed Embodiments​

[0078] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0079] Combined with Figures 2 to 3 As shown, a method for intelligently extracting the point cloud of stacked construction waste particles of the present invention includes the following steps:

[0080] S1. Acquisition of construction waste particle point cloud data: The original point cloud data of the construction waste particles on the loading surface is scanned by a 3D camera.

[0081] S2. Conversion of the stacked point cloud into a depth image: Taking the loading surface as the reference plane, a particle depth image is generated based on the Z-axis coordinate values of the particle point cloud data.

[0082] The core of the method for acquiring construction waste particle point cloud data is to calculate the Z coordinate of each point in the point cloud and project the point cloud according to the Z value. Specifically, it includes:

[0083] S21. Read the particle point cloud in PLY format through the Open3D library, extract the Z coordinates therein, and construct a histogram of Z values.

[0084] S22. By calculating the region with the minimum histogram density, determine the height of the projection reference plane, and calculate the distance from each point to the reference plane, and finally obtain the depth value of each point.

[0085] S23. Use the binned_statistic_2d function in the SciPy library for two-dimensional binning statistics to generate a depth image.

[0086] S24. Normalize the generated depth image, apply the Viridis color map for coloring, map the depth value to the interval [0, 1], generate a color depth image, and save it in PNG format for subsequent analysis and processing.

[0087] The advantage of this method lies in its high automation and batch processing capabilities, which can provide high-quality depth images for subsequent image segmentation and particle analysis.

[0088] In the acquisition of construction waste particle point cloud data, the Python programming language and related libraries (such as Open3D, NumPy, Matplotlib, and SciPy) are used to implement the reading, processing, and conversion of point cloud data.

[0089] The depth image is formed by projecting the particle point cloud onto a reference plane. First, it is necessary to calculate and determine the projection reference plane. Specifically, use the Open3D library to read the particle point cloud P in PLY format, extract the Z coordinates of all the point clouds in P, and denote it as the set Uniformly divide 50 bins in the Z direction and calculate the histogram of the Z values. hist ( Z ) as shown in Equation (1).

[0090] (1)

[0091] where B is the number of bins of the histogram, which is set to 50 here, b i is the i th bin boundary.

[0092] Then find the bin index with the minimum point density as shown in Equation (2). The height of the reference plane for the final projection can be calculated by Equation (3). Finally, for each point , calculate its distance d i to the reference plane as shown in Equation (4).

[0093] (2)

[0094] (3)

[0095] (4)

[0096] In the formulas, x i , y i and z i represent the coordinates of each point in the point cloud: x i represents the coordinate of the i th point on the X-axis, y i represents the coordinate of the i th point on the Y-axis, z i represents the coordinate of the i th point on the Z-axis.

[0097] hist ( Z ) i represents the number of points in the i th bin in the Z direction, that is, the number of points falling within the range of the i th bin in the Z direction. hist ( Z ) i reflects the density of points in this interval. By counting the number of points in each bin, the region with the minimum density can be further calculated to determine the position of the reference plane.

[0098] Z base Represents the height of the projection reference plane on the Z-axis. and [[ID=�]] are the two boundaries of the bin with the lowest density respectively. The height of this reference plane is determined by finding the region with the lowest density, so Z base represents the middle height of this region. It is the reference height for projection calculation.

[0099] After determining the projection reference plane, the present invention uses the binned_statistic_2d function in the SciPy library to perform two-dimensional binning statistics on the points in the point cloud, calculate the average distance value in each bin, and thus generate a two-dimensional distance image, that is, a depth image. Then, the generated depth image is normalized and colored using the Viridis colormap of Matplotlib to visually represent different distance values. During the normalization process, the values of the depth image are mapped to the interval [0, 1], and then multiplied by 255 to be converted to the uint8 type for easy image storage and display. The purpose of the above process is to convert the point cloud data into a depth image. The following is a brief description of the four main steps:

[0100] (1) Calculate the average depth of each region: First, each point in the point cloud is divided into different regions (bins) according to their x and y coordinates. Then, the average value of the z values (i.e., depth) of all points in each region is calculated.

[0101] (2) Normalize the depth values: Then, the depth values of each region are standardized, and their range is adjusted to between 0 and 1, so that all depth values are on the same scale for easy processing and display.

[0102] (3) Adjust the depth values to a suitable range: Then, the standardized depth values are converted to integers from 0 to 255, so that they can be adapted to common image formats (such as PNG or JPEG) for storage and display.

[0103] (4) Color the depth values: Finally, use the Viridis colormap to map each depth value to a different color, from dark blue to bright yellow, so that the depth image can not only display depth information, but also visually express different depth levels through colors.

[0104] Finally, the generated color depth image is saved in PNG format for subsequent image processing and analysis. The advantage of this method lies in its high degree of automation, which can process multiple point cloud files in batches and generate high-quality depth images. In addition, through distance calculation and normalization processing, this method can intuitively represent the positions at different heights in the point cloud, resulting in obvious color changes between the depth images of different particle point clouds, creating conditions for accurate subsequent image segmentation. Figure 4 shows the particle point cloud and the resulting depth image after transformation. Figure 4 (a) in it represents the particle point cloud. Figure 4 (b) in it represents the particle depth image. It can be clearly seen that compared with the particle point cloud, the depth image of the particles forms an obvious color transition at the junction of the particles, which creates conditions for automatic segmentation.

[0105] S3. The Waste-Net model extracts the segmentation mask of the particles: Based on the U-Net network, combined with the Laplacian operator, the particle boundary attention mechanism module, and the convolutional block attention module, the Waste-Net model is constructed to process the input particle depth image and finally output the segmentation mask of the completely visible particles.

[0106] Overview of the Waste-Net network: After the generation of the particle depth image, the present invention further proposes an intelligent segmentation mask generation method based on the U-Net model. This generation method automatically extracts the segmentation mask of the particles through the deep learning model Waste-Net. Waste-Net is an optimized version based on the classic U-Net architecture, introducing the Laplacian operator and the convolutional block attention module (CBAM) to enhance the accuracy of the model in segmenting particles. After training, Waste-Net can process the input depth image, accurately distinguish completely visible particles, extract deep features through convolutional layers and pooling layers, and restore the spatial resolution through upsampling and skip connections in the decoding stage, thereby generating a segmentation mask with pixel-level accuracy. Through this method, the boundaries of the particles can be accurately extracted, providing a precise segmentation result for subsequent analysis of the grading characteristics of the particles. The innovation of this method lies in automatically completing particle segmentation through deep learning technology, significantly improving the segmentation efficiency and accuracy.

[0107] As Figure 5 shown, the Waste-Net model uses the U-Net network as the basic architecture and enhances it. Its module framework is as follows:

[0108] (1) U-Net basic architecture:

[0109] Encoder: The encoder part of Waste-Net uses the classic U-Net architecture, including multiple convolutional layers and pooling operations, which gradually extract image features and reduce the image size.

[0110] Bottleneck: At the end of the encoder, it enters the bottleneck layer, which further processes the extracted high-level features and connects to the decoder module.

[0111] Decoder: Through transposed convolution operations, it gradually restores the spatial resolution of the image and finally restores it to the original image size.

[0112] Skip Connections: A key feature of the U-Net architecture, skip connections fuse the low-level features of the encoder with the high-level features of the decoder, which helps to refine edge information and details.

[0113] (2) Laplacian Operator:

[0114] The Laplacian operator is used in image processing to detect the edges of an image by calculating the second derivative of pixels to highlight the edge information in the image. In Waste-Net, the Laplacian operator is used to enhance the edge features in the image, especially the boundaries of particles, enabling the model to better capture the subtle differences and details between particles.

[0115] In the two-dimensional case, the Laplacian operator is defined as the sum of the second derivatives of the neighborhood around each pixel point in the image, as shown in the formula where, is the pixel intensity value of the image, and are the second derivatives of this point in the horizontal and vertical directions respectively.

[0116] By convolving the image, the Laplacian operator helps to accelerate edge detection and improve the attention of the segmentation model to the detailed parts.

[0117] (3) PEA Module (Particle Edge Attention Mechanism Module):

[0118] As Figure 6 shown, the PEA (Particle Edge attention) module is an innovative module in Waste-Net, mainly used to enhance the ability to capture details of particle edges. The working process of the PEA module is as follows:

[0119] Combination of Input Feature Map and Laplacian Feature Map: The PEA module receives the output feature map from the encoder and the edge feature map processed by the Laplacian operator. The Laplacian operator enhances the edge information of the image, especially the boundary part of the particles.

[0120] Extraction of Edge-Enhanced Features: By combining the Laplacian feature map with the output feature map of the encoder, the PEA module emphasizes the particle edge features, which helps to improve the segmentation accuracy, especially in the details.

[0121] Feature Fusion: The combined feature map is processed by element-wise multiplication and element-wise addition to fuse the detailed information of the edges and particles.

[0122] As Figure 9 shown, the picture stored in the computer can be represented as Figure 9 the matrix shown, where each square represents a pixel point, and the value of the pixel point corresponds to the corresponding color. In the computer, the value range of the color is usually set to (0, 255) to represent different color intensities. The process of element-wise addition is Figure 9 the process shown, where the elements at each position are added one by one. The same applies to element-wise multiplication.

[0123] Further Optimization by CBAM: Finally, the output feature map of the PEA module is input into the CBAM module to further optimize the feature map and enhance the attention to the key regions of the particles.

[0124] (4) CBAM (Convolutional Block Attention Module):

[0125] Channel Attention: Through adaptive pooling operations, CBAM first captures the global context information and uses a multi-layer perceptron (MLP) to establish the relationship between channels, giving more attention weights to the important channels in the network.

[0126] Spatial Attention: Based on the channel attention, CBAM further analyzes in the spatial dimension and uses convolutional operations to generate a spatial mask, thereby highlighting the key regions in the image.

[0127] The CBAM module helps the model to pay more attention to the key parts of the particles (such as the edges and the regions with more prominent features), thereby improving the segmentation performance, especially in the segmentation effect of the particle boundaries.

[0128] The working process of Waste-Net can be divided into the following main steps:

[0129] (1)Input data processing:

[0130] S31. Conversion from point cloud to depth image: The particle point cloud data collected from the actual scene is preprocessed and converted into the corresponding depth image. The Z coordinate of each point cloud represents its distance to the reference plane, and each pixel value of the depth image represents the distance between the particle and the reference plane.

[0131] S32. Dataset construction: Pair the particle depth image with the corresponding annotation mask (the area representing the completely visible particles) to form the training dataset. Manually annotate the particle mask using tools such as Labelme, and apply data augmentation techniques (such as brightness and contrast adjustment) to expand the training data.

[0132] (2)Feature extraction and fusion:

[0133] S33. Encoder module: The encoder part gradually extracts the features of the image through multiple convolutional layers, and at the same time reduces the size of the image through max-pooling operations and increases the number of channels of the feature map.

[0134] S34. Laplacian operator processing: The Laplacian operator performs edge enhancement processing on the input image to highlight the particle boundary information. This step helps the model better identify the details in the image.

[0135] S35. Particle boundary attention mechanism module processing: Feature fusion is performed on the output feature map of the encoder and the edge feature map processed by the Laplacian operator;

[0136] S35a. Feature map input: Receive the output feature map from the encoder and the edge feature map processed by the Laplacian operator;

[0137] S35b. Extraction of edge-enhanced features: Obtain the particle depth image information extracted by the encoder H i × W i × N i ;

[0138] S35c. Feature fusion: Multiply the particle depth image information H i × W i × N i directly with the edge feature map processed by the Laplacian operator element-wise to obtain the intermediate process quantity of the edge-enhanced feature, and then multiply this intermediate process quantity of the edge-enhanced feature with the particle depth image information H i × W i ×N i Perform element-wise addition to obtain a new intermediate feature map. Through element-wise multiplication and element-wise addition operations, fuse the detailed information of the edges and particles. Combine the Laplacian feature map with the output feature map of the encoder, and emphasize the particle edge features through the PEA module.

[0139] S36. CBAM Module: The CBAM module generates a weighted image of important feature regions by applying channel and spatial attention mechanisms to the new intermediate feature map. H i1 × W i1 × N i1 , enhancing the boundary and detailed features of the particles.

[0140] In the CBAM module, the generation of the weighted image is first processed through the channel attention mechanism. The channel attention mechanism performs two pooling operations on the input feature map: average pooling and max pooling, generating the average value and the maximum value of the feature map respectively, so as to extract global information and significant information. Then, these two pooling results are processed through 1x1 convolution, and the ReLU activation function is used. Finally, each channel is weighted through the Sigmoid activation function to generate the weighted channel feature map. Then, the spatial attention mechanism processes the weighted channel feature map. The spatial attention mechanism performs average pooling and max pooling operations on the feature map in the spatial dimension to generate a feature map representing spatial importance, and then performs convolution operations on it. Finally, it is normalized using the Sigmoid activation function to obtain the weighted coefficients for each spatial position. The CBAM module combines channel attention and spatial attention, and applies spatial weighting to the channel-weighted feature map to generate a weighted image. This weighted image can highlight the important regions in the image, especially the boundary and detailed features of the particles, helping the model to better perform image segmentation and improve the segmentation accuracy.

[0141] S37. Skip Connection and Decoder Module: The feature map processed by the CBAM module is skip-connected with the low-level feature map of the encoder and enters the decoder module. The decoder restores the spatial resolution of the image through deconvolution operations and generates the final segmentation mask.

[0142] In the Waste-Net model, the innovations of the skip connection and decoder module are as follows:

[0143] ①Improvement of skip connections: In the traditional U-Net architecture, skip connections directly transfer the low-level features of the encoder layer to the corresponding layer of the decoder. In this Waste-Net model, however, the low-level feature maps are processed by the CBAM module before skip connections are made. The CBAM module enhances the key information in the image by calculating the channel and spatial attention of the image, enabling the decoder to focus more on important regions, especially the boundary parts of the particles, when restoring the image.

[0144] ②Processing in the decoder: In the processing of the decoder, traditional transposed convolution operations are used for upsampling to restore the spatial resolution of the image. The improvement of this Waste-Net model lies in that the decoder not only receives the low-level features from the encoder, but also uses the feature maps processed by CBAM, and combines them with the upsampled image for splicing, thereby improving the accuracy of restoration. In addition, the decoder introduces the Laplacian image, and performs upsampling and element-wise multiplication operations on it, further enhancing the segmentation accuracy, especially in the particle boundary parts.

[0145] ③Differences from traditional encoders: In the traditional encoder-decoder architecture, the decoder only relies on transposed convolution operations to restore the spatial resolution of the image and directly transfers the low-level features through skip connections. In this Waste-Net model, however, the low-level features are processed by the CBAM module before being transferred, and combined with the Laplacian image, so as to better utilize the edge information in the decoder and improve the segmentation accuracy.

[0146] (3)Output of the segmentation result:

[0147] Generation of the segmentation mask: The output of Waste-Net is a segmentation mask image of the same size as the input image, where the regions with predicted values of 1 represent the particle regions and 0 represents the background regions.

[0148] Enhancement of boundary details: Through the combined action of the Laplacian operator and the CBAM module, Waste-Net can more accurately segment the boundary and detail regions of the particles, improving the segmentation accuracy.

[0149] S4. Extraction of the depth image of completely visible particles: Based on the segmentation mask generated by the Waste-Net model, the depth image of completely visible particles is extracted.

[0150] Waste-Net generates segmentation masks for the particle depth images, and these masks represent the position information of the fully visible particles recognized by Waste-Net. It should be noted that Waste-Net reads the depth image and outputs a binary (black and white) mask image. The positions in these mask images correspond to the particles at the same positions in the depth image, indicating that the particles at these positions are fully visible. Through the one-to-one pixel correspondence between the depth image and the segmentation mask, the depth image of the fully visible particles can be quickly obtained. This process is shown clearly in Figure 7 as follows. Figure 7 In (a) of Figure 7 is the particle depth image, Figure 7 in (b) of

[0151] is the segmentation mask generated by Waste-Net,

[0152] and in (c) of

[0153] is the depth image extracted from the segmentation mask.

[0154] Correspondence between the depth image and the segmentation mask: The segmentation mask corresponds to the pixel positions in the depth image, which is the classification information of each pixel. Each pixel value (0 or 1) in the mask indicates whether the pixel belongs to the target area (e.g., particles). Through the mapping relationship from pixels to point clouds, each pixel position in the depth image can be accurately mapped to the corresponding position in the point cloud, thus ensuring the one-to-one correspondence between the depth image and the segmentation mask.

[0155] The one-to-one pixel correspondence between the depth image and the segmentation mask is mainly established through the following steps:

[0153] S41. Generation of the depth image: Each point in the point cloud is mapped to the pixel position of the depth image through its (x, y) coordinates, and the z value is used as the pixel value of the depth image.

[0154] S42. Generation of the segmentation mask: The segmentation mask generated by the deep learning model corresponds one-to-one with the pixel positions of the depth image, representing the target area in the image.

[0155] S43. Mapping from pixels to point clouds: The digitize function is used to map the (x, y) coordinates of the point cloud to the pixel positions of the depth image, establishing an accurate mapping relationship between pixels and point clouds.

[0156] S44. Connection between the mask and the depth image: Finally, through these mapping relationships, each pixel value in the segmentation mask is corresponded to that in the depth image, realizing accurate segmentation and analysis.

[0157] In this way, the one-to-one pixel correspondence between the depth image and the segmentation mask ensures the accuracy and effectiveness of the particle point cloud segmentation.

[0158] S5. Complete Visible Particle Point Cloud Segmentation and Extraction: Extract the three-dimensional point cloud data of completely visible particles based on the depth image of completely visible particles.

[0159] The segmentation mask of completely visible particles generated by Waste-Net can be used to extract the corresponding particle depth image. Based on this depth image, the three-dimensional point cloud of completely visible particles can be quickly extracted through reverse indexing. Specifically, using a preset pixel-point cloud mapping function, each selected pixel point in the image is mapped to the corresponding position in the three-dimensional point cloud space. Through this mapping relationship, the three-dimensional point cloud set corresponding to each completely visible particle can be accurately located.

[0160] In this way, the model accurately extracts the point cloud data of these particles in the three-dimensional space based on the segmentation result of the two-dimensional depth image, providing support for subsequent three-dimensional particle analysis, pose estimation, and other spatial applications. This reverse indexing method based on the segmentation mask and mapping relationship makes the extraction of three-dimensional information of particles more efficient and accurate.

[0161] To clearly illustrate this process, the concept of pixel-to-point mapping is introduced, which aims to establish the correspondence between the pixels in the depth image segmentation mask and the point indices in the point cloud.

[0162] Suppose the scanned particle point cloud , each point p i has coordinates ( x i , y i , z i ), and the size of the depth image is n×n , where n is the image height and width. Each point of the particle point cloud is assigned to an image pixel, thus establishing a functional relationship between the X and Y coordinates of the point cloud and the X and Y coordinates of the image pixel grid.

[0163] Specifically, first, the X and Y indices i x and i y of each point in the particle point cloud need to be calculated, as shown in equations (5) and (6).

[0164] (5)

[0165] (6)

[0166] Here, digitize is to x i andy i are mapped into predefined bins, X bins and Y bins are the bin boundaries of the point cloud in the X and Y directions respectively.

[0167] Next is to calculate the flat index of each point, that is, convert the 2D index into a 1D index to adapt to the storage and calculation methods of the computer, as shown in Equation (7). Finally, a mapping function or data structure is used to store all the points corresponding to each pixel. In this study, the defaultdict data structure in Python is used for storage, so that the corresponding point cloud can be quickly retrieved according to the image pixels, as shown in Equation (8).

[0168] (7)

[0169] (8)

[0170] In the formula, j is an integer from 0 to n 2 -1, representing each pixel in the depth image, and the set is the index of the points in the point cloud.

[0171] index flat,i is a variable used to represent the 1D index of an image in some image processing tasks. It converts the coordinates of a 2D image into a 1D index, so that each pixel of the image can be represented by a single scalar value. In some tasks, especially when dealing with the pixel positions in an image, using index flat,i to avoid the complexity of dealing with 2D matrices. Specifically in the formula, i usually represents the index of the current pixel in the 1D array, used to find the corresponding pixel in the 1D array representation. i y,i represents the y coordinate (row index) of the pixel in the image; i x,i represents the x coordinate (column index) of the pixel in the image.

[0172] Mapping pixel to point j represents the mapping of the pixel position in the image to the corresponding point in 3D space. Mapping pixel to point j means mapping the pixel in the image ( pixel ) to the point in 3D space ( point ). Here, point j ​​​] usually represents the first j Specifically, each pixel in the image represents a location or point in space, and the mapping process converts the two-dimensional pixel coordinates of the image ( i x , i y ) is converted into the corresponding three-dimensional space coordinate points.

[0173] This mapping often appears in 3D reconstruction or depth image processing tasks, where each pixel in the image has a corresponding 3D spatial position. Through this mapping, the pixels in the 2D image can be mapped one-to-one to points in the real space, thereby obtaining a point cloud or depth information in space.

[0174] Through the above method, this study established a pixel mapping relationship between particle point cloud and depth image. This mapping relationship enables the rapid finding of corresponding point cloud according to the pixel position of depth image, thus quickly completing the point cloud segmentation task. This process not only improves the efficiency of segmentation, but also provides accurate segmentation results for subsequent particle grading detection and analysis, providing important technical support for resource utilization and particle grading detection. This process is Figure 8 Described in detail in Figure 8 (a) is the original point cloud, Figure 8 (b) in the figure is the depth image of the fully visible particles extracted by the Waste-Net segmentation mask. Figure 8 (c) represents the point cloud data for fully visible particles. The input raw point cloud is converted into a depth image, which is then processed in step S4 to obtain a depth image of the fully visible particles. Finally, by mapping the depth image pixels to the particle point cloud, the point cloud data for the fully visible particles is rapidly extracted. This completes the intelligent and automated extraction of complex stacked particle point clouds. This entire process does not involve complex computational segmentation of 3D point cloud data; it relies solely on a simple method, such as projection, image segmentation, and pixel-to-point cloud mapping, to rapidly segment the fully visible particle point cloud.

[0175] The overall innovation of the method for intelligently extracting point clouds of stacked slag particles in the present invention lies in:

[0176] The innovation of this invention lies in significantly improving the efficiency and image quality of converting 3D particle point clouds to depth images through automated reference plane calculation, efficient 2D binning statistics, and image normalization and colorization. This conversion method also offers significant advantages for particle image segmentation, resulting in more distinct boundaries between particles, providing a robust data foundation for image segmentation models.

[0177] ② The core innovation of the present invention is the Waste-Net network architecture, which introduces the Laplacian operator and the Convolutional Block Attention Module (CBAM) on the basis of the classical U-Net, greatly improving the accuracy of particulate depth image segmentation, especially in the processing of particulate boundaries and details. Although the commonly used U-Net architecture in the prior art is widely applied in image segmentation, it is often difficult to handle complex boundaries and details in particulate image segmentation. The Laplacian operator enhances the edge features of the image, especially the details of particulate boundaries, thus significantly improving the accuracy of the model in boundary processing. At the same time, the CBAM module helps the model focus on the key regions of the particles through channel and spatial attention mechanisms, further improving the segmentation accuracy.

[0178] ③ The present invention provides an automated and efficient registration mechanism through the segmentation mask generated by Waste-Net and the precise mapping relationship between the depth image and the segmentation mask. Different from traditional methods that rely on manually set parameters or complex image processing steps, the present invention can accurately map the positions of fully visible particles in the segmentation mask image to the depth image through an optimized mapping function. This mechanism ensures that the model can maintain high accuracy and stability when dealing with different particulate morphologies and environmental conditions.

[0179] ④ The precise mapping method from the depth image to the point cloud of the present invention uses a preset pixel-point cloud mapping function to correspond each selected pixel point in the image to the corresponding position in the three-dimensional point cloud space. Compared with the threshold segmentation based on the depth image and the point cloud registration based on the Iterative Closest Point (ICP) algorithm, it provides higher accuracy, stronger robustness and higher automation. Especially in the case of complex particulate morphologies and occlusions, it can ensure the accurate extraction and annotation of the point cloud of each particle. This innovation solves the problems of insufficient accuracy, inaccurate segmentation and high computational complexity existing in traditional methods, and significantly improves the quality and processing efficiency of particulate point cloud data.

[0180] The present invention has the following remarkable advantages:

[0181] High efficiency and automation: The present invention significantly reduces manual intervention and improves the efficiency of particulate point cloud processing through a highly automated workflow. The conversion from the point cloud to the depth image, the generation of the particulate segmentation mask and the final particulate analysis process are all completed automatically, and multiple point cloud files can be processed batch by batch. This automated process not only improves the processing speed, reduces human errors, but also ensures the consistency and standardization of the processing results, greatly improving the overall work efficiency, and is especially suitable for application scenarios that require a large amount of data processing and real-time detection.

[0182] High efficiency: The particle segmentation method provided by the present invention has significant advantages in processing speed and efficiency. By optimizing the point cloud conversion, segmentation mask generation, and particle segmentation steps, the entire processing process is accelerated, enabling efficient processing of large-scale data, especially suitable for real-time detection requirements such as engineering sites. The Waste-Net model demonstrates excellent real-time performance in processing particle segmentation tasks, ensuring the high efficiency and low latency of the particle segmentation process, thus meeting the requirements of rapid change and large-scale data processing.

[0183] High segmentation accuracy: Through the Waste-Net deep learning model of the present invention, pixel-level accuracy is achieved in particle segmentation. Combining the optimization of the Laplacian operator and the convolutional block attention module (CBAM), the model can accurately extract the particle boundaries and effectively segment fully visible particles. Through the fine processing of depth images, high-precision particle segmentation is ensured, providing a reliable basis for subsequent particle gradation analysis, quality calculation, etc., and ensuring the accuracy and reliability of the overall analysis results.

[0184] Wide applicability: The method of the present invention is not only applicable to common particle analysis tasks such as muck particles and construction sand and gravel, but can also be widely applied to other industries such as mineral particle screening, agricultural particle fertilizer grading, and food particle screening. Through the strong adaptability of the Waste-Net model and the accuracy of particle segmentation, the present invention can process particles of various shapes, sizes, and densities, with excellent generalization ability. Whether in industry, agriculture, or other fields, the present invention can achieve accurate particle analysis, meet the needs of different industries, and provide a flexible and efficient particle segmentation solution.

[0185] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. An intelligent method for extracting the point cloud of stacked muck particles, characterized in that, It includes the following steps: S1. Acquisition of muck particle point cloud data: The original point cloud data of muck particles on the loading surface is scanned by a 3D camera; S2. Conversion of stacked point cloud into depth image: Taking the loading surface as the reference plane, generating a particle depth image based on the Z-axis coordinate values of the particle point cloud data; S3. Extracting the segmentation mask of particles by the Waste-Net model: Based on the U-Net network, constructing the Waste-Net model by combining the Laplacian operator, the particle boundary attention mechanism module, and the convolutional block attention module, processing the input particle depth image, and finally outputting the segmentation mask of completely visible particles; S4. Extracting the depth image of completely visible particles: Based on the segmentation mask generated by the Waste-Net model, extracting the depth image of completely visible particles; S5. Segmenting and extracting the point cloud of completely visible particles: Extracting the three-dimensional point cloud data of completely visible particles based on the depth image of completely visible particles.

2. The method for intelligently extracting the point cloud of stacked muck particles according to claim 1, wherein The step S2 includes: S21. Reading the particle point cloud in PLY format through the Open3D library, extracting the Z coordinates of all particle point clouds, and constructing a histogram of Z values; S22. Determining the height of the reference plane projected by each point in the point cloud data by calculating the region with the minimum histogram density, calculating the distance from each point to the reference plane, and finally obtaining the depth value of each point; S23. Using the binned_statistic_2d function in the SciPy library to perform two-dimensional binning statistics on the points in the point cloud data, calculating the average distance value in each bin, and generating a depth image; S24. Normalizing the generated depth image, coloring it with the Viridis colormap of Matplotlib, mapping the depth values to the specified interval, generating a color depth image, and saving it in image format.

3. The method for intelligently extracting the point cloud of stacked muck particles according to claim 2, wherein, The step S21 specifically includes: Read the particle point cloud P in PLY format using the Open3D library, extract the Z coordinates of all particle point clouds in P, and denote them as a set , evenly divide multiple bins in the Z direction, and calculate the histogram of Z values hist ( Z ), as shown in Equation (1): (1) where B is the number of bins of the histogram, b i is the i boundary of the 4. The method for intelligently extracting the point cloud of stacked muck particles according to claim 3, wherein The step S22 specifically includes: Next, find the bin index with the minimum point density, as shown in Equation (2); finally, the height of the reference plane for projection is calculated by Equation (3); for each point , calculate its distance to the reference plane d i , as shown in Equation (4); (2) (3) (4) Among them, x i represents the coordinate of the i -th point on the X-axis, y i represents the coordinate of the i -th point on the Y-axis, z i represents the coordinate of the i -th point on the Z-axis; hist ( Z ) i represents the number of point clouds in the i -th bin in the Z direction; Z base represents the height of the projection reference plane on the Z-axis, and are the two boundaries of the bin with the minimum density respectively.

5. The method for intelligently extracting the point cloud of stacked muck particles according to claim 1, wherein, In the step S3, the Waste-Net model processes the input particle depth image and finally outputs the segmentation mask of completely visible particles, which specifically includes: S31. Conversion from point cloud to depth image: Preprocessing the collected particle point cloud data and converting it into the corresponding depth image. The Z coordinate of each point cloud represents the distance from the point cloud to the reference plane, and each pixel value of the depth image represents the distance between the particle and the reference plane; S32. Dataset construction: Pairing the particle depth image with the annotation mask of the completely visible particle region to form a training dataset; S33. Encoder module processing: The encoder gradually extracts the features of the image through multiple convolutional layers, and at the same time reduces the size of the image through max-pooling operations and increases the number of channels of the feature map; S34. Laplacian operator processing: The Laplacian operator performs edge enhancement processing on the input image, and highlights the particle boundary information by calculating the second-order derivative of the pixels; S35. Processing of the particle boundary attention mechanism module: Feature fusion of the output feature map of the encoder and the edge feature map processed by the Laplacian operator; S36. CBAM Module Processing: The CBAM module applies channel and spatial attention mechanisms to the feature map processed by the granular boundary attention mechanism module to generate a weighted image of the important feature region; S37. Skip Connection and Decoder Module Processing: The feature map processed by the CBAM module is skip-connected to the low-level feature map of the encoder and enters the decoder module. The decoder module restores the spatial resolution of the image through deconvolution operations and generates the final segmentation mask.

6. The method for intelligently extracting the point cloud of stacked muck particles according to claim 5, characterized in that, In the step S35, the processing process of the granular boundary attention mechanism module specifically includes: S35a. Feature Map Input: Receive the output feature map from the encoder and the edge feature map processed by the Laplacian operator; S35b. Extraction of Edge-Enhanced Features: Obtain the granular depth image information extracted by the encoder; S35c. Feature Fusion: Multiply the granular depth image information element-wise directly with the edge feature map processed by the Laplacian operator to obtain an intermediate process quantity of the edge-enhanced features. Then, add this intermediate process quantity of the edge-enhanced features to the granular depth image information element-wise to obtain a new intermediate quantity of the feature map.

7. The method for intelligently extracting the point cloud of stacked muck particles according to claim 5, wherein In the step S36, in the CBAM module, the process of generating the weighted image specifically includes: S36a. The channel attention mechanism performs two pooling operations on the input feature map: average pooling and max pooling, respectively generating the average value and the maximum value of the feature map, and extracting global information and significant information; S36b. The two pooling results are processed by a 1x1 convolution, and the ReLU activation function is used. Each channel is weighted by the Sigmoid activation function to generate a weighted channel feature map; S36c. The spatial attention mechanism performs average pooling and max pooling operations on the feature map in the spatial dimension to generate a feature map representing spatial importance. Convolution operations are performed on the feature map representing spatial importance, and normalization is performed using the Sigmoid activation function to obtain the weighted coefficient for each spatial position; S36d. The CBAM module combines channel attention and spatial attention, and applies spatial weighting to the channel-weighted feature map to generate a weighted image.

8. The method for intelligently extracting the point cloud of stacked muck particles according to claim 1, wherein The step S4 includes: Obtaining the depth image of completely visible particles through the one-to-one pixel correspondence between the depth image and the segmentation mask; Among them, the establishment process of the one-to-one pixel correspondence between the depth image and the segmentation mask includes: S41. Generation of the Depth Image: Each point in the point cloud is mapped to the pixel position of the depth image through its (x, y) coordinates, and the z value is used as the pixel value of the depth image; S42. Generation of the Segmentation Mask: The segmentation mask generated by the deep learning model corresponds one-to-one to the pixel positions of the depth image, representing the target region in the image; S43. Pixel-to-Point Cloud Mapping: Use the digitize function to map the (x, y) coordinates of the point cloud to the pixel positions of the depth image to establish the mapping relationship between pixels and the point cloud; S44. Connection between the Segmentation Mask and the Depth Image: Through the mapping relationship between pixels and the point cloud, the segmentation mask is corresponding to each pixel value in the depth image.

9. The method for intelligently extracting the point cloud of stacked muck particles according to claim 1, wherein, The step S5 specifically includes: S51. Based on the depth image of completely visible particles, using a preset pixel-point cloud mapping function, map each selected pixel point in the image to the corresponding position in the three-dimensional point cloud space, and extract the three-dimensional point cloud data of the completely visible particles through reverse indexing.

10. The method for intelligently extracting the stacked muck particle point cloud according to claim 9, wherein The specific steps of step S51 include: The pixel-point cloud mapping function is realized by establishing the correspondence between the pixels in the depth image segmentation mask and the point indices in the point cloud; assuming the scanned particle point cloud , each point p i has coordinates ([[]] x i , y i , z i ). The size of the depth image is n×n , where n is the image height and width; each point of the particle point cloud is assigned to an image pixel, and a functional relationship is established between the X and Y coordinates of the point cloud and the X and Y coordinates of the pixel grid of the image: Calculate the x and y coordinates of each point in the particle point cloud in pixels in the image i x,i and i y,i , as shown in Eqs. (5) and (6); (5) (6) digitize is to map x i and y i into predefined bins, X bins and Y bins which are the bin boundaries of the point cloud in the X and Y directions respectively; Calculate the flat index of each point, convert the 2D index to a one-dimensional index, as shown in Equation (7); use a mapping function or data structure to store all points corresponding to each pixel, as shown in Equation (8); (7) (8) Among them, j ranges from 0 to n 2 -1, representing each pixel in the depth image, and the indices in the set are the indices of the points in the point cloud; index flat,i is a variable used to represent the one-dimensional index of an image in an image processing task, i y,i represents the y coordinate of the pixel in the image, i x,i represents the x coordinate of the pixel in the image; Mapping pixel to point j represents the corresponding point in the three-dimensional space where the pixel position in the image is mapped.​

Images