Method for intelligently extracting point cloud of stacked muck particles

Abstract

The invention relates to the technical field of intelligent detection of building materials, in particular to a method for intelligently extracting point clouds of stacked muck particles. Comprising the following steps: S1, muck particle point cloud data acquisition; s2, converting the stacked point cloud into a depth image; s3, the Wast-Net model is used for extracting segmentation masks of the particles; s4, extracting a completely visible particle depth image; and S5, segmenting and extracting the completely visible particle point cloud. According to the method, rapid and accurate segmentation of completely visible particle point cloud of the stacked residue soil can be realized, meanwhile, incomplete visible particle point cloud data is effectively filtered, and the problem that automatic segmentation cannot be realized in a complex stacking environment by a traditional segmentation method is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent detection of building materials, and in particular to a method for intelligently extracting a point cloud of stacked slag particles. Background Art

[0002] Figure 1 The slag particle feature detection system based on three-dimensional scanning is demonstrated. The slag particles present a complex stacking shape, which is consistent with the working conditions faced in actual engineering applications. Patent CN119360363A discloses a particle gradation detection method for evenly dispersed particles. The method uses a clustering algorithm to segment particle point cloud data. However, this method is obviously not suitable for the complex stacking conditions involved in this case. Therefore, it is particularly important to develop a particle point cloud fast extraction algorithm that adapts to complex stacking conditions.

[0003] There is no relevant research on the point cloud segmentation of soil particles under complex stacking conditions. Common point cloud segmentation methods include clustering-based algorithms, geometric feature-based methods, and deep learning-based methods. Existing technologies face many challenges, especially in the processing of complex stacking and partially occluded particles.

[0004] (1) The DBSCAN algorithm is a representative clustering-based algorithm and is widely used in the segmentation of uniformly dispersed particles. However, in the complex environment of stacked particles, due to the influence of the mutual adhesion and overlapping occlusion of particle point clouds, the DBSCAN algorithm will appear over-segmentation or under-segmentation, or even unable to segment, and the computational complexity is high, which makes it difficult to meet the efficiency requirements of real-time detection.

[0005] (2) Segmentation methods based on geometric features (such as region growing and edge detection) can be used to extract particle boundaries. However, when particles overlap or are occluded, it is difficult to accurately distinguish the boundaries of the particles. In particular, when the particle shape is complex or the surface is irregular, the segmentation accuracy is greatly reduced and the computational complexity is also high.

[0006] (3) Deep learning-based segmentation methods (such as PointNet) can effectively segment point cloud data with obvious features, such as the ground, vehicles, or point clouds of indoor target objects such as tables, walls, beams, etc. The principle is to extract the geometric features of the object point cloud and use deep learning algorithms for segmentation. It is a method to directly segment the point cloud. However, these methods rely on a large amount of labeled data for training. The training process requires a lot of computing power to process complex point cloud data, with high computing resource requirements and poor real-time performance. On the other hand, the complex stacked particles in this case have complex and changeable geometric features and do not have a fixed geometric shape. Therefore, the deep learning-based segmentation method is not suitable for the point cloud segmentation of such complex stacked particles.

[0007] The complex stacking of soil particles targeted by the present invention brings two major challenges to point cloud segmentation and analysis: ① The adhesion and occlusion problems between particles are the core difficulties of the segmentation task. In actual engineering scenarios, slag particles are often stacked in a disordered state, and the contact surface boundaries are blurred, which makes it difficult for traditional segmentation methods based on geometric features (such as region growing and watershed algorithms) to accurately distinguish adjacent particles. If the residual cloud of partially obscured particles is misjudged as a complete particle, it will significantly affect the accuracy of subsequent grading analysis. Although existing studies have attempted to improve clustering algorithms, such as variants of the density-based clustering algorithm DBSCAN (Density-Based Spatial Clustering of Applications with Noise), or introduce morphological processing, their adaptability to complex stacking conditions is still limited.

[0008] ② Balancing algorithm efficiency and accuracy is another major challenge. Engineering applications require segmentation algorithms to have real-time processing capabilities (such as processing millions of point clouds per second), while existing high-precision methods (such as voxel-based three-dimensional convolutional networks PointNet) often have high computational complexity and are difficult to meet real-time requirements. At the same time, the diversity of particle morphology (such as pebbles, gravel, flakes, etc.) requires the algorithm to have strong generalization capabilities, but the actual labeled data is scarce, which restricts the performance of supervised learning models. Some studies have attempted to alleviate data dependence through transfer learning or weak supervision methods, but their stability in cross-scenario applications still needs to be verified.

[0009] In summary, how to quickly and automatically extract and segment completely visible particle point clouds from complex stacked particle point clouds to provide an algorithmic basis for online monitoring of soil slag particle characteristics in engineering projects is still a problem that needs to be solved urgently in this field. Summary of the invention

[0010] The present invention provides a method for intelligently extracting point clouds of stacked slag particles, aiming to solve the technical problem of segmenting point clouds of slag particles under complex stacking conditions.

[0011] The present invention provides a method for intelligently extracting a point cloud of stacked slag particles, comprising the following steps: S1. Acquisition of soil particle point cloud data: Scanning the original point cloud data of soil particles on the loading surface by using a 3D camera; S2. Converting the stacked point cloud into a depth image: Using the loading surface as the reference surface, the particle depth image is generated using the Z-axis coordinate value of the particle point cloud data; S3. Waste-Net model extracts the segmentation mask of particles: Based on the U-Net network, the Waste-Net model is constructed by combining the Laplacian operator, the particle boundary attention mechanism module, and the convolution block attention module to process the input particle depth image and finally output the segmentation mask of the fully visible particles; S4. Extraction of depth image of fully visible particles: Based on the segmentation mask generated by the Waste-Net model, the depth image of fully visible particles is extracted; S5. Segmentation and extraction of point cloud of completely visible particles: Extract three-dimensional point cloud data of completely visible particles based on the depth image of completely visible particles.

[0012] As a further improvement of the present invention, step S2 comprises: S21. Read the particle point cloud in PLY format through the Open3D library, extract the Z coordinates of all particle point clouds, and construct a histogram of Z values; S22. Determine the height of the reference plane projected by each point in the point cloud data by calculating the area with the smallest histogram density, and calculate the distance from each point to the reference plane, and finally obtain the depth value of each point; S23. Use the binned_statistic_2d function in the SciPy library to perform two-dimensional bin statistics on the points in the point cloud data, calculate the average distance value in each bin, and generate a depth image; S24. Normalize the generated depth image and apply the Viridis color map of Matplotlib to color it, map the depth value to the specified interval, generate a color depth image, and save it in image format.

[0013] As a further improvement of the present invention, the step S21 specifically includes: Use the Open3D library to read the particle point cloud P in PLY format, extract the Z coordinates of all particle point clouds in P, and record them as a set , divide multiple bins evenly in the Z direction and calculate the histogram of the Z value hist ( Z ), as shown in formula (1): (1) in B is the number of bins of the histogram, b i It is i The boundaries of the bins.

[0014] As a further improvement of the present invention, the step S22 specifically includes: Then find the bin index with the smallest point density, as shown in formula (2); the height of the final projection reference plane can be calculated by formula (3); for each point , calculate its distance to the reference surface d i , as shown in formula (4); (2) (3) (4) in, x i Indicates i The coordinates of a point on the X axis, y i Indicates i The coordinate of a point on the Y axis, z i Indicates i The coordinates of a point on the Z axis; hist ( Z ) i It indicates the first i The number of point clouds in a bin; Z base Indicates the height of the projection reference plane on the Z axis. and They are the two boundaries of the bin with the smallest density.

[0015] As a further improvement of the present invention, in step S3, the Waste-Net model processes the input particle depth image and finally outputs a segmentation mask of completely visible particles, specifically including: S31. Conversion from point cloud to depth image: The collected particle point cloud data is preprocessed and converted into a corresponding depth image, where the Z coordinate of each point cloud represents the distance from the point cloud to the reference plane, and each pixel value of the depth image represents the distance between the particle and the reference plane; S32. Dataset construction: Pair the particle depth image with the annotated mask of the fully visible particle area to form a training dataset; S33. Encoder module processing: The encoder gradually extracts the features of the image through multiple convolutional layers, and reduces the size of the image through the maximum pooling operation and increases the number of channels of the feature map; S34. Laplace operator processing: The Laplace operator performs edge enhancement processing on the input image and highlights the particle boundary information by calculating the second-order derivative of the pixel; S35. Granular boundary attention mechanism module processing: feature fusion of the encoder output feature map and the edge feature map processed by the Laplacian operator; S36.CBAM module processing: The CBAM module applies channel and spatial attention mechanisms on the feature map processed by the particle boundary attention mechanism module to generate a weighted image of important feature areas; S37. Skip connection and decoder module processing: The feature map processed by the CBAM module is skip-connected with the low-level feature map of the encoder and enters the decoder module. The decoder module restores the spatial resolution of the image through deconvolution operation and generates the final segmentation mask.

[0016] As a further improvement of the present invention, in step S35, the particle boundary attention mechanism module processing process specifically includes: S35a feature map input: receiving the output feature map from the encoder and the edge feature map after Laplacian operator processing; S35b edge enhancement feature extraction: Get the encoder to extract the particle depth image information; S35c. Feature fusion: directly multiply the particle depth image information and the edge feature map processed by the Laplace operator element by element to obtain the edge enhancement feature intermediate process quantity, and then add the edge enhancement feature intermediate process quantity and the particle depth image information element by element to obtain a new feature map intermediate quantity.

[0017] As a further improvement of the present invention, in step S36, in the CBAM module, the process of generating the weighted image specifically includes: S36a. The channel attention mechanism performs two pooling operations on the input feature map: average pooling and maximum pooling, generating the average value and maximum value of the feature map respectively, extracting global information and salient information; S36b. The two pooling results are processed by 1x1 convolution, and the ReLU activation function is used. Each channel is weighted by the Sigmoid activation function to generate a weighted channel feature map; S36c. The spatial attention mechanism generates a feature map representing spatial importance by performing average pooling and maximum pooling operations on the feature map in the spatial dimension, performs convolution operations on the feature map representing spatial importance, and normalizes it using the Sigmoid activation function to obtain the weighting coefficient for each spatial position; The S36d.CBAM module combines channel attention and spatial attention, applies spatial weighting on the channel-weighted feature map, and generates a weighted image.

[0018] As a further improvement of the present invention, the step S4 includes: obtaining a depth image of completely visible particles through a one-to-one correspondence between pixels of the depth image and the segmentation mask; The process of establishing the pixel-to-pixel correspondence between the depth image and the segmentation mask includes: S41. Generation of depth image: Each point in the point cloud is mapped to a pixel position of the depth image through its (x, y) coordinates, and the z value is used as the pixel value of the depth image; S42. Generation of segmentation mask: The segmentation mask generated by the deep learning model corresponds to the pixel position of the depth image one by one, indicating the target area in the image; S43. Pixel to point cloud mapping: Use the digitize function to map the (x, y) coordinates of the point cloud to the pixel position of the depth image, and establish a mapping relationship between pixels and point clouds; S44. Connection between segmentation mask and depth image: Through the mapping relationship between pixels and point clouds, the segmentation mask is corresponded to each pixel value in the depth image.

[0019] As a further improvement of the present invention, step S5 specifically includes: S51. Based on the depth image of the fully visible particles, each screened pixel in the image is mapped to a corresponding position in the three-dimensional point cloud space using a preset pixel-point cloud mapping function, and the three-dimensional point cloud data of the fully visible particles is extracted through reverse indexing.

[0020] As a further improvement of the present invention, the step S51 specifically includes: The pixel-point cloud mapping function is implemented by establishing the correspondence between the pixels in the depth image segmentation mask and the point index in the point cloud; Assume that the particle point cloud obtained by scanning , each point p i The coordinates of are ( x i , y i , z i ), the size of the depth image is n×n ,in n is the image height and width; each point of the particle point cloud is assigned to an image pixel, and a functional relationship is established between the X, Y coordinates of the point cloud and the X, Y coordinates of the pixel grid of the image: Calculate the X and Y index of each point in the particle point cloud in the image i x and i y , as shown in equations (5) and (6); (5) (6) digitize is to x i and y iMapped to predefined bins, X bins and Y bins They are the bin boundaries of the point cloud in the X and Y directions respectively; Calculate the flat index of each point and convert the 2D index into a one-dimensional index, as shown in formula (7); use a mapping function or data structure to store all points corresponding to each pixel, as shown in formula (8); (7) (8) in, j From 0 to n 2 An integer of -1 represents each pixel in the depth image, and the set is the index of the point in the point cloud; index flat,i is a variable used to represent the one-dimensional index of an image in image processing tasks. i y,i represents the y coordinate of a pixel in the image, i x,i Represents the x-coordinate of a pixel in the image; Mapping pixel to point [ j ] indicates that the pixel positions in the image are mapped to corresponding points in three-dimensional space.

[0021] The beneficial effects of the present invention are as follows: the method is suitable for accurate segmentation of slag particles and real-time detection of grading characteristics, and can efficiently handle the stacking, overlapping and occlusion problems between particles. The present invention can realize the rapid and accurate segmentation of the fully visible particle point cloud of stacked slag, while effectively filtering the incompletely visible particle point cloud data, solving the problem that the traditional segmentation method cannot automatically segment in a complex stacking environment. The method can also be widely used in other fields, such as mineral particle screening, construction sand and gravel detection, agricultural granular fertilizer grading, etc. Its intelligent segmentation capability can meet the needs of different industries in the segmentation of stacked particle point clouds, and is particularly suitable for large-scale point cloud data processing and real-time monitoring of engineering sites, and has significant engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of the application background of the method for intelligently extracting the point cloud of stacked slag particles of the present invention; Figure 2 It is a schematic diagram of the automated segmentation process of the complex stacked particle point cloud in the present invention; Figure 3 It is a flow chart of the method for intelligently extracting point clouds of stacked slag particles of the present invention; Figure 4 It is a comparison diagram of the particle point cloud and the depth image of the present invention; Figure 5 It is a structural diagram of the Waste-Net network model in the present invention; Figure 6 It is a module structure diagram of the particle boundary attention mechanism (PEA) in the present invention; Figure 7 It is a comparison diagram of the particle depth image, Waste-Net segmentation mask and the depth image after segmentation in the present invention; Figure 8 This is a comparison diagram of the fully visible particle point cloud extraction of the present invention; Fig. 9 It is a schematic diagram of the feature graph in the present invention being processed by element-by-element addition. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0024] Combination Figures 2 to 3 As shown, a method for intelligently extracting a point cloud of stacked slag particles of the present invention comprises the following steps: S1. Acquisition of point cloud data of slag particles: Scan the original point cloud data of slag particles on the loading surface through a 3D camera.

[0025] S2. Converting the stacked point cloud into a depth image: Using the loading surface as the reference surface, the particle depth image is generated using the Z-axis coordinate value of the particle point cloud data.

[0026] The core of the method for acquiring soil particle point cloud data is to calculate the Z coordinate of each point in the point cloud and project the point cloud according to the Z value. Specifically, it includes: S21. Read the particle point cloud in PLY format through the Open3D library, extract the Z coordinates therein, and construct a histogram of the Z values.

[0027] S22. Determine the height of the projection reference plane by calculating the area with the minimum histogram density, and calculate the distance from each point to the reference plane, and finally obtain the depth value of each point.

[0028] S23. Use the binned_statistic_2d function in the SciPy library to perform two-dimensional binning statistics and generate a depth image.

[0029] S24. Normalize the generated depth image and apply the Viridis color map to color it, map the depth value to the interval [0, 1], generate a color depth image, and save it in PNG format for subsequent analysis and processing.

[0030] The advantage of this method is its high automation and batch processing capabilities, which can provide high-quality depth images for subsequent image segmentation and particle analysis.

[0031] In the acquisition of soil particle point cloud data, Python programming language and related libraries (such as Open3D, NumPy, Matplotlib and SciPy) are used to realize the reading, processing and conversion of point cloud data.

[0032] The depth image is formed by projecting the particle point cloud onto a reference plane. First, the projection reference plane needs to be calculated and determined. Specifically, the Open3D library is used to read the particle point cloud P in PLY format, and the Z coordinates of all point clouds in P are extracted and recorded as the set . Divide 50 bins evenly in the Z direction and calculate the histogram of the Z value hist ( Z ), as shown in formula (1).

[0033] (1) in B is the number of bins in the histogram, which is set to 50 here. b i It is i The boundaries of the bins.

[0034] Then find the bin index with the smallest point density, as shown in formula (2). Finally, the height of the projected reference plane can be calculated by formula (3). Finally, for each point , calculate its distance to the reference surface d i , as shown in formula (4).

[0035] (2) (3) (4) In the formula, x i , y i and z i Represents the coordinates of each point in the point cloud: x i Indicates i The coordinates of a point on the X axis, y i Indicates i The coordinate of a point on the Y axis, z i Indicates i The coordinate of a point on the Z axis.

[0036] hist ( Z ) i It indicates the first i The number of point clouds in the bin, that is, the number of points in the Z direction that fall in the bin i The number of points within the bin range. hist ( Z ) i It reflects the density of points in the interval. By counting the number of point clouds in each bin, the area with the smallest density can be further calculated to determine the position of the reference plane.

[0037] Z base Indicates the height of the projection reference plane on the Z axis. and are the two boundaries of the bin with the lowest density. The height of this reference plane is determined by finding the area with the lowest density, so Z base It represents the middle altitude of the area. It is the reference altitude used for projection calculations.

[0038] After determining the projection reference plane, the present invention uses the binned_statistic_2d function in the SciPy library to perform two-dimensional binning statistics on the points in the point cloud, and calculates the average distance value in each bin, thereby generating a two-dimensional distance image, i.e., a depth image. Then, the generated depth image is normalized and colored using the Viridis color map of Matplotlib to intuitively represent different distance values. During the normalization process, the value of the depth image is mapped to the [0, 1] interval, and then multiplied by 255 to convert it to the uint8 type for image storage and display. The purpose of the above process is to convert the point cloud data into a depth image. The following is a brief description of the four main steps: (1) Calculate the average depth of each region: First, each point in the point cloud is divided into different regions (bins) according to their x and y coordinates. Then, in each region, the average z value (i.e., depth) of all points is calculated.

[0039] (2) Normalized depth values: The depth values ​​of each region are then normalized and their range is adjusted to between 0 and 1, so that all depth values ​​are on the same scale for easy processing and display.

[0040] (3) Adjust the depth value to the appropriate range: Then, convert the standardized depth value into an integer from 0 to 255 so that it can be saved and displayed in common image formats (such as PNG or JPEG).

[0041] (4) Coloring the depth values: Finally, the Viridis color map is used to map each depth value to a different color, from dark blue to bright yellow. In this way, the depth image can not only display the depth information, but also intuitively express different depth levels through color.

[0042] Finally, the generated color depth image is saved in PNG format for subsequent image processing and analysis. The advantage of this method is that it has a high degree of automation and can process multiple point cloud files in batches to generate high-quality depth images. In addition, through distance calculation and normalization processing, this method can intuitively represent the positions of different heights in the point cloud, so that the depth images of different particle point clouds have obvious color changes, creating conditions for subsequent accurate image segmentation. Figure 4 The particle point cloud and the converted depth image are shown. Figure 4 (a) in the figure represents the particle point cloud. Figure 4 (b) in the figure shows the particle depth image. It can be clearly seen that compared with the particle point cloud, the particle depth image forms an obvious color transition at the junction of particles, which creates conditions for automated segmentation.

[0043] S3. Waste-Net model extracts the segmentation mask of particles: Based on the U-Net network, the Waste-Net model is constructed in combination with the Laplacian operator, the particle boundary attention mechanism module, and the convolution block attention module to process the input particle depth image and finally output the segmentation mask of the fully visible particles.

[0044] Waste-Net Network Overview: After the particle depth image is generated, the present invention further proposes an intelligent segmentation mask generation method based on the U-Net model. This generation method automatically extracts the segmentation mask of the particle through the deep learning model Waste-Net. Waste-Net is an optimized version based on the classic U-Net architecture, which introduces the Laplacian operator and the convolutional block attention module (CBAM) to enhance the accuracy of the model in segmenting particles. After training, Waste-Net can process the input depth image, accurately distinguish the fully visible particles, and extract deep features through convolutional layers and pooling layers. In the decoding stage, the spatial resolution is restored through upsampling and jump connections, thereby generating a segmentation mask with pixel-level accuracy. In this way, the boundaries of the particles can be accurately extracted, providing accurate segmentation results for the subsequent particle grading feature analysis. The innovation of this method is that the particle segmentation is automatically completed through deep learning technology, which significantly improves the segmentation efficiency and accuracy.

[0045] like Figure 5 As shown in the figure, the Waste-Net model uses the U-Net network as the basic architecture and enhances it. Its module framework is as follows: (1) U-Net basic architecture: Encoder: The encoder part of Waste-Net uses the classic U-Net architecture, including multiple convolutional layers and pooling operations, which gradually extracts image features and reduces the image size.

[0046] Bottleneck layer: At the end of the encoder, it enters the bottleneck layer to further process the extracted high-level features and connect to the decoder module.

[0047] Decoder: Through deconvolution (transposed convolution) operations, the spatial resolution of the image is gradually restored and finally restored to the original image size.

[0048] Skip Connections: A key feature of the U-Net architecture, skip connections fuse low-level features from the encoder with high-level features from the decoder, helping to refine edge information and details.

[0049] (2) Laplacian Operator: The Laplacian operator is used in image processing to detect the edges of images and to highlight the edge information in the image by calculating the second-order derivative of pixels. In Waste-Net, the Laplacian operator is used to enhance the edge features in the image, especially the boundaries of particles, so that the model can better capture the subtle differences and details between particles.

[0050] In the two-dimensional case, the Laplace operator is defined as the sum of the second-order derivatives of the neighborhood around each pixel in the image, as shown in the formula As shown, is the pixel intensity value of the image, and are the second-order derivatives of the point in the horizontal and vertical directions respectively.

[0051] By convolving the image, the Laplacian operator helps speed up edge detection and improve the segmentation model's attention to details.

[0052] (3) PEA module (granule boundary attention mechanism module): like Figure 6 As shown in the figure, the PEA (Particle Edge Attention) module is an innovative module in Waste-Net, which is mainly used to enhance the ability to capture details of particle edges. The workflow of the PEA module is as follows: Combination of input feature map and Laplacian feature map: The PEA module receives the output feature map from the encoder and the edge feature map processed by the Laplacian operator. The Laplacian operator enhances the edge information of the image, especially the boundary part of the particles.

[0053] Extraction of edge enhancement features: By combining the Laplacian feature map with the output feature map of the encoder, the PEA module emphasizes the particle edge features, which helps to improve the segmentation accuracy, especially the details.

[0054] Feature fusion: The combined feature map is processed by element-wise multiplication and element-wise addition to fuse the edge and particle detail information.

[0055] like Fig. 9 As shown, the pictures stored in the computer can be represented as Fig. 9 In the matrix shown in the figure, each square represents a pixel, and the value of the pixel corresponds to the corresponding color. In computers, the value range of color is usually set to (0, 255) to represent different color intensities. The process of element-by-element addition is Fig. 9 In the process shown, the elements at each position are added one by one. The same is true for step-by-step multiplication.

[0056] CBAM further optimization: Finally, the output feature map of the PEA module will be input into the CBAM module to further optimize the feature map and enhance the focus on the key areas of the particles.

[0057] (4) CBAM (Convolutional Block Attention Module): Channel Attention: Through adaptive pooling operations, CBAM first captures global contextual information and uses a multi-layer perceptron (MLP) to establish the relationship between channels, giving more attention weights to important channels of the network.

[0058] Spatial Attention: Based on channel attention, CBAM further analyzes in the spatial dimension and uses convolution operations to generate spatial masks to highlight key areas in the image.

[0059] The CBAM module helps the model focus more on the key parts of particles (such as edges and areas with more prominent features), thereby improving the segmentation performance, especially the segmentation effect of particle boundaries.

[0060] Waste-Net's workflow can be divided into the following main steps: (1) Input data processing: S31. Conversion from point cloud to depth image: The particle point cloud data collected from the actual scene is preprocessed and converted into the corresponding depth image. The Z coordinate of each point cloud represents its distance to the reference plane, and each pixel value of the depth image represents the distance between the particle and the reference plane.

[0061] S32. Dataset construction: Pair the particle depth images with the corresponding annotated masks (indicating the areas where the particles are fully visible) to form a training dataset. Manually annotate the particle masks using tools such as Labelme, and apply data enhancement techniques (such as brightness and contrast adjustment) to expand the training data.

[0062] (2) Feature extraction and fusion: S33. Encoder module: The encoder part gradually extracts the features of the image through multiple convolutional layers, and at the same time reduces the size of the image through the maximum pooling operation and increases the number of channels of the feature map.

[0063] S34. Laplacian operator processing: The Laplacian operator performs edge enhancement on the input image to highlight the particle boundary information. This step helps the model better identify details in the image.

[0064] S35. Granular boundary attention mechanism module processing: feature fusion of the encoder output feature map and the edge feature map processed by the Laplacian operator; S35a feature map input: receiving the output feature map from the encoder and the edge feature map after Laplacian operator processing; S35b. Extraction of edge enhancement features: Obtaining particle depth image information extracted by the encoder H i × W i × N i ; S35c. Feature fusion: integrating particle depth image information H i × W i × N i Directly multiply the edge feature map processed by the Laplace operator element by element to obtain the edge enhancement feature intermediate process quantity, and then the edge enhancement feature intermediate process quantity is combined with the particle depth image information H i × W i × N iThe new feature map intermediate quantity is obtained by element-by-element multiplication and element-by-element addition. The edge and particle detail information is integrated, the Laplacian feature map is combined with the output feature map of the encoder, and the particle edge features are emphasized through the PEA module.

[0065] S36.CBAM module: The CBAM module generates a weighted image of important feature areas by applying channel and spatial attention mechanisms on the intermediate quantities of the new feature map. H i1 × W i1 × N i1 , enhance the boundary and detail features of particles.

[0066] In the CBAM module, the generation of weighted images is first processed by the channel attention mechanism. The channel attention mechanism extracts global information and salient information by performing two pooling operations on the input feature map: average pooling and maximum pooling, respectively, to generate the average and maximum values ​​of the feature map. Then, the two pooling results are processed by 1x1 convolution, and the ReLU activation function is used. Finally, each channel is weighted by the Sigmoid activation function to generate a weighted channel feature map. Then, the weighted channel feature map is processed by the spatial attention mechanism. The spatial attention mechanism generates a feature map representing spatial importance by performing average pooling and maximum pooling operations on the feature map in the spatial dimension, and then performs convolution operations on it. Finally, it is normalized using the Sigmoid activation function to obtain the weighting coefficient of each spatial position. The CBAM module combines channel attention and spatial attention, applies spatial weighting to the channel-weighted feature map, and generates a weighted image. This weighted image can highlight the important areas in the image, especially the boundaries and detail features of the particles, helping the model to better segment the image and improve segmentation accuracy.

[0067] S37. Skip connection and decoder module: The feature map processed by the CBAM module is skip-connected with the low-level feature map of the encoder and enters the decoder module. The decoder restores the spatial resolution of the image through deconvolution operations and generates the final segmentation mask.

[0068] In the Waste-Net model, the innovations of the skip connection and decoder module are: ① Improvement of skip connection: In the traditional U-Net architecture, the skip connection directly passes the low-level features of the encoder layer to the corresponding layer of the decoder. In this Waste-Net model, the low-level feature map is processed by the CBAM module before the skip connection. The CBAM module strengthens the key information in the image by calculating the channel and spatial attention of the image, so that the decoder can focus more on important areas when restoring the image, especially the boundary part of the particles.

[0069] ② Processing in the decoder: In the processing of the decoder, the traditional deconvolution operation is used to upsample and restore the spatial resolution of the image. The improvement of this Waste-Net model is that the decoder not only receives the low-level features from the encoder, but also uses the feature map processed by CBAM and combines it with the upsampled image for splicing, thereby improving the accuracy of recovery. In addition, the decoder also introduces the Laplacian image and performs upsampling and element-by-element multiplication operations on it, further enhancing the segmentation accuracy, especially at the particle boundary.

[0070] ③Difference from traditional encoders: In the traditional encoder-decoder architecture, the decoder only relies on deconvolution operations to restore the spatial resolution of the image and directly transmits low-level features through jump connections. In this Waste-Net model, low-level features are processed by the CBAM module before being transmitted and combined with the Laplacian image, so that edge information can be better utilized in the decoder and the segmentation accuracy can be improved.

[0071] (3) Output segmentation results: Segmentation mask generation: The output of Waste-Net is a segmentation mask image of the same size as the input image, where the area with a predicted value of 1 represents the particle area and 0 represents the background area.

[0072] Boundary detail enhancement: Through the combined effect of the Laplacian operator and the CBAM module, Waste-Net can more accurately segment the boundaries and detail areas of particles and improve segmentation accuracy.

[0073] S4. Depth image extraction of fully visible particles: Based on the segmentation mask generated by the Waste-Net model, the depth image of fully visible particles is extracted.

[0074] Waste-Net generates segmentation masks for the particle depth image. These masks represent the position information of the fully visible particles identified by Waste-Net. It should be noted that Waste-Net reads the depth image and outputs a binary (black and white) mask image. The positions in these mask images correspond to the particles at the same position in the depth image, indicating that the particles at that position are fully visible. Through the one-to-one correspondence between the depth image and the segmentation mask, the depth image of the fully visible particles can be quickly obtained. This process is Figure 7 was clearly demonstrated in. Figure 7 (a) is the particle depth image. Figure 7 (b) in the figure is the segmentation mask generated by Waste-Net. Figure 7 (c) in the figure is the depth image extracted by the segmentation mask.

[0075] Correspondence between depth image and segmentation mask: The segmentation mask corresponds to the pixel position in the depth image, that is, the classification information of each pixel. Each pixel value (0 or 1) in the mask indicates whether the pixel belongs to the target area (for example, a particle). Through the mapping relationship from pixels to point clouds, each pixel position in the depth image can be accurately mapped to the corresponding position in the point cloud, thus ensuring a one-to-one correspondence between the depth image and the segmentation mask.

[0076] The pixel-to-pixel correspondence between the depth image and the segmentation mask is established mainly through the following steps: S41. Generation of depth image: Each point in the point cloud is mapped to the pixel position of the depth image through its (x, y) coordinates, and the z value is used as the pixel value of the depth image.

[0077] S42. Generation of segmentation mask: The segmentation mask generated by the deep learning model corresponds one-to-one to the pixel position of the depth image, representing the target area in the image.

[0078] S43. Pixel to point cloud mapping: Use the digitize function to map the (x, y) coordinates of the point cloud to the pixel positions of the depth image to establish an accurate mapping relationship between pixels and point clouds.

[0079] S44. Relationship between mask and depth image: Ultimately, through these mapping relationships, the segmentation mask is matched to each pixel value in the depth image to achieve accurate segmentation and analysis.

[0080] In this way, the pixel-to-pixel correspondence between the depth image and the segmentation mask ensures the accuracy and effectiveness of the particle point cloud segmentation.

[0081] S5. Segmentation and extraction of point cloud of completely visible particles: Extract three-dimensional point cloud data of completely visible particles based on the depth image of completely visible particles.

[0082] The fully visible particle segmentation mask generated by Waste-Net can be used to extract the corresponding particle depth image. Based on this depth image, the 3D point cloud of fully visible particles can be quickly extracted through reverse indexing. Specifically, each screened pixel in the image is mapped to the corresponding position in the 3D point cloud space using the preset pixel-point cloud mapping function. Through this mapping relationship, the 3D point cloud set corresponding to each fully visible particle can be accurately located.

[0083] In this way, the model uses the segmentation results of the two-dimensional depth image to accurately extract the point cloud data of these particles in three-dimensional space, thereby providing support for subsequent three-dimensional particle analysis, posture estimation and other spatial applications. This reverse indexing method based on segmentation masks and mapping relationships makes the extraction of three-dimensional information of particles more efficient and accurate.

[0084] In order to clearly explain this process, the concept of pixel-to-point mapping is introduced, which aims to establish the correspondence between the pixels in the depth image segmentation mask and the point index in the point cloud.

[0085] Assume that the particle point cloud obtained by scanning , each point p i The coordinates of are ( x i , y i , z i ), the size of the depth image is n×n ,in n is the image height and width. Each point of the particle point cloud is assigned to an image pixel, thus establishing a functional relationship between the X, Y coordinates of the point cloud and the X, Y coordinates of the pixel grid of the image.

[0086] Specifically, we first need to calculate the X and Y index of each point in the particle point cloud in the image. i x and i y , as shown in equations (5) and (6).

[0087] (5) (6) Here digitize is to x i and y i Mapped to predefined bins, X bins and Ybins are the bin boundaries of the point cloud in the X and Y directions, respectively.

[0088] The next step is to calculate the flat index of each point, that is, convert the 2D index into a one-dimensional index to adapt to the computer's storage and calculation method, as shown in formula (7). Finally, a mapping function or data structure is used to store all the points corresponding to each pixel. This study uses Python's defaultdict data structure storage, so that the corresponding point cloud can be quickly retrieved according to the image pixels, as shown in formula (8).

[0089] (7) (8) In the formula, j From 0 to n 2 An integer of -1 represents each pixel in the depth image, and the set is the index of the point in the point cloud.

[0090] index flat,i It is a variable used in some image processing tasks to represent the one-dimensional index of an image. It converts the coordinates of a two-dimensional image into a one-dimensional index so that each pixel of the image can be represented by a single scalar value. In some tasks, especially when processing pixel locations in an image, it is useful to use index flat,i To avoid the complexity of dealing with two-dimensional matrices. i Usually represents the index of the current pixel in a one-dimensional array, and is used to find the corresponding pixel in the one-dimensional array representation. i y,i Represents the y coordinate (row index) of a pixel in the image; i x,i Represents the x-coordinate (column index) of a pixel in the image.

[0091] Mapping pixel to point [ j ] indicates that the pixel positions in the image are mapped to corresponding points in three-dimensional space. Mapping pixel to point [ j ] refers to the pixels in the image ( pixel ) is mapped to a point in three-dimensional space ( point ). Here point [ j ] usually represents the first j Specifically, each pixel in the image represents a location or point in space, and the mapping process converts the two-dimensional pixel coordinates of the image ( i x ,i y ) are converted into corresponding three-dimensional space coordinate points.

[0092] This mapping usually appears in 3D reconstruction or deep image processing tasks, where each pixel in the image has a corresponding 3D spatial position. Through this mapping, the pixels in the 2D image can be mapped one-to-one to the points in the real space, thus obtaining a point cloud or depth information in space.

[0093] Through the above method, this study established the pixel mapping relationship between the particle point cloud and the depth image. This mapping relationship enables the corresponding point cloud to be quickly found according to the pixel position of the depth image, thereby quickly completing the point cloud segmentation task. This process not only improves the efficiency of segmentation, but also provides accurate segmentation results for subsequent particle grading detection and analysis, providing important technical support for resource utilization and particle grading detection. This process is Figure 8 Described in detail in Figure 8 (a) is the original point cloud, Figure 8 (b) in the figure is the depth image of the fully visible particles extracted by the Waste-Net segmentation mask. Figure 8 (c) is the point cloud data of completely visible particles. After the input original point cloud is converted into a depth image, the depth image of the completely visible particles is obtained through step S4. Finally, the point cloud data of the completely visible particles is quickly extracted through the mapping relationship between the pixels of the depth image and the particle point cloud. From then on, the intelligent and automatic extraction of complex stacked particle point clouds is completed. The whole process does not involve complex computational segmentation of three-dimensional point cloud data, but only relies on simple methods such as projection-image segmentation-pixel to point cloud mapping to quickly complete the segmentation task of the completely visible particle point cloud.

[0094] The overall innovation of the method for intelligently extracting the point cloud of stacked slag particles of the present invention lies in: ① The innovation of the present invention is that the conversion efficiency and image quality of three-dimensional particle point cloud to depth image are significantly improved through automatic calculation of reference surface, efficient two-dimensional binning statistics, image normalization and coloring processing. At the same time, this conversion method brings significant advantages to particle image segmentation, that is, the boundaries between particles are more obvious, providing a good data foundation for image segmentation model.

[0095] ②The core innovation of the present invention is the Waste-Net network architecture, which introduces the Laplacian operator and the convolutional block attention module (CBAM) on the basis of the classic U-Net, greatly improving the accuracy of particle depth image segmentation, especially in the processing of particle boundaries and details. Although the U-Net architecture commonly used in the prior art is widely used in image segmentation, it is often difficult to process complex boundaries and details in particle image segmentation. The Laplacian operator enhances the edge features of the image, especially the details of the particle boundaries, thereby significantly improving the accuracy of the model in boundary processing. At the same time, the CBAM module helps the model focus on the key areas of the particles through the channel and spatial attention mechanism, further improving the segmentation accuracy.

[0096] ③ The present invention provides an automated and efficient registration mechanism by combining the segmentation mask generated by Waste-Net with the precise mapping relationship between the depth image and the segmentation mask. Unlike traditional methods that rely on manually set parameters or complex image processing steps, the present invention can accurately map the fully visible particle positions in the segmentation mask image to the depth image through an optimized mapping function. This mechanism ensures that the model can maintain high accuracy and stability when processing different particle morphologies and environmental conditions.

[0097] ④ The accurate mapping method of the depth image to the point cloud of the present invention uses a preset pixel-point cloud mapping function to correspond each screened pixel point in the image to the corresponding position in the three-dimensional point cloud space. Compared with the threshold segmentation based on the depth image and the point cloud registration based on the iterative closest point (ICP) algorithm, it provides higher accuracy, stronger robustness and higher degree of automation, especially in the case of complex particle morphology and occlusion, and can ensure the accurate extraction and annotation of the point cloud of each particle. This innovation solves the problems of insufficient accuracy, inaccurate segmentation and high computational complexity in traditional methods, and significantly improves the quality and processing efficiency of particle point cloud data.

[0098] The present invention has the following significant advantages: Efficient automation: The present invention significantly reduces manual intervention and improves the efficiency of particle point cloud processing through a highly automated workflow. The conversion of point cloud to depth image, particle segmentation mask generation and final particle analysis process are completely automated, and multiple point cloud files can be processed in batches. This automated process not only improves processing speed and reduces human errors, but also ensures consistency and standardization of processing results, greatly improving overall work efficiency, and is particularly suitable for application scenarios that require large amounts of data processing and real-time detection.

[0099] High efficiency: The particle segmentation method provided by the present invention has significant advantages in processing speed and efficiency. By optimizing the point cloud conversion, segmentation mask generation and particle segmentation steps, the entire processing process is accelerated, and large-scale data can be processed efficiently, which is particularly suitable for real-time detection needs such as engineering sites. The Waste-Net model exhibits excellent real-time performance when processing particle segmentation tasks, ensuring the efficiency and low latency of the particle segmentation process, thereby adapting to the needs of rapid changes and large-scale data processing.

[0100] High segmentation accuracy: The present invention achieves pixel-level accuracy in particle segmentation through the Waste-Net deep learning model. Combined with the optimization of the Laplacian operator and the convolutional block attention module (CBAM), the model can accurately extract particle boundaries and effectively segment fully visible particles. Through the fine processing of the depth image, the high accuracy of particle segmentation is ensured, which provides a reliable foundation for subsequent particle grading analysis, quality calculation, etc., and ensures the accuracy and reliability of the overall analysis results.

[0101] Wide applicability: The method of the present invention is not only applicable to common particle analysis tasks such as slag particles, construction sand and gravel, but can also be widely used in other industries such as mineral particle screening, agricultural particle fertilizer grading, and food particle screening. Through the powerful adaptability of the Waste-Net model and the accuracy of particle segmentation, the present invention can handle particles of various shapes, sizes, and densities, and has excellent generalization capabilities. Whether in industry, agriculture or other fields, the present invention can achieve accurate particle analysis, meet the needs of different industries, and provide flexible and efficient particle segmentation solutions.

[0102] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A method for intelligently extracting point clouds of stacked slag particles, characterized in that: The following steps are involved: S1. Acquisition of soil particle point cloud data: Scanning the original point cloud data of soil particles on the loading surface by using a 3D camera; S2. Converting the stacked point cloud into a depth image: Using the loading surface as the reference surface, the particle depth image is generated using the Z-axis coordinate value of the particle point cloud data; S3. Waste-Net model extracts the segmentation mask of particles: Based on the U-Net network, the Waste-Net model is constructed by combining the Laplacian operator, the particle boundary attention mechanism module, and the convolution block attention module to process the input particle depth image and finally output the segmentation mask of the fully visible particles; S4. Extraction of depth image of fully visible particles: Based on the segmentation mask generated by the Waste-Net model, the depth image of fully visible particles is extracted; S5. Segmentation and extraction of point cloud of completely visible particles: Extract three-dimensional point cloud data of completely visible particles based on the depth image of completely visible particles.

2. The method for intelligently extracting point clouds of stacked slag particles according to claim 1 is characterized in that: The step S2 comprises: S21. Read the particle point cloud in PLY format through the Open3D library, extract the Z coordinates of all particle point clouds, and construct a histogram of Z values; S22. Determine the height of the reference plane projected by each point in the point cloud data by calculating the area with the smallest histogram density, and calculate the distance from each point to the reference plane, and finally obtain the depth value of each point; S23. Use the binned_statistic_2d function in the SciPy library to perform two-dimensional bin statistics on the points in the point cloud data, calculate the average distance value in each bin, and generate a depth image; S24. Normalize the generated depth image and apply the Viridis color map of Matplotlib to color it, map the depth value to the specified interval, generate a color depth image, and save it in image format.

3. The method for intelligently extracting point clouds of stacked soil particles according to claim 2 is characterized in that: The step S21 specifically includes: Use the Open3D library to read the particle point cloud P in PLY format, extract the Z coordinates of all particle point clouds in P, and record them as a set , divide multiple bins evenly in the Z direction and calculate the histogram of the Z value hist ( Z ), as shown in formula (1): (1) in B is the number of bins of the histogram, b i It is i The boundaries of the bins.

4. The method for intelligently extracting point clouds of stacked slag particles according to claim 3 is characterized in that: The step S22 specifically includes: Then find the bin index with the smallest point density, as shown in formula (2); finally, the height of the projected reference plane is calculated by formula (3); for each point , calculate its distance to the reference surface d i , as shown in formula (4); (2) (3) (4) in, x i Indicates i The coordinates of a point on the X axis, y i Indicates i The coordinate of a point on the Y axis, z i Indicates i The coordinates of a point on the Z axis; hist ( Z ) i It indicates the first i The number of point clouds in a bin; Z base Indicates the height of the projection reference plane on the Z axis. and They are the two boundaries of the bin with the smallest density.

5. The method for intelligently extracting point clouds of stacked slag particles according to claim 1 is characterized in that: In step S3, the Waste-Net model processes the input particle depth image and finally outputs a segmentation mask of completely visible particles, which specifically includes: S31. Conversion from point cloud to depth image: The collected particle point cloud data is preprocessed and converted into a corresponding depth image, where the Z coordinate of each point cloud represents the distance from the point cloud to the reference plane, and each pixel value of the depth image represents the distance between the particle and the reference plane; S32. Dataset construction: Pair the particle depth image with the annotated mask of the fully visible particle area to form a training dataset; S33. Encoder module processing: The encoder gradually extracts the features of the image through multiple convolutional layers, and reduces the size of the image through the maximum pooling operation and increases the number of channels of the feature map; S34. Laplace operator processing: The Laplace operator performs edge enhancement processing on the input image and highlights the particle boundary information by calculating the second-order derivative of the pixel; S35. Granular boundary attention mechanism module processing: feature fusion of the encoder output feature map and the edge feature map processed by the Laplacian operator; S36.CBAM module processing: The CBAM module applies channel and spatial attention mechanisms on the feature map processed by the particle boundary attention mechanism module to generate a weighted image of important feature areas; S37. Skip connection and decoder module processing: The feature map processed by the CBAM module is skip-connected with the low-level feature map of the encoder and enters the decoder module. The decoder module restores the spatial resolution of the image through deconvolution operation and generates the final segmentation mask.

6. The method for intelligently extracting point clouds of stacked slag particles according to claim 5 is characterized in that: In step S35, the particle boundary attention mechanism module processing process specifically includes: S35a feature map input: receiving the output feature map from the encoder and the edge feature map after Laplacian operator processing; S35b edge enhancement feature extraction: Get the encoder to extract the particle depth image information; S35c. Feature fusion: directly multiply the particle depth image information and the edge feature map processed by the Laplace operator element by element to obtain the edge enhancement feature intermediate process quantity, and then add the edge enhancement feature intermediate process quantity and the particle depth image information element by element to obtain a new feature map intermediate quantity.

7. The method for intelligently extracting point clouds of stacked slag particles according to claim 5 is characterized in that: In step S36, in the CBAM module, the process of generating a weighted image specifically includes: S36a. The channel attention mechanism performs two pooling operations on the input feature map: average pooling and maximum pooling, generating the average value and maximum value of the feature map respectively, extracting global information and salient information; S36b. The two pooling results are processed by 1x1 convolution, and the ReLU activation function is used. Each channel is weighted by the Sigmoid activation function to generate a weighted channel feature map; S36c. The spatial attention mechanism generates a feature map representing spatial importance by performing average pooling and maximum pooling operations on the feature map in the spatial dimension, performs convolution operations on the feature map representing spatial importance, and normalizes it using the Sigmoid activation function to obtain the weighting coefficient for each spatial position; The S36d.CBAM module combines channel attention and spatial attention, applies spatial weighting on the channel-weighted feature map, and generates a weighted image.

8. The method for intelligently extracting point clouds of stacked slag particles according to claim 1 is characterized in that: The step S4 comprises: obtaining a depth image of completely visible particles through a one-to-one correspondence between pixels of the depth image and the segmentation mask; The process of establishing the pixel-to-pixel correspondence between the depth image and the segmentation mask includes: S41. Generation of depth image: Each point in the point cloud is mapped to a pixel position of the depth image through its (x, y) coordinates, and the z value is used as the pixel value of the depth image; S42. Generation of segmentation mask: The segmentation mask generated by the deep learning model corresponds to the pixel position of the depth image one by one, indicating the target area in the image; S43. Pixel to point cloud mapping: Use the digitize function to map the (x, y) coordinates of the point cloud to the pixel position of the depth image, and establish a mapping relationship between pixels and point clouds; S44. Connection between segmentation mask and depth image: Through the mapping relationship between pixels and point clouds, the segmentation mask is corresponded to each pixel value in the depth image.

9. The method for intelligently extracting point clouds of stacked soil particles according to claim 1, characterized in that: The step S5 specifically includes: S51. Based on the depth image of the fully visible particles, each screened pixel in the image is mapped to a corresponding position in the three-dimensional point cloud space using a preset pixel-point cloud mapping function, and the three-dimensional point cloud data of the fully visible particles is extracted through reverse indexing.

10. The method for intelligently extracting point clouds of stacked slag particles according to claim 9, characterized in that: The step S51 specifically includes: The pixel-point cloud mapping function is implemented by establishing the correspondence between the pixels in the depth image segmentation mask and the point index in the point cloud; assuming that the particle point cloud obtained by scanning , each point p i The coordinates of are ( x i , y i , z i ), the size of the depth image is n×n ,in n is the image height and width; each point of the particle point cloud is assigned to an image pixel, and a functional relationship is established between the X, Y coordinates of the point cloud and the X, Y coordinates of the pixel grid of the image: Calculate the X and Y index of each point in the particle point cloud in the image i x and i y , as shown in equations (5) and (6); (5) (6) digitize is to x i and y i Mapped to predefined bins, X bins and Y bins They are the bin boundaries of the point cloud in the X and Y directions respectively; Calculate the flat index of each point and convert the 2D index into a one-dimensional index, as shown in formula (7); use a mapping function or data structure to store all points corresponding to each pixel, as shown in formula (8); (7) (8) in, j From 0 to n 2 An integer of -1 represents each pixel in the depth image, and the set is the index of the point in the point cloud; index flat,i is a variable used to represent the one-dimensional index of an image in image processing tasks. i y,i represents the y coordinate of a pixel in the image, i x,i Represents the x-coordinate of a pixel in the image; Mapping pixel to point [ j ] indicates that the pixel positions in the image are mapped to corresponding points in three-dimensional space.

Images