Coarse aggregate depth image segmentation method based on deep learning
Through the deep learning depth image segmentation method based on deep learning, the RD U-NET network and feature extraction module are used to solve the problem of data scarcity and edge recognition in the collection of materials, and achieve high accuracy and stability collection material segmentation Effect.
Patent Information
- Application Number
- CN202411517374.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-01-28
AI Technical Summary
The existing aggregate depth image segmentation methods have problems such as scarcity of data sets, difficulty in accurately capturing the details of aggregate profiles, poor performance when dealing with complex geometric structures, and weak robustness to small-particle-sized aggregate edges.
Adopt the deep image division method based on deep learning, collect deep image data through the structural optical camera system, perform dual-value and ASF interchange sequence filtering to remove noise, and use the RD U-Net convolutional neural network for segmentation. DENSE BLOCK and asymmetric convolutional blocks are extracted to extract multi -scale characteristics.
The accurate identification and segmentation of the complex edges of the coarse collection is realized, and the problem of multiple materials division is solved, and the accuracy, the recall rate and accuracy are superior.
Smart Images
Figure CN119379723A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of coarse aggregate depth image segmentation methods, and in particular relates to a coarse aggregate depth image segmentation method based on deep learning. Background Art
[0002] Aggregates play a vital role in the raw materials of construction and are widely used in the preparation of building materials such as concrete and asphalt. The quality and characteristics of aggregates directly affect the performance and durability of the final building materials. Therefore, accurate evaluation and testing of aggregate characteristics is crucial to the design and construction of civil engineering projects.
[0003] In recent years, with the rapid development of 3D vision technology, line structured light cameras have become one of the key technologies in the field of depth data acquisition. Compared with traditional two-dimensional images, line structured light cameras project light stripes through the principle of structured light to capture the deformation of the surface of the target object and obtain high-precision three-dimensional depth information. This height information not only provides more comprehensive aggregate morphology information, but also enhances the accuracy of aggregate property detection. Especially in the case of complex edges and low differentiation, three-dimensional depth data can reveal the shape and structure of the aggregate surface in more detail, providing a more reliable basis for pavement and material performance analysis and application.
[0004] At present, traditional segmentation methods in the field of aggregate depth image segmentation have a series of obvious defects, which limit their effectiveness and reliability in practical applications. First, there are currently few data sets related to aggregate depth images, and there is a lack of data support. At the same time, due to the complexity of aggregate contours, traditional methods are difficult to accurately capture and segment the details of aggregate contours. Traditional depth data segmentation methods perform poorly when dealing with complex geometric structures on aggregate surfaces. In addition, due to the small height variation of small-size aggregates and the low distinction between contours and backgrounds, traditional methods are less robust and less stable to such edges. Summary of the invention
[0005] The technical problem solved by the present invention is to provide a coarse aggregate depth image segmentation method based on deep learning with reliable analysis performance and good stability.
[0006] The present invention adopts the following technical solution to solve the above technical problems, a method for deep image segmentation of coarse aggregate based on deep learning, characterized by the specific steps of:
[0007] Step S1: Keep the structured light camera system unloaded for calibration, and set the background height data to 0;
[0008] Step S2: Collect multiple sets of aggregate depth image data through a line structured light system, and the three channels of the image respectively store x, y, and z axis data in pixel format, wherein the z axis data of the third channel is the height data required by the method, i.e., the depth image;
[0009] Step S3: Binarize the depth data in step S2 and perform ASF alternating sequence filtering on the binary image to remove noise generated during the acquisition process due to impurity reflections, sharp targets, etc., and then use the binarized image as a mask to extract the ROI area from the original depth map;
[0010] Step S4: obtaining an aggregate depth image dataset, performing the preprocessing of step S3 on the original aggregate depth image dataset to obtain an aggregate depth image dataset; labeling the aggregate area of each image in the dataset based on the labelme tool to obtain a labeled mask image set;
[0011] Step S5: Based on the U-Net network, an RD U-Net convolutional neural network is improved and established. U-Net is used as the backbone, and the Dense Block module is used to replace the two consecutive 3×3 convolutions of the convolution layer in the U-Net network. The connection module of ResBlock based on the residual structure is used to replace the four skip connections. In addition, all 3×3 convolutions are replaced with asymmetric convolutions based on the above.
[0012] Step S6: Use the aggregate depth image set as a training set, use the annotated mask image set as a label set to train the convolutional neural network, and use the trained model as the aggregate depth image segmentation model.
[0013] Specifically, the no-load calibration in step S1 means that when the background plate remains unloaded, the structured light scans its height data and sets the z value of the lowest point to 0, and the remaining three-dimensional points maintain a fixed relative height with the lowest point.
[0014] Specifically, the data in step S2 is stored in CSV format during the acquisition process, including the camera internal parameters such as XYZ three-axis resolution and the number of encoder nodes, which are used to calculate the actual height of the aggregate.
[0015] Specifically, the ASF alternating sequence filtering in step S3 means first using a small-sized structural unit to perform an opening and closing operation; then gradually increasing the size of the structural unit; and finally terminating the operation at a most suitable structural unit size. It usually includes two filtering methods: ASF first opening and then closing operation OC and close-before-open operation ASF CO , whose expression is
[0016]
[0017] Specifically, in the marking process of step S4, the outermost contour of the aggregate with adhesion is marked as a whole.
[0018] Specifically, the Dense Block module in step S5 refers to using several lightweight 3×3 convolution kernels to split and reconstruct the complex 5×5 and 7×7 convolution kernels, and adopts the idea of DenseNet to concatenate the outputs obtained from three consecutive convolution blocks, combining the output results of each convolution to ensure that more feature information is extracted with the same number of parameters.
[0019] Specifically, the asymmetric convolution block in step S5 refers to three parallel branches with convolution kernel sizes of 3×3, 1×3 and 3×1. Batch normalization is performed after each layer of convolution, and the results of the three parallel branches are combined as the output of the asymmetric module. In order to avoid the problems of gradient explosion and gradient disappearance in the neural network, 1×1 convolution needs to be added to the network as a residual connection.
[0020] Specifically, the convolution kernel size of the Res Block module in step S5 is 3×3, and a 1×1 residual connection is introduced to replace the skip connection. The number of Res Blocks decreases layer by layer, that is, from top to bottom, there are 4, 3, 2, and 1 ResBlocks, respectively. At the same time, in order to control the number of model parameters, 48, 96, 192, and 384 filters are used in the four Res Blocks, respectively. The normalization involved in the entire network adopts batch normalization, and the Sigmoid function is used to activate the output layer, and the ReLU function is used to activate all other convolutional layers.
[0021] The present invention has the following beneficial effects and advantages: the present invention requires fewer training sets and can effectively solve the gradient vanishing and gradient exploding problems and the advantages that Dense Block and asymmetric convolution blocks can learn features of multiple scales. At the same time, the semantic gap between the feature information obtained by the encoder and the decoder will also be optimized, that is, it will gradually decrease toward the inside of the network. The present invention can more accurately identify and segment the complex edges of coarse aggregates, and can solve the problem of segmenting multiple aggregates. At the same time, the algorithm of the present invention has the best performance in three performance indicators, which proves the superiority of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Figure 1 This is the detailed diagram of the RD U-Net network structure.
[0023] Figure 2 This is the RD U-Net network structure diagram.
[0024] Figure 3 It is the result image of aggregate image segmentation model.
[0025] Figure 4 It is a diagram of the model training process.
[0026] Figure 5 It is the performance evaluation of aggregate image segmentation model. DETAILED DESCRIPTION
[0027] The above contents of the present invention are further described in detail below through examples, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following examples, and all technologies implemented based on the above contents of the present invention belong to the scope of the present invention.
[0028] A coarse aggregate depth image segmentation method based on deep learning, the specific steps are:
[0029] Step S1: Keep the structured light camera system unloaded for calibration, and set the background height data to 0;
[0030] Step S2: Collect multiple sets of aggregate depth image data through a line structured light system, and the three channels of the image respectively store x, y, and z axis data in pixel format, wherein the z axis data of the third channel is the height data required by the method, i.e., the depth image;
[0031] Step S3: Binarize the depth data in step S2 and perform ASF alternating sequence filtering on the binary image to remove noise generated during the acquisition process due to impurity reflections, sharp targets, etc., and then use the binarized image as a mask to extract the ROI area from the original depth map;
[0032] Step S4: obtaining an aggregate depth image dataset, performing the preprocessing of step S3 on the original aggregate depth image dataset, and obtaining an aggregate depth image dataset; labeling the aggregate area of each image in the dataset based on the labelme tool, and obtaining a labeled mask image set;
[0033] Step S5: Improve and build the RD U-Net convolutional neural network based on the U-Net network, use U-Net as the backbone, use the Dense Block module to replace the two consecutive 3×3 convolutions in the convolution layer of the U-Net network, and use the ResBlock connection module based on the residual structure to replace the four skip connections. And based on the above, replace all 3×3 convolutions with asymmetric convolutions;
[0034] Step S6: Use the aggregate depth image set as a training set, use the annotated mask image set as a label set to train the convolutional neural network, and use the trained model as the aggregate depth image segmentation model.
[0035] Specifically, the no-load calibration in step S1 means that when the background plate remains unloaded, the structured light scans its height data and sets the z value of the lowest point to 0, and the remaining three-dimensional points maintain a fixed relative height with the lowest point;
[0036] Specifically, the data in step S2 is stored in CSV format during the acquisition process, including the camera internal parameters such as XYZ three-axis resolution, encoder node number, etc., which are used to calculate the actual height of the aggregate;
[0037] Specifically, the ASF alternating sequence filtering in step S3 means first using a small-sized structural unit to perform an opening and closing operation; then gradually increasing the size of the structural unit; and finally terminating the operation at a certain most suitable structural unit size. It usually includes two filtering methods: ASF first opening and then closing operation OC and close-before-open operation ASF CO , whose expression is
[0038]
[0039] Specifically, in the marking process of step S4, the outermost contour of the aggregate with adhesion is marked as a whole;
[0040] Specifically, the Dense Block module in step S5 refers to using several lightweight 3×3 convolution kernels to split and reconstruct the complex 5×5 and 7×7 convolution kernels, and adopts the idea of DenseNet to concatenate the outputs obtained from three consecutive convolution blocks, combining the output results of each convolution to ensure that more feature information is extracted with the same number of parameters.
[0041] Specifically, the asymmetric convolution block in step S5 refers to three parallel branches with convolution kernel sizes of 3×3, 1×3 and 3×1. Batch normalization is performed after each layer of convolution, and the results of the three parallel branches are combined as the output of the asymmetric module. In order to avoid the problems of gradient explosion and gradient disappearance in the neural network, 1×1 convolution needs to be added to the network as a residual connection.
[0042] Specifically, the convolution kernel size of the Res Block module in step S5 is 3×3, and a 1×1 residual connection is introduced to replace the skip connection. The number of Res Blocks decreases layer by layer, that is, from top to bottom, there are 4, 3, 2, and 1 ResBlocks, respectively. At the same time, in order to control the number of model parameters, 48, 96, 192, and 384 filters are used in the four Res Blocks, respectively. The normalization involved in the entire network adopts batch normalization, using the Sigmoid function to activate the output layer and the ReLU function to activate all other convolutional layers.
[0043] Example 1
[0044] This embodiment discloses a method for establishing a coarse aggregate depth image segmentation model based on deep learning. On the basis of the above embodiment, the following technical features are also disclosed: a three-dimensional device based on laser triangulation is used to shoot the aggregate to obtain three-dimensional aggregate data, and the aggregate height map is extracted using the data, and the ASF alternating sequence filtering method mentioned in step 3 is used to filter it to obtain a filtered height map. A total of 100 data sets required for this example are obtained through the above processing method. The image size is 576×832, where the ratio of the training set to the test set is 8:2, and the edge contours of the aggregate particles are manually labeled using the image annotation tool labelme.
[0045] In order to illustrate the effectiveness of the method of the present invention, the inventors conducted a test of aggregate segmentation based on depth images. The operating system used was Win1064 and the CPU was Intel(R) Core(TM) i7-11800H CPU@2.30GHz.
[0046] The evaluation indicators of the aggregate segmentation model of the present invention mainly include: Precision, Recall and Accuracy, and the corresponding calculation formulas are as follows:
[0047]
[0048] Among them, TP represents the number of pixels with a correctly detected pixel value of 1; FP represents the number of pixels with a falsely detected pixel value of 1; TN represents the number of pixels with a correctly detected pixel value of 0; and FN represents the number of pixels with a missed pixel value of 1. Among them, Precision represents the proportion of pixels with a correctly detected pixel value of 1 in all the results of pixels with a pixel value of 1. The larger its value, the fewer the falsely detected targets; Recall represents the proportion of pixels with a correctly detected pixel value of 1 in all the pixels with a pixel value of 1. The larger its value, the fewer the missed targets; Accuracy represents the proportion of all the pixels with a correctly detected pixel value. The larger its value, the higher the accuracy and the better the segmentation effect.
[0049] The loss function curve and accuracy curve in the training and testing of the RD U-Net aggregate segmentation model are as follows: Figure 4 As shown in the figure. From the loss curve, we can see that the network gradually stabilizes in the later stage of training; from the accuracy curve, we can see that the accuracy in the later stage of training also gradually stabilizes and has a higher accuracy. In terms of accuracy, the test set is higher than the training set, indicating that the RD U-Net model does not have the problem of overfitting and has good generalization ability.
[0050] In order to further verify the performance of the RD U-Net network proposed in this invention, 20 test set images were selected and segmented using RD U-Net, traditional U-Net and watershed algorithm respectively. The test results were evaluated using the three groups of performance indicators (precision, recall and accuracy) mentioned above. Figure 5 To evaluate the performance of the aggregate image segmentation model, Figure 3 These are the results of the three aggregate image segmentation models. Figure 3 It can be seen that the performance of the watershed algorithm is poor in all aspects. Compared with the traditional U-Net, RD U-Net has a significant improvement in all three performance indicators, with an increase of 1.97% in precision, 1.24% in recall, and 2.52% in accuracy on average. Figure 4 It can be seen that the traditional U-Net network and watershed algorithm are prone to segmenting multiple aggregates into one when segmenting, and cannot perform accurate segmentation. The RD U-Net used in the present invention is more accurate in identifying the edges of aggregates and is less likely to over-segment.
[0051] The above embodiments describe the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for illustrating the principles of the present invention. Without departing from the scope of the principles of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of protection of the present invention.
Claims
1. A coarse aggregate depth image segmentation method based on deep learning, characterized in that The specific steps are: Step S1: Keep the structured light camera system unloaded for calibration, and set the background height data to 0; Step S2: Collect multiple sets of aggregate depth image data through a line structured light system, and the three channels of the image respectively store x, y, and z axis data in pixel format, wherein the z axis data of the third channel is the height data required by the method, i.e., the depth image; Step S3: Binarize the depth data in step S2 and perform ASF alternating sequence filtering on the binary image to remove noise generated during the acquisition process due to impurity reflections, sharp targets, etc., and then use the binarized image as a mask to extract the ROI area from the original depth map; Step S4: obtaining an aggregate depth image dataset, performing the preprocessing of step S3 on the original aggregate depth image dataset to obtain an aggregate depth image dataset; labeling the aggregate area of each image in the dataset based on the labelme tool to obtain a labeled mask image set; Step S5: Based on the U-Net network, an RD U-Net convolutional neural network is improved and established. U-Net is used as the backbone, and the Dense Block module is used to replace the two consecutive 3×3 convolutions of the convolution layer in the U-Net network. The connection module of ResBlock based on the residual structure is used to replace the four skip connections. In addition, all 3×3 convolutions are replaced with asymmetric convolutions based on the above. Step S6: Use the aggregate depth image set as a training set, use the annotated mask image set as a label set to train the convolutional neural network, and use the trained model as the aggregate depth image segmentation model.
2. The method for coarse aggregate depth image segmentation based on deep learning according to claim 1, characterized in that: The no-load calibration in step S1 means that when the background plate remains unloaded, the structured light scans its height data and sets the z value of the lowest point to 0, and the remaining three-dimensional points maintain a fixed relative height with the lowest point.
3. The method for coarse aggregate depth image segmentation based on deep learning according to claim 1, characterized in that: The data in step S2 are stored in CSV format during the acquisition process, including the camera parameters such as the XYZ three-axis resolution and the number of encoder nodes, which are used to calculate the actual height of the aggregate.
4. The method for coarse aggregate depth image segmentation based on deep learning according to claim 1, characterized in that: In step S3, the ASF alternating sequence filtering means first using a small-sized structural unit to perform an opening and closing operation; then gradually increasing the size of the structural unit; and finally terminating the operation at a certain most suitable structural unit size. It usually includes two filtering methods: ASF first opening and then closing operation OC and close-before-open operation ASF CO , whose expression is 5. The method for coarse aggregate depth image segmentation based on deep learning according to claim 1, characterized in that: In the marking process of step S4, the outermost contour of the aggregate with adhesion is marked as a whole.
6. The method for coarse aggregate depth image segmentation based on deep learning according to claim 1, characterized in that: The Dense Block module in step S5 refers to the use of several lightweight 3×3 convolution kernels to split and reconstruct the complex 5×5 and 7×7 convolution kernels, and adopts the idea of DenseNet to concatenate the outputs obtained from three consecutive convolution blocks and combine the output results of each convolution to ensure that more feature information is extracted with the same number of parameters.
7. The method for coarse aggregate depth image segmentation based on deep learning according to claim 1, characterized in that: The asymmetric convolution block in step S5 refers to three parallel branches with convolution kernel sizes of 3×3, 1×3 and 3×1. Batch normalization is performed after each layer of convolution, and the results of the three parallel branches are combined as the output of the asymmetric module. In order to avoid the problems of gradient explosion and gradient disappearance in the neural network, 1×1 convolution needs to be added to the network as a residual connection.
8. The method for coarse aggregate depth image segmentation based on deep learning according to claim 1, characterized in that: The convolution kernel size of the Res Block module in step S5 is 3×3, and a 1×1 residual connection is introduced to replace the skip connection. The number of ResBlocks decreases layer by layer, that is, from top to bottom, there are 4, 3, 2, and 1 Res Blocks, respectively. At the same time, in order to control the number of model parameters, 48, 96, 192, and 384 filters are used in the four Res Blocks, respectively. The normalization involved in the entire network adopts batch normalization, and the Sigmoid function is used to activate the output layer, and the ReLU function is used to activate all other convolutional layers.