Rock slag three-dimensional form perception method and system based on linear array laser and deep learning

Abstract

This invention provides a method and system for three-dimensional morphological perception of rock debris based on linear laser array and deep learning. The method acquires point cloud images of rock debris using a linear laser array. The rock debris point cloud images are preprocessed by trimming useless segments, eliminating error points, segmenting valid rock debris point cloud data, and labeling rock debris features. A trained detection and recognition network model is then used to process the preprocessed rock debris point cloud images. This network model converts the rock debris point cloud images into feature maps, extracts rock debris features including curvature features, performs learning mining, filters redundant and overlapping detection bounding boxes, and obtains the rock debris particle size and volume along with the identified bounding box results. This invention enables the detection and recognition of rock debris particle size distribution. Based on the rock debris point cloud data, it can automatically and online detect and extract real-time rock debris particle size distribution features, thereby providing data support for TBM tunneling decisions.

Description

Technical Field

[0001] This invention belongs to the field of tunnel boring machine construction technology, specifically relating to a method and system for perceiving the three-dimensional morphology of rock debris based on linear laser array and deep learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Tunnel Boring Machines (TBMs) are widely used due to their safety, efficiency, and environmental friendliness, and have become one of the most important construction equipment for water conservancy and hydropower diversion tunnels, transportation tunnels, and mine roadways.

[0004] However, TBMs have limited adaptability to rock geological conditions. The control of tunneling parameters mainly relies on qualitative understanding of geological conditions and human experience, making it difficult to effectively match the optimization and adjustment of TBM control parameters with changes in rock geological conditions. During TBM tunneling, the surrounding rock in front of the tunnel face is isolated by the cutterhead, making it difficult to conduct in-situ tests directly and lacking timely rock mass information as a reference. This is one of the key reasons why TBM tunneling relies on human experience and even blindly proceeds.

[0005] Rock debris is a direct product of TBM rock-machine interaction and has high reference value in characterizing surrounding rock conditions. Its morphological characteristics are significantly affected by the integrity of the surrounding rock and the wear of the cutter head, making it an important basis for operators to judge surrounding rock conditions in actual construction. However, traditional in-situ detection methods require manual sieving of large amounts of rock debris. Although relatively accurate statistical information can be obtained, it is time-consuming, labor-intensive, and has poor timeliness. Identifying rock debris characteristics through photography is limited to obtaining two-dimensional information and cannot fully extract three-dimensional characteristics such as rock debris particle size and volume.

[0006] Currently, there are still some difficulties in achieving accurate three-dimensional morphology perception of rock slag:

[0007] (1) The tunnel construction environment is dusty and dimly lit. The conveyor belt speed is as high as 2.5 m / s or more. The original photo data obtained by taking pictures is of poor quality, difficult to process later, and lacks more comprehensive three-dimensional feature information.

[0008] (2) The rock debris conveyed by the TBM belt conveyor contains different volumes and a large amount of crushed stone powder. The rock debris is piled up and obscured, making it difficult to apply ordinary target detection algorithms. At the same time, it is difficult to obtain recognition results and push them to the TBM main control room in a short time. Summary of the Invention

[0009] To address the aforementioned problems, this invention proposes a method and system for three-dimensional morphological perception of rock debris based on linear laser array and deep learning. This invention can detect and identify the particle size distribution of rock debris, and can automatically and online detect and extract real-time rock debris particle size distribution features based on point cloud data of rock debris, thereby providing data support for TBM tunneling decisions.

[0010] According to some embodiments, the present invention adopts the following technical solution:

[0011] A method for perceiving the three-dimensional morphology of rock slag based on linear laser array and deep learning includes the following steps:

[0012] Acquire point cloud images of rock debris based on linear laser array acquisition;

[0013] The rock debris point cloud image is preprocessed by trimming useless fragments, eliminating error points in the data, segmenting the valid rock debris point cloud data, and marking the rock debris features.

[0014] The pre-processed rock debris point cloud image is processed using the trained detection and recognition network model. The detection and recognition network model converts the rock debris point cloud image into a feature map and extracts rock debris features containing curvature features. It performs learning mining, filters redundant and overlapping detection bounding boxes, and obtains the results of rock debris particle size and volume and the identified bounding boxes.

[0015] As an alternative implementation method, rock debris point cloud images are acquired according to a set three-dimensional interval, wherein the three-dimensional interval is the ratio of rock debris moving speed to image scanning frequency.

[0016] As an alternative implementation, the Crop Section algorithm is used to trim useless fragments outside the point cloud data.

[0017] As an alternative implementation method, the SOR filtering algorithm is used to eliminate error points in the data caused by dust.

[0018] As an alternative implementation, the Crop Section algorithm is used to overlap and segment the effective rock debris point cloud data, and Sustech Points is used to mark the bounding box for each rock debris with a coverage rate below a set threshold, and the marking information is saved.

[0019] As an alternative implementation, the detection and recognition network model is a MuckNet neural network model based on YOLO3D, which includes an input layer, an encoder, a decoder and an output layer connected in sequence. The input layer is used to receive point cloud images and extract features. The encoder and decoder of the YOLO3D network are used to transform rock debris features and learn and mine them through convolutional neural networks to obtain structural representations at different scales. The output layer is used to filter out redundant and overlapping detection bounding boxes using non-maximum suppression.

[0020] As an alternative implementation, curvature features are introduced as one of the features in the rock debris characteristics. For each rock debris, there is a feature mapping relationship:

[0021] H,D,C=P(x)

[0022] Wherein, H is the height map, D is the density map, C is the curvature, and P represents the mapping function. These three features are used together for the detection and identification of rock debris.

[0023] Furthermore, the curvature is:

[0024] C = Cur(Patch) N(i) (i x i y i z ))

[0025] Where Patch represents the best-fit quadratic curve around the neighboring points N(i) of point i, and Cur represents the curve around point (i). x i y i z The function of the curvature of the quadratic curve at point ().

[0026] As a further step, k-means clustering is used to obtain representative types of rock debris bounding boxes. The distance between bounding boxes A and B is defined as:

[0027] Distance(A,B) = 1 - IOU(A,B)

[0028] Where IOU(A,B) represents the intersection of bounding boxes A and B.

[0029] As a further step, the detection and recognition network model uses GIoU as the loss function and defines each bounding box using seven degrees of freedom: length, width, height, center point x, y, z, and eccentricity angle. When detecting and recognizing rock debris, the error in predicting the bounding box is calculated using mean square error, and the mean square error is used to regress the sum of the above-mentioned degree-of-freedom components.

[0030] As an alternative implementation, the detection and recognition network model is trained using training samples. During training, the training samples are enhanced by randomly deleting some data, setting the data segmentation probability and segmentation ratio, and filling the segmentation area with the value 1.

[0031] A three-dimensional morphology perception system for rock slag based on linear laser array and deep learning, comprising:

[0032] In-situ acquisition device, used to acquire point cloud images of rock debris at the target location;

[0033] The preprocessing module is used to preprocess the rock debris point cloud image, trim useless fragments, eliminate error points in the data, segment the valid rock debris point cloud data, and perform rock debris feature labeling.

[0034] The perception and recognition module is used to process the pre-processed rock debris point cloud image using the trained detection and recognition network model. The detection and recognition network model converts the rock debris point cloud image into a feature map, extracts rock debris features containing curvature features, performs learning mining, filters redundant and overlapping detection bounding boxes, and obtains the rock debris particle size and volume and the recognized bounding box results.

[0035] As an alternative implementation, the in-situ acquisition device includes a belt conveyor, a support frame, and a linear laser scanner. The belt conveyor is mounted on a TBM, the support frame is located next to the belt conveyor, and the linear laser scanner is mounted on the support frame, located above the belt of the belt conveyor.

[0036] A TBM includes the system described above or includes a processor that performs the steps of the method described above.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] This invention first designs an in-situ acquisition device that can be mounted on a TBM conveyor belt to acquire raw 3D point cloud data of rock debris in situ. Based on the single-stage 3D target detection network YOLO3D, and combined with the 3D morphological features of rock debris, curvature features are introduced to distinguish adjacent stacked rocks. The GIoU loss function is reconstructed to improve the prediction accuracy of the rock debris boundary range. Finally, a MuckNet neural network model for rock debris identification is reconstructed, effectively realizing the detection and identification of rock debris particle size distribution. This invention can automatically and online detect and extract real-time rock debris particle size distribution features based on rock debris point cloud data, with high identification accuracy, thereby providing data support for TBM tunneling decisions.

[0039] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0040] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0041] Figure 1 This is a schematic diagram of the in-situ acquisition device in this embodiment;

[0042] Figure 2 This is a schematic diagram of the MuckNet network architecture in this embodiment;

[0043] Figure 3 This is a schematic diagram of the clustering algorithm code in this embodiment;

[0044] Figure 4 This is a flowchart illustrating this embodiment;

[0045] Figure 5 This is the identification result compared with other methods in this embodiment.

[0046] Among them, 1 is the TBM belt conveyor, 2 is the rock debris transported on the belt conveyor, 3 is the linear laser scanner, and 4 is the steel frame for fixing the linear laser scanner on the belt conveyor. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0048] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0049] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0050] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0051] Example 1

[0052] A method for three-dimensional morphology perception and recognition of rock slag based on linear laser array and deep learning, such as... Figure 4 As shown, it includes the following steps:

[0053] (1) Obtain three-dimensional point cloud data of rock debris.

[0054] The three-dimensional point cloud data of rock slag in this embodiment can be utilized Figure 1 The on-site in-situ data acquisition device shown collects and uploads data. The TBM conveyor belt 1 is used to transport rock debris. A steel frame is installed on both sides or one side of the TBM conveyor belt 1, and a linear array laser scanner 3 is mounted on it. In this embodiment, the steel frame 4 for the fixed linear array laser scanner mounted on the conveyor belt is 150cm high. The steel frame can be bolted to the side frame of the conveyor belt or can be detached and connected to the scanner. In this embodiment, the scanner frequency is set to 2200Hz / s, and after startup, it can quickly scan and acquire point cloud images of rock debris in a fixed section.

[0055] (2) Processing of three-dimensional point cloud data of rock slag.

[0056] This step mainly includes:

[0057] To eliminate the tensile deformation of rock debris point cloud data in the conveyor belt direction, a three-dimensional interval (d) for data acquisition needs to be set. The calculation formula is as follows:

[0058]

[0059] Where v is the belt conveyor speed and f is the scanning frequency.

[0060] The Crop Section algorithm is used to trim useless fragments outside the point cloud data and extract subsets of rock debris point clouds in space. During the calculation process, the valid and invalid data segments are divided by specifying the spatial region where the valid point cloud data is located.

[0061] The SOR filtering algorithm is used to eliminate error points caused by dust in the data.

[0062] Similarly, the Crop Section algorithm is used to overlap and segment the effective rock debris point cloud data to improve computational efficiency. At the same time, Sustech Points is used to annotate 7-DOF bounding boxes for each rock debris with a coverage of less than 20%, and the annotation information is saved in an ASCII text file.

[0063] (3) Use MuckNet to detect and identify network models and process them.

[0064] The MuckNet detection and recognition network model was constructed and pre-trained.

[0065] First, a MuckNet neural network model based on YOLO3D is constructed. Given labeled point cloud data is converted into 2D feature maps. The encoder and decoder of the YOLO3D network then convert the rock debris features, which are learned and mined through a convolutional neural network. To improve target recognition accuracy, non-maximum suppression (NMS) is used to filter out redundant and overlapping detection bounding boxes, leaving only the bounding boxes that most accurately cover the 3D target. The MuckNet network architecture is as follows: Figure 2 As shown.

[0066] In this embodiment, the MuckNet neural network model includes an input layer, an encoder, a decoder, and an output layer connected in sequence. The input layer is used to receive point cloud images and extract features. The encoder and decoder of the YOLO3D network are used to transform rock debris features and learn and mine them through a convolutional neural network to obtain structural representations at different scales. The output layer is used to filter out redundant and overlapping detection bounding boxes using non-maximum suppression.

[0067] In this embodiment, for the target rock debris, the curvature feature introduced in YOLO3D can be used as one of the features of the rock debris. For each rock debris, the following feature mapping relationship exists:

[0068] H,D,C=P(x)

[0069] Where H is the height map, D is the density map, C is the curvature, and P represents the mapping function. These three features are used together for the detection and identification of rock debris. For a point i on the rock debris, the curvature is calculated using the following formula:

[0070] C = Cur(Patch) N(i) (i x i y i z ))

[0071] Where Patch represents the best-fit quadratic curve around the neighboring points N(i) of point i, and Cur represents the curve around point (i). x i y i z The function of the curvature of the quadratic curve at point ().

[0072] In this embodiment, for the specific detection target of rock slag, the k-means clustering method is used to obtain representative rock slag bounding box types:

[0073] In the k-means algorithm, the distance between bounding boxes A and B is defined as:

[0074] Distance(A,B) = 1 - IOU(A,B)

[0075] Where IOU(A,B) represents the intersection of bounding boxes A and B. IOU can be calculated using the following formula:

[0076]

[0077] As an alternative implementation, for the specific detection target of rock slag, the MuckNet network uses GIoU as the loss function.

[0078] Each bounding box is defined using seven degrees of freedom: length, width, height, center point (x, y, z), and eccentricity angle. For the rock debris detection and identification regression problem, the error in predicting the bounding boxes is expressed as mean squared error (MSE). In the MSE regression loss, the loss function is defined as the sum of several components:

[0079]

[0080]

[0081]

[0082] Where L coor L raw and L conf These represent the losses in the 7-DOF coordinate, rotation, and object confidence tasks, respectively. In the above formulas, x... i y i z i Let i and w represent the coordinates of the center of the selected point i (i.e., the true value); i l i h i θ represents the width, length, and height of the bounding box, respectively; i C is the rotation angle; i , where is the confidence level. These represent the predicted center coordinates of the points; These represent the predicted bounding box width, length, and height, respectively; The predicted rotation angle; S represents the confidence level of the prediction. S represents the number of grid cells in the image segmentation; B represents the number of bounding boxes to be identified. It is a variable whose value of 0 or 1 depends on whether there is a true value at the i-th and j-th positions. If there is a bounding box, it is 1; otherwise, it is 0.

[0083] The key feature of this invention, which uses specific anchor points to delineate the boundary range of rock debris, is that it has better adaptability compared to the general identification algorithm that uses regression bounding boxes to delineate the length, width, and height range of the target object. Specifically, the method employed is k-means spatial clustering. Figure 3 These are the steps for implementing clustering.

[0084] The final loss function in the MuckNet network is calculated using the following formula:

[0085]

[0086] Where a and b are the ground truth bounding box and the predicted bounding box, respectively.

[0087] For the specific target of rock debris detection, the training data needs to be augmented before the MuckNet network learns:

[0088] To make the detection and recognition results more stable, the input data needs to be randomly transformed to enhance the diversity and quantity of training data, improve the model's adaptability to a wider range of scenarios, and reduce the risk of overfitting. Data augmentation methods include, but are not limited to, scaling within the range of (0.9, 1.1), angular rotation from -20° to 20°, and horizontal axis mirroring.

[0089] In this embodiment, the enhancement method used in the feature mapping space of the rock debris is as follows: randomly delete some data, set the data segmentation probability to 1, the segmentation ratio to 0.3, and fill the segmented region with the value 1. The enhanced data is then used as the input to the MuckNet model.

[0090] In this embodiment, the input of the MuckNet model is a three-dimensional point cloud image of rock slag, and the output is the rock slag particle size and volume within the image range and the identified bounding box results.

[0091] 150 training set samples, 54 validation set samples, and 59 test set samples were obtained from an engineering site. The MuckNet model was used with 300 iterations, and validation was performed after every 50 iterations. All input data was divided into 10 batches for training. An Adam optimizer with a learning rate of 0.0025 and a cosine optimization scheduler were used. The final prediction results are as follows: Figure 5 As shown, MuckNet-C indicates that the MuckNet network does not include rock slag curvature features; MuckNet-CA indicates that the MuckNet network does not include rock slag curvature features and bounding box classification; MuckNet-MSE indicates that the MuckNet network uses the Lmse loss function; and finally, the results obtained by the method of this invention are shown. The comparison shows that adding rock slag curvature features and reconstructing the GIoU loss function can improve the recognition accuracy, proving the reliability of the proposed scheme in this embodiment.

[0092] Example 2

[0093] A three-dimensional morphology perception system for rock slag based on linear laser array and deep learning, comprising:

[0094] In-situ acquisition device, used to acquire point cloud images of rock debris at the target location;

[0095] The preprocessing module is used to preprocess the rock debris point cloud image, trim useless fragments, eliminate error points in the data, segment the valid rock debris point cloud data, and perform rock debris feature labeling.

[0096] The perception and recognition module is used to process the pre-processed rock debris point cloud image using the trained detection and recognition network model. The detection and recognition network model converts the rock debris point cloud image into a feature map, extracts rock debris features containing curvature features, performs learning mining, filters redundant and overlapping detection bounding boxes, and obtains the rock debris particle size and volume and the recognized bounding box results.

[0097] The in-situ acquisition device includes a belt conveyor, a support frame, and a linear laser scanner. The belt conveyor is mounted on a TBM, the support frame is located next to the belt conveyor, and the linear laser scanner is mounted on the support frame, located above the belt of the belt conveyor.

[0098] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for slag three-dimensional form perception based on linear array laser and deep learning, characterized in that, Includes the following steps: Acquire point cloud images of rock debris based on linear laser array acquisition; The rock debris point cloud image is preprocessed by trimming useless fragments, eliminating error points in the data, segmenting the valid rock debris point cloud data, and marking the rock debris features. The pre-processed rock debris point cloud image is processed using the trained detection and recognition network model. The detection and recognition network model converts the rock debris point cloud image into a feature map and extracts rock debris features containing curvature features. The model performs learning mining, filters redundant and overlapping detection bounding boxes, and obtains the results of rock debris particle size and volume and the identified bounding boxes. The detection and recognition network model is a MuckNet neural network model based on YOLO3D, which includes an input layer, an encoder, a decoder and an output layer connected in sequence. The input layer is used to receive point cloud images and extract features. The encoder and decoder of the YOLO3D network are used to transform rock debris features and learn and mine them through convolutional neural networks to obtain structural representations at different scales. The output layer is used to filter out redundant and overlapping detection bounding boxes using non-maximum suppression. Curvature is introduced as one of the features in the rock debris characteristics. For each rock debris, there is a feature mapping relationship: in, H For height map, D For density plots, C For curvature, P These three features, representing the mapping function, are used together for the detection and identification of rock debris. The curvature is: in Patch Point i neighboring points N ( i The best-fit quadratic curve around ) Cur Representing a point ( i x , i y , i z The function of the curvature of the quadratic curve at point ().

2. The method for three-dimensional morphological perception of rock slag based on linear laser array and deep learning as described in claim 1, characterized in that, Rock debris point cloud images are acquired according to a set three-dimensional interval, where the three-dimensional interval is the ratio of the rock debris moving speed to the image scanning frequency.

3. The method for three-dimensional morphological perception of rock slag based on linear laser array and deep learning as described in claim 1, characterized in that, The Crop Section algorithm is used to overlap and segment the effective rock debris point cloud data, and Sustech Points is used to mark the bounding box for each rock debris with a coverage rate below a set threshold, and the marking information is saved.

4. The method for three-dimensional morphological perception of rock slag based on linear laser array and deep learning as described in claim 1, characterized in that, The k-means clustering method was used to obtain representative types of rock debris bounding boxes. A and B The distance between them is defined as: in, IOU ( A , B ) represents the bounding box A and B The intersection of.

5. The method for three-dimensional morphological perception of rock slag based on linear laser array and deep learning as described in claim 4, characterized in that, The detection and recognition network model uses GIoU as the loss function and employs length, width, height, and center point. x , y , z Each bounding box is defined by 7 degrees of freedom, including the eccentricity angle. When detecting and identifying rock debris, the error in predicting the bounding box is calculated using the mean square error, which is then used to regress the loss of the sum of the aforementioned degree-of-freedom components.

6. The method for three-dimensional morphological perception of rock slag based on linear laser array and deep learning as described in claim 1, characterized in that, When training the detection and recognition network model, training samples are used for training. During training, the training samples are enhanced by randomly deleting some data, setting the data segmentation probability and segmentation ratio, and filling the segmentation area with the value 1.

7. A three-dimensional morphology perception system for rock slag based on linear laser array and deep learning, employing the method described in claim 1, characterized in that, include: In-situ acquisition device, used to acquire point cloud images of rock debris at the target location; The preprocessing module is used to preprocess the rock debris point cloud image, trim useless fragments, eliminate error points in the data, segment the valid rock debris point cloud data, and perform rock debris feature labeling. The perception and recognition module is used to process the pre-processed rock debris point cloud image using the trained detection and recognition network model. The detection and recognition network model converts the rock debris point cloud image into a feature map, extracts rock debris features containing curvature features, performs learning mining, filters redundant and overlapping detection bounding boxes, and obtains the rock debris particle size and volume and the recognized bounding box results.

Images