A shield machine construction early warning method and device
Through convolutional neural network and CPU+FPGA heterogeneous calculation, real-time monitoring of cutter disc wear and surrounding rock integrity in tunnel boring is achieved, solving the problems of difficulty in detection and impact of interfering objects in traditional methods, and improving construction warning Accuracy and efficiency.
Patent Information
- Application Number
- CN202210047086.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-14
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-01-14
AI Technical Summary
During tunnel excavation, traditional shield machine tool blade wear detection is difficult to detect in time, affecting construction efficiency and quality, and the surrounding rock grading is rough, making it difficult to obtain accurate support measures; at the same time, the construction environment is complex, and interfering objects affect image recognition .
Using a computer vision method based on convolutional neural network, rock slag and surrounding rock images are collected through image sensors, features are extracted using convolutional layers, and combined with full connection layers and decoder for classification and segmentation to achieve rock slag classification and surrounding rock integrity Monitor and perform calculations through the CPU+FPGA heterogeneous computing acceleration architecture.
Real-time monitoring of tool disc wear and surrounding rock integrity detection is achieved, the accuracy and efficiency of construction warning is improved, the impact of interfering objects during construction is reduced, and reasonable tool maintenance and support design is guided.
Smart Images

Figure CN114320316B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of tunnel excavation, and in particular relates to a shield machine construction early warning method and device. Background Art
[0002] The full-face rock tunnel boring machine (TBM) has the advantages of fast construction speed, high safety and good economy. It has been widely used in major projects such as highways, railway transportation, and urban subways. Due to the harsh working environment, the cutter disc is very susceptible to abnormal wear and damage. If the damage to the cutter cannot be discovered and handled in time during the construction process, it will not only reduce the excavation efficiency, but also cause abnormal wear of the cutter disc, affecting the progress and quality of the project. Therefore, it is particularly necessary to study the wear causes of shield cutters and monitor the wear of the cutters in real time, and to reasonably select, use, maintain and replace the cutters. According to the rock breaking mechanism of the TBM cutter disc, when the cutter disc is worn or damaged, the size of the rock debris fragments generated by excavation will increase. Therefore, the state of the cutter disc can be indirectly monitored by monitoring the rock debris fragments on the conveyor belt, and the on-site construction personnel can be guided to check and replace the cutter in time. The camera collects images above the conveyor belt, avoiding the harsh production environment. It also has the advantages of simple equipment, low loss, low cost, and long-term monitoring.
[0003] At the same time, the classification of tunnel surrounding rock grade plays a vital role in tunnel construction. It is an important factor in determining the stability of the tunnel after excavation and selecting corresponding support measures. The traditional surrounding rock classification is based on the strength of the rock, the integrity of the rock mass, crack fillings, groundwater and ground stress. Due to the large workload and difficulty in obtaining, the surrounding rock classification in the early survey is relatively rough. The tunnel construction environment and procedures are complex. The TBM during construction may affect the shooting line of sight. At the same time, certain support measures will be adopted after the surrounding rock is excavated to ensure the stability of the surrounding rock. For example, anchor rods, steel meshes, steel arch frames, etc. will interfere with image recognition. The surrounding rock integrity detection network based on the fully convolutional neural network can effectively avoid the influence of interference. At the same time, the image segmentation network can separate the cracks from the background, and more intuitively display the current surrounding rock integrity, thereby guiding construction personnel to choose appropriate support design and construction according to the type of surrounding rock.
[0004] Based on the above background, the present invention proposes a shield machine construction early warning method and device based on convolutional neural network. Rock slag classification and surrounding rock integrity monitoring are realized through convolutional neural network, and image acquisition and network calculation are realized through the proposed device. Summary of the invention
[0005] The present invention provides a shield machine early warning method and device based on computer vision, including a number of image sensors for collecting image information, a heterogeneous computing device, and a display for displaying the results. To achieve the above purpose, the present invention adopts the following technical solutions:
[0006] Step 1: Collect images of rock slag and surrounding rock during excavation by using image acquisition devices distributed at different positions of the shield machine.
[0007] Step 2: The image is transmitted to the heterogeneous computing terminal, and after preprocessing, it is input into different convolutional neural networks to perform network calculations on the input rock slag image and surrounding rock image respectively.
[0008] Step 3: Feature extraction. The convolutional layer is used for feature extraction branch. For the rock debris classification network, the convolutional layer is connected to the fully connected layer for classification. For the surrounding rock integrity detection network, the decoding layer is connected for classification and anomaly segmentation.
[0009] Step 4: Feature classification: The slag image is divided into normal and abnormal categories through the Softmax function, and the surrounding rock image is divided into complete and incomplete areas through the decoder.
[0010] Step 5: Output the classification results to a display device for display, thereby helping construction personnel determine the current construction status. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 The overall flow chart of the slag classification method based on convolutional neural network involved in the present invention is as follows
[0012] Figure 2 This is an example of a rock slag dataset.
[0013] Figure 3 A surrounding rock integrity monitoring method based on a fully convolutional network
[0014] Figure 4 Schematic diagram of surrounding rock integrity detection network calculation results
[0015] Figure 5 Convolutional neural network heterogeneous computing architecture based on CPU+FPGA
[0016] Figure 6 Overall system architecture DETAILED DESCRIPTION
[0017] The present invention is used to provide a shield machine early warning method and device based on computer vision. In order to make the technical scheme and effect of the present invention clearer and more specific, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.
[0018] like Figure 1As shown, a flow chart of a rock slag classification algorithm based on a convolutional neural network designed by the present invention. The rock slag classification network is composed of three network layers, namely a convolutional layer, a pooling layer, and a fully connected layer. The function of the convolutional layer is to extract features, and it is the most important calculation layer of the convolutional neural network. High-level features in the image can be extracted. For each convolution calculation, each output eigenvalue is obtained by adding a bias after the corresponding convolution kernel and the input eigenvalue are multiplied and added. There is no data dependency between the operations of different eigenvalues, which means that the convolution calculations can be calculated in parallel.
[0019] The pooling layer is usually located after the convolution layer to reduce the amount of calculation and control overfitting. Usually, pooling downsampling is performed after each feature map to reduce the dimension of the feature map. There are generally two pooling methods: average pooling and maximum pooling. Maximum pooling is used in the algorithm in this article because maximum pooling can retain more information such as edges and textures of the image, and the extracted features can better realize the classification function.
[0020] The fully connected layer plays the role of a classifier in the entire network. Since the result of the network convolution operation is the feature information of the image, it cannot be manually filtered and processed. Therefore, the fully connected layer is needed to integrate the feature information, classify the features extracted by the convolution layer through the SoftMax classifier, and establish a connection with the sample label space.
[0021] The network training process uses rock slag images collected during production, such as Figure 2 As shown, there are three categories in total, two of which are normal categories and one is abnormal category. The normal category represents that the hob works normally, and the abnormal category represents that the hob works abnormally.
[0022] In the actual production process, rock slag may be randomly distributed in direction, and the camera may be affected by the environment and collect noisy images. Therefore, based on the above characteristics, the data set is augmented by random rotation and adding Gaussian noise, and transfer learning is used to retrain the rock slag classification network. The weights of the pre-trained network are learned into the rock slag classification network, thereby achieving a high-accuracy rock slag classification network.
[0023] like Figure 3As shown, the present invention designs a surrounding rock integrity monitoring method based on a fully convolutional network. A fully convolutional network is a convolutional neural network in deep learning, which is specially used for semantic segmentation of images. For a fully convolutional network, the network input is an image, and the output is also an image. The function of the network is to learn pixel-to-pixel mapping to achieve pixel-level recognition. Using the classic fully convolutional network SegNet, on the basis of the original network, a network structure for surrounding rock integrity recognition is constructed, and the original image and label image in the surrounding rock image sample data set are input into the network. Through the encoder part of the network, the input image is subjected to multiple convolution operations and pooling operations to extract the high-level features of the input image. Then, through the decoder part, convolution and de-pooling operations are performed to restore the high-level features to the same size as the input image. At the same time, the probability of each pixel belonging to an incomplete area is calculated and compared with the corresponding forward direction to achieve a forward reasoning operation. Then, the back propagation algorithm is used to update and learn the network weights. After continuous iterations, the surrounding rock network is designed and trained, thereby obtaining a usable surrounding rock integrity monitoring network. As shown Figure 4 As shown in the figure, after the surrounding rock image is calculated by the network, the surrounding rock cracks and the background are represented by pixels of different colors.
[0024] For the determination of rock mass integrity, Table 1 provides a detailed introduction to different surrounding rock types.
[0025] Table 1: Classification of surrounding rock integrity
[0026]
[0027] According to the degree of development of cracks and the degree of combination of main cracks, the surrounding rock types can be divided into five categories: complete, relatively complete, relatively broken, broken, and extremely broken. Through the calculation of the surrounding rock integrity monitoring network, the cracks and background of the input surrounding rock image are segmented, thereby guiding the construction personnel to classify the surrounding rock for subsequent support design and construction.
[0028] During production, images of surrounding rocks are collected through image sensors, which input the images into heterogeneous terminals and use the terminals for network operations. In terms of algorithm deployment acceleration, most convolutional neural networks are currently operated on CPU / GPU platforms, but there are not many computing units inside the CPU, and the computing power cannot meet the computing needs; GPUs have huge advantages in parallel computing and are mostly used in the training stage of neural networks, but in the inference stage, GPUs cannot give full play to their high bandwidth and multi-computing core characteristics, and face challenges in hardware power consumption, size and other aspects in actual edge application deployment. FPGAs themselves integrate rich storage hardware resources, flexible programmable logic resources and high-performance computing resources. They can not only meet the needs of parallel computing and low power consumption at the same time, but also have high flexibility and reconfigurability, providing an effective solution for the edge deployment of convolutional neural networks.
[0029] Therefore, the terminal designed by this method is based on CPU+FPGA heterogeneous design, and an FPGA convolutional neural network acceleration architecture based on OpenCL development process is proposed to accelerate the slag classification network, such as Figure 5 As shown in the figure. According to the independence of each computing layer of the convolutional neural network, a configurable OpenCL kernel is designed to complete the convolution, pooling and fully connected layer operations of the network respectively. The entire architecture is mainly divided into two parts: on-chip and off-chip. The off-chip global memory is mainly used to store eigenvalues, model parameters and quantization parameter information. The on-chip memory is used to store and transmit the eigenvalue data, weight parameters, quantization parameters and calculation results of the current layer of the current computing layer.
[0030] In this architecture, cores are connected to each other through configurable pipelines. Pipelines are an efficient, non-blocking, first-in, first-out queue data transmission module built using on-chip resources. Data between cores is transmitted directly on-chip, which can reduce the number of accesses to off-chip memory and improve the efficiency of data transmission. By flexibly combining core connections through control words, complex network computing acceleration can be achieved through hardware resources.
[0031] When the control word is 0, the convolution or full connection operation is completed, and the connection mode of the pipeline and the kernel is: off-chip memory - on-chip cache - data pipeline - convolution kernel - data pipeline - on-chip cache - off-chip memory.
[0032] When the control word is 1, the convolution + pooling operation is completed, and the pipeline and kernel connection mode is: off-chip memory - on-chip cache - data pipeline - convolution kernel - data pipeline - pooling kernel - data pipeline - on-chip cache - off-chip memory.
[0033] When the control word is 2, the convolution + depooling operation is completed, and the pipeline and kernel connection mode is: off-chip memory - on-chip cache - data pipeline - convolution kernel - depooling kernel - data pipeline - on-chip cache - off-chip memory.
[0034] When the data of the related connection mode is piped, each core is executed only once, and there will be no contention for core resources, thus ensuring the smooth flow of the overall pipeline. Through the combination of control words, the control architecture can be used to calculate the rock slag classification network or the surrounding rock integrity monitoring network.
[0035] Therefore, combining the method and the device, the complete system is as follows Figure 6 As shown, the system working steps are as follows:
[0036] 1. The image sensor above the slag conveying belt and the image sensor outside the shield machine collect the slag image and the surrounding rock image respectively.
[0037] 2. The image sensor transmits the collected rock debris images and surrounding rock images to the heterogeneous computing device, and the device CPU crops the images to suit the network input requirements.
[0038] 3. The slag image and the surrounding rock image are respectively input into the network acceleration architecture proposed in the present invention. Through the scheduling on the CPU side and the intensive calculation on the FPGA side, the flow of the feature map data is controlled according to the set control word, thereby realizing the operation of the slag classification network and the surrounding rock integrity monitoring network.
[0039] 4. After the calculation is completed, the original image and output results are output from the FPGA to the CPU and displayed after processing. Among them, the output of the rock slag classification network displays the current rock slag image and its type; the output of the surrounding rock integrity network displays the current surrounding rock image and its binary image after image segmentation.
Claims
1. A shield machine construction early warning device, characterized in that: include: Visual sensors, heterogeneous computing devices, and operating room displays are used to identify the state of rock slag and the integrity of surrounding rock; among them: The visual sensor uses an industrial lens that is resistant to high temperature and high humidity, and is equipped with a zoom lens and a fill light to achieve image acquisition. The visual sensor is set above the slag conveyor belt and in front of the shield machine, and is used to collect images of slag on the slag conveyor belt and surrounding rocks outside the shield machine. In order to cooperate with subsequent network calculations, the image sensor also has the function of acquiring images of different resolutions; A heterogeneous computing device is connected to a visual sensor and is used to receive and process images collected by the visual sensor. The device includes a CPU and an FPGA. The CPU and the FPGA collaborate in heterogeneous computing to implement an early warning method based on a convolutional neural network. Specifically, in the OpenCL process, the CPU is responsible for data preprocessing and scheduling, and the FPGA is responsible for intensive data computing. According to the independence between the computing layers of the CNN, a configurable OpenCL kernel is designed to complete the network's convolution, pooling, de-pooling, and fully connected layer operations respectively. The entire architecture is divided into two parts: on-chip and off-chip. The off-chip global memory is used to store eigenvalues, model parameters, and quantization parameter information. The on-chip memory is used to store and transmit the eigenvalue data, weight parameters, quantization parameters, and calculation results of the current computing layer. In this architecture, the kernels are connected to each other through a configurable pipeline. The pipeline is an efficient, non-blocking, first-in, first-out queue (FIFO) data transmission module built using on-chip resources, so that data between kernels is directly transmitted on-chip, which can reduce the number of accesses to the off-chip memory and improve the efficiency of data transmission. The kernel connection can be flexibly combined through control words to achieve complex network computing acceleration through hardware resources. The operating room display is used to display the acquired image information and its corresponding calculation results; For rock slag images, a rock slag classification algorithm based on convolutional neural network was designed and deployed; for surrounding rock images, a surrounding rock integrity monitoring network based on image segmentation was designed and deployed, and network computing was realized through heterogeneous computing architecture. The rock slag classification network was modified and trained by VGG16 network. The rock slag images collected in engineering scenes were used to transfer learning of the pre-trained network to obtain the rock slag classification network. The rock slag classification result was positively correlated with the current state of the shield machine cutter disc. By monitoring the rock slag state, the purpose of indirectly monitoring the cutter state can be achieved. The surrounding rock integrity monitoring network uses the image segmentation network SegNet network to realize semantic segmentation of the input image, classify the pixels of the input surrounding rock image into incomplete surrounding rock area and background area, and obtain the specific location and accurate geometric information of the incomplete area. The method for constructing the surrounding rock integrity monitoring network is as follows: on the basis of the fully convolutional network SegNet, a network structure for surrounding rock integrity identification is constructed, the original image and the label image in the surrounding rock image sample data set are input into the network, and the input image is subjected to multiple convolution operations and pooling operations through the encoder part of the network to extract the high-level features of the input image, and then the high-level features are restored to the same size as the input image through the decoder part through convolution and de-pooling operations, and the high-level features are restored to the same size as the input image, and at the same time, the probability of each pixel belonging to the incomplete area is calculated and compared with the corresponding forward direction to realize a forward reasoning operation, and then the network weights are updated and learned by the back propagation algorithm, and the surrounding rock network is designed and trained through continuous iterations, thereby obtaining a usable surrounding rock integrity monitoring network.
2. The shield machine construction early warning device according to claim 1 is characterized in that: The rock slag classification algorithm based on convolutional neural network includes convolutional layer, pooling layer and fully connected layer. The rock slag image features are extracted through the convolutional layer, the network calculation amount is reduced and the overfitting of the network is controlled through the pooling layer, the role of the classifier is realized through the fully connected layer, and the connection between feature information and label space is established.
3. The shield machine construction early warning device according to claim 2 is characterized in that: The surrounding rock complete monitoring network includes an encoder and a decoder, wherein the encoder includes a convolution layer and a pooling layer. In the encoder, the input surrounding rock image is subjected to multiple convolution operations and pooling operations to generate a high-level feature map, which is then passed through the decoder to restore the abstract high-level features to the same size as the input image through convolution and de-pooling operations. At the same time, after network operations, each pixel point in the output image is set with a prediction value to indicate whether cracks or water seepage occur in the surrounding rock at the corresponding position of the pixel point. The stability judgment during the construction process and the corresponding support measures selection strategy are guided according to the segmented image.
