A real-time early warning method and system for face collapse based on machine vision technology

Abstract

The present invention discloses a real-time early warning method and system for face collapse based on machine vision technology. The method includes obtaining rock ballast point cloud data collected by a laser scanner and rock ballast image data collected by a camera per unit time; performing polynomial fitting and facet integration processing on the rock ballast point cloud data to determine the output volume per unit time of the rock ballast; using a deep learning model to identify the rock ballast image data to determine the morphological characteristics of the rock ballast per unit time; and using a preset model to perform real-time early warning of face collapse according to the changes in the output volume per unit time of the rock ballast and the morphological characteristics of the rock ballast in different unit times. The solution provided by the present invention can monitor the morphological characteristics of rock ballast and the output volume per unit time in real time, and issue a warning in time when the morphology of the rock ballast changes and the output volume per unit time of the rock ballast changes suddenly.

Description

Technical Field

[0001] The present invention relates to the field of intelligent early warning technology, and in particular to a real-time early warning method and system for tunnel face collapse based on machine vision technology. Background Art

[0002] The full-face rock tunnel boring machine (TBM) is a highly mechanized and automated large-scale tunnel excavation and lining equipment. It uses the cutter on the cutter head to break the rock and then transports the rock chips out through a belt conveyor. In TBM excavation operations, face collapse is a typical geological disaster, which has a significant correlation with the morphological changes of rock chips and the changes in the output volume of rock chips per unit time. First, in terms of the morphology of rock chips, under normal excavation conditions, rock chips present a regular or broken shape determined by the crushing action of the cutter, and its characteristics are affected by many factors such as rock physical properties and TBM excavation parameters. However, once the face collapses, this morphology will change dramatically. During the collapse process, the rock mass suddenly breaks and collapses due to instability, resulting in the size of the rock chips increasing, the shape becoming more irregular, and even containing large pieces of rock, which directly reflects the dynamic destruction process of the rock mass at the face. Secondly, with regard to the changes in the output volume of rock chips per unit time, face collapse will cause a significant increase in this volume. This is because during a landslide, the rock mass fragmentation rate accelerates dramatically, and a large amount of rock mass breaks and falls into the TBM's mucking system in a short period of time. Compared with normal excavation, the amount of rock slag generated during a landslide increases significantly, which in turn increases the volume of rock slag output per unit time.

[0003] At present, the main method for obtaining the morphological characteristics of rock slag and the change of output volume per unit time is manual observation. Manual observation is highly subjective, limited by the experience of the observer, and has disadvantages such as time-consuming and labor-intensive, and strong data lag, which cannot meet the TBM's demand for real-time acquisition of rock slag information. Summary of the invention

[0004] In order to solve the technical problems of time-consuming and labor-intensive manual observation of rock chip morphological characteristics and output volume changes per unit time and data lag, an embodiment of the present invention provides a real-time early warning method and system for face collapse based on machine vision technology.

[0005] The technical solution of the embodiment of the present invention is achieved as follows:

[0006] An embodiment of the present invention provides a real-time early warning method for tunnel face collapse based on machine vision technology, the method comprising: acquiring rock ballast piece point cloud data collected by a laser scanner and rock ballast piece image data collected by a camera per unit time; performing polynomial fitting and surface element integration processing on the rock ballast piece point cloud data to determine the output volume of the rock ballast piece per unit time; using a deep learning model to identify the rock ballast piece image data to determine the morphological characteristics of the rock ballast piece per unit time; and using a preset model to perform real-time early warning of tunnel face collapse according to changes in the output volume of the rock ballast piece per unit time and the morphological characteristics of the rock ballast piece in different unit times.

[0007] In one embodiment, polynomial fitting and bin integration processing are performed on the point cloud data of the rock ballast piece to determine the output volume per unit time of the rock ballast piece, including:

[0008] The three-dimensional coordinate system is constructed with the scanner installation position as the origin, the direction perpendicular to the belt conveyor as the z-axis, the moving direction of the belt conveyor as the y-axis, and the direction of the y-axis rotated 90° counterclockwise as the x-axis;

[0009] Based on the three-dimensional coordinate system, determining the coordinates of each point cloud in the rock ballast piece point cloud data;

[0010] According to the coordinates of each point cloud, the point cloud is cropped with the coordinates specified on the x-axis as the boundary, the edge contour point cloud of the belt conveyor is obtained, the contour edge point cloud is fitted with a polynomial based on the least squares method, the belt conveyor contour calibration curve is obtained, and the first reference area of ​​the area formed by the origin and the belt conveyor contour calibration curve is calculated;

[0011] According to the coordinates of each point cloud, every two adjacent point clouds in the rock ballast piece point cloud data are connected with the origin to construct a plurality of tiny triangles, and the areas of all the constructed tiny triangles are accumulated to obtain a second reference area of ​​the region formed by the origin and the rock ballast contour curve;

[0012] Subtract the second reference area from the first reference area to obtain the cross-sectional area of ​​the rock ballast piece;

[0013] The running distance of the belt conveyor per unit time is obtained, and the cross-sectional area of ​​the rock ballast piece profile is integrated over the running distance to obtain the output volume of the rock ballast piece per unit time.

[0014] In one embodiment, the areas of all constructed tiny triangles are accumulated to obtain a second reference area of ​​the region formed by the origin and the rock ballast contour curve, including:

[0015] The second reference area of ​​the region formed by the origin and the rock ballast contour curve is obtained using the following calculation formula:

[0016]

[0017] in, is the second reference area, is the number of points scanned at one time, is the first The distance from a point to the origin, is the first The distance from a point to the origin, For line segment and line segments The angle between.

[0018] In one embodiment, after determining the coordinates of each point cloud in the rock chip point cloud data, the method further includes:

[0019] The outliers and abnormal points in the rock chip point cloud data are removed using the following calculation formula:

[0020]

[0021] in, is the post-filter point The new value of Yes Points in the neighborhood, Indicate point The neighborhood of A three-dimensional sphere or cube is centered on the neighborhood. The median function takes the median after sorting the points in the neighborhood.

[0022] In one embodiment, the rock ballast image data is identified using a deep learning model to determine the morphological features of the rock ballast per unit time, including: bilateral filtering and adaptive histogram equalization processing of the rock ballast image data to eliminate noise in the rock ballast image data and enhance the contrast of the rock ballast image data; annotating and data enhancement processing of the processed rock ballast image data to obtain a data set; using the data set to train a deep learning model to obtain a trained deep learning model; using the trained deep learning model to obtain a trained deep learning model; and using the trained deep learning model to obtain a trained deep learning model. The deep learning model is used to identify the rock ballast piece image data to obtain the mask result output by the model; the rectangular bounding box of each rock ballast piece mask is obtained by the bounding box method, and the length and width of the rectangular bounding box are used as the major axis parameter and minor axis parameter of the rock ballast piece, the pixel area of ​​the rock ballast piece mask is calculated, and its actual area is obtained by the calibration algorithm, and each preset picture is used as a cycle to statistically analyze the distribution of the major and minor axis parameters and areas of all rock ballast pieces as the morphological characteristics of the rock ballast piece per unit time; wherein the deep learning model is an improved Mask R-CNN model; the improved Mask R-CNN model is obtained by: replacing the ResNet network in the Mask R-CNN model with the ResNeXt network; and adding a bottom-up path enhancement module and a feature fusion operation module to the feature pyramid network in the Mask R-CNN model.

[0023] In one embodiment, bilateral filtering and adaptive histogram equalization processing are performed on the rock chip image data, including:

[0024] The following calculation formula is used to perform bilateral filtering on the rock chip image data:

[0025]

[0026] in, is the pixel value after filtering, is the pixel value of the original image, is a collection of pixels within a window, the window is Centered on Represents the pixel coordinates within the window, is a spatial domain weight function, which is used to measure the spatial distance between pixels. is a grayscale domain weight function, which is used to measure the difference between pixel values. is a normalization constant, which is used to ensure that the pixel value after filtering is between 0 and 255. It is obtained by summing the weights of all pixels in the window;

[0027] The following formula is used to perform adaptive histogram equalization processing on the rock chip image data:

[0028]

[0029] in, is the grayscale of the output image, is the grayscale of the input image, T Represents an adaptive histogram equalization operation.

[0030] In one embodiment, a preset model is used to perform a real-time early warning of face collapse according to the changes in the output volume per unit time of the rock ballast piece and the morphological characteristics of the rock ballast piece in different unit times, including: calculating the roughness index of the rock ballast according to the morphological characteristics of the rock ballast piece; obtaining the total thrust and cutterhead torque of the full-face rock tunnel boring machine during operation; inputting the total thrust, the cutterhead torque, the roughness index of the rock ballast and the output volume per unit time into the preset model to obtain the face collapse result output by the preset model; the preset model is a SVM model; and performing a real-time early warning of face collapse according to the face collapse result.

[0031] In one embodiment, the roughness index of the rock ballast is calculated according to the morphological characteristics of the rock ballast piece, including:

[0032] The roughness index of rock ballast is calculated using the following formula:

[0033]

[0034] in, is the roughness index of rock ballast; is the area residual screening rate of particle size; It is the total area of ​​rock debris with minor axis larger than a certain particle size; is the total area of ​​each group of rock ballast.

[0035] An embodiment of the present invention also provides a real-time warning system for face collapse based on machine vision technology, comprising: a processor and a memory for storing a computer program that can be run on the processor; wherein the processor executes the steps of the above-mentioned method when running the computer program.

[0036] An embodiment of the present invention further provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0037] This embodiment has the following beneficial effects:

[0038] This embodiment can monitor the morphological characteristics and output volume per unit time of rock slag in real time based on machine vision technology. When the morphology of rock slag changes and the output volume per unit time of rock slag changes suddenly, a warning is issued in time and an early warning is carried out. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a real-time early warning method for tunnel face collapse based on machine vision technology according to an embodiment of the present invention;

[0040] Figure 2 It is a schematic diagram of the process of the real-time early warning method for tunnel face collapse according to an embodiment of the present invention;

[0041] Figure 3 This is a schematic diagram of the structure of a data acquisition submodule according to an embodiment of the present invention;

[0042] Figure 4 Schematic diagram of the calculation principle of the output volume per unit time of rock chip according to an embodiment of the present invention; wherein (a) is the reference area Schematic diagram of calculation principle; (b) is the airspace area Schematic diagram of calculation principle;

[0043] Figure 5 This is a schematic diagram of improving the deep learning model according to an embodiment of the present invention;

[0044] Figure 6 1 is a diagram showing the internal structure of a computer device according to an embodiment of the present invention.

[0045] Figure 3 Middle: 11-mounting bracket, 22-laser scanner, 33-industrial camera, 44-LED light source, 55-speed encoder. DETAILED DESCRIPTION

[0046] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0047] The embodiment of the present invention provides a real-time early warning method for tunnel face collapse based on machine vision technology, such as Figure 1 As shown, the method includes:

[0048] Step 101: acquiring rock chip point cloud data collected by a laser scanner and rock chip image data collected by a camera per unit time;

[0049] Step 102: performing polynomial fitting and surface element integration processing on the point cloud data of the rock ballast piece to determine the output volume of the rock ballast piece per unit time; using a deep learning model to identify the image data of the rock ballast piece to determine the morphological characteristics of the rock ballast piece per unit time;

[0050] Step 103: Based on the changes in the output volume of the rock chip per unit time and the morphological characteristics of the rock chip per unit time, a preset model is used to perform a real-time early warning of tunnel face collapse.

[0051] This embodiment adopts a non-contact intelligent recognition method based on machine vision to observe the morphological characteristics of rock chips and the change in output volume per unit time, which has the advantages of real-time, objectivity, and easy integration.

[0052] This embodiment first obtains rock ballast point cloud data through a combination of a laser scanner and an incremental encoder, and uses a combination of polynomial fitting and surface element integration method to obtain the output volume per unit time of the rock ballast above the belt conveyor; then, the rock ballast image data is obtained through an array camera, and a multi-scale fusion mechanism and a path aggregation network are added to the original deep learning model to enhance the model's perception of rock ballast of different scales and improve recognition accuracy. The improved deep learning model is used to perform rock ballast image segmentation to obtain the morphological characteristics of the rock ballast; finally, based on the obtained output volume per unit time of the rock ballast and the morphological characteristics of the rock ballast, when the output volume is compared with the theoretical excavation volume and there is an obvious increasing trend, and the size of the rock ballast increases and the shape becomes more irregular, a face collapse warning is issued.

[0053] Specifically, in order to obtain the output volume and morphological characteristics of the rock ballast pieces above the belt conveyor per unit time and realize the real-time early warning of tunnel face collapse, this embodiment combines the point cloud data and image data of the rock ballast pieces and proposes a real-time early warning method for tunnel face collapse based on the morphological characteristics of the rock ballast pieces and the output volume per unit time. The main process of this method is as follows Figure 2 As shown:

[0054] Firstly, the rock slag identification and measurement system integrating laser scanner, camera, encoder, light source and industrial computer is installed on the TBM slag conveyor. Then, the point cloud and image data of the rock slag sheet above the belt conveyor are synchronously collected by the laser scanner and camera, and the data is transmitted to the industrial computer. For the point cloud data, the output volume per unit time of the rock slag sheet on the belt conveyor is obtained by means of preprocessing, belt conveyor contour fitting, and surface element integration method. For the image data, the morphological characteristic parameters of the rock slag sheet on the belt conveyor are obtained by means of preprocessing, deep learning model recognition, and directional bounding box. Finally, the obtained data is used to complete the real-time early warning of face collapse.

[0055] The rock chip identification and measurement system consists of three subsystems: data acquisition, communication control, and data analysis.

[0056] The data acquisition subsystem uses machine vision technology to collect rock ballast data on the TBM ballast conveyor. Figure 3As shown, the subsystem includes a mounting bracket 11, a laser scanner 22, an industrial camera 33, an LED light source 44 and a speed encoder 55. The industrial camera 33 is fixed above the belt conveyor and can continuously capture rock slag image data. The speed encoder 55 is connected to the belt conveyor drive shaft to provide belt conveyor movement information. The LED light source 44 provides light compensation to ensure clear image quality. The laser scanner 22 is fixed above the belt conveyor to continuously capture rock slag point cloud data.

[0057] The communication and control subsystem is responsible for connecting the camera, scanner and industrial computer, and controlling the opening and closing of the camera and scanner, parameter adjustment and data transmission through the industrial computer.

[0058] The data analysis subsystem is located on an industrial computer. It obtains the output volume per unit time and the morphological characteristic parameters of the rock chips above the belt conveyor through calculations, stores the data, and finally uses these data to complete real-time early warning of face collapse.

[0059] For the acquired point cloud data, in the preprocessing stage, we first need to build a three-dimensional coordinate system based on the scanner, with the scanner installation position as the starting origin O, the z-axis perpendicular to the belt conveyor direction, the movement direction of the belt conveyor as the y-axis, and rotate it 90° counterclockwise to establish the x-axis. Then, use the median filter to remove the outliers and abnormal points in the original point cloud. The principle of median filtering is to sort the coordinate values ​​of the point cloud in a window and select the middle value as the filtering result. This method can remove noise and outliers in the point cloud data while retaining the overall shape and structure of the data. The formula is as follows:

[0060]

[0061] In the formula, is the post-filter point The new value of Yes Points in the neighborhood, Indicate point The neighborhood of A three-dimensional sphere or cube is centered on the neighborhood. The median function takes the median after sorting the points in the neighborhood.

[0062] Since only the surface of the rock slag on the belt conveyor is exposed, in order to measure the volume of the rock slag passing through in a period of time, it is necessary to first measure the contour of the empty belt conveyor surface, determine the reference contour, and then obtain the surface contour of the rock slag pile, so as to obtain the rock slag volume section under the scanner for each scan, and thus obtain the volume and volume flow rate within a period of time. First, a three-dimensional spatial coordinate system is established, and the scanner installation position point is determined in the coordinate system. Then, the coordinates of all feature points of the rock slag on the surface of the belt conveyor are obtained through the point cloud data acquisition system composed of the LMS511 scanner and the speed sensor, that is, the rock slag surface contour point cloud (point clouds), and the point cloud data is constructed into a series of regular volume microelements that can be calculated. Finally, the volume accumulation can calculate the volume of the rock slag passing through in one scanning time, and the volume flow rate of the rock slag is the volume of the rock slag passing through the scanning section of the scanner per unit time.

[0063] For the preprocessed point cloud data, since the part of the belt blocked by rock slag will sink due to the weight of the rock slag, the point cloud is further cropped with the specified coordinates on the x-axis as the boundary to obtain the edge contour point cloud of the belt conveyor. The belt conveyor contour calibration curve L is obtained by performing polynomial fitting on the contour edge point cloud based on the least squares method.

[0064] In order to obtain the output volume per unit time of the rock slag above the belt conveyor, it is necessary to convert the point cloud data per unit time into calculable volume elements and accumulate them. The volume element is calculated by multiplying the cross-sectional area of ​​the rock slag obtained by each scan by the distance between the two scanned sections. To obtain the cross-sectional area of ​​the rock slag for each scan, the surface element integration method can be used to calculate it. For the rock slag contour L obtained by a certain scan per unit time, k The calculation method of the cross-sectional area of ​​rock ballast is as follows, and its principle diagram is as follows Figure 4 As shown: First, take the scanner installation position as the origin O, and calculate the reference area composed of the origin and the belt conveyor profile calibration curve L , is the calculation origin O and the rock ballast contour L k The airspace area , L k Every two adjacent points on the slag sheet (i.e. every two adjacent points in the slag sheet point cloud data) are connected with the origin O to construct a small triangle. By adding up the areas of the constructed small triangles, the airspace area can be obtained. , the calculation formula is as follows: is the number of points scanned at one time, For L k On The distance from a point to the origin, is the angle between two points:

[0065]

[0066] In getting the reference area and airspace area Then, subtract the two to get the cross-sectional area of ​​the ballast sheet. .

[0067] Use the following formula to obtain the cross-sectional area of ​​the ballast sheet: Along the coordinate axis z The output volume per unit time of rock ballast can be obtained by integration .

[0068]

[0069] The distance that the belt conveyor travels in one second is After calculating the cross-sectional area of ​​the ballast sheet profile for each scan, By integrating, we get the output volume within this second.

[0070] In the process of image recognition based on supervised deep learning, the model needs a large number of well-annotated image data sets for training and learning. In the field of mature recognition technology, there are often many open source data sets available to enhance the training effect of the model, but there is no well-annotated open source data set in the field of rock chip recognition. Therefore, it is very important to obtain enough rock chip image data and annotate the data set to achieve rock chip recognition.

[0071] In order to improve the learning ability of the deep learning model, the collected rock chip image data needs to be preprocessed before making the data set and performing image annotation. The present invention uses bilateral filtering, adaptive histogram equalization and other methods for preprocessing to eliminate the noise contained in the image and enhance the image contrast. The formula of bilateral filtering is as follows:

[0072]

[0073] In the formula is the pixel value after filtering, is the pixel value of the original image, is a collection of pixels within a window, the window is As the center, Represents the pixel coordinates within the window, is a spatial domain weight function that measures the spatial distance between pixels. Usually, this function is represented by a Gaussian function, whose weight decreases as the distance between pixels increases. is a grayscale domain weight function that measures the difference between pixel values. Similarly, this function is also represented by a Gaussian function, and its weight decreases as the difference in pixel values ​​increases. is a normalization constant, which is used to ensure that the pixel values ​​after filtering are between 0 and 255. It can be obtained by summing the weights of all pixels in the window.

[0074] The formula for adaptive histogram equalization is as follows:

[0075]

[0076] In the formula is the grayscale of the output image, is the grayscale of the input image.

[0077] Then, the pre-processed rock chip image is labeled. Since the image segmentation performed in the present invention is based on instance segmentation, the outline of each rock chip in the image needs to be drawn and the rock chip label needs to be written. In order to improve the generalization ability of the model, the labeled data set needs to be enhanced by adding noise, randomly setting occlusion and other non-geometric transformation operations, as well as rotation, translation, mirror and other geometric transformation operations to expand the original data set.

[0078] The collected rock chip dataset is divided into training set, validation set and test set in the ratio of 6:2:2.

[0079] The labeled image data is input into a deep learning model improved based on the Mask R-CNN model. The deep model is improved on the basis of the original deep learning model in the following ways: the ResNet network in the original model is replaced by the ResNeXt network. The ResNeXt network replaces the ordinary convolution in the ResNet network with the grouped convolution and increases the number of channels, which is equivalent to increasing the width of the network, improving the network's expression ability, and improving the accuracy of the model; in order to enhance the recognition of the edge of the rock chip during the image segmentation of the rock chip, the present invention adopts a path enhancement feature pyramid network, by adding a bottom-up path enhancement module ( Figure 5 The process of generating Ni+1 from the feature map Ni through the 3×3 convolution layer is the bottom-up path enhancement module) and the feature fusion operation module ( Figure 5 In the figure, feature maps P2-P5 are added to feature maps N2-N5 by horizontal connection, which is the feature fusion operation, to enhance the feature hierarchy with rich low-level features. The feature maps generated by the added bottom-up path enhancement module are represented by N2, N3, N4, and N5, which correspond to P2-P5 respectively, where N2 is P2 without any processing, and the calculation process of each newly generated feature map is as follows Figure 5As shown in the figure. In the figure, Ni (specifically N2-N5) represents a higher-resolution feature map. Each feature map Ni first passes through a 3×3 convolution layer with a stride of 2 to reduce the spatial size, and then is horizontally connected to add the coarser feature map Pi+1 to the downsampled feature map. Then, the added feature map passes through a 3×3 convolution layer to obtain the final feature map Ni+1. Finally, the higher-level feature map is merged into the lower-level feature map to complete the feature fusion. The model improvement content is shown in the figure below. Figure 5 shown.

[0080] The deep learning model described in this embodiment is built based on NVIDIA's Pytorch1.13.1 deep learning framework and trained on a Windows system computer equipped with a single GTX4070 graphics card. An epoch-based training cycle is used, and a total of 100 epochs are trained. A verification (val) is performed after each epoch. During the training process, the present invention uses stochastic gradient descent (SGD) as the optimizer, with a momentum of 0.9 and a weight decay of 0.0001. The loss function of the model mainly consists of three parts: classification loss, bounding box regression loss, and mask loss, and its formula is as follows:

[0081]

[0082] The setting of parameters such as learning rate and batch size will affect the efficiency of model training and the final result. When the learning rate is high, the training speed of the model is faster, but the loss will also increase sharply. If the learning rate is set too low, the model convergence speed will decrease, which may cause overfitting. Therefore, the initial learning rate of the model should be set larger to increase the model convergence speed, and the learning rate of the model in the later stage should be appropriately reduced to avoid overfitting. The initial learning rate set by the present invention is 0.0025, and the learning rate is reduced to the original 0.1 at the 80th and 90th epochs to meet the needs of the model.

[0083] Similarly, batch size (batch_size) will also affect the efficiency and accuracy of the model. Larger batch settings can better utilize hardware resources (such as GPU) to accelerate the training process, while reducing the randomness in the training process and making the model converge more stable, but it may also consume more memory and may cause the model to overfit in some cases. On the contrary, a smaller batch_size may require more iterations to complete the training, and the training speed is relatively slow, but it can better avoid overfitting, especially when the data set is small. Taking into account the performance and requirements of the computer, the present invention sets the batch_size of the model to 4.

[0084] For the mask results recognized by the model, the rectangular bounding box of each rock chip mask is obtained by the bounding box method, and the length and width of the rectangular bounding box are used as the major axis parameter and minor axis parameter of the rock chip. The pixel area of ​​the rock chip mask is calculated, and its actual area is obtained through the calibration algorithm. Every twenty pictures are taken as a cycle (the camera shoots twenty images per second), and the distribution of the major and minor axis parameters and areas of all rock chips are counted.

[0085] The rock slag roughness index is a dimensionless number that characterizes the distribution of rock slag particle size generated during TBM rock cutting. It was first proposed by Roxboroug. It reflects the degree of rock slag crushing and its particle size distribution by counting the relative content of particles of different particle sizes in the rock slag, especially the ratio of large-size rock blocks to small-size rock powder. When the proportion of large-size rock blocks is large, the roughness index is large; conversely, when the proportion of small-size rock powder is large, the roughness index is small. Its calculation formula is as follows:

[0086]

[0087] In the formula is the roughness index; is the cumulative screening rate of rock slag of a certain particle size; The total mass of rock slag larger than a certain particle size obtained by screening; is the total mass of each group of rock ballast.

[0088] Since the two-dimensional image can only identify the area of ​​rock debris, CI It is calculated based on the area residual screening rate of the particle size. is the total area of ​​rock debris with a minor axis larger than a certain particle size, is the total area of ​​each group of rock slag. The particle size grades used in this paper are divided according to the standard "Standard for Particle Size Distribution of Pebbles and Crushed Stones for Construction GB / T 14685-2022", which are divided into 11 particle sizes: 4.75, 9.50, 16.0, 19.0, 26.5, 31.5, 37.5, 53.0, 63.0, 75.0 and 90.0 (unit: mm).

[0089] According to existing research, mechanical parameters such as total thrust and cutterhead torque can reflect the effect of the cutterhead on the surrounding rock. When the tunnel face collapses, the thrust will increase and the cutterhead torque will rise.

[0090] The SVM model was established, and the total thrust, cutterhead torque, rock roughness index, and output volume per unit time were used as input parameters of the SVM model. Through the machine learning method, whether the tunnel face collapsed (0 or 1) was used as the output value as the early warning result of the model.

[0091] This embodiment is mainly used for real-time early warning of tunnel face collapse, and its advantages are:

[0092] (1) A rock chip identification and measurement system based on machine vision technology was designed. The system consists of three subsystems: data acquisition, communication control, and data analysis.

[0093] (2) A method for obtaining the output volume of rock slag per unit time based on the surface element integration method is proposed. By performing polynomial fitting on the belt conveyor profile per unit time, a more accurate belt conveyor profile calibration curve is obtained. This reduces the settlement error of the belt conveyor caused by the weight of rock slag during the slag discharge process and improves the calculation accuracy of the output volume of rock slag per unit time.

[0094] (3) In order to address the difficulties in the image recognition of rock debris above the conveyor belt, a deep learning model based on the improved Mask R-CNN model was proposed. By replacing the backbone network and feature pyramid structure, the model's multi-scale feature recognition ability and mask positioning ability were improved, thereby improving the model's recognition accuracy.

[0095] (4) A real-time early warning method for face collapse based on the morphological characteristics of rock shavings and the output volume per unit time was proposed. The method uses industrial cameras and scanners to monitor the morphological characteristics of rock shavings and the output volume per unit time in real time with a cycle of one second. The roughness index, unit output volume, cutterhead torque, total thrust and other parameters of the rock shavings are comprehensively analyzed through the SVM model to determine the collapse situation of the face ahead and issue a warning.

[0096] In order to implement the method of an embodiment of the present invention, an embodiment of the present invention also provides a real-time warning system for face collapse based on machine vision technology, including: a processor and a memory for storing a computer program that can be run on the processor; wherein the processor is used to execute the steps of the above-mentioned method when running the computer program.

[0097] The above-mentioned system provided in this embodiment belongs to the same concept as the above-mentioned method embodiment. The specific implementation process thereof is detailed in the method embodiment and will not be repeated here.

[0098] In order to implement the method of the embodiment of the present invention, the embodiment of the present invention also provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of the above method.

[0099] Based on the hardware implementation of the above program modules, and in order to implement the method of the embodiment of the present invention, the embodiment of the present invention further provides an electronic device (computer device). Specifically, in one embodiment, the computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, the method of any one of the above embodiments is implemented. The display screen A04 of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0100] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0101] The device provided by the embodiment of the present invention includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the method of any one of the above embodiments is implemented.

[0102] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0103] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0106] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0107] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0108] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0109] It can be understood that the memory of the embodiment of the present invention can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAMbus random access memory (DRRAM, Direct Rambus Random Access Memory).The memories described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memories.

[0110] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0111] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A real-time early warning method for tunnel face collapse based on machine vision technology, characterized in that: The method comprises: Obtaining the point cloud data of the rock chip collected by the laser scanner and the image data of the rock chip collected by the camera per unit time; Performing polynomial fitting and surface element integration processing on the point cloud data of the rock ballast piece to determine the output volume of the rock ballast piece per unit time; using a deep learning model to identify the image data of the rock ballast piece to determine the morphological characteristics of the rock ballast piece per unit time; According to the changes in the output volume per unit time of the rock ballast pieces and the morphological characteristics of the rock ballast pieces in different unit times, a preset model is used to provide a real-time early warning of tunnel face collapse; The step of performing polynomial fitting and surface element integration processing on the rock ballast piece point cloud data to determine the output volume per unit time of the rock ballast piece includes: The three-dimensional coordinate system is constructed with the scanner installation position as the origin, the direction perpendicular to the belt conveyor as the z-axis, the moving direction of the belt conveyor as the y-axis, and the direction of the y-axis rotated 90° counterclockwise as the x-axis; Based on the three-dimensional coordinate system, determining the coordinates of each point cloud in the rock ballast piece point cloud data; According to the coordinates of each point cloud, the point cloud is cropped with the coordinates specified on the x-axis as the boundary, the edge contour point cloud of the belt conveyor is obtained, the contour edge point cloud is fitted with a polynomial based on the least squares method, the belt conveyor contour calibration curve is obtained, and the first reference area of ​​the area formed by the origin and the belt conveyor contour calibration curve is calculated; According to the coordinates of each point cloud, every two adjacent point clouds in the rock ballast piece point cloud data are connected with the origin to construct a plurality of tiny triangles, and the areas of all the constructed tiny triangles are accumulated to obtain a second reference area of ​​the region formed by the origin and the rock ballast contour curve; Subtract the second reference area from the first reference area to obtain the cross-sectional area of ​​the rock ballast piece; The running distance of the belt conveyor per unit time is obtained, and the cross-sectional area of ​​the rock ballast piece profile is integrated over the running distance to obtain the output volume of the rock ballast piece per unit time.

2. The real-time early warning method for tunnel face collapse based on machine vision technology according to claim 1 is characterized in that: The areas of all constructed tiny triangles are accumulated to obtain a second reference area of ​​the region formed by the origin and the rock ballast contour curve, including: The second reference area of ​​the region formed by the origin and the rock ballast contour curve is obtained using the following calculation formula: in, is the second reference area, is the number of points scanned at one time, is the first The distance from a point to the origin, is the first The distance from a point to the origin, For line segment and line segments The angle between.

3. The real-time early warning method for tunnel face collapse based on machine vision technology according to claim 1 is characterized in that: After determining the coordinates of each point cloud in the rock ballast piece point cloud data, the method further includes: The outliers and abnormal points in the rock chip point cloud data are removed using the following calculation formula: in, is the post-filter point The new value of Yes Points in the neighborhood, Indicate point The neighborhood of A three-dimensional sphere or cube centered at The function means sorting the points in the neighborhood and taking the median.

4. The real-time early warning method for tunnel face collapse based on machine vision technology according to claim 1 is characterized in that: The rock chip image data is identified using a deep learning model to determine the morphological characteristics of the rock chip per unit time, including: Performing bilateral filtering and adaptive histogram equalization processing on the rock ballast image data to eliminate noise in the rock ballast image data and enhance the contrast of the rock ballast image data; Annotating and data enhancing the processed rock chip image data to obtain a data set; Using the data set to train the deep learning model to obtain a trained deep learning model; Using the trained deep learning model to identify the rock chip image data, and obtaining a mask result output by the model; The rectangular bounding box of each rock chip mask is obtained by the bounding box method, and the length and width of the rectangular bounding box are used as the major axis parameter and minor axis parameter of the rock chip. The pixel area of ​​the rock chip mask is calculated, and its actual area is obtained by the calibration algorithm. Each preset picture is taken as a cycle, and the distribution of the major and minor axis parameters and areas of all rock chips is counted as the morphological characteristics of the rock chip per unit time. Among them, the deep learning model is an improved Mask R-CNN model; the improved Mask R-CNN model is obtained by: replacing the ResNet network in the Mask R-CNN model with the ResNeXt network; adding a bottom-up path enhancement module and a feature fusion operation module to the feature pyramid network in the Mask R-CNN model.

5. The real-time early warning method for tunnel face collapse based on machine vision technology according to claim 4 is characterized in that: The rock chip image data is subjected to bilateral filtering and adaptive histogram equalization processing, including: The following calculation formula is used to perform bilateral filtering on the rock chip image data: in, is the pixel value after filtering, is the pixel value of the original image, is a collection of pixels within a window, the window is As the center, Represents the pixel coordinates within the window, is a spatial domain weight function, which is used to measure the spatial distance between pixels. is a grayscale domain weight function, which is used to measure the difference between pixel values. is a normalization constant, which is used to ensure that the pixel value after filtering is between 0 and 255. It is obtained by summing the weights of all pixels in the window; The following formula is used to perform adaptive histogram equalization processing on the rock chip image data: in, is the grayscale of the output image, is the grayscale of the input image, Represents an adaptive histogram equalization operation.

6. The real-time early warning method for tunnel face collapse based on machine vision technology according to claim 4 is characterized in that: According to the changes in the output volume per unit time of the rock ballast pieces and the morphological characteristics of the rock ballast pieces in different unit times, a preset model is used to perform a real-time early warning of tunnel face collapse, including: Calculating the roughness index of the rock ballast according to the morphological characteristics of the rock ballast piece; Obtain the total thrust and cutterhead torque during full-face rock tunnel boring machine operation; The total thrust, the cutter head torque, the roughness index of the rock ballast and the output volume per unit time are input into the preset model to obtain the tunnel face collapse result output by the preset model; the preset model is a SVM model; A real-time early warning of tunnel face collapse is performed according to the tunnel face collapse result.

7. The real-time early warning method for tunnel face collapse based on machine vision technology according to claim 6 is characterized in that: Calculating the roughness index of the rock ballast according to the morphological characteristics of the rock ballast piece includes: The roughness index of rock ballast is calculated using the following formula: in, is the roughness index of rock ballast; is the area residual screening rate of particle size; It is the total area of ​​rock debris with minor axis larger than a certain particle size; is the total area of ​​each group of rock ballast.

8. A real-time early warning system for tunnel face collapse based on machine vision technology, characterized in that: include: A processor and a memory for storing a computer program that can be run on the processor; wherein, when the processor is used to run the computer program, the steps of the method according to any one of claims 1 to 7 are executed.

9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images