Roof fall prediction and early warning method integrating video monitoring, numerical simulation and deep learning

Abstract

The invention discloses a roof caving prediction and early warning method integrating video monitoring, numerical simulation and deep learning. The method comprises the following steps: obtaining monitoring video data of the top of a roadway; recognizing the displacement condition of a top plate at the top of the roadway based on the monitoring video data to form displacement monitoring data; the method comprises the following steps: constructing a big data set of a relationship between physical and mechanical parameters of different rock masses and roof displacement on the basis of an FDEM numerical simulation method, carrying out dimension reduction on the physical parameters by adopting an improved 3d-dct method, and on the basis of dimension-reduced data, establishing a roof fall and wall caving disaster-causing process agent model based on a U-Net deep learning technology; and by taking displacement monitoring data as constraint conditions and the roof caving disaster-causing process agent model as a solver, dynamically solving optimal parameters of the roof caving disaster-causing process agent model by adopting a genetic algorithm, and predicting stress and damage states of the roadway roof. According to the method, prediction and early warning results of roof caving and wall caving are disclosed from multiple aspects of data and mechanisms, and important data support is provided for prevention and control of disasters.

Description

Technical Field

[0001] The present invention relates to the field of mine roof monitoring and early warning, and in particular to a roof fall prediction and early warning method integrating video monitoring, numerical simulation and deep learning. Background Art

[0002] Roof falls and spalling are the accidents that cause the most casualties in underground mines. As more mines enter deep mining in the future, under the conditions of high ground stress and surrounding rock fragmentation, the probability of roof falls and spalling will be higher and the harm will be greater, which seriously threatens the safe and efficient mining of deep mineral resources. The prediction and early warning technology of mine roof falls and spalling disasters induced by mining is the basis of ground pressure control and disaster prevention and control. Real-time monitoring of the mechanical response of the tunnel surrounding rock is an important means to achieve the prediction and early warning of roof falls and spalling. Among them, multi-point displacement meters, distributed optical fibers and other technologies are currently commonly used means of roof monitoring, but there is a problem of low monitoring density. The application of three-dimensional laser scanning technology solves the above problems, but there are problems of high monitoring cost and poor timeliness.

[0003] With the development of computer vision technology, digital image correlation technology has also been applied in deformation monitoring. Monitoring equipment has a high monitoring range and timeliness, as well as low cost. It is a perfect solution to the above problems and can provide important data support for the prediction and early warning of roof falls. However, it is difficult to characterize the catastrophic mechanism of roof falls by simply using displacement monitoring data, and it is impossible to achieve the accurate prediction and early warning of roof falls in terms of "time, space, and strength". As an important means to reveal the catastrophic mechanism, numerical simulation technology has problems such as poor timeliness and strong uncertainty, which restricts the application of this method in the dynamic prediction and early warning of roof falls. The rapid development of deep learning technology in recent years has provided a new idea for solving the timeliness uncertainty problem of numerical simulation. To this end, the present invention aims to develop a high-timeliness and high-precision prediction and early warning method for the "time, space, and strength" catastrophic characteristics of roof falls by integrating the advantages of video monitoring, numerical simulation, and deep learning, so as to provide theoretical and technical support for the safe production of mines. Summary of the invention

[0004] In order to solve the above problems, the technical solution adopted by the present invention is: a roof fall prediction and early warning method integrating video monitoring, numerical simulation and deep learning, comprising the following steps:

[0005] Obtain surveillance video data from the top of the tunnel;

[0006] Based on the monitoring video data, the displacement of the roof at the top of the tunnel is identified to form displacement monitoring data;

[0007] Based on the FDEM numerical simulation method, a large data set of the relationship between different rock mass physical and mechanical parameters and roof displacement is constructed. The improved 3D-DCT method is used to reduce the dimension of the physical parameters. On the basis of the reduced dimension data, a proxy model of the roof fall and slab disaster process based on U-Net deep learning technology is established.

[0008] Taking the displacement monitoring data as the constraint condition and the proxy model of the roof fall and spalling disaster process as the solver, a genetic algorithm is used to dynamically solve the optimal parameters of the proxy model of the roof fall and spalling disaster process and predict the stress and damage state of the tunnel roof.

[0009] Further: the process of identifying the displacement of the roof at the top of the tunnel based on the monitoring video data is as follows:

[0010] S21: determining an area of ​​interest, and performing real-time displacement monitoring on the area of ​​interest;

[0011] S22: Select the clearest frame image from the video stream as the reference image, use the SIFT algorithm on the reference image to extract the key points and feature vectors of the reference image, and extract the key points and feature vectors of each image based on the key points and feature vectors of the reference image;

[0012] S23: using Euclidean distance to calculate the similarity between the key points extracted at each moment and the key points of the reference image, for each key point feature vector, using the nearest neighbor method to find the most similar feature point in the reference image, and according to the matched feature point pairs, calculating its deformation (dx, dy) in the pixel coordinate system, and converting it into the deformation (dx, dy, dz) in the world coordinate system through the intrinsic and extrinsic parameters of the camera;

[0013] S24: Combining the coordinate vectors (x, y, z) of each key point and the deformation increment vector (dx, dy, dz), the Kriging interpolation method is used to interpolate the deformation increments of each position in the region of interest.

[0014] S25: Using Euclidean distance as the basis for similarity feature point matching, calculate the deformation dx, dy, dz and total displacement of the key points transferred to the world coordinate system, and use the Kriging interpolation algorithm on this basis to restore the actual displacement in the ROI area.

[0015] Further: the large data set of the relationship between the physical and mechanical parameters of different rock masses and the displacement of the roof is constructed based on the FDEM numerical simulation method, and the dimensionality reduction of the physical parameters is performed using the improved three-dimensional discrete cosine transform method. On the basis of the dimensionality reduction data, a proxy model of the roof fall and slab disaster-causing process based on the U-Net deep learning technology is established, which includes the following steps:

[0016] S31: Obtaining point cloud data of the inner wall of the tunnel for preprocessing and constructing a geometric model of the tunnel;

[0017] Using meshing software, the tunnel geometry model is converted into a mesh model for hybrid finite element-discrete element numerical simulation;

[0018] The geometric model is converted into a block model, different values ​​are used to mark the rock mass area and non-rock mass area of ​​the roadway, and the centroid coordinates and attribute values ​​of the ore body model are exported as a TXT file;

[0019] S32: using the HB criterion to preliminarily estimate the physical and mechanical parameters of the rock mass, and obtaining the coordinates of the center point of the grid model to generate random physical and mechanical parameters with position attributes;

[0020] S33: combining the generated inhomogeneous physical and mechanical parameters, establishing a numerical simulation model, and performing numerical simulation under different mechanical parameter conditions, to construct a big data sample set of the relationship between different physical and mechanical parameters and roof displacement;

[0021] S34: extracting the coordinates, physical parameters and mechanical response data of each unit of the numerical simulation model, combining the extracted block model coordinates and attribute information, using the Kriging interpolation algorithm to convert the numerical simulation input and output results into a block model, and combining the coordinate information to convert it into a three-dimensional matrix; calculating the three-dimensional gradient amplitude of each attribute three-dimensional matrix, calculating the weight matrix of each attribute three-dimensional matrix, multiplying the three-dimensional matrix by the weight coefficient, enhancing the three-dimensional matrix data, setting the weight threshold and the three-dimensional sliding window, and when there is a situation in the three-dimensional sliding window where the enhanced data is greater than the threshold, using the 3D-DCT algorithm to compress the enhanced data, and when it is not satisfied, using the average value of the region to represent the result of this region, and obtaining a sparse matrix;

[0022] S35: Design a data structure that includes the coordinates of the upper left corner and the lower right corner of the sliding window, the compression type, and the eigenvalue type, calculate the sparse matrix, and convert the sparse matrix into the COO form;

[0023] S36: Using a column-first filling method, storing the COO form sparse matrix data into a two-dimensional matrix;

[0024] S37: Aiming at the task of dynamically solving the displacement field and characterizing the disaster-causing process of tunnel surrounding rock roof fall and slab spalling, a deeply improved U-Net network structure is adopted. The two-dimensional matrix of physical and mechanical parameters is used as input, and the two-dimensional matrix of damage field and deformation velocity field in the next N days is used as output to construct an agent model for characterizing the disaster-causing process of tunnel surrounding rock roof fall and slab spalling.

[0025] Further: The construction process of the improved U-Net network structure is as follows:

[0026] A parallel convolution module with SENet attention mechanism is embedded in the jump connection of U-Net network. Specifically, SENet attention mechanism introduces two operations, Squeeze and Excitation, so that the model can explicitly model the dependency between feature channels and learn the importance of each feature channel.

[0027] In the Squeeze operation, the feature map is compressed by global average pooling to obtain the global information of each feature channel;

[0028] Next, in the Excitation operation, two fully connected layers and a Sigmoid activation function are used to learn and generate a weight vector with the same number of feature channels. Each element in the vector corresponds to the weight of a feature channel.

[0029] Finally, the feature channel weights are recalibrated by multiplying this weight vector with the original feature map channel by channel;

[0030] The SENet attention mechanism is integrated into the dilated convolution branch to adjust the channel relationship of the large-scale global information feature map to obtain better global information.

[0031] The SENet attention mechanism is integrated into the common convolution branch to adjust the channel relationship of the small-scale local information feature map to obtain better local information;

[0032] After the global information and local information feature map channels are concatenated, the SENet attention mechanism is integrated to further adjust the channel relationship of the dual-scale information feature map;

[0033] The SENet attention mechanism is added to the three positions in the parallel convolution module to obtain the improved U-Net network.

[0034] Furthermore, the process of constructing a proxy model for characterizing the process of tunnel surrounding rock collapse and spalling disasters is as follows:

[0035] Based on the improved U-Net network, with sparse matrix as data, two-dimensional matrix of basic physical and mechanical parameters as input, and two-dimensional matrix of current displacement field as output, a proxy model for dynamic solution of displacement field is constructed. At the same time, based on the improved U-Net network, with two-dimensional matrix of physical and mechanical parameters as input, and two-dimensional matrix of damage field and deformation velocity field of the next N days as output, a proxy model for characterizing the disaster process caused by roof fall and spalling in tunnel surrounding rock is constructed.

[0036] Further: the displacement monitoring data is used as a constraint condition, the proxy model of the roof fall and spalling disaster process is used as a solver, and a genetic algorithm is used to dynamically solve the optimal parameters of the proxy model of the roof fall and spalling disaster process to predict the stress and damage state of the tunnel roof, including the following steps:

[0037] S41: Based on the mass center information of the block model and the physical and mechanical parameters estimated under the Hawke-Brown strength criterion, an adaptive compression method is used to convert the physical and mechanical parameters into a two-dimensional image of sparse matrix row features, column features and value features;

[0038] S42: using a tunnel surrounding rock roof fall and spalling disaster-causing process characterization agent model, dynamically solving the sparse matrix row feature image, column feature image, value feature image and feature vector length used to characterize the compressed numerical simulation results;

[0039] S43: using the feature vector length to constrain the feature image extraction range, using the reverse column priority principle, converting the feature vector into a sparse matrix in the form of COO, converting the sparse matrix into a sparse matrix in the form of a large matrix, combining the sparse matrix definition, converting it into a block model, and for different compression modes, respectively using the 3D-IDCT algorithm and the same value assignment method to assign the final result, and obtaining a three-dimensional mechanical response feature;

[0040] S44: Based on the coordinate information of the defined region of interest, combined with the reconstruction result, the numerical simulation result of the region of interest is calculated by interpolation, and further based on the numerical simulation result d pre The displacement monitoring result d driven by the monitoring video obtained in module 1 true As the basis for calculating the error function, the error is calculated by using;

[0041] S45: When the error is greater than the preset range, the genetic algorithm is used to optimize the mechanical parameter matrix, and step S42 is further executed to solve the error. When the error is less than the preset range or the number of iterations meets the requirement, S46 is performed;

[0042] S46: Taking the optimal two-dimensional matrix of mechanical parameters as input, based on the agent model for characterizing the disaster-causing process of tunnel surrounding rock roof collapse, solving the two-dimensional characteristic matrix of the damage field and deformation velocity field of the roof in the next 10 days, and reconstructing the three-dimensional mechanical response characteristics of the surrounding rock using the method of step S43;

[0043] S47: Based on the damage field data, a higher damage threshold is set, and through the graph theory shortest path method, a damage inverse weighted shortest and free surface penetration path search algorithm is established to complete the delineation of the collapse area, and the average damage value of the path is used to quantify the catastrophic probability of roof fall; on this basis, the deformation speed is regarded as the movement speed of the collapsed block, and the intensity index of the roof fall disaster is evaluated in combination with the volume and mass of the collapsed block; when the catastrophic probability exceeds 90%, it is considered to be a sign of roof fall, and it is used as the time prediction result of roof fall, and finally the "time-space-strength" characteristic prediction of roof fall disaster is made.

[0044] Further: the error calculation adopts the following formula:

[0045]

[0046] Where: d pre represents the numerical simulation results of the ROI area, d true Indicates the displacement monitoring result driven by the surveillance video.

[0047] A roof fall prediction and warning device integrating video monitoring, numerical simulation and deep learning, comprising:

[0048] Acquisition module: used to obtain surveillance video data at the top of the tunnel;

[0049] Identification module: used to identify the displacement of the roof at the top of the tunnel based on the monitoring video data to form displacement monitoring data;

[0050] Establishing module: It is used to construct a large data set of the relationship between different rock mass physical and mechanical parameters and roof displacement based on the FDEM numerical simulation method, and use the improved 3D-DCT method to reduce the dimension of physical parameters. On the basis of the reduced dimension data, an agent model of the roof fall and slab disaster-causing process based on U-Net deep learning technology is established;

[0051] Prediction module: It is used to use the displacement monitoring data as constraints, the proxy model of the roof fall and slab disaster process as a solver, and adopt genetic algorithms to dynamically solve the optimal parameters of the proxy model of the roof fall and slab disaster process to predict the stress and damage state of the tunnel roof.

[0052] A computer device comprises: a processor and a memory, wherein the memory stores a program module, and wherein the program module runs on the processor to implement any of the methods described above.

[0053] The present invention provides a method for predicting and warning of roof falls by integrating video monitoring, numerical simulation and deep learning, which has the following advantages:

[0054] By using machine vision methods, real-time monitoring of roof displacement under low-cost conditions is achieved, thus reducing monitoring costs. Furthermore, by integrating deep learning and numerical simulation technology, efficient agency of numerical simulation disaster mechanism models at a three-dimensional scale is achieved, thus solving the problem of poor timeliness of numerical simulation methods in disaster prediction and early warning. Finally, the two are further integrated to propose a practical monitoring data-driven intelligent optimization method for numerical simulation parameters, which solves the problem of strong uncertainty in information such as mechanical parameters of numerical simulation models in disaster prediction and early warning methods. Based on the fusion results of the two, a practical method for predicting and warning roof falls with “time-space-strength” characteristics is proposed, which can reveal the prediction and warning results of roof falls from multiple levels of data and mechanism, thus providing important data support for disaster prevention and control. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 is a flow chart of the method;

[0057] Figure 2 It is a schematic diagram of the ROI area;

[0058] Figure 3 It is the main view of the surpac ore body model;

[0059] Figure 4 These are the renderings of different filling solutions;

[0060] Figure 5 It is a data structure diagram;

[0061] Figure 6 is a sparse matrix attempt;

[0062] Figure 7 (a) is a schematic diagram of sparse matrix row features, (b) is a schematic diagram of sparse matrix column features, and (c) is a schematic diagram of sparse matrix value features;

[0063] Figure 8 It is a schematic diagram of the network structure of the agent model for characterizing the process of tunnel surrounding rock collapse and wall collapse disaster. DETAILED DESCRIPTION

[0064] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0065] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] Figure 1 is a flow chart of the method;

[0067] A roof fall prediction and early warning method integrating video monitoring, numerical simulation and deep learning comprises the following steps:

[0068] S1: Obtain monitoring video data at the top of the tunnel;

[0069] S2: Identify the displacement of the roof at the top of the tunnel based on the monitoring video data to form displacement monitoring data;

[0070] S3: Based on the FDEM numerical simulation method, a large data set of the relationship between different rock mass physical and mechanical parameters and roof displacement is constructed, and the improved 3D-DCT method is used to reduce the dimension of the physical parameters. On the basis of the reduced dimension data, a proxy model of the roof fall and slab disaster-causing process based on the U-Net deep learning technology is established;

[0071] S4: Taking the displacement monitoring data as the constraint condition and the proxy model of the roof fall and spalling disaster process as the solver, a genetic algorithm is used to dynamically solve the optimal parameters of the proxy model of the roof fall and spalling disaster process to predict the stress and damage state of the tunnel roof.

[0072] Steps S1 / S2 / S3 / S4 are executed sequentially;

[0073] The process of identifying the displacement of the roof at the top of the tunnel based on the monitoring video data is as follows:

[0074] S21: Install a camera on the tunnel roof to determine the area of ​​interest and perform real-time displacement monitoring on the area of ​​interest; Figure 2 It is a schematic diagram of the ROI area;

[0075] S22: Use RTSP (Real-Time Streaming Protocol) for real-time transmission of video streams to ensure low latency and high stability of video data. After the data transmission is stable, capture images multiple times and select the clearest frame as the benchmark. Apply the SIFT (Scale-Invariant Feature Transform) algorithm to the benchmark image. The SIFT algorithm is an algorithm used for image feature point extraction and description. It has scale invariance and rotation invariance, and extracts key points and their feature vectors. These key points will serve as the benchmark for subsequent deformation monitoring. Thereafter, extract a frame of image from the video stream every 5 minutes to ensure the timeliness and continuity of data acquisition. Apply the SIFT algorithm to each frame of the image to extract the key points and feature vectors of the image;

[0076] S23: using Euclidean distance to calculate the similarity between the key points extracted at each moment and the key points of the reference image, for each key point feature vector, using the nearest neighbor method to find the most similar feature point in the reference image, and according to the matched feature point pairs, calculating its deformation (dx, dy) in the pixel coordinate system, and converting it into the deformation (dx, dy, dz) in the world coordinate system through the intrinsic and extrinsic parameters of the camera;

[0077] S24: Combining the coordinate vectors (x, y, z) of each key point and the deformation increment vector (dx, dy, dz), the Kriging interpolation method is used to interpolate the deformation increments of each position in the region of interest.

[0078] S25: Using Euclidean distance as the basis for similarity feature point matching, calculate the deformation dx, dy, dz and total displacement of the key points transferred to the world coordinate system, and on this basis use the Kriging interpolation algorithm to interpolate the deformation increments of each position in the ROI area. The selection of the deformation function in the Kriging interpolation needs to be analyzed in combination with the actual results.

[0079] The Euclidean distance is used as the basis for similarity feature point matching. On this basis, the deformation dx, dy, dz and total displacement of the key points transferred to the world coordinate system are calculated, and the Kriging interpolation algorithm is used on this basis to restore the actual displacement in the ROI area.

[0080] Based on the FDEM numerical simulation method, a large data set of the relationship between different rock mass physical and mechanical parameters and roof displacement is constructed. The improved 3D-DCT method is used to reduce the dimension of the data. On this basis, a proxy model of the roof fall and slab disaster process based on U-Net deep learning technology is established. In this way, a fast solution model between the physical and mechanical parameters of different rock masses and roof displacement is constructed, which provides an important basis for the subsequent optimization estimation of mechanical parameters.

[0081] The method of constructing a large data set of the relationship between different rock mass physical and mechanical parameters and roof displacement based on the FDEM numerical simulation method, using an improved 3D-DCT (three-dimensional discrete cosine transform) method to reduce the dimension of the physical parameters, and establishing a proxy model of the roof fall and slab disaster-causing process based on the U-Net deep learning technology on the basis of the reduced dimension data includes the following steps:

[0082] S31: Use a high-precision 3D laser scanner to conduct a comprehensive scan of the tunnel and the monitoring tunnel to obtain point cloud data of the inner wall of the tunnel. The resolution and accuracy of the scanner should be able to capture subtle geometric features. Use point cloud processing software (such as CloudCompare) to denoise, filter and align the scanned data to build a refined tunnel geometry model. On this basis, first use meshing software such as Rhino to convert the geometric model into a mesh model suitable for FDEM numerical simulation. Furthermore, use Surpac software to convert the geometric model into a block model, use different values ​​to mark the rock mass area and non-rock mass area of ​​the tunnel, and further export the centroid coordinates and attribute values ​​of the ore body model as a TXT file, thereby creating conditions for exporting the numerical simulation results into a block model and constructing the data structure required for deep learning; Figure 3 It is the main view of the surpac ore body model;

[0083] S32: Integrate geological drilling, seismic exploration and laboratory uniaxial compression, shear test and other data to obtain the geological and physical and mechanical properties of the tunnel surrounding rock, and further use the HB criterion to preliminarily estimate the physical and mechanical parameters of the rock mass. On this basis, considering the error range of ±20%, combined with the grid center point coordinates of the above grid model, generate random physical and mechanical parameters with position attributes, thereby creating conditions for constructing a three-dimensional model of the heterogeneous tunnel.

[0084] S33: Based on the HFDEM solver developed by Professor Liu Hongyuan of the University of Tasmania, Australia, each grid is regarded as an independent material. Combined with the non-homogeneous physical and mechanical parameters generated in S32, a numerical simulation model is established, and numerical simulation under different mechanical parameter conditions is performed to construct a big data sample set of the relationship between different physical and mechanical parameters and top plate displacement.

[0085] S34: Extract the coordinates, physical parameters and mechanical response data of each unit of the numerical simulation model, combine the block model coordinates and attribute information extracted in the above step S31, use the Kriging interpolation algorithm to convert the numerical simulation input and output results into a block model, and further combine the coordinate information to convert it into a three-dimensional matrix. On this basis, calculate the three-dimensional gradient amplitude of each attribute three-dimensional matrix Formula (1) is used to calculate the weight matrix of each attribute three-dimensional matrix, and formula (2) is used to enhance the data, set the weight threshold and the three-dimensional sliding window, and when the enhanced data in the three-dimensional sliding window is greater than the threshold, the three-dimensional matrix data is enhanced, the weight threshold and the three-dimensional sliding window are set, and when the enhanced data in the three-dimensional sliding window is greater than the threshold, the 3D-DCT algorithm is used to compress the enhanced data. When it is not satisfied, the average value of the area is used to represent the result of this area, and a sparse matrix is ​​obtained;

[0086] On the one hand, this can enhance the contrast of the data and facilitate feature extraction, and on the other hand, it can provide a basis for selecting data compression schemes.

[0087]

[0088] d′=w×d (2)

[0089] Based on the above-mentioned enhanced data d′, the weight threshold w>1.5 is set, and a three-dimensional sliding window is introduced to partition the three-dimensional matrix, where the window size is set to 18*18*18. The specific window size can be adjusted according to actual needs. When w>1.5, it means that there is a strong variability or stress concentration state in the area, and it is necessary to save more information. For this reason, the 3D-DCT algorithm is used to compress the data. On the contrary, it means that the area has weak variability or weak stress concentration phenomenon, and the impact on disaster assessment is not significant. The average value is used to replace the attributes of the area.

[0090] It is worth noting that the compression rate of data is closely related to the distribution of data. For non-rock areas, there is no real attribute information. Therefore, it is necessary to fill it with appropriate values. After research and analysis of this method, it is determined that filling with the average value is the best choice. The effects of different filling schemes are as follows Figure 4 shown.

[0091] S35: A data structure is designed that combines the coordinates of the upper left corner and lower right corner of the sliding window, the compression type, and the eigenvalue type, such as Figure 5 As shown, a sparse matrix is ​​calculated, and the sparse matrix is ​​as follows Figure 6 As shown, the sparse matrix is ​​converted into COO (coordinate format); Figure 7 (a) is the sparse matrix row feature, (b) is the sparse matrix column feature, and (c) is the sparse matrix value feature;

[0092] S36: Use column-first filling to store the COO form sparse matrix data into a two-dimensional matrix; it is worth noting that the size of the two-dimensional matrix in this example is n^2, where the value of n is the result of the square root of the above vector and the result of taking the value upward. When the number is insufficient, it is filled with 0. Because the spatial variability of the results of different numerical simulations is different, this model also uses the length of the vector as a parameter as an important input for deep learning.

[0093] S37: Aiming at the task of dynamically solving the displacement field and characterizing the disaster-causing process of tunnel surrounding rock roof fall and slab spalling, a deeply improved U-Net network structure is adopted. The two-dimensional matrix of physical and mechanical parameters is used as input, and the two-dimensional matrix of damage field and deformation velocity field in the next N days is used as output to construct an agent model for characterizing the disaster-causing process of tunnel surrounding rock roof fall and slab spalling.

[0094] Furthermore, a deeply improved U-Net network structure was adopted, with a two-dimensional matrix of physical and mechanical parameters as input and a two-dimensional matrix of damage field and deformation velocity field in the next N days (considering the accuracy of the model, N<5 is recommended) as output. The process of constructing a proxy model for characterizing the process of tunnel surrounding rock collapse and spalling disasters is as follows:

[0095] The original U-Net network has shown strong performance in the field of image segmentation with its unique encoder-decoder architecture and jump connection mechanism. In the present invention, considering the heterogeneity of rock mechanical parameters, the nonlinearity of numerical simulation input and output is extremely high. It is difficult for the traditional U-Net algorithm to extract and characterize the key features in the numerical simulation results. For this reason, the present invention embeds a parallel convolution module that incorporates the SENet (Squeeze-and-Excitation Network) attention mechanism in the jump connection of U-Net. Specifically, SENet introduces the two operations of Squeeze and Excitation, so that the model can explicitly model the dependencies between feature channels and learn the importance of each feature channel. In the Squeeze operation, the model compresses the feature map through global average pooling to obtain the global information of each feature channel. Then, in the Excitation operation, the model uses two fully connected layers and a Sigmoid activation function to learn and generate a weight vector with the same number of feature channels, and each element in the vector corresponds to the weight of a feature channel. Finally, by multiplying this weight vector with the original feature map channel by channel, the recalibration of the feature channel weight is achieved. The SENet attention mechanism is integrated into the hole convolution branch to adjust the channel relationship of the large-scale global information feature map to obtain better global information; the SENet attention mechanism is integrated into the ordinary convolution branch to adjust the channel relationship of the small-scale local information feature map to obtain better local information; after the global information and local information feature map channels are spliced, the SENet attention mechanism is integrated to further adjust the channel relationship of the dual-scale information feature map. The SENet attention mechanism is added to three positions in the parallel convolution module. The model pays more attention to those features that have an important impact on the prediction results. This improvement not only enhances the model's ability to capture key features, but also improves the model's prediction accuracy and robustness.

[0096] Based on the improved U-Net network, the above sparse matrix is ​​used as data, the two-dimensional matrix of basic physical and mechanical parameters is used as input, and the two-dimensional matrix of the current displacement field is used as output to construct a displacement field dynamic solution agent model; at the same time, based on the improved U-Net network, the two-dimensional matrix of physical and mechanical parameters is used as input, and the two-dimensional matrix of the damage field and deformation velocity field in the next N days is used as output to construct a tunnel surrounding rock roof fall and spalling disaster process characterization agent model;

[0097] Figure 8 It is a schematic diagram of the network structure of the agent model for characterizing the process of tunnel surrounding rock collapse and caving disaster;

[0098] Further: the displacement monitoring data is used as a constraint condition, the proxy model of the roof fall and spalling disaster process is used as a solver, and a genetic algorithm is used to dynamically solve the optimal parameters of the proxy model of the roof fall and spalling disaster process to predict the stress and damage state of the tunnel roof, including the following steps:

[0099] S41: Based on the mass center information of the block model and the physical and mechanical parameters estimated under the HB criterion (Hawk-Brown strength criterion), an adaptive compression method is used to convert the physical and mechanical parameters into a two-dimensional image of sparse matrix row features, column features and value features;

[0100] S42: using a tunnel surrounding rock roof fall and spalling disaster-causing process characterization agent model, dynamically solving the sparse matrix row feature image, column feature image, value feature image and feature vector length used to characterize the compressed numerical simulation results;

[0101] S43: using the feature vector length to constrain the feature image extraction range, using the reverse column priority principle, converting the feature vector into a sparse matrix in the form of COO, converting the sparse matrix into a sparse matrix in the form of a large matrix, combining the sparse matrix definition, converting it into the form of a block model, and using the 3D-IDC T algorithm and the same value assignment method to assign the final result according to the compression method of different modes;

[0102] S44: Based on the coordinate information of the defined region of interest, combined with the reconstruction result, the numerical simulation result of the region of interest is calculated by interpolation, and further based on the numerical simulation result d pre The displacement monitoring result d driven by the monitoring video obtained in module 1 true As the basis for calculating the error function, the error is calculated by using;

[0103] S45: When the error is greater than the preset range, a genetic algorithm is used to optimize the mechanical parameter matrix, and step S42 is further executed to solve the error. When the error is less than the preset range or the number of iterations meets the requirements, S46 is performed; the error range and the number of iterations need to be set according to the actual situation. In this example, the error range is selected as 90% and the number of iterations is selected as 100,000.

[0104] S46: Taking the optimal two-dimensional matrix of mechanical parameters as input, based on the agent model for characterizing the process of tunnel surrounding rock roof fall and spalling disaster, solving the two-dimensional characteristic matrix of the damage field and deformation velocity field of the roof in the next N days, and reconstructing the three-dimensional mechanical response characteristics of the surrounding rock by using the method of step S43;

[0105] S47: Based on the damage field data, a higher damage threshold of 0.9 is set, and the shortest path search algorithm with inverse weighted damage and connected to the free surface is established through the graph theory shortest path method, so as to complete the delineation of the collapse area, and use the average damage value of the path to quantify the catastrophic probability of roof fall; on this basis, the deformation speed is regarded as the movement speed of the collapsed block, and the intensity index of the roof fall disaster is evaluated in combination with the volume and mass of the collapsed block; when the catastrophic probability exceeds 90%, it is considered to be a sign of roof fall, which is used as the time prediction result of roof fall, and finally the "time-space-strength" characteristic prediction of roof fall disaster is made.

[0106] Furthermore, the error calculation adopts the following formula:

[0107]

[0108] Where: d pre represents the numerical simulation results of the ROI area, d true Indicates the displacement monitoring result driven by the surveillance video.

[0109] A roof fall prediction and warning device integrating video monitoring, numerical simulation and deep learning, characterized in that it includes:

[0110] Acquisition module: used to obtain surveillance video data at the top of the tunnel;

[0111] Identification module: used to identify the displacement of the roof at the top of the tunnel based on the monitoring video data to form displacement monitoring data;

[0112] Establishing module: It is used to construct a large data set of the relationship between different rock mass physical and mechanical parameters and roof displacement based on the FDEM numerical simulation method, and use the improved 3D-DCT method to reduce the dimension of physical parameters. On the basis of the reduced dimension data, an agent model of the roof fall and slab disaster-causing process based on U-Net deep learning technology is established;

[0113] Prediction module: It is used to use the displacement monitoring data as constraints, the proxy model of the roof fall and slab disaster process as a solver, and adopt genetic algorithms to dynamically solve the optimal parameters of the proxy model of the roof fall and slab disaster process to predict the stress and damage state of the tunnel roof.

[0114] A computer device comprises: a processor and a memory, wherein the memory stores a program module, and the program module runs on the processor to implement any one of the methods described above.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A roof fall prediction and early warning method integrating video monitoring, numerical simulation and deep learning, characterized in that: The steps include: Obtain surveillance video data from the top of the tunnel; Based on the monitoring video data, the displacement of the roof at the top of the tunnel is identified to form displacement monitoring data; Based on the FDEM numerical simulation method, a large data set of the relationship between different rock mass physical and mechanical parameters and roof displacement is constructed. The improved 3D-DCT method is used to reduce the dimension of the physical parameters. On the basis of the reduced dimension data, a proxy model of the roof fall and slab disaster process based on U-Net deep learning technology is established. Taking the displacement monitoring data as the constraint condition and the proxy model of the roof fall and spalling disaster process as the solver, a genetic algorithm is used to dynamically solve the optimal parameters of the proxy model of the roof fall and spalling disaster process and predict the stress and damage state of the tunnel roof.

2. A roof fall prediction and early warning method integrating video monitoring, numerical simulation and deep learning according to claim 1, characterized in that: The process of identifying the displacement of the roof at the top of the tunnel based on the monitoring video data is as follows: S21: determining an area of ​​interest, and performing real-time displacement monitoring on the area of ​​interest; S22: Select the clearest frame image from the video stream as the reference image, use the SIFT algorithm on the reference image to extract the key points and feature vectors of the reference image, and extract the key points and feature vectors of each image based on the key points and feature vectors of the reference image; S23: using Euclidean distance to calculate the similarity between the key points extracted at each moment and the key points of the reference image, for each key point feature vector, using the nearest neighbor method to find the most similar feature point in the reference image, and according to the matched feature point pairs, calculating its deformation (dx, dy) in the pixel coordinate system, and converting it into the deformation (dx, dy, dz) in the world coordinate system through the intrinsic and extrinsic parameters of the camera; S24: Combining the coordinate vectors (x, y, z) of each key point and the deformation increment vector (dx, dy, dz), the Kriging interpolation method is used to interpolate the deformation increments of each position in the region of interest. S25: Using Euclidean distance as the basis for similarity feature point matching, calculate the deformation dx, dy, dz and total displacement of the key points transferred to the world coordinate system, and use the Kriging interpolation algorithm on this basis to restore the actual displacement in the ROI area.

3. The method for predicting and warning of roof fall integrating video monitoring, numerical simulation and deep learning according to claim 1 is characterized in that: The method of constructing a large data set of the relationship between the physical and mechanical parameters of different rock masses and the displacement of the roof based on the FDEM numerical simulation method, using an improved three-dimensional discrete cosine transform method to reduce the dimension of the physical parameters, and establishing a proxy model of the roof fall and slab disaster-causing process based on the U-Net deep learning technology on the basis of the reduced dimension data includes the following steps: S31: Obtaining point cloud data of the inner wall of the tunnel for preprocessing and constructing a geometric model of the tunnel; Using meshing software, the tunnel geometry model is converted into a mesh model for hybrid finite element-discrete element numerical simulation; The geometric model is converted into a block model, different values ​​are used to mark the rock mass area and non-rock mass area of ​​the roadway, and the centroid coordinates and attribute values ​​of the ore body model are exported as a TXT file; S32: using the HB criterion to preliminarily estimate the physical and mechanical parameters of the rock mass, and obtaining the coordinates of the center point of the grid model to generate random physical and mechanical parameters with position attributes; S33: combining the generated inhomogeneous physical and mechanical parameters, establishing a numerical simulation model, and performing numerical simulation under different mechanical parameter conditions, to construct a big data sample set of the relationship between different physical and mechanical parameters and roof displacement; S34: extracting the coordinates, physical parameters and mechanical response data of each unit of the numerical simulation model, combining the extracted block model coordinates and attribute information, using the Kriging interpolation algorithm to convert the numerical simulation input and output results into a block model, and combining the coordinate information to convert it into a three-dimensional matrix; calculating the three-dimensional gradient amplitude of each attribute three-dimensional matrix, calculating the weight matrix of each attribute three-dimensional matrix, multiplying the three-dimensional matrix by the weight coefficient, enhancing the three-dimensional matrix data, setting the weight threshold and the three-dimensional sliding window, and when there is a situation in the three-dimensional sliding window where the enhanced data is greater than the threshold, using the 3D-DCT algorithm to compress the enhanced data, and when it is not satisfied, using the average value of the region to represent the result of this region, and obtaining a sparse matrix; S35: Design a data structure that includes the coordinates of the upper left corner and the lower right corner of the sliding window, the compression type, and the eigenvalue type, calculate the sparse matrix, and convert the sparse matrix into the COO form; S36: Using a column-first filling method, storing the COO form sparse matrix data into a two-dimensional matrix; S37: Aiming at the task of dynamically solving the displacement field and characterizing the disaster-causing process of tunnel surrounding rock roof fall and slab spalling, a deeply improved U-Net network structure is adopted. The two-dimensional matrix of physical and mechanical parameters is used as input, and the two-dimensional matrix of damage field and deformation velocity field in the next N days is used as output to construct an agent model for characterizing the disaster-causing process of tunnel surrounding rock roof fall and slab spalling.

4. The method for predicting and warning of roof fall integrating video monitoring, numerical simulation and deep learning according to claim 1 is characterized in that: The construction process of the improved U-Net network structure is as follows: A parallel convolution module with SENet attention mechanism is embedded in the jump connection of U-Net network. Specifically, SENet attention mechanism introduces two operations, Squeeze and Excitation, so that the model can explicitly model the dependency between feature channels and learn the importance of each feature channel. In the Squeeze operation, the feature map is compressed by global average pooling to obtain the global information of each feature channel; Next, in the Excitation operation, two fully connected layers and a Sigmoid activation function are used to learn and generate a weight vector with the same number of feature channels. Each element in the vector corresponds to the weight of a feature channel. Finally, the feature channel weights are recalibrated by multiplying this weight vector with the original feature map channel by channel; The SENet attention mechanism is integrated into the dilated convolution branch to adjust the channel relationship of the large-scale global information feature map to obtain better global information. The SENet attention mechanism is integrated into the common convolution branch to adjust the channel relationship of the small-scale local information feature map to obtain better local information; After the global information and local information feature map channels are concatenated, the SENet attention mechanism is integrated to further adjust the channel relationship of the dual-scale information feature map; The SENet attention mechanism is added to the three positions in the parallel convolution module to obtain the improved U-Net network.

5. The method for predicting and warning of roof fall integrating video monitoring, numerical simulation and deep learning according to claim 1 is characterized in that: The process of constructing a proxy model for characterizing the process of tunnel surrounding rock collapse and spalling disasters is as follows: Based on the improved U-Net network, with sparse matrix as data, two-dimensional matrix of basic physical and mechanical parameters as input, and two-dimensional matrix of current displacement field as output, a proxy model for dynamic solution of displacement field is constructed. At the same time, based on the improved U-Net network, with two-dimensional matrix of physical and mechanical parameters as input, and two-dimensional matrix of damage field and deformation velocity field of the next N days as output, a proxy model for characterizing the disaster process caused by roof fall and spalling in tunnel surrounding rock is constructed.

6. The method for predicting and warning of roof fall integrating video monitoring, numerical simulation and deep learning according to claim 1 is characterized in that: The method uses displacement monitoring data as a constraint condition, uses a proxy model of the roof fall and spalling disaster process as a solver, and adopts a genetic algorithm to dynamically solve the optimal parameters of the proxy model of the roof fall and spalling disaster process, and predicts the stress and damage state of the tunnel roof, including the following steps: S41: Based on the mass center information of the block model and the physical and mechanical parameters estimated under the Hawke-Brown strength criterion, an adaptive compression method is used to convert the physical and mechanical parameters into a two-dimensional image of sparse matrix row features, column features and value features; S42: using a tunnel surrounding rock roof fall and spalling disaster-causing process characterization agent model, dynamically solving the sparse matrix row feature image, column feature image, value feature image and feature vector length used to characterize the compressed numerical simulation results; S43: using the feature vector length to constrain the feature image extraction range, using the reverse column priority principle, converting the feature vector into a sparse matrix in the form of COO, converting the sparse matrix into a sparse matrix in the form of a large matrix, combining the sparse matrix definition, converting it into a block model, and for different compression modes, respectively using the 3D-IDCT algorithm and the same value assignment method to assign the final result, and obtaining a three-dimensional mechanical response feature; S44: Based on the coordinate information of the defined region of interest, combined with the reconstruction result, the numerical simulation result of the region of interest is calculated by interpolation, and further based on the numerical simulation result d pre The displacement monitoring result d driven by the monitoring video obtained in module 1 true As the basis for calculating the error function, the error is calculated by using; S45: When the error is greater than the preset range, the genetic algorithm is used to optimize the mechanical parameter matrix, and step S42 is further executed to solve the error. When the error is less than the preset range or the number of iterations meets the requirement, S46 is performed; S46: Taking the optimal two-dimensional matrix of mechanical parameters as input, based on the agent model for characterizing the disaster-causing process of tunnel surrounding rock roof collapse, solving the two-dimensional characteristic matrix of the damage field and deformation velocity field of the roof in the next 10 days, and reconstructing the three-dimensional mechanical response characteristics of the surrounding rock using the method of step S43; S47: Based on the damage field data, a higher damage threshold is set, and through the graph theory shortest path method, a damage inverse weighted shortest and free surface penetration path search algorithm is established to complete the delineation of the collapse area, and the average damage value of the path is used to quantify the catastrophic probability of roof fall; on this basis, the deformation speed is regarded as the movement speed of the collapsed block, and the intensity index of the roof fall disaster is evaluated in combination with the volume and mass of the collapsed block; when the catastrophic probability exceeds 90%, it is considered to be a sign of roof fall, which is used as the time prediction result of roof fall, and finally the "time-space-strength" characteristic prediction of roof fall disaster is made.

7. The method for predicting and warning of roof falls integrating video monitoring, numerical simulation and deep learning according to claim 1 is characterized in that: The error calculation adopts the following formula: Where: d pre represents the numerical simulation results of the ROI area, d true Indicates the displacement monitoring result driven by the surveillance video.

8. A roof fall prediction and warning device integrating video monitoring, numerical simulation and deep learning, characterized in that: include: Acquisition module: used to obtain monitoring video data at the top of the tunnel; Identification module: used to identify the displacement of the roof at the top of the tunnel based on the monitoring video data to form displacement monitoring data; Establishing module: It is used to construct a large data set of the relationship between different rock mass physical and mechanical parameters and roof displacement based on the FDEM numerical simulation method, and use the improved 3D-DCT method to reduce the dimension of physical parameters. On the basis of the reduced dimension data, an agent model of the roof fall and slab disaster-causing process based on U-Net deep learning technology is established; Prediction module: It is used to use the displacement monitoring data as constraints, the proxy model of the roof fall and slab disaster process as a solver, and adopt genetic algorithms to dynamically solve the optimal parameters of the proxy model of the roof fall and slab disaster process to predict the stress and damage state of the tunnel roof.

9. A computer device comprising: A processor and a memory, wherein the memory stores a program module, wherein the program module runs on the processor to implement the method according to any one of claims 1 to 7.

Images