Dangerous rock ground disaster monitoring and early warning system and method based on multi-modal data analysis

Abstract

The invention relates to the technical field of dangerous rock and ground disaster monitoring and early warning, and particularly discloses a dangerous rock and ground disaster monitoring and early warning system and method based on multi-modal data analysis, and the system comprises a data obtaining module which obtains the multi-modal data of a monitoring area; the data fusion processing module is used for carrying out preprocessing, data alignment, abnormal value processing and standardized analysis on the acquired data; and the multi-modal data analysis module is used for carrying out dimension reduction processing on the data set by applying principal component analysis, then inputting the data set into an input gate of the LSTM, dynamically regulating and controlling the write-in proportion of new information to the cell state through a gating mechanism, and optimizing the capturing capability of the model on time sequence characteristics and the like. Classification of dangerous rock state change information can be obtained, the change process of dangerous rock disaster pregnancy can be evaluated, the missing report rate of dangerous rock ground disaster is reduced, and the early warning success rate is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of dangerous rock and geological disaster monitoring and early warning, and particularly relates to a dangerous rock and geological disaster monitoring and early warning system and method based on multi-modal data analysis. BACKGROUND

[0002] With the acceleration of urbanization, geological disasters such as landslides and rock avalanches occur frequently, posing a serious threat to people's life and property safety and infrastructure.

[0003] Traditional monitoring methods rely on a single data source and often cannot timely and accurately identify potential geological disaster risks, and face multiple problems:

[0004] Firstly, the limitations of single monitoring methods are obvious. Traditional methods such as GPS displacement monitoring and inclination sensors mainly rely on a single physical quantity such as displacement or deformation, which is difficult to fully reflect the internal structural damage and instability mechanism of dangerous rock mass;

[0005] Secondly, environmental noise (such as temperature changes and mechanical vibrations) can easily lead to misjudgment, and there is a lack of multi-parameter cross-validation mechanism, which leads to a high false alarm rate.

[0006] Thirdly, there is a high delay in data collection and processing, which cannot timely capture sudden crack propagation or rock mass loosening;

[0007] Finally, the application of shallow sensors makes it difficult to effectively detect hidden disaster-causing factors such as fracture development and groundwater infiltration in the rock mass, further increasing the difficulty of monitoring.

[0008] Therefore, it is necessary to integrate multi-modal data to break through the limitations of a single mode and improve the comprehensiveness and analysis efficiency of dangerous rock early warning monitoring information. SUMMARY

[0009] Therefore, the present application provides a dangerous rock and geological disaster monitoring and early warning system based on multi-modal data analysis to improve the comprehensiveness and analysis efficiency of dangerous rock early warning monitoring information.

[0010] To achieve the above purpose, the basic scheme of the present application provides a dangerous rock and geological disaster monitoring and early warning system based on multi-modal data analysis, comprising,

[0011] a data acquisition module for acquiring multi-modal data of a monitoring area;

[0012] a data fusion processing module for pre-processing, data alignment, outlier processing and standardization analysis of the acquired data;

[0013] The multi-modal data analysis module applies principal component analysis dimension reduction processing to the data set, and then inputs into the input gate of the LSTM, dynamically regulates the writing proportion of new information to the cell state through the gating mechanism, and optimizes the capture ability of the model to the time sequence characteristics.

[0014] The dangerous rock state change information grading module integrates historical monitoring data and newly built model data, and comprehensively determines the dangerous rock body state in combination with the attribute characteristics of the specific dangerous rock body and the geological environment information thereof.

[0015] In a possible design, the data acquisition module acquires multi-modal data of the monitoring area through sensors, which include but are not limited to GNSS, rain gauge, crack gauge and wide-area electromagnetic equipment.

[0016] In a possible design, the dangerous rock body state is determined according to the Technical Guide for Geological Disaster Monitoring and Early Warning (Trial Version), which includes red warning of alarm level, orange warning of alert level, yellow warning of warning level and blue warning of attention level.

[0017] The application further provides a dangerous rock and disaster monitoring and early warning method based on multi-modal data analysis, which is based on the dangerous rock and disaster monitoring and early warning system based on multi-modal data analysis as described above, and includes the following steps,

[0018] Step 1, data acquisition: acquiring multi-modal data of the monitoring area;

[0019] Step 2, data fusion: pre-processing, data alignment, outlier processing and standardization analysis are performed on the multi-modal data;

[0020] Step 3, data analysis modeling: principal component analysis dimension reduction is performed on the data set to extract key features, and then the dimension-reduced data is input into the input gate of the long short-term memory network, the writing proportion of new information to the cell state is dynamically regulated through the gating mechanism, and the capture ability of the model to the time sequence characteristics is optimized;

[0021] Step 4, dangerous rock state change information grading: integrating historical monitoring data and newly built model data, and comprehensively determining the dangerous rock body state in combination with the attribute characteristics of the specific dangerous rock body and the geological environment information thereof.

[0022] In a possible design, in step 3, the data after data fusion and principal component analysis dimension reduction is input into the input gate of the LSTM, and the data proportion of the training set, the test set and the validation set is adjusted for model training.

[0023] In a possible design, in step 3, principal component analysis includes standardizing the data to eliminate the influence of dimension; calculating the covariance matrix to reflect the correlation between variables; performing eigendecomposition on the matrix to obtain eigenvalues ​​and corresponding eigenvectors; sorting by eigenvalue size, selecting the first k principal components to form a new coordinate system, and projecting the original data into this space to achieve the purpose of explaining most of the data variation with fewer variables, achieving the purpose of de-redundancy, noise reduction and feature extraction. Specifically,

[0024] Step a: Standardization of raw data

[0025] For a P-dimensional random variable X=(X1,X2,...,X p ) T n samples x i =(x i1 ,x i2 ,...,x ip ) T Where i = 1, 2, ..., n; n > p, construct the sample matrix and perform the normalization transformation, where i = 1, 2, ..., n; j = 1, 2, ..., p;

[0026]

[0027] Z ij : The standard score of the i-th sample on the j-th feature,

[0028] x ij : The original value of the i-th sample at the j-th feature,

[0029] Step b, solve the correlation coefficient matrix

[0030] Since the data table has been standardized, the covariance matrix of the indicator data is exactly equal to the correlation coefficient matrix of the original variable indicators, where i = 1, 2, ..., n; j = 1, 2, ..., p;

[0031]

[0032] R: Correlation matrix, showing the correlation between different variables,

[0033] z ij : The standardized value of the i-th sample at the j-th feature,

[0034] n: total number of samples,

[0035] Z: The standardized data matrix, where each row is a sample and each column is a feature;

[0036] Step c, obtain the eigenvalues ​​of the correlation matrix

[0037] Compute the correlation coefficient matrix of the standardized data and solve for the eigenvalues, where i = 1, 2,..., n; j = 1, 2,..., p;

[0038] R - λI p = 0 (7)

[0039] R: correlation matrix, showing the correlation between different variables,

[0040] λ: eigenvalue, a scalar, representing the characteristic property of the matrix,

[0041] I p : identity matrix, dimension p, where p is the number of characteristics,

[0042] |·|: determinant of the matrix,

[0043] Step d, index variable conversion

[0044] Convert the standardized index variables to principal components, where i = 1, 2,..., n; j = 1, 2,..., p;

[0045]

[0046] U ij : represents the measurement or calculation result,

[0047] z i : is a vector representing the eigenvalue of the ith sample,

[0048] A vector or matrix representing a certain parameter or reference value, usually related to the jth feature or category,

[0049] represents the transpose of the vector z i , changing it from a column vector to a row vector.

[0050] In one possible design, step 3, the data analysis modeling process also involves the following operations,

[0051] i t = σ(W i [h t-1 ,x t ]+b i ) (5)

[0052] f t = σ(W f [h t-1 ,x t ]+b f ) (6)

[0053] gt = tanh(W g [h t-1 ,x t ]+b g ) (7)

[0054] o t = σ(W o [h t-1 ,x t ]+b o ) (8)

[0055] c t = f t ⊙c t-1 +i t ⊙g t (9)

[0056] h t = o t ⊙tanh(c t ) (10)

[0057]

[0058] where i t is the input gate, f t is the forget gate, o t is the output gate, W i W f W o are the weight matrices of the forget gate, input gate, and output gate, respectively; W g is used to calculate the updated weight matrix, b i , b f , b g , and b o are bias vectors; ⊙ is a kind of matrix operation, called Hadamard product, g t represents the candidate state at time t; x t represents the input at time t; h t-1 represents the hidden state output at the previous time; and c t-1 represents the memory information flow at the previous time.

[0059] In a possible design, in step 3, after data alignment, outlier processing, and normalization analysis, the data after fusion is combined into a data matrix, a correlation coefficient matrix is solved, eigenvalues and standard orthogonal eigenvectors are obtained, a comprehensive index value is obtained after rotating the component matrix, the data after principal component is output, and a reduced data matrix is obtained. Normalization processing is needed to accelerate the convergence of the LSTM model and improve the numerical stability. The normalized data is divided into a training set, a validation set, and a test set. The division ratio needs to be adjusted according to the data set and the accuracy of the model. The data matrix is input into the input gate of the LSTM as the value at the starting time. The value at the next time is obtained through formula summation operation. The current value and the value at the next time are operated, and the final value is obtained by repeatedly using the current value and the value at the next time. ι and the forgetting gate f t The core of the long short-term memory network lies in its cell state. This structure serves as the memory hub of the network and is responsible for storing and transmitting long-term dependence information of the sequence. The cell state is dynamically adjusted by the input gate i t and the forgetting gate f t to achieve precise control of memory: the input gate generates a weight between 0 and 1 using a sigmoid activation function, filters the information in the current input x g that needs to be retained for a long time, and controls the proportion of new information generated by the candidate memory g t-1 = tanh(W t [h g , x t-1 ]+b t ) written into the cell state; the forgetting gate determines the proportion of old information to be erased from the cell state h t-1 at the previous time step based on a similar mechanism. The calculation logic at each time step is based on three inputs: the current input x t-1 , the hidden state h t at the previous time step, and the cell state c t-1 at the previous time step. The state update is completed through weighted summation and gating mechanism coordination.

[0060] In a possible design, first, the forgetting gate output f t selectively retains historical memory, and then the input gate output i t injects new candidate memory, and finally the cell state is updated iteratively through c t = f t ⊙ c t-1 + i t ⊙ g t , and the output gate o t further implements h t = o t ⊙ tanh(c t) to generate the current hidden state, which is passed as output to the next time step, while the updated cell state c t is directly carried over to the next time step, forming a memory continuity across time steps, in which the weighted sum operation linearly fuses the current input with the history state (h t-1 ,c t-1 ) to generate the gating signal and the candidate memory, the synergy of the three enables LSTM to adaptively balance the maintenance of long-term memory and the capture of short-term features, while the output gate regulates the degree of information exposure of the cell state according to the task requirements.

[0061] In one possible design, LSTM uses the optimization algorithm Adam to prevent gradient explosion in the process of processing the descending gradient.

[0062] Compared with the prior art, the principles and effects of the present application are as follows:

[0063] The present application solves the limitations of traditional dangerous rock monitoring methods, such as insufficient coverage of single modal data, difficulty in comprehensively reflecting the true state of the dangerous rock mass, lack of effective data sharing and cooperation between monitoring systems, leading to serious information island problem, and weak generalization ability of the warning model based on limited data, making it difficult to adapt to changing geological environment.

[0064] The system of the present application is integrated with multi-modal monitoring as the core, which integrates various dangerous rock monitoring index factors, carries out systematic data analysis, comprehensively improves the integrity and relevance of the monitoring data, and further enhances the adaptability and accuracy of the warning model, realizes more scientific and reliable monitoring and warning of the stability of the dangerous rock mass. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0066] Figure 1 A multi-modal data analysis flowchart proposed by an embodiment of a dangerous rock and disaster monitoring and warning system based on multi-modal data analysis of the present application is shown;

[0067] Figure 2 A multi-modal data fusion processing flowchart proposed by an embodiment of a dangerous rock and disaster monitoring and warning system based on multi-modal data analysis of the present application is shown;

[0068] Figure 3A multi-modal data analysis and model flowchart proposed by an embodiment of the dangerous rock and geological disaster monitoring and early warning system based on multi-modal data analysis of the application is shown.

[0069] Figure 4 A whole framework diagram proposed by an embodiment of the dangerous rock and geological disaster monitoring and early warning system based on multi-modal data analysis of the application is shown.

[0070] Figure 5 The accuracy rate and loss change curve of model training and verification in an embodiment of the dangerous rock and geological disaster monitoring and early warning system based on multi-modal data analysis of the application is shown, wherein (a) is the loss change curve of model training and verification in an embodiment of the dangerous rock and geological disaster monitoring and early warning system based on multi-modal data analysis of the application, and (b) is the accuracy rate curve of model training and verification in an embodiment of the dangerous rock and geological disaster monitoring and early warning system based on multi-modal data analysis of the application. DETAILED DESCRIPTION

[0071] To further illustrate the embodiments, the present application provides drawings which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can explain the operating principle of the embodiments in conjunction with the related description of the specification. With reference to these contents, those skilled in the art should understand other possible implementations and advantages of the present application. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0072] The prior art, the Chinese invention patent with application number 202411432048.X provides a dangerous rock displacement data monitoring system based on Internet of Things. The system includes a plurality of terminal nodes, a signal receiving and sending module, a base station and a visual interface. Each terminal node includes a CC2530 module, a vibration sensor, a displacement sensor and a power supply module. The signal receiving and sending module includes a 4G communication module and a CC2530 module, which is responsible for receiving data transmitted by the terminal node and transmitting data to the base station. The visual interface can be a PC, an APP and a visual large screen. Although it realizes remote, non-contact and real-time monitoring and early warning of slope dangerous rock, which is better than the traditional slope dangerous rock monitoring method and has good application prospect, it only monitors displacement index data, the monitoring data source is single, and the identification accuracy of the dangerous rock body is low.

[0073] The prior art, the Chinese invention patent publication with application number 202510000619.0 provides a recognition evaluation method and system based on multi-modal data analysis. The application relates to the technical field of image data processing. Image data and sequence data are obtained through a data acquisition module. An attribute analysis module determines the existence of a target through image data and analyzes the target attributes in the image data and the sequence attributes in the sequence data, thereby converting the original multi-modal data into attributes that can be used for analysis, reducing the complexity of multi-modal data, and improving the accuracy of analysis and judgment. A fusion recognition module fuses the target attributes and sequence attributes to obtain target sequence attributes to identify the comprehensive state, which can more comprehensively reflect the comprehensive state. An identification evaluation module compares the identified comprehensive state with the actual comprehensive state to obtain a feedback signal, optimizes the identification of the comprehensive state and the judgment of the existence of the target according to the feedback signal, and can adapt to the dynamic changes of the comprehensive state. The identification evaluation method of the application fuses image data and sequence data to improve the accuracy and reliability of identification, but does not consider real-time data of identification monitoring and geological conditions of the identification area.

[0074] The prior art, the Chinese invention patent publication with application number 202410046420.7 provides a dangerous rock collapse evaluation method and system based on artificial intelligence. The application relates to the technical field of collapse evaluation. The target rock state information of the target to be processed is spliced with the main feature information of the target to be processed to comprehensively extract the potential feature information of the target to be processed and the main feature information of the target to be processed, so as to accurately determine the stability of the rock and improve the accuracy and reliability of rock collapse evaluation. In addition, the target to be processed is evaluated through the target rock state information of the target to be processed to obtain the dangerous rock collapse evaluation result. The artificial intelligence dangerous rock collapse evaluation method of the application can ensure the accuracy of dangerous rock disaster evaluation and accurately prevent in advance to reduce the loss caused by rock collapse, but does not consider multiple influencing factors and multi-modal data sources of dangerous rock collapse.

[0075] Based on the technical problems existing in the above prior art, in the embodiments of the present application:

[0076] Among them, the multi-modal data acquisition strategy includes that the data of the application is composed of database attribute data and sensor acquisition equipment. The database attribute data, i.e. multi-modal data, includes dangerous rock body attribute data, geological attribute data, and human activity data. The sensor acquisition equipment includes a rain gauge, a wide-area electric shock amplitude frequency monitoring device, a crack meter, and GNSS.

[0077] The multimodal data fusion processing process includes the following: First, the multimodal data of dangerous rock geological hazards requires preliminary data cleaning to remove noise and fill missing values. This process can eliminate accidental data errors. Secondly, the data is standardized to eliminate the influence of the data's dimension. Then, the data is aligned in time and space, which facilitates the identification of abnormal areas and abnormal numerical time points. Finally, the data is corrected and combined with existing geological survey reports to determine the high-risk areas for rock mass hazards in the region. After eliminating the influence of dimension in the data processing, the multidimensional feature fusion method of principal component analysis (PCA) is used to combine multiple single-dimensional data columns into a multidimensional data set.

[0078] The multimodal data analysis and model building process includes:

[0079] The data indicators that affect the damage of dangerous rock masses have multiple dimensions. First, they need to be reduced in dimension using principal component analysis. After dimensionality reduction, they are predicted using a deep learning time series model.

[0080] (1) Principal component analysis dimensionality reduction

[0081] Principal component analysis (PCA) is a dimensionality reduction technique that aims to simplify the data structure by projecting high-dimensional data onto low-dimensional orthogonal principal components through linear transformation, preserving the maximum variance. Its principles are as follows: Normalizing the data to eliminate dimensionality effects; calculating the covariance matrix to reflect the correlations between variables; performing eigendecomposition on the matrix to obtain eigenvalues ​​(reflecting component importance) and corresponding eigenvectors (directions of the principal components); sorting by eigenvalue, selecting the first k principal components to form a new coordinate system, and projecting the original data onto this new coordinate system. This allows fewer variables to explain most of the data variation, achieving the goals of de-redundancy, noise reduction, and feature extraction.

[0082] ① Standardization of raw data

[0083] For a P-dimensional random variable X=(X1,X2,...,X p ) T n samples x i =(x i1 ,x i2 ,...,x ip ) T Where i = 1, 2, ..., n; n > p, construct the sample matrix and perform normalization transformation. Where i = 1, 2, ..., n; j = 1, 2, ..., p;

[0084]

[0085] Z ij : The standard score of the i-th sample on the j-th feature,

[0086] x ij : the original value of the i-th sample in the j-th feature,

[0087] ②Solving the correlation coefficient matrix

[0088] Since the data table has been standardized, the covariance matrix of the index data is exactly equal to the correlation coefficient matrix of the original variable index. Where i = 1, 2,..., n; j = 1, 2,..., p;

[0089]

[0090] R: correlation matrix, showing the correlation between different variables,

[0091] z ij : the standardized value of the i-th sample in the j-th feature,

[0092] n: total number of samples,

[0093] Z: standardized data matrix, where each row is a sample and each column is a feature;

[0094] ③Obtaining the eigenvalues of the correlation matrix

[0095] Calculate the correlation coefficient matrix of the standardized data and solve the eigenvalues. Where i = 1, 2,..., n; j = 1, 2,..., p;

[0096] |R-λI p | = 0 (11)

[0097] R: correlation matrix, showing the correlation between different variables,

[0098] λ: eigenvalue, a scalar representing the characteristic properties of the matrix,

[0099] I p : identity matrix, dimension p, where p is the number of features,

[0100] |·| : represents the determinant of the matrix,

[0101] ④Index variable conversion

[0102] Convert the standardized index variables to principal components, where i = 1, 2,..., n; j = 1, 2,..., p;

[0103]

[0104] U ij : represents the measurement or calculation result,

[0105] z i: is a vector, representing the feature value of the i-th sample,

[0106] a vector or matrix, representing a certain parameter or benchmark value, usually related to the j-th feature or category,

[0107] denotes the transpose of the vector z i , which changes it from a column vector to a row vector

[0108] (2) Time series model prediction

[0109] After data fusion, the data after dimensionality reduction by principal component analysis is input into the input gate of LSTM, and the data ratio of training set, test set and validation set is adjusted for model training.

[0110] The time series model selects LSTM, which is a kind of deep neural network. Its core mechanism lies in its unique internal memory unit (i.e. cell), which realizes long-term dependence modeling of sequence data through dynamic adjustment of information flow. Its core architecture is controlled by three types of gate mechanisms: i t The input gate is responsible for screening the effective information of the current input and writing it into the cell, f t The forget gate quantifies the retention proportion of historical memory through the activation function, and o t The output gate generates the final output for external transmission according to the current state. In the operation of each time step, the model is based on the current input data, the output state of the previous time step and the cell memory, and through weighted summation and nonlinear transformation, the dynamic calculation of the gate parameters is completed. In this process, the input gate and the forget gate jointly control the update of the cell state-the former decides the fusion strength of new information, and the latter controls the decay degree of historical memory; the output gate maps the updated cell state to the prediction output at the current time through the activation function, and passes it to the subsequent time step. This selective memory mechanism based on gate makes LSTM able to effectively capture key features in long sequences while suppressing noise interference, and finally realize efficient modeling and stable learning of complex time series patterns. This process will apply sigmoid and

[0111] tanh activation functions to constantly adjust the model and feedback the model results.

[0112] i t = σ(W i [h t-1 , x t ] + b i ) (5)

[0113] f t = σ(W f [h t-1 , x t ] + bf ) (6)

[0114] g t = tanh(W g [h t-1 , x t ] + b g ) (7)

[0115] o t = σ(W o [h t-1 , x t ] + b o ) (8)

[0116] c t = f t ⊙c t-1 + i t ⊙g t (9)

[0117] h t = o t ⊙tanh(c t ) (10)

[0118]

[0119] In the formula: i t is the input gate, f t is the forget gate, o t is the output gate, W i W f W o are the weight matrices of the forget gate, the input gate and the output gate respectively; W g is used to calculate the updated weight matrix, b i , b f , b g , b o are bias vectors; ⊙ is a kind of matrix operation, called Hadamard product, g t represents the candidate state at time t; x t represents the input at time t; h t-1 represents the hidden state output at the previous time; c t-1 represents the memory information flow at the previous time.

[0120] The comprehensive state value of the dangerous rock warning can be determined by historical data and new model data, combined with the attribute information and geological information of the specific dangerous rock mass. According to the “Guidelines for Monitoring and Early Warning of Geological Hazards” (Trial Version), the state change information of the dangerous rock is divided into red warning (alarm level), orange warning (alert level), yellow warning (warning level) and blue warning (attention level).

[0121] The fourth aspect provides a dangerous rock and disaster monitoring and early warning system, comprising: after the lower computer collects data, it is uploaded to the cloud platform through MQTT and HTTP communication, the cloud platform is composed of front-end visualization and back-end logic processing, the front-end can realize the business function of monitoring, the back-end realizes the interaction with the front-end and the transmission communication of data, and stores data.

[0122] As shown in Figure 1 The dangerous rock and disaster monitoring and early warning system multi-modal data analysis process is shown, which includes steps 1-4, including multi-modal data acquisition, multi-modal data fusion processing, multi-modal data fusion processing, and dangerous rock state change information classification.

[0123] In step 1, multi-modal data acquisition is through sensor data collection, including GNSS, rain gauge, crack gauge, wide area electromagnetic equipment. During installation, attention should be paid to the power supply of the monitoring area, and the integrity and real-time of data communication should be considered during data uploading. In addition, in order to more comprehensively analyze the dangerous rock mass rupture condition, the database stores the dangerous rock mass attribute data, geological attribute data and human activity data.

[0124] In step 2, multi-modal data fusion processing, the data needs to be preprocessed, data alignment, outlier processing, and standardized analysis. In multi-modal data fusion processing, the original data needs to be preprocessed first to ensure the quality and consistency of the data. The preprocessing steps include denoising, missing value filling and format standardization, which aims to lay a solid foundation for subsequent processing. Then, data alignment operation is needed to solve the inconsistency of different modal data in time, space or structure, so as to realize the synchronous integration of multi-source information. On this basis, it is also necessary to identify and process the outliers in the data to prevent extreme data from interfering with the accuracy of the fusion result. Finally, through standardized analysis, each modal data is converted to a unified scale range to eliminate the influence of dimension difference on the fusion effect, so as to improve the reliability of data fusion and the generalization ability of the model.

[0125] The data processing logic is as follows Figure 2As shown, the multi-dimensional data set may appear empty mining and other phenomena due to the influence of factors such as the setting of acquisition instructions, equipment working time, surface environment of dangerous rock mass area, etc., so there will be null values in the database, so data preprocessing is needed. Because different types of sensors produce different data dimensions during the sensor collection process, in order to exclude the influence of dimension, the data will be standardized. Standardization is to transform the data into a distribution with a mean of 0 and a standard deviation of 1. Through standardization processing, all features are on the same scale, making the model more stable and efficient, especially suitable for normally (Gaussian) distributed data. The standardized data has the same scale, reduces the influence of inconsistent dimensions between features, and helps to improve the performance of the model.

[0126] In step 3, multi-modal data analysis and model, first use principal component analysis dimensionality reduction processing on the data set, and then input to the input gate of LSTM. First, principal component analysis (PCA) is used to reduce the dimension of the data set to extract key features, and then the reduced data is input into the input gate of the long short-term memory network (LSTM), which dynamically controls the proportion of new information written to the cell state through the gating mechanism, and optimizes the model's ability to capture time series features.

[0127] Principal component analysis (PCA) is a dimensionality reduction method based on linear transformation, whose core goal is to map high-dimensional data to low-dimensional space through orthogonal projection while maximizing the preservation of original variation information. The specific implementation consists of four steps: first, standardize the data to eliminate dimension differences; second, calculate the covariance matrix to quantify the correlation between variables; then, perform eigenvalue decomposition on the matrix to obtain eigenvalues and their corresponding eigenvectors, where eigenvalues represent the ability of each principal component to explain variance, and eigenvectors define the orthogonal direction of data projection; finally, sort the eigenvalues from large to small, select the first k principal components to construct a new feature space, and convert the original data to this space to achieve dimension compression. Through this process, PCA can effectively eliminate redundant features, reduce noise interference, and extract key dimensions with the most information in the data, thus providing simple and efficient data representation for subsequent analysis.

[0128] It can be understood that in the existing multi-dimensional data analysis process, the dimension caused by high-dimensional data will significantly reduce the accuracy of the analysis model. On the one hand, the exponential growth of redundant features and noise data will dilute the key information density, making it difficult for the model to capture the core law; on the other hand, the interference of non-important factors may cause multicollinearity problem, leading to the increase of model parameter estimation bias, and finally affecting the reliability of the prediction result. In view of this challenge, the dimension reduction technology maps the high-dimensional space to the low-dimensional manifold through mathematical transformation, realizes the focus on the core influencing factors, and uses principal component analysis (PCA) to extract the characteristic combination of the maximum variance direction through covariance matrix decomposition. Such method not only enhances the information representation strength of key factors, but also effectively alleviates the overfitting problem and improves the generalization ability of the model.

[0129] The long short-term memory network (LSTM) is an improved recurrent neural network (RNN) that solves the long-term dependence problem of traditional RNN by introducing a gating mechanism. Its core unit includes a forgetting gate (determining the proportion of discarded old memories), an input gate (filtering new information to join the memory), and an output gate (controlling the proportion of current memory output), which generates a 0-1 weight value through a sigmoid function to adjust the information flow. The key structure of cell state as the "main line of memory" uses selective updating and transmission through gating to make the gradient flow stable and avoid gradient vanishing or explosion. LSTM effectively captures the time sequence dependence in long sequences by dynamically retaining key historical information.

[0130] It can be understood that the long short-term memory network (Long Short-Term Memory, LSTM) is an important upgrade of deep neural network architecture, which effectively solves the gradient explosion problem faced by traditional recurrent neural network (RNN) in the gradient descent process through innovative control gate mechanism. Traditional RNN has difficulty in converging when processing long sequence data due to the continuous multiplication operation of time steps, which causes the gradient to exponentially increase or disappear in the backpropagation process. LSTM addresses this deficiency by designing a three-gate system consisting of a forgetting gate, an input gate, and an output gate within the neuron: the forgetting gate dynamically determines the retention proportion of historical information through a sigmoid function, the input gate regulates the fusion degree of current input, and the output gate generates the final memory result by integrating internal and external states. This gating structure not only realizes the fine regulation of data flow, but also restricts the gradient amplitude to a reasonable range through the product of gate function derivatives. The product term of error terms in the gradient propagation path of LSTM changes from exponential growth to linear superposition compared to traditional RNN, significantly improving the model's ability to model long-range dependencies.

[0131] The processing and judgment logic is as shown in Figure 3 , and specifically includes:

[0132] On the basis of the above, after data alignment, outlier processing, normalization analysis, data fusion is combined into a data matrix, the correlation coefficient matrix is solved, the eigenvalue and standard orthogonal characteristic vector are obtained, and the comprehensive index value can be obtained after rotating the component matrix. After the principal component data, the dimensionality reduced data matrix is output, which needs to be normalized to speed up the convergence of the LSTM model and improve the numerical stability. The normalized data is divided into training set, validation set and test set, and the division ratio needs to be adjusted according to the data set and the accuracy of the model. The data matrix is input into the input gate of LSTM as the value at the starting time, and the formula summation operation is used to obtain the value at the next time. The current value and the next time value are operated, and the current value and the next time value are used in a loop to obtain the final value. The core of long short-term memory network (LSTM) is its cell state (Cell State), which is the memory hub of the network and is responsible for storing and transmitting long-term dependent information of the sequence. The cell state is dynamically adjusted by the input gate i ι and the forgetting gate f t to accurately control memory: the input gate generates a weight between 0 and 1 with a sigmoid activation function, filters the information that needs to be long-term retained in the current input x t , and controls the proportion of new information (generated by candidate memory g t =tanh(W g [h t-1 ,x t ]+b g ) written into the cell state; the forgetting gate determines the proportion of old information erased from the cell state h t-1 at the previous time step. The calculation logic at each time step is based on three inputs: the current input x t , the hidden state h t-1 at the previous time step (i.e. the output) and the cell state c t-1 at the previous time step, which completes the state update through weighted summation and gating mechanism: first, the forgetting gate output f t ⊙c t-1 selectively retains historical memory, then the input gate output i t ⊙g t injects new candidate memory, and finally c t =f t ⊙c t-1 +i t ⊙g t implements iterative update of the cell state. The output gate o t further generates the current hidden state h t =o t ⊙tanh(c t ), which is transmitted to the next time step as output, and the updated cell state ct Then directly continue to the next moment, forming a memory continuity across time steps. In this process, the weighted sum operation linearly fuses the current input and the historical state (h t-1 ,c t-1 ) to generate the gating signal and the candidate memory, and the synergy of the three enables LSTM to adaptively balance the maintenance of long-term memory (filtering invalid information through the forget gate) and the capture of short-term features (writing key information through the input gate), while the output gate controls the degree of information exposure of the cell state according to the task requirements. LSTM uses the optimization algorithm Adam to prevent gradient explosion in the process of processing the descending gradient.

[0133] In step 4, the dangerous rock state change information is graded, and the dangerous rock early warning comprehensive state value is determined based on sufficient integration of historical monitoring data and newly built model data, combined with the attribute characteristics of the specific dangerous rock body and the geological environment information thereof. The warning level of the state value is divided according to the “Technical Guidelines for Geological Disaster Monitoring and Early Warning (Trial Version)”, and the state change of the dangerous rock body is divided into four levels, namely red warning (alarm level), orange warning (alert level), yellow warning (warning level) and blue warning (attention level), to reflect the disaster risk degree at different stages, and provide basis for dynamic monitoring and scientific early warning of geological disasters.

[0134] Red warning (alarm level): The possibility of geological disaster is very large, and various short-term precursor characteristics are obvious, and the probability of large-scale occurrence within a few hours or days is very large.

[0135] Orange warning (alert level): The possibility of geological disaster is large, and there are certain macro-precursor characteristics, and the probability of large-scale occurrence within a few days or weeks is large.

[0136] Yellow warning (warning level): The possibility of geological disaster is relatively large, and there are obvious deformation characteristics, and the probability of large-scale occurrence within a few weeks or months is relatively large.

[0137] Blue warning (attention level): The possibility of geological disaster is small, and there are certain deformation characteristics, and the possibility of geological disaster within a year is not large.

[0138] Figure 4 As shown in the overall framework diagram of the dangerous rock geological disaster monitoring and early warning system of the present application, it is a cloud platform relying on the multi-modal data analysis of the present application, and the system is introduced as follows:

[0139] The input of multi-modal data requires the communication protocol of the Internet of Things system, and the data acquisition stage requires issuing instructions on the page end for data acquisition. The cloud platform needs to consider storage capacity, communication rate, and data transmission integrity during long-term monitoring. The system uses a layered architecture to manage the entire data flow. The lower machine first completes the raw data acquisition, and then transmits the data to the cloud platform through the MQTT (lightweight Internet of Things protocol) and HTTP (general protocol) dual channels. The cloud platform is composed of a front-end visualization module and a back-end logic processing module: the front-end focuses on user interaction and business function implementation, providing real-time data monitoring, historical trend display, and abnormal alarm notification visualization services; the back-end undertakes the core responsibility of bottom layer data processing, including dynamic interaction with the front-end through API interfaces, managing Internet of Things device communication links, receiving and analyzing uploaded data streams, and persistently storing structured data to the database. The front-end and back-end are decoupled through standardized protocols, the front-end focuses on business layer interaction logic, and the back-end ensures the stability of data reception, storage, and transmission links, both of which ensure efficient processing of massive data while maintaining scalability and operational reliability.

[0140] The page display of the front-end is divided into App and Web. App is used for the early testing stage, which can be used for debugging and testing the data of installed devices and monitoring areas. Web is used for the actual monitoring stage of the device, which is used for long-term monitoring. The back-end logic processing interacts with the front-end, and the data is transmitted and stored. The transmission needs to ensure that the downlink and uplink data bandwidth is sufficient, and the storage method is stored in the database and the web page according to the amount of data.

[0141] In one embodiment, the model is tested to improve the stability and convergence speed of the training. After multiple rounds of experimental verification, the final training round is set to 60 rounds. To comprehensively evaluate the performance of the model, the experiment uses training and loss value changes, precision, and other indicators to ensure the scientificity and comprehensiveness of the evaluation.

[0142] Figure 5 (a), Figure 5 (b) shows the change trend of accuracy and loss function of the model during training. It can be observed that the training accuracy rises rapidly at the beginning and tends to saturate after about the 60th round, eventually exceeding 80%; at the same time, the loss value decreases steadily and approaches 0.4, indicating that the model has been well fitted on the training set. The validation accuracy shows an overall upward trend, although there are some fluctuations in some iteration stages, the fluctuation amplitude is small, and the overall trend is stable, eventually converging to 81.22%. This kind of slight jitter phenomenon is common in deep learning training, which may be caused by the imbalance of data distribution or the fluctuation in the model parameter fine-tuning process, but it does not affect the overall convergence effect. This trend indicates that the model does not have obvious overfitting during training, and has good fitting ability and stability.

[0143] Although the above-described methods are illustrated and described as a series of acts for simplicity, it is to be appreciated that the methods are not limited by the order of acts, as some acts can, in accordance with one or more embodiments, occur concurrently or in different orders than shown and described herein. In addition, not all illustrated acts can be required to implement the methods in accordance with one or more embodiments.

[0144] The above description is only preferred embodiments of the present application, not any form of limitation on the present application, although the present application has been disclosed as above with preferred embodiments, however, not to define the present application, any person skilled in the art, within the scope of the technical scheme of the present application, can make some changes or modifications as equivalent embodiments with the above disclosed technical content, but as long as it does not deviate from the technical scheme of the present application, any brief introduction, modification, equivalent change and modification of the above embodiments according to the technical essence of the present application, still belongs to the scope of the technical scheme of the present application.

Claims

1. A dangerous rock disaster monitoring and early warning system based on multimodal data analysis, characterized in that: include, Data acquisition module, which acquires multimodal data of the monitoring area; The data fusion processing module performs preprocessing, data alignment, outlier processing, and standardization analysis on the acquired data; The multimodal data analysis module applies principal component analysis to the dataset for dimensionality reduction, then feeds the data into the LSTM input gate. This gating mechanism dynamically regulates the rate at which new information is written into the cell state, optimizing the model's ability to capture temporal features. The dangerous rock status change information classification module integrates historical monitoring data and newly created model data, and comprehensively determines the dangerous rock status based on the attribute characteristics of the specific dangerous rock body and the geological environment information in which it is located.

2. The dangerous rock disaster monitoring and early warning system based on multimodal data analysis according to claim 1 is characterized in that: The data acquisition module acquires multimodal data of the monitoring area through sensors, including but not limited to GNSS, rain gauges, crack meters, and wide-area electromagnetic equipment.

3. The dangerous rock disaster monitoring and early warning system based on multimodal data analysis according to claim 1 or 2, characterized in that: The status of dangerous rock masses is defined in accordance with the "Technical Guidelines for Monitoring and Early Warning of Geological Hazards by Specialized Groups (Trial Version)", including: red warning at the alarm level, orange warning at the alert level, yellow warning at the caution level, and blue warning at the attention level.

4. A dangerous rock disaster monitoring and early warning method based on multimodal data analysis, characterized in that: Based on the dangerous rock disaster monitoring and early warning system based on multimodal data analysis as described in any one of claims 1 to 3, The following steps are included: Step 1, data acquisition: obtaining multimodal data of the monitoring area; Step 2, data fusion: preprocessing, data alignment, outlier processing, and standardization analysis of multimodal data; Step 3: Data analysis and modeling: Principal component analysis is performed on the dataset to reduce its dimensionality and extract key features. The reduced data is then fed into the input gate of the long short-term memory network. The gating mechanism dynamically regulates the rate at which new information is written into the cell state, optimizing the model's ability to capture temporal features. Step 4: Classification of dangerous rock status change information: Integrate historical monitoring data with newly created model data, and comprehensively determine the status of the dangerous rock mass based on the attribute characteristics of the specific dangerous rock mass and its geological environment information.

5. The dangerous rock disaster monitoring and early warning method based on multimodal data analysis according to claim 4 is characterized in that: In step 3, the data after data fusion and principal component analysis dimensionality reduction is input into the input gate of LSTM, and the data ratio of the training set, test set, and validation set is adjusted for model training.

6. The dangerous rock disaster monitoring and early warning method based on multimodal data analysis according to claim 4 or 5, characterized in that: In step 3, principal component analysis involves standardizing the data to eliminate dimensional effects; Calculate the covariance matrix to reflect the correlation between variables; perform eigendecomposition on the matrix to obtain eigenvalues ​​and corresponding eigenvectors; sort by eigenvalue size, select the first k principal components to form a new coordinate system, and project the original data into this space to achieve the purpose of explaining most of the data variation with fewer variables, achieving the purpose of de-redundancy, noise reduction and feature extraction. Specifically, Step a: Standardization of raw data For a P-dimensional random variable X=(X1,X2,...,X p ) T n samples x i =(x i1 ,x i2 ,...,x ip ) T Where i = 1, 2, ..., n; n > p, construct the sample matrix and perform the normalization transformation, where i=1,2,...,n; j=1,2,...,p; Z ij : The standard score of the i-th sample on the j-th feature, x ij : The original value of the i-th sample at the j-th feature, Step b, solve the correlation coefficient matrix Since the data table has been standardized, the covariance matrix of the indicator data is exactly equal to the correlation coefficient matrix of the original variable indicators, where i = 1, 2, ..., n; j = 1, 2, ..., p; R: Correlation matrix, showing the correlation between different variables, z ij : The standardized value of the i-th sample at the j-th feature, n: total number of samples, Z: The standardized data matrix, where each row is a sample and each column is a feature; Step c, obtain the eigenvalues ​​of the correlation matrix Calculate the correlation coefficient matrix of the standardized data and solve for the characteristic roots, where i = 1, 2, ..., n; j = 1, 2, ..., p; |R-λI p |=0 (3) R: Correlation matrix, showing the correlation between different variables, λ: Eigenvalue, a scalar that represents the characteristic properties of the matrix. I p : identity matrix, dimension p, where p is the number of features, |·|: represents the determinant of the matrix, Step d, indicator variable conversion The standardized indicator variables are converted into principal components, where i = 1, 2, ..., n; j = 1, 2, ..., p; U ij : Indicates measurement or calculation results, z i : is a vector representing the eigenvalue of the i-th sample, A vector or matrix representing some parameter or reference value, usually associated with the jth feature or category, Represents vector z i Transpose of , changing it from a column vector to a row vector.

7. The dangerous rock disaster monitoring and early warning method based on multimodal data analysis according to claim 6 is characterized in that: In step 3, the data analysis and modeling process also involves the following operations: i t =σ(W i [h t-1 ,x t ]+b i ) (5) f t =σ(W f [h t-1 ,x t ]+b f ) (6) g t =tanh(W g [h t-1 ,x t ]+b g ) (7) the t =σ(W o [h t-1 ,x t ]+b o ) (8) c t =f t ⊙c t-1 +i t ⊙g t (9) h t =o t ⊙tanh(c t ) (10) Where: i t is the input gate, f t It's the forget gate, t is the output gate, W i W f W o are the weight matrices of the forget gate, input gate, and output gate respectively; W g Used to calculate the updated weight matrix, b i 、b f 、b g 、b o Refers to the bias vector; ⊙ is a matrix operation called the Hadamard product, g t Represented as the candidate state at time t; x t Represented as the input at time t; h t-1 Represented as the hidden state output of the previous moment; c t-1 Represented as the memory information flow of the previous moment.

8. The dangerous rock disaster monitoring and early warning method based on multimodal data analysis according to claim 7 is characterized in that: Step 3 also includes, after data alignment, outlier processing, and normalization analysis, the data is fused and combined into a data matrix, the correlation coefficient matrix is ​​solved, the eigenvalues ​​and standard orthogonal eigenvectors are obtained, and the comprehensive index value can be obtained after rotating the component matrix. The data after the principal component is output as the data matrix after dimensionality reduction, which needs to be normalized to accelerate the convergence of the LSTM model and improve the stability of the values. The normalized data is divided into training set, validation set, and test set. The division ratio needs to be adjusted according to the accuracy of the data set and the model. The data matrix is ​​input to the input gate of the LSTM as the value at the starting moment, and the value at the next moment is obtained through the formula summation operation. The current value and the value at the next moment are calculated, and the current value and the value at the next moment are cyclically applied to obtain the final value. The core of the long short-term memory network lies in its cell state. This structure serves as the memory center of the network and is responsible for storing and transmitting long-term dependency information of the sequence. The cell state is transmitted through the input gate i ι and forget gate f t Dynamic adjustment realizes precise control of memory: the input gate generates weights between 0 and 1 with sigmoid activation function, and filters the current input x t The information that needs to be retained for a long time is controlled by the candidate memory g t =tanh(W g [h t-1 ,x t ]+b g ) is written into the cell state; the forget gate uses a similar mechanism to determine the proportion of new information generated by the cell state h from the previous moment. t-1 The ratio of old information erased in the calculation logic of each time step is based on three parts of input: the current input x t , the hidden state h at the previous moment t-1 and the cell state c at the previous moment t-1 , the state update is completed through the collaboration of weighted summation and gating mechanism.

9. The dangerous rock disaster monitoring and early warning method based on multimodal data analysis according to claim 7 or 8, characterized in that: First, the forget gate outputs f t ⊙c t-1 Selectively retain historical memory, then input gate output i t ⊙g t Inject new candidate memory and finally pass c t =f t ⊙c t-1 +i t ⊙g t Implement iterative update of cell state, output gate o t Further h t =o t ⊙tanh(c t ) generates the current hidden state, which is passed to the next time step as output, and the updated cell state c t It continues directly to the next moment, forming a memory continuity across time steps. In this process, the weighted sum operation combines the current input with the historical state (h t-1 ,c t-1 ) linear fusion to generate gating signals and candidate memories. The collaboration of the three enables LSTM to adaptively balance the maintenance of long-term memory and the capture of short-term features. At the same time, the output gate regulates the degree of information exposure of the cell state according to task requirements.

10. The dangerous rock disaster monitoring and early warning method based on multimodal data analysis according to claim 9 is characterized in that: LSTM uses the Adam optimization algorithm to prevent gradient explosion when processing the descending gradient.

Images