Tunnel equipment positioning method based on multi-modal data and deep learning

Abstract

The application relates to the technical field of tunnel construction positioning, and discloses a tunnel equipment positioning method based on multi-modal data and deep learning, which aims to solve the problems of insufficient positioning accuracy, easy error accumulation and poor environmental adaptability of the prior art under the dynamic disturbance environment of tunnel construction, and mainly comprises the following steps: real-time synchronous acquisition of multi-modal sensor data of tunnel equipment; through a prediction step of extended Kalman filtering, an a priori estimation value of the equipment state and a covariance thereof are obtained; through an extended Kalman filtering update link, a Kalman gain is calculated, the a priori estimation value is corrected by using an observation vector composed of observation variables, and a posteriori estimation value is obtained; an optimized positioning result of the posteriori estimation value is obtained by using a deep neural model; and based on Lie derivatives, observability analysis is carried out, and if the equipment is in an unobservable state, the covariance matrix of the extended Kalman filtering is adjusted. The application improves the positioning accuracy under a strong disturbance environment, and is particularly suitable for tunnel construction in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of tunnel construction positioning, and particularly relates to a tunnel equipment positioning method based on multi-modal data and deep learning. BACKGROUND

[0002] With the continuous expansion of tunnel and underground engineering construction scale, the requirements for construction precision, automation level and safety are increasingly improved. Accurate real-time positioning is the key foundation to realize intelligent construction and unmanned operation.

[0003] At present, the positioning technology in tunnel construction mainly faces the following challenges: first, the global navigation satellite system (GNSS) signal cannot be effectively covered in the limited space such as tunnel, underground, etc., resulting in the failure of positioning methods relying on external signals. Second, although the widely used inertial navigation system (INS) or inertial measurement unit (IMU) does not rely on external signals, its inherent sensor errors (such as zero offset, drift) will be continuously accumulated through the integration process, resulting in a significant decrease in positioning accuracy over time, which is difficult to meet the requirements of long-distance, long-time tunnel construction. In addition, the tunnel construction environment is complex, often accompanied by high ground stress rock burst, strong vibration generated by large mechanical equipment operation and other dynamic disturbances. These disturbances will further pollute the sensor data and exacerbate the errors of traditional positioning algorithms.

[0004] To overcome the error accumulation problem of inertial navigation system, the existing technology proposes a multi-sensor fusion scheme, such as combining IMU, visual sensor and laser radar. These methods improve the accuracy to a certain extent, but when facing the severe and unstructured dynamic disturbance in tunnel construction, such as sudden rock burst and strong impact vibration, the positioning accuracy will still decrease significantly. The fusion algorithm relies on fixed mathematical models and parameters, such as extended Kalman filter, which is difficult to adapt to the rapidly changing environment, resulting in insufficient system robustness. SUMMARY

[0005] The present application aims to solve the problems of insufficient positioning accuracy, easy error accumulation and poor environmental adaptability of the existing technology in the dynamic disturbance environment of tunnel construction, and proposes a tunnel equipment positioning method based on multi-modal data and deep learning.

[0006] The technical solution adopted by the present application to solve the above technical problems is:

[0007] The tunnel equipment positioning method based on multi-modal data and deep learning comprises:

[0008] Deploy a sensor array on the tunnel equipment, the sensor array comprising an inertial measurement unit, a vibration sensor, a force sensor and an ultra-wideband module, control each sensor to collect IMU data, vibration data, force data and UWB ranging data of the tunnel equipment in real time, and perform timestamp alignment to generate time-synchronized multi-modal sensor data;

[0009] Input the IMU data into the system state equation, and obtain the prior estimate value of the device state and its covariance through the prediction step of the extended Kalman filter, wherein the device state comprises the position, velocity and attitude parameters of the tunnel equipment;

[0010] The vibration data, force data and UWB ranging data are used as observation variables to construct an observation model containing dynamic disturbance variables, and the Kalman gain is calculated through the update link of the extended Kalman filter, and the observation vector composed of the observation variables is used to correct the prior estimate value to obtain the posterior estimate value of the device state and its updated covariance;

[0011] A deep learning model based on deep neural network and long short-term memory network is constructed, the posterior estimate value and the multi-modal sensor data corresponding to its time sequence are input into the deep neural network to obtain the first optimized positioning result of the posterior estimate value; the time sequence of the first optimized result is input into the long short-term memory network to obtain the second optimized positioning result;

[0012] Based on the Lie derivative, the observability of the current state is analyzed, and the rank of the observability matrix is calculated, if the rank of the observability matrix is less than the dimension of the system state, it is determined that the system is in an unobservable state, at this time, the process noise covariance matrix and the observation noise covariance matrix of the extended Kalman filter are adjusted.

[0013] Further, the real-time synchronous collection of each sensor is controlled, specifically including:

[0014] A unified clock source is configured for the inertial measurement unit, vibration sensor, force sensor and ultra-wideband module, based on the unified clock source, a unified timestamp is stamped on the data collected by each sensor, and an interpolation algorithm is used to align the sensor data of different sampling rates to the same time sequence.

[0015] Further, the observation model containing dynamic disturbance variables is constructed, specifically:

[0016] The time-domain amplitude feature and the frequency spectrum feature of the vibration data are extracted as the first observation variable, the gradient change quantity of the force data is calculated as the second observation variable, and the UWB ranging data is used as the third observation variable, and the first observation variable, the second observation variable and the third observation variable are used to form an observation vector.

[0017] Further, in the extended Kalman filter update link, the calculation formula of the Kalman gain is as follows:

[0018] ;

[0019] wherein, denotes the Kalman gain, denotes the covariance matrix of the prior estimate value, denotes the observation model matrix, denotes the observation noise covariance matrix, denotes the matrix transpose;

[0020] The calculation formula of the posterior estimate value is as follows:

[0021] ;

[0022] wherein, denotes the posterior estimate value, denotes the prior estimate value, denotes the observation vector, , denotes the first observation vector, denotes the second observation vector, denotes the third observation vector.

[0023] Further, the training method of the deep learning model comprises:

[0024] Collecting a training data set containing real position labels at a tunnel construction site, taking multi-modal sensor data and corresponding posterior estimate values as input features, and taking the deviation of the real position and the posterior estimate value as the training target, and using the back propagation algorithm to optimize the parameters of the deep learning model.

[0025] Further, based on the Lie derivative, the observability of the current state is analyzed, including:

[0026] Constructing the state equation and the observation equation of the system, calculating the Lie derivative of the state variable in the state equation with respect to the observation variable in the observation equation, constructing the observability matrix from the Lie derivative, and calculating the rank of the observability matrix, when the rank of the observability matrix is less than the system state dimension, it is determined that the system is unobservable, and the system state dimension is the number of state variables in the state equation.

[0027] Further, the process noise covariance matrix and the observation noise covariance matrix of the extended Kalman filter are adjusted, including:

[0028] When the system is determined to be in an unobservable state, the element value corresponding to the unobservable state in the process noise covariance matrix is increased to reduce the fusion weight corresponding to the inertial measurement unit, and the element value in the observation noise covariance matrix corresponding to the vibration sensor and the force sensor is reduced to increase the fusion weight corresponding to the vibration sensor and the force sensor.

[0029] Further, the method further comprises:

[0030] The disturbance event in the tunnel environment is identified and predicted, and when a preset disturbance event is identified, the process noise covariance matrix and the observation noise covariance matrix of the extended Kalman filter are adjusted according to the disturbance type and intensity.

[0031] Further, adjusting the process noise covariance matrix and the observation noise covariance matrix of the extended Kalman filter according to the disturbance type and intensity comprises:

[0032] When a high-frequency vibration disturbance or rock burst impact disturbance is identified, the element value corresponding to the acceleration and speed state in the process noise covariance matrix is increased, and the element value corresponding to the inertial measurement unit in the observation noise covariance matrix is increased, and the element value corresponding to the vibration sensor, the force sensor and the ultra-wideband module in the observation noise covariance matrix is reduced.

[0033] Further, the method further comprises:

[0034] According to the second optimized positioning result of the tunnel equipment and the relative position data between the tunnel equipment provided by the UWB ranging data, and based on the pre-constructed multi-agent reinforcement learning model, the collaborative path planning and dynamic obstacle avoidance between multiple tunnel equipment are performed, and the optimized collaborative motion trajectory and control instruction of each tunnel equipment are generated.

[0035] The beneficial effects of the present application are: the tunnel equipment positioning method based on multi-modal data and deep learning provided by the present application fuses multi-modal data such as inertial measurement units, vibration sensors, force sensors and ultra-wideband modules to construct a more comprehensive observation model, effectively suppresses the error drift of the inertial navigation system caused by integral accumulation, combines the extended Kalman filter with the deep learning model, further optimizes the state estimation, realizes high-precision positioning in a strong disturbance environment, and significantly improves the positioning accuracy and stability. For sudden rock burst, strong vibration and other unstructured disturbances in tunnel construction, a dynamic observation model is constructed by introducing vibration data and force data, and a deep learning model is used to adaptively learn disturbance features from data, which significantly improves the robustness of the system in extreme environments and enhances the adaptive ability to dynamic disturbance. The observability is analyzed by using Lie derivative, and it is dynamically judged whether the system is in an unobservable state, and the process noise and observation noise covariance matrix of the extended Kalman filter are adaptively adjusted to ensure the continuity of positioning when the signal is missing or the disturbance is severe. Attached Figure Description

[0036] Figure 1 A flowchart illustrating a tunnel equipment positioning method based on multimodal data and deep learning, provided as an example;

[0037] Figure 2 This is a flowchart illustrating another tunnel equipment positioning method based on multimodal data and deep learning, provided as an example. Detailed Implementation

[0038] To address the problem of low positioning accuracy in tunnel construction due to the inability of current positioning technologies to adapt to the severe, unstructured dynamic disturbances encountered during tunnel construction, this invention proposes a technical solution. This invention utilizes a three-layer architecture of multi-sensor fusion, deep learning, and observability analysis to form a closed-loop high-precision positioning system with self-sensing, self-calibration, and adaptive capabilities. Specifically, firstly, the initial position and attitude of the equipment are calculated using IMU data, and absolute position reference is provided using UWB ranging data. Simultaneously, vibration and force sensors detect environmental disturbances, and this information is used as correction parameters. An extended Kalman filter algorithm is then used to continuously correct the accumulated error of the inertial prediction using the observed values, resulting in a preliminary, improved positioning result. By introducing vibration and force data, the extended Kalman filter can detect and partially compensate for abnormal state changes caused by dynamic disturbances, improving the robustness of the basic fusion result. Then, a deep learning model is used to analyze the current sensor data and the preliminary positioning result from the extended Kalman filter, learning the nonlinear error patterns caused by complex environments, and performing the first positioning optimization. The system is optimized by analyzing long-term positioning trajectories and learning the patterns and trends of error accumulation over time. A second positioning optimization is then performed, outputting the final high-precision positioning result. The combination of these two optimizations achieves fine compensation for errors in both the spatiotemporal dimensions. Finally, Lie derivatives are used to perform observability analysis on the nonlinear system. By calculating the rank of the observability matrix, the system's current state is judged in real time to determine whether all states are observable. When the system is found to be in an unobservable state, it indicates insufficient observation information. At this time, the process noise covariance matrix and observation noise covariance matrix of the extended Kalman filter are adaptively adjusted. That is, when the signal is poor, the system enters a conservative mode to prevent the extended Kalman filter from becoming overconfident and causing the positioning result to collapse. This buys time for the device to regain a valid signal and enhances the robustness of the system.

[0039] The technical solutions in this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0040] Please see Figure 1 andFigure 2 The tunnel equipment positioning method based on multi-modal data and deep learning provided in the embodiment includes the following steps:

[0041] Step 1, deploying a sensor array on the tunnel equipment, the sensor array including an inertial measurement unit, a vibration sensor, a force sensor and an ultra-wideband module, controlling the sensors to synchronously collect IMU data, vibration data, force data and UWB ranging data of the tunnel equipment in real time, and performing timestamp alignment to generate time-synchronized multi-modal sensor data.

[0042] Specifically, by deploying a sensor array on each tunnel equipment, different types of data are collected in real time, wherein the tunnel equipment can be a tunnel boring machine, a transport vehicle, etc.

[0043] It can be understood that the inertial measurement unit measures the acceleration and angular velocity of the equipment through an accelerometer and a gyroscope, and calculates the attitude and displacement of the equipment therefrom. The IMU data provided by the inertial measurement unit can track the state of the equipment in real time, especially during tunnel construction, which can provide real-time position and direction estimation for the system. However, when relying solely on IMU data for positioning, it is easy to be affected by error accumulation during equipment movement, especially when running for a long time or in a strong environmental disturbance, this error accumulation will gradually become significant.

[0044] The vibration sensor is used to detect vibration data caused by rock burst, equipment operation and other dynamic disturbances. Through the vibration data detected by the vibration sensor, the system can realize real-time sensing of the dynamic disturbance of the equipment movement. When rock burst or strong vibration occurs, the feedback provided by the vibration sensor will help the positioning system to correct the IMU data in time, thereby avoiding further accumulation of errors.

[0045] The force sensor is mainly used to detect the force data of the tunnel equipment, and to evaluate the interaction between the equipment and the tunnel wall or the soil. It can provide data related to soil pressure and equipment load, helping to calculate the actual position of the equipment, especially in high ground stress environment, the load change of the equipment can effectively provide necessary correction information for the positioning system.

[0046] The ultra-wideband module is used to detect the precise relative position between each equipment, which is crucial for the cooperative work of multiple equipment, especially in the tunnel construction process where precise cooperation between equipment is required. The ultra-wideband module can provide stable and high-precision relative positioning information.

[0047] In the embodiment, the sensors are controlled to collect data in real time synchronously, specifically including: configuring a unified clock source for the inertial measurement unit, the vibration sensor, the force sensor and the ultra-wideband module, based on the unified clock source, stamping a unified time stamp on the data collected by each sensor, and using an interpolation algorithm to align the sensor data of different sampling rates to the same time sequence.

[0048] Through the above scheme, all sensor data is strictly aligned in the time dimension, avoiding fusion errors caused by different time synchronization, and providing high-quality, internally time-sequential input data for subsequent extended Kalman filtering and deep learning models.

[0049] Step 2, input the IMU data into the system state equation, and obtain the prior estimate value and covariance of the device state through the prediction step of the extended Kalman filter, the device state including the position, velocity and attitude parameters of the tunnel device.

[0050] The embodiment adopts an extended Kalman filter (EKF) algorithm for multi-sensor data fusion. The extended Kalman filter is a widely used filtering algorithm in nonlinear systems, and is particularly suitable for estimating system states and correcting them through uncertain observation data. However, in tunnel construction, multiple sensors are used and the data types are complex, and the quality of sensor data in different environments differs greatly. For example, the IMU is greatly affected by inertial error, the force sensor is strongly affected by external interference, and the UWB may fail when the signal is blocked. Based on this, the embodiment performs weighted fusion on different sensor data according to the current environmental noise, sensor reliability and collection accuracy. During system operation, the fusion weight is adjusted according to the external disturbance, and the sensor data with higher signal quality is preferentially used.

[0051] Specifically, when performing the prediction step of the extended Kalman filter, the acceleration and angular velocity data of the IMU data are substituted into the state equation based on Newtonian mechanics and attitude dynamics to predict the prior estimate value and covariance of the device state at the next time. The acceleration and angular velocity data provided by the inertial measurement unit can calculate the state of the device at each time (such as position, velocity, direction, etc.). EKF uses the state estimate at the previous time as the initial input, and combines the dynamic equation of the device to predict the state. At the same time, it can also perform prediction update of the IMU data. Since the velocity, acceleration and other data provided by the inertial measurement unit are calculated based on the inertial characteristics of the device, error accumulation may occur over time. At this time, EKF will update the prediction data of the inertial measurement unit and compare it with the measurement results of other sensors (such as the ultra-wideband module, the force sensor and the vibration sensor) to ensure the accuracy of the positioning system.

[0052] Step 3, the vibration data, stress data and UWB ranging data are taken as observation variables, an observation model containing dynamic disturbance variables is constructed, the Kalman gain is calculated through an extended Kalman filter update link, and the prior estimate value is corrected by using an observation vector composed of the observation variables to obtain the posterior estimate value of the device state and the updated covariance thereof.

[0053] In the embodiment, the time-domain amplitude feature and the frequency-domain spectrum feature of the vibration data are extracted as the first observation variable, the gradient change of the stress data is calculated as the second observation variable, and the UWB ranging data is taken as the third observation variable. The first observation variable, the second observation variable and the third observation variable are collectively taken as an observation vector, wherein the first observation variable and the second observation variable are disturbance variables.

[0054] Specifically, the vibration and stress data are taken as observation variables to form a more abundant observation vector together with the UWB ranging data, so as to realize the expansion of the observation vector. That is, the disturbance is no longer regarded as noise, but as an observable input affecting the system state. The observation model establishes the physical or statistical relationship between the device state and the external disturbance (vibration, stress).

[0055] In the embodiment, in the extended Kalman filter update link, the calculation formula of the Kalman gain is as follows:

[0056] ;

[0057] wherein, K represents the Kalman gain, P represents the covariance matrix of the prior estimate value, H represents the observation model matrix, R represents the observation noise covariance matrix, T represents the matrix transpose;

[0058] The calculation formula of the posterior estimate value is as follows:

[0059] ;

[0060] wherein, X represents the posterior estimate value, X represents the prior estimate value, Z represents the observation vector, , Z1 represents the first observation vector, Z2 represents the second observation vector, Z3 represents the third observation vector.

[0061] In the extended Kalman filter, the observation model matrix is the first-order partial derivative matrix of the observation function at the current state, which describes how each observation variable changes with a small change in each state variable. The observation noise covariance matrix is a diagonal matrix or a nearly diagonal matrix, and the element values on the diagonal represent the uncertainty (noise size) of each sensor. Since the observation vector contains vibration and force information, the observation residual can sensitively capture the difference between the state prediction and the disturbed reality. The Kalman gain will feed back this difference to the state estimation with the optimal weight, thereby directly compensating for the impact of part of the disturbance, achieving optimal correction of inertial navigation prediction using multi-modal sensor data, and thereby significantly improving the robustness and accuracy of the positioning system in harsh environments such as strong vibration and rock burst.

[0062] Step 4, constructing a deep learning model based on a deep neural network and a long short-term memory network, inputting the posterior estimate value and the time sequence corresponding multi-modal sensor data into the deep neural network to obtain a first optimized positioning result of the posterior estimate value; inputting the time sequence of the first optimized positioning result into the long short-term memory network to obtain a second optimized positioning result.

[0063] It can be understood that although the EKF provides a good preliminary positioning result through multi-sensor fusion, it has inherent limitations. First, the model linearization error: EKF needs to linearize the approximation of the nonlinear system, which will introduce errors; second, the fixed noise assumption: EKF usually assumes that the process noise and observation noise are fixed and conform to the Gaussian distribution, but the actual disturbance in the tunnel (such as rock burst and vibration) is often sudden and non-Gaussian, and EKF is difficult to handle perfectly.

[0064] To solve the above problems, the posterior estimate value of the extended Kalman filter is regarded as a preliminary solution in the embodiment, and then the powerful capabilities of the deep neural network (DNN) and the long short-term memory network (LSTM) are used to learn and compensate for the complex and nonlinear residual errors that EKF cannot accurately describe. DNN and LSTM do not depend on specific physical models, but directly learn the complex mapping relationship between input (sensor data + EKF posterior estimate value) and output (true position) from data, capturing subtle patterns ignored or improperly handled by EKF.

[0065] In the embodiment, the training method of the deep learning model includes:

[0066] A training data set containing true position labels is collected at the tunnel construction site, and multi-modal sensor data and corresponding posterior estimate values are used as input features, and the deviation between the true position and the posterior estimate value is used as the training target. The back propagation algorithm is used to optimize the parameters of the deep learning model.

[0067] In the offline training phase, the DNN learns the complex nonlinear relationship between different sensor data and the final positioning error by analyzing a large amount of tunnel construction data (sensor data and corresponding EKF posteriori estimates) containing real labels. In essence, it is automatically determining a set of relative importance of each sensor data in fusion under various typical working conditions. When the model is deployed online, the DNN has the ability to dynamically adjust the fusion strategy according to the characteristics of real-time input data, that is, each layer of neurons in the DNN will perform feature extraction and transformation on the input data. When the input data (such as the noise characteristics of IMU data, the frequency spectrum characteristics of vibration signals) indicates that the current environment or sensor reliability has changed, the activation state of different neurons within the network will change accordingly, ultimately leading to the output result focusing on relying on those more reliable sensor information in the current mode, for example, if the quality of IMU data decreases (such as error accumulation during long-term operation), the DNN will reduce its weight and provide more positioning information through vibration sensors or force sensors. Through this adaptive adjustment mechanism, the DNN enhances the robustness of the system. The DNN can assign appropriate weights to the data of different sensors through the weights and features obtained through training, so that the error can be effectively corrected in the data fusion process, thereby predicting a more accurate error correction value in actual application according to the current sensor mode and EKF state.

[0068] LSTM is particularly suitable for processing time series data with long-term dependencies. In the tunnel construction process, the device state of the tunnel device changes over time, and these changes are often influenced by previous states. For example, error accumulation may occur during device operation, and the error of the inertial measurement unit will increase over time, which requires the system to "remember" the previous state and use this historical information to correct the current positioning error. LSTM handles this long-term dependency through its special gating mechanism, which helps the system eliminate problems caused by IMU error accumulation. In actual application, LSTM is input with the positioning result time series optimized by DNN, and its learning goal is to predict the residual error between the current positioning result and the true position. By analyzing the historical trajectory of the device, it learns the accumulation rule of IMU error. LSTM can also learn the relationship between the time series characteristics of vibration, force, and other sensor data and the change in positioning error. In the tunnel construction environment, disturbances (such as rock burst, device vibration, etc.) affect the positioning of the device, and LSTM can predict possible disturbance events in advance by learning the time series patterns of these disturbances, and make error corrections before these events occur, adjust the positioning strategy, and reduce the impact of disturbances on device positioning in advance.

[0069] In actual application, DNN and LSTM work together in the following ways:

[0070] Firstly, after the DNN input layer of each neuron receives a part of the input features after the posteriori estimation value and the corresponding multi-modal sensor data of the time sequence are input into the DNN, the data is linearly weighted and summed in each layer and a bias term is added, and then a nonlinear activation function is introduced to introduce a nonlinear transformation, so that the network can fit a complex mapping relationship, realize spatial feature fusion and instantaneous error correction. The DNN outputs an error correction value, and based on the error correction value, the posteriori estimation value is optimized for the first time to obtain the first optimized positioning result. Then the time sequence of the first optimized positioning result of the DNN is input into the LSTM, and the LSTM learns the long-term dependency relationship through the memory cell, the forgetting gate, the input gate and the output gate, outputs the second error correction value based on the time sequence optimization, and based on the second error correction value, the first optimized positioning result is optimized for the second time to obtain the second optimized positioning result. Through the coordination of the DNN for processing spatial complexity and the LSTM for processing time dependence, the system realizes high-precision and high-robustness estimation of the tunnel equipment position.

[0071] Step 5, performing observability analysis on the current state based on Lie derivative, calculating the rank of the observability matrix, and if the rank of the observability matrix is less than the system state dimension, determining that the system is in an unobservable state, at which time the process noise covariance matrix and the observation noise covariance matrix of the extended Kalman filter are adjusted.

[0072] Specifically, the purpose of the observability analysis is to evaluate whether the state of the equipment can be effectively observed in real time. If the state of the system at a certain moment is unobservable, the system cannot provide reliable positioning information. In a state estimation system, if some states of the system cannot be measured by the sensor, the system is unobservable. By analyzing the state equation and the sensor data of the system, it is determined whether the system is in an observable state, and if it is unobservable, the system cannot accurately estimate the position and speed of the equipment.

[0073] In this embodiment, the observability analysis on the current state based on Lie derivative includes:

[0074] The state equation and the observation equation of the system are constructed, the Lie derivative of the state variable in the state equation with respect to the observation variable in the observation equation is calculated, the observability matrix is constructed from the Lie derivative, and the rank of the observability matrix is calculated. When the rank of the observability matrix is less than the system state dimension, it is determined that the system is unobservable, and the system state dimension is the number of state variables in the state equation.

[0075] The embodiment utilizes Lie derivative for observability analysis. Lie derivative is a tool for analyzing dynamic changes of a system, and can be used to derive the relationship between the system state and the measurement. By analyzing the Lie derivative of the system, the system can determine whether the current state can be inferred from the existing observation data. It is derived from the state equation and the observation equation of the system, and the rate of change of the system state variable is calculated to evaluate whether the state has an impact on the observation value. If the value of the Lie derivative is zero, it means that the state is unobservable. At the same time, the corresponding observability matrix is calculated by the Lie derivative, which is used to describe the observability of all system state variables. The system determines whether the current state is observable according to the observability matrix. If the rank of the matrix is full rank, it means that the system state is observable. At each time, the system calculates the observability of the current device state. If it is found that some state variables are unobservable, the system will dynamically adjust the fusion weight of the multi-modal sensor data, that is, adjust the process noise covariance matrix and the observation noise covariance matrix of the extended Kalman filter.

[0076] In the embodiment, adjusting the process noise covariance matrix and the observation noise covariance matrix of the extended Kalman filter comprises:

[0077] When it is determined that the system is in an unobservable state, the element value corresponding to the unobservable state in the process noise covariance matrix is increased to reduce the fusion weight corresponding to the inertial measurement unit, and the element value in the observation noise covariance matrix corresponding to the vibration sensor and the force sensor is reduced to increase the fusion weight corresponding to the vibration sensor and the force sensor.

[0078] The process noise covariance matrix quantifies the uncertainty of the system model (i.e., the IMU's dynamic equations). A larger value indicates a less accurate model. When positions, attitudes, and other states become unobservable, significantly increasing the diagonal elements in the process noise covariance matrix corresponding to these states increases the state prediction covariance in subsequent calculations, thus reducing the EKF's fusion weight on the IMU's inertial extrapolation results. The observation noise covariance matrix quantifies the measurement noise of the observation sensors (vibration and force sensors). A smaller value indicates that the observation is considered more accurate and reliable. Reducing the diagonal elements in the observation noise covariance matrix corresponding to vibration and force sensor data is equivalent to improving the reliability rating of the vibration and force sensor data; the system will consider these still usable observations reliable. The final effect of these two adjustments is centrally reflected in the Kalman gain, a trade-off coefficient that determines the strength of the observation's correction of the state. After adjustment, the Kalman gain will significantly increase, and during state updates, the system will rely more heavily on the observation information provided by vibration and force sensors to correct the state estimate. By dynamically adjusting the strategy, that is, putting the system into a conservative mode when the signal is poor, the extended Kalman filter is prevented from becoming overconfident and causing the positioning results to collapse. This gives the device time to regain a valid signal and enhances the robustness of the system.

[0079] In this embodiment, the method further includes:

[0080] The system identifies and predicts disturbance events in the tunnel environment. Once a preset disturbance event is identified, the process noise covariance matrix and observation noise covariance matrix of the extended Kalman filter are adjusted according to the disturbance type and intensity.

[0081] Tunnel construction environments are typically affected by various factors such as rockbursts, equipment vibrations, and climate change, which can severely impact the positioning accuracy of equipment. To proactively address these disturbances, this embodiment utilizes a machine learning classifier to analyze and identify the sources of disturbances using real-time data from vibration and force sensors, and combines this with historical data to identify disturbance events. The machine learning classifier learns patterns of environmental change from historical data to predict potential future disturbance events (such as rockbursts). By pre-identifying specific disturbance events and proactively adjusting the trust strategy of the EKF filter based on their physical characteristics, the robustness of the system is improved before the disturbances truly affect positioning accuracy.

[0082] In this embodiment, adjusting the process noise covariance matrix and observation noise covariance matrix of the extended Kalman filter according to the type and intensity of the disturbance includes:

[0083] When a high-frequency vibration disturbance or rockburst impact disturbance is identified, the values of the elements in the process noise covariance matrix corresponding to the acceleration and velocity states are increased, and the values of the elements in the observation noise covariance matrix corresponding to the inertial measurement unit are increased, and the values of the elements in the observation noise covariance matrix corresponding to the vibration sensor, the force sensor and the ultra-wideband module are reduced.

[0084] By increasing the values of the elements on the diagonals of the process noise covariance matrix corresponding to the acceleration and velocity state variables, the prediction uncertainty of the model for these states is increased. By increasing the values of the elements in the observation noise covariance matrix corresponding to the inertial measurement unit, the trust of the EKF in the IMU observation values is reduced. By reducing the values of the elements in the observation noise covariance matrix corresponding to the vibration sensor, the force sensor and the ultra-wideband module, the fusion weights of the vibration sensor, the force sensor and the ultra-wideband module are increased, so that the system changes the filtering strategy in advance when the disturbance just starts to affect the sensor readings but has not caused catastrophic positioning errors, effectively avoiding the jumping and divergence of the positioning results; and can judge who's data is more reliable according to the current environment, and perform weighted fusion. For example, when a rockburst occurs, the system will automatically tend to trust the information provided by the UWB and vibration sensor, and temporarily ignore the contaminated IMU data.

[0085] In the embodiment, the method further comprises:

[0086] According to the second optimized positioning result of the tunnel device and the relative position data between the tunnel devices provided by the UWB ranging data, and based on the pre-constructed multi-agent reinforcement learning model, cooperative path planning and dynamic obstacle avoidance between multiple tunnel devices are performed to generate an optimized cooperative motion trajectory and control instruction for each tunnel device.

[0087] Reinforcement Learning (RL) is a machine learning method that learns the optimal policy through the interaction between the agent and the environment. The agent interacts with the environment, obtains state information (State), and takes action (Action), and obtains reward (Reward). The goal is to maximize the cumulative reward (or reward), usually referred to as the reward function or reward signal. In deep reinforcement learning, neural networks (usually deep neural networks) are used to approximate the optimal policy or value function. Through the Multi-Agent Reinforcement Learning (MARL) model, combined with the Ultra-Wideband module, the collaborative work between multiple tunnel devices can be realized. In the tunnel construction environment, multiple tunnel devices need to work accurately in coordination to complete complex construction tasks. The system uses the MARL model to achieve real-time collaboration and synchronization between devices through information sharing and mutual learning between devices. Specifically, each device can obtain real-time information from other devices when performing tasks, and adjust its motion trajectory according to the current environment and task requirements. Through this information sharing and learning mechanism, devices can dynamically adjust their paths to avoid collisions with other devices. Each device in the system can learn how to collaborate with other devices in a complex environment according to the feedback of the reinforcement learning model, thereby improving the efficiency and safety of the entire construction process. When device A and device B work in the same space, the system will automatically adjust their paths to avoid crossing and ensure that the construction work is not affected. This mechanism can significantly reduce conflicts between devices and unnecessary waiting time, thereby improving construction efficiency.

[0088] In practical applications, each tunnel device equipped with a high-precision positioning system is considered as an agent, and the multi-agent reinforcement learning model uses the final high-precision positioning results and UWB ranging data of each device to perform global optimal collaborative path planning and dynamic obstacle avoidance, and generates optimized collaborative motion trajectories and control instructions for each device, guiding the device group to safely and efficiently complete the construction task.

[0089] To sum up, the tunnel equipment positioning method based on multi-modal data and deep learning provided by the embodiment fuses multi-modal data such as inertial measurement units, vibration sensors, force sensors and ultra-wideband modules to construct a more comprehensive observation model, effectively suppresses the error drift of the inertial navigation system caused by integral accumulation, combines the extended Kalman filter and the deep learning model to further optimize the state estimation, realizes high-precision positioning in a strong disturbance environment, and significantly improves the positioning accuracy and stability. In view of the sudden rock burst, strong vibration and other unstructured disturbances in tunnel construction, a dynamic observation model is constructed by introducing vibration data and force data, and a deep learning model is used to adaptively learn the disturbance characteristics from the data, which significantly improves the robustness of the system in extreme environments and enhances the adaptive ability to dynamic disturbance. The observability is analyzed by using the Lie derivative, and it is dynamically judged whether the system is in an unobservable state, and the process noise and observation noise covariance matrix of the extended Kalman filter are adaptively adjusted to ensure the positioning continuity when the signal is missing or the disturbance is severe.

Claims

1. A method for tunnel equipment positioning based on multi-modal data and deep learning, characterized in that, The method includes: A sensor array is deployed on the tunnel equipment. The sensor array includes an inertial measurement unit, a vibration sensor, a force sensor, and an ultra-wideband module. Each sensor is controlled to synchronously collect IMU data, vibration data, force data, and UWB ranging data of the tunnel equipment in real time, and timestamp alignment is performed to generate time-synchronized multimodal sensor data. The vibration data is obtained by vibration sensors deployed on the tunnel equipment, and the force data is obtained by force sensors deployed on the tunnel equipment. The IMU data is input into the system state equation, and the prior estimate of the equipment state and its covariance are obtained through the prediction step of extended Kalman filtering. The equipment state includes the position, velocity and attitude parameters of the tunnel equipment. The vibration data, force data, and UWB ranging data are used as observation variables to construct an observation model that includes dynamic disturbance variables. The Kalman gain is calculated by extending the Kalman filter update loop, and the prior estimate is corrected by the observation vector composed of the observation variables to obtain the posterior estimate of the equipment state and its updated covariance. The construction of the observation model including dynamic disturbance variables specifically involves: The time-domain amplitude features and frequency-domain spectrum features of the vibration data are extracted and used as the first observation variable. The gradient change of the force data is calculated and used as the second observation variable. The UWB ranging data is used as the third observation variable. The first observation variable, the second observation variable and the third observation variable are combined to form an observation vector, where the first observation variable and the second observation variable are dynamic disturbance variables. A deep learning model based on a deep neural network and a long short-term memory network is constructed. The posterior estimate and the corresponding multimodal sensor data of its time series are input into the deep neural network to obtain the first optimized localization result of the posterior estimate. The time series of the first optimized localization result is input into the long short-term memory network to obtain the second optimized localization result. Based on the Lie derivative, an observability analysis is performed on the current state, and the rank of the observability matrix is ​​calculated. If the rank of the observability matrix is ​​less than the system state dimension, the system is determined to be in an unobservable state. At this time, the process noise covariance matrix and the observation noise covariance matrix of the extended Kalman filter are adjusted. 2.The multi-modal data and deep learning based tunnel equipment positioning method according to claim 1, characterized in that, Controlling the real-time synchronous acquisition of data from each sensor, specifically including: A unified clock source is configured for the inertial measurement unit, vibration sensor, force sensor and ultra-wideband module. Based on the unified clock source, a unified timestamp is applied to the data collected by each sensor, and an interpolation algorithm is used to align sensor data with different sampling rates to the same time series. 3.The multi-modal data and deep learning based tunnel equipment positioning method according to claim 1, wherein, In the extended Kalman filter update stage, the Kalman gain is calculated using the following formula: ； in, Indicates Kalman gain, This represents the covariance matrix of the prior estimates. Represents the observation model matrix, Represents the observation noise covariance matrix. Indicates matrix transpose; The formula for calculating the posterior estimate is as follows: ； in, Represents the posterior estimate. Represents the prior estimate. Represents the observation vector. , Denotes the first observation vector. Represents the second observation vector. This represents the third observation vector.

4. The tunnel equipment positioning method based on multimodal data and deep learning according to claim 1, characterized in that, The training methods for the deep learning model include: A training dataset containing real location labels was collected at the tunnel construction site. Multimodal sensor data and corresponding posterior estimates were used as input features, and the deviation between the real location and the posterior estimate was used as the training objective. The parameters of the deep learning model were optimized using the backpropagation algorithm.

5. The tunnel equipment positioning method based on multimodal data and deep learning according to claim 1, characterized in that, Observability analysis of the current state based on Lie derivatives includes: Construct the system's state equation and observation equation, calculate the Lie derivatives of the state variables in the state equation with respect to the observed variables in the observation equation, construct an observability matrix from the Lie derivatives, and calculate the rank of the observability matrix. When the rank of the observability matrix is ​​less than the system state dimension, the system is determined to be unobservable, where the system state dimension is the number of state variables in the state equation.

6. The tunnel equipment positioning method based on multimodal data and deep learning according to claim 1, characterized in that, Adjusting the process noise covariance matrix and observation noise covariance matrix of the extended Kalman filter includes: When the system is determined to be in an unobservable state, the value of the element corresponding to the unobservable state in the process noise covariance matrix is ​​increased to reduce the fusion weight corresponding to the inertial measurement unit. At the same time, the value of the element in the observation noise covariance matrix corresponding to the vibration sensor and the force sensor is decreased to increase the fusion weight corresponding to the vibration sensor and the force sensor.

7. The tunnel equipment positioning method based on multimodal data and deep learning according to claim 1, characterized in that, The method further includes: The system identifies and predicts disturbance events in the tunnel environment. Once a preset disturbance event is identified, the process noise covariance matrix and observation noise covariance matrix of the extended Kalman filter are adjusted according to the disturbance type and intensity.

8. The tunnel equipment positioning method based on multimodal data and deep learning according to claim 7, characterized in that, The process noise covariance matrix and observation noise covariance matrix of the extended Kalman filter are adjusted according to the type and intensity of the disturbance, including: When high-frequency vibration disturbances or rockburst impact disturbances are detected, the element values ​​corresponding to acceleration and velocity states in the process noise covariance matrix are increased, and the element values ​​corresponding to the inertial measurement unit in the observation noise covariance matrix are increased, while the element values ​​corresponding to the vibration sensor, force sensor, and ultra-wideband module in the observation noise covariance matrix are decreased.

9. The tunnel equipment positioning method based on multimodal data and deep learning according to claim 1, characterized in that, The method further includes: Based on the second optimized positioning results of the tunnel equipment and the relative position data between the tunnel equipment provided by UWB ranging data, and based on the pre-built multi-agent reinforcement learning model, the cooperative path planning and dynamic obstacle avoidance between the multiple tunnel equipment are carried out, and the optimized cooperative motion trajectory and control commands of each tunnel equipment are generated.

Images