Shield tunneling machine attitude deviation measurement method based on laser-vision deep learning heterogeneous fusion

Abstract

The invention discloses a shield tunneling machine attitude deviation measurement method based on laser-vision deep learning heterogeneous fusion, and the method comprises the steps: employing double sensors of a laser target and a vision camera, carrying out the deployment of the sensors, and collecting data, which comprises laser data and vision data; preprocessing the data, wherein the preprocessing comprises laser data preprocessing and visual data preprocessing; the laser data preprocessing comprises laser data coordinate conversion, laser target attitude deviation calculation and noise filtering; the visual data preprocessing comprises image preprocessing, mark point detection, camera calibration and world coordinate calculation, and visual measurement attitude deviation calculation; a deep learning heterogeneous fusion model is constructed, the deep learning heterogeneous fusion model comprises a laser absolute feature branch, a visual spatial-temporal feature branch, an attention fusion layer and an output layer, and the model is trained; and inputting the preprocessed real-time data into the trained deep learning heterogeneous fusion model, and carrying out dynamic calibration on an output result.

Description

Technical Field

[0001] This invention belongs to the field of tunnel boring machine (TBM) construction measurement technology, and particularly relates to a method for measuring TBM attitude deviation based on heterogeneous fusion of laser-visual deep learning. Background Technology

[0002] Existing shield tunneling machine attitude measurement technologies suffer from several core flaws, making it difficult to meet the high-precision and high-reliability requirements of complex construction environments. Mainstream laser-vision linear fusion schemes rely on linear algorithms such as fixed weights or extended Kalman filters, resulting in rigid fusion logic. This makes it impossible to adjust sensor contributions based on dynamic conditions such as changes in dust concentration and vibration intensity within the tunnel, leading to significant accuracy degradation when visual marker recognition is interfered with or when laser aiming is jittery. Furthermore, these schemes only perform shallow processing on single frames of data, failing to explore the temporal dependencies of visual data and the spatial correlation features of laser data. Consequently, their dynamic response speed and accuracy are ill-suited to scenarios involving sudden changes in shield tunneling machine attitude.

[0003] Single-sensor solutions and simple multi-sensor stacking also have limitations: gyroscope measurements are prone to drift due to time accumulation, requiring frequent calibration; total station laser guidance systems have stringent requirements for line-of-sight conditions, dust and water mist can interfere with laser beam transmission, and segment offset may cause changes in the station benchmark; multi-sensor combination solutions often fail to address the nonlinear coupling problem of heterogeneous data, and simple weighting can easily introduce errors when data conflicts occur. Furthermore, the additional deployment of equipment increases system complexity, cost, and maintenance difficulty, limiting their engineering practicality.

[0004] In summary, it can be seen that the existing solution cannot adjust the sensor contribution when the visual accuracy decreases due to increased dust concentration or when laser noise increases due to increased vibration, resulting in a decrease in fusion accuracy. Furthermore, traditional solutions only perform shallow fusion of single-frame laser and visual data, ignoring the temporal dependencies of visual data and the spatial correlation characteristics of laser data, thus failing to match the dynamic changes in the attitude of the tunnel boring machine. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method for measuring the attitude deviation of tunnel boring machines based on the heterogeneous fusion of laser-vision deep learning. This method fully extracts and utilizes the spatiotemporal information of the two types of data, thereby improving the dynamic measurement accuracy and real-time response speed.

[0006] The technical solution of this invention is implemented as follows: A method for measuring the attitude deviation of a tunnel boring machine based on laser-visual deep learning heterogeneous fusion includes the following steps: S1. A dual-sensor system, consisting of a laser target and a vision camera, is used to deploy the sensors and collect data, including laser data and vision data. S2. Preprocess the data, including laser data preprocessing and visual data preprocessing; laser data preprocessing includes laser data coordinate transformation, laser target attitude deviation calculation and noise filtering; visual data preprocessing includes image preprocessing, marker detection, camera calibration and world coordinate solution, and visual measurement attitude deviation calculation. S3. Construct a deep learning heterogeneous fusion model, which includes a laser absolute feature branch, a visual spatiotemporal feature branch, an attention fusion layer, and an output layer, and train the model. S4. Input the preprocessed real-time data into the trained deep learning heterogeneous fusion model and dynamically calibrate the output results.

[0007] Furthermore, in S1, the specific method for sensor deployment and data acquisition using a laser target in conjunction with an electronic total station and a visual camera dual sensor includes: S1.1 Install the laser target at the center axis of the front end of the shield of the tunnel boring machine to ensure stable incident of the laser beam from the electronic total station; install the vision camera in an area of ​​the shield with little dust and no obstruction, facing the photoelectric marker point preset on the tunnel wall, with the photoelectric marker point and the laser target in the same designed tunnel axis DTA coordinate system; S1.2 The electronic total station collects the absolute attitude data of the laser target in the station coordinate system at a preset cycle, including position and angle; the visual camera collects the images of the photoelectric markers at a preset frame rate, which is the visual perception data. S1.3. Using the electronic total station timestamp as a reference, the visual perception data is synchronized and aligned to ensure that the time synchronization error meets the measurement accuracy requirements.

[0008] Furthermore, in S1.3, time synchronization employs a linear interpolation algorithm or a timestamp alignment algorithm to ensure that the temporal correlation between laser data and visual data meets the input requirements of the deep learning heterogeneous fusion model, specifically including: Employing a timestamp alignment algorithm, both the electronic total station and the visual camera enable high-precision timestamp recording. Using the system time of the electronic total station as the reference clock, the visual camera timestamps undergo initial calibration via NTP protocol or hardware synchronization trigger. After data acquisition, the timestamp sequence of the visual perception data is traversed, and the visual image closest to each laser data timestamp is found through the nearest neighbor matching rule. If no direct match exists, the corresponding laser data or visual image is discarded to ensure the temporal correlation of the paired data. A linear interpolation algorithm is used to complete visual pose deviation data whose timestamps are not directly aligned when the laser data acquisition period does not match the visual camera frame rate. Let the timestamps of two adjacent valid visual data points be... , The corresponding attitude deviation is , The laser data timestamp is ,and The corresponding interpolation result is The same applies to the deviations on the Y-axis and Z-axis.

[0009] Furthermore, in step S2, the laser data preprocessing specifically includes: S2.11. Laser data coordinate transformation: Using at least three known control points, solve the 4×4 homogeneous transformation matrix using the least squares algorithm to transform the coordinates of the laser target from the station coordinate system to the DTA coordinate system. The formula is as follows: ; in, Let X be the X-axis coordinate of the laser target in the DTA coordinate system. Let Y be the Y-axis coordinate of the laser target in the DTA coordinate system. The Z-axis coordinate of the laser target in the DTA coordinate system It is the homogeneous transformation matrix from the station coordinate system to the DTA coordinate system. It is a 3×3 rotation matrix that describes the attitude rotation relationship of the station coordinate system relative to the DTA coordinate system, and satisfies orthogonality. , It is the identity matrix. It is a 3×1 translation vector, describing the spatial translation of the origin of the station coordinate system relative to the origin of the DTA coordinate system, including the offset distances in the X, Y, and Z directions. It is a 3-dimensional zero vector, a standard component of the homogeneous transformation matrix, used to fill the matrix dimension, has no actual physical meaning, and only ensures the standardization of matrix operations; It is the original measured coordinate of the laser target along the X-axis in the station coordinate system established by the electronic total station. It is the original measured Y-axis coordinate of the laser target in the station coordinate system; These are the original measured coordinates of the laser target along the Z-axis in the station coordinate system; S2.12 Calculation of laser target attitude deviation: The formula for calculating the deviation between the actual attitude of the laser target and the DTA-designed attitude is as follows: ; ; ; ; ; ; in, It is the attitude deviation of the laser target in the X-axis direction. It is the attitude deviation of the laser target in the Y-axis direction. It is the attitude deviation of the laser target in the Z-axis direction. Rolling attitude deviation of the laser target Pitch attitude deviation of the laser target Yaw attitude deviation of laser target , , , respectively t Design coordinates of DTA at any time , , for t The angle of DTA at any given moment; This is the measured roll angle of the laser target in the DTA coordinate system. It is the measured value of the elevation angle of the laser target in the DTA coordinate system. It is the measured value of the yaw angle of the laser target in the DTA coordinate system; S2.13 Noise filtering: The moving average filtering algorithm is used to suppress the aiming jitter of the electronic total station and obtain the filtered laser target attitude deviation data.

[0010] Furthermore, in step S2, the preprocessing of the visual data specifically includes: S2.21. Image preprocessing: The visual perception data is subjected to noise suppression and contrast enhancement processes sequentially. Noise suppression uses a Gaussian filtering algorithm, the formula of which is: ; in, The dimension parameter corresponding to the filter kernel size. These are the original image pixel values. These are the filtered pixel values. m For the filter kernel in x The length of half a side along the axial direction, n For the filter kernel in y The length of half a side along the axial direction, The coordinates of the target pixel to be filtered in the image are integers. x For pixel column coordinates, y These are the pixel row coordinates; S2.22, Marker point detection: A target detection algorithm is used to identify photoelectric marker points in the image and output pixel coordinates. ; S2.23. Camera calibration: Obtain camera intrinsic parameters using Zhang Zhengyou's calibration method. and extrinsic parameter matrix The expression for the intrinsic parameter matrix is: ; in, 、 For camera focal length, 、 Principal point coordinates; S2.24, World coordinate calculation: Based on the principle of perspective projection, the pixel coordinates are transformed to the DTA coordinate system. The formula is as follows: ; in, Let be the depth of the photoelectric marker point in the camera coordinate system. These are the pixel column coordinates of the photoelectric markers in the visual image. Let be the pixel row coordinates of the photoelectric marker in the visual image. It is the attitude rotation matrix of the camera coordinate system relative to the DTA coordinate system. It is the spatial translation vector of the camera coordinate system origin relative to the DTA coordinate system origin. 、 、 These are the world coordinates of the visual marker points in the DTA coordinate system; S2.25. Attitude Deviation Calculation: Based on the world coordinates of the visual markers and the DTA design coordinates, calculate the attitude deviation measured visually. .

[0011] Furthermore, in S2.11, the coordinates of the known control point in the DTA coordinate system and the coordinates of the station coordinate system are obtained through static precision measurement, using a homogeneous transformation matrix. The solution is achieved through least squares iteration, and the convergence condition of the iteration is that the coordinate transformation residuals satisfy the preset accuracy threshold. In S2.22, the target detection algorithm uses the YOLO series algorithm or an improved target detection network, and filters photoelectric markers by setting a pre-set confidence threshold and a non-maximum suppression algorithm to ensure detection accuracy. In S2.24, the depth of the visual marker point in the camera coordinate system It is obtained through binocular vision measurement, calculation of the spacing between preset marker points, or measurement assisted by a depth sensor.

[0012] Furthermore, S3 specifically includes: S3.1, Laser Absolute Feature Branch: Input the filtered 6-dimensional laser target attitude deviation data, perform feature encoding using multiple fully connected layers, and use ReLU as the activation function to output the laser feature vector. The formula is: ; in, This represents the total number of fully connected layers in the laser absolute feature branch. Here are the weight matrices for each layer. For each layer's bias vector, This is the filtered laser target attitude deviation feature vector; S3.2, Visual Spatiotemporal Feature Branch: The input consists of a temporal sequence of multiple consecutive frames of visual pose deviation data. First, a Convolutional Neural Network (CNN) extracts spatial features. Then, a Bidirectional Long Short-Term Memory (BiLSTM) network captures temporal dependencies. Finally, a fully connected layer outputs a visual feature vector. The formula is: ; ; ; in, It is a visual temporal feature sequence, composed of multiple consecutive frames of visual pose deviation data spliced ​​together in chronological order. It is the visual space feature vector extracted by the convolutional neural network (CNN). It is the output feature vector of the Bidirectional Long Short-Term Memory (BiLSTM) network. This is the weight matrix of the fully connected layer. This is the bias vector for the fully connected layer; S3.3 Attention Fusion Layer, which integrates laser feature vectors With visual feature vectors The weights are concatenated and fused adaptively through a fully connected layer and a sigmoid function. The formula is: ; Feature fusion is achieved based on fusion weights, and the formula is as follows: ; in, This is the attention layer weight matrix. This is the attention layer bias vector. This indicates that the feature vectors are concatenated horizontally. S3.4, Output Layer: Outputs the final 6-dimensional attitude deviation through a fully connected layer. The angular deviation is calculated by the geometric relationship between the positional deviation and the characteristic length of the shield body, using the following formula: ; ; ; in, Pre-set characteristic lengths for the tunnel boring machine; The X-axis position deviation in the final attitude deviation, The Y-axis position deviation in the final attitude deviation, The Z-axis position deviation in the final attitude deviation, The roll angle deviation is part of the final attitude deviation. The pitch angle deviation is part of the final attitude deviation. The yaw angle deviation in the final attitude deviation; S3.5 Model Training: Collect synchronous data from dual sensors of the tunnel boring machine under different working conditions to construct a dataset. Use the filtered laser target attitude deviation as a label, and train a deep learning heterogeneous fusion model using a gradient descent optimizer and a mean squared error loss function. The loss function formula is as follows: ; in, The number of training samples, To account for model prediction bias, For laser deviation labels.

[0013] Furthermore, in S3.2, the Convolutional Neural Network (CNN) adopts a structure of alternating convolutional and activation layers, and maintains the consistency of feature dimensions through zero padding or adaptive pooling; the input dimension of the Bidirectional Long Short-Term Memory Network (BiLSTM) matches the output feature dimension of the CNN, and captures temporal dependencies through forward and backward hidden state fusion. In S3.3, the fusion weights The value range is [0,1]. The contribution of absolute laser features and spatiotemporal visual features is dynamically adjusted through the attention mechanism to achieve adaptive fusion under different working conditions. In S3.4, the tunnel boring machine has a preset characteristic length. The distance from the center of the cutter head to the center of the laser target installation is obtained through actual precision measurement. In step S3.5, the dataset is divided into a training set, a validation set, and a test set according to a preset ratio. The training process uses an early stopping strategy or regularization method to avoid model overfitting and ensure the model's generalization ability.

[0014] Furthermore, S4 specifically includes: S4.1 Input the preprocessed real-time laser data and visual temporal data into the trained deep learning heterogeneous fusion model, and output the attitude deviation prediction results. S4.2. The model prediction results are dynamically calibrated using the latest filtered laser target attitude deviation at a preset cycle to obtain the final attitude deviation. The calibration formula is as follows: ; in, This is the final 6-dimensional attitude deviation vector output after dynamic calibration, containing 3 position deviations ( ) and 3 angular deviations ( ), These are the weighting coefficients. This represents the vector of attitude deviation predictions output by the deep learning heterogeneous fusion model. This is a vector of tag values ​​based on the filtered laser target attitude deviation.

[0015] Furthermore, in S4.2, the calibration cycle is dynamically adjusted according to the working conditions of the tunnel boring machine. When the vibration intensity or environmental interference exceeds a preset threshold, the calibration cycle is shortened; when the working conditions are stable, the calibration cycle is extended.

[0016] Compared with the prior art, the present invention achieves the following beneficial effects: This invention provides a method for measuring the attitude deviation of a tunnel boring machine (TBM) based on heterogeneous fusion of laser and vision deep learning. The accuracy of angle deviation measurement is significantly improved compared to existing single-sensor systems, meeting the accuracy requirements for autonomous steering control of the TBM. The method provided by this invention has strong anti-interference capabilities. Through image preprocessing, laser filtering, and attention weight adjustment, it effectively resists environmental interference such as tunnel dust, vibration, and darkness, solving the environmental adaptability defects of single vision systems. At the same time, it balances accuracy and real-time performance. Laser provides an absolute accuracy benchmark, vision provides a high-frequency dynamic response, and deep learning fusion achieves "1+1>2", satisfying both high accuracy requirements and ensuring real-time output. Attached Figure Description

[0017] Figure 1 This is a flowchart of a shield machine attitude deviation measurement method based on laser-visual deep learning heterogeneous fusion provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] Example like Figure 1 A method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning includes the following steps: S1. A dual-sensor system, consisting of a laser target and a vision camera, is used to deploy the sensors and collect data, including laser data and vision data. In step S1, the specific method for sensor deployment and data acquisition using a laser target in conjunction with an electronic total station and a dual-sensor visual camera includes: S1.1 Install the laser target at the center axis of the front end of the shield of the tunnel boring machine to ensure stable incident of the laser beam from the electronic total station; install the vision camera in an area of ​​the shield with little dust and no obstruction, facing the photoelectric marker point preset on the tunnel wall, with the photoelectric marker point and the laser target in the same designed tunnel axis DTA coordinate system; S1.2 The electronic total station collects the absolute attitude data of the laser target in the station coordinate system at a preset cycle, including position and angle (roll, pitch, yaw); the visual camera collects images of photoelectric markers at a preset frame rate, which are the visual perception data. S1.3. Using the electronic total station timestamp as a reference, the visual perception data is synchronized and aligned to ensure that the time synchronization error meets the measurement accuracy requirements.

[0020] In step S1.3, time synchronization employs a linear interpolation algorithm or a timestamp alignment algorithm to ensure that the temporal correlation between laser data and visual data meets the input requirements of the deep learning heterogeneous fusion model. Specifically, this includes: Employing a timestamp alignment algorithm, both the electronic total station and the visual camera enable high-precision timestamp recording. Using the system time of the electronic total station as the reference clock, the visual camera timestamps undergo initial calibration via NTP protocol or hardware synchronization trigger. After data acquisition, the timestamp sequence of the visual perception data is traversed, and the visual image closest to each laser data timestamp is found through the nearest neighbor matching rule. If no direct match exists, the corresponding laser data or visual image is discarded to ensure the temporal correlation of the paired data. Visual pose deviation data is target result data obtained from visual perception data through multi-step calculations. A linear interpolation algorithm is used to complete visual pose deviation data whose timestamps are not directly aligned when the laser data acquisition period does not match the visual camera frame rate. Let the timestamps of two adjacent valid visual data points be... , The corresponding attitude deviation is , The laser data timestamp is ,and The corresponding interpolation result is The same applies to the deviations on the Y-axis and Z-axis.

[0021] S2. Preprocess the data, including laser data preprocessing and visual data preprocessing; laser data preprocessing includes laser data coordinate transformation, laser target attitude deviation calculation and noise filtering; visual data preprocessing includes image preprocessing, marker detection, camera calibration and world coordinate solution, and visual measurement attitude deviation calculation. In step S2, the laser data preprocessing specifically includes: S2.11. Laser data coordinate transformation: Using at least three known control points, solve the 4×4 homogeneous transformation matrix using the least squares algorithm to transform the coordinates of the laser target from the station coordinate system to the DTA coordinate system. The formula is as follows: ; in, Let X be the X-axis coordinate of the laser target in the DTA coordinate system. Let Y be the Y-axis coordinate of the laser target in the DTA coordinate system. The Z-axis coordinate of the laser target in the DTA coordinate system It is the homogeneous transformation matrix from the station coordinate system to the DTA coordinate system. It is a 3×3 rotation matrix that describes the attitude rotation relationship of the station coordinate system relative to the DTA coordinate system (including roll, pitch, and yaw three-dimensional rotation information), and satisfies orthogonality. , It is the identity matrix. It is a 3×1 translation vector, describing the spatial translation of the origin of the station coordinate system relative to the origin of the DTA coordinate system, including the offset distance in the X, Y, and Z directions (unit: meters). It is a 3-dimensional zero vector, a standard component of the homogeneous transformation matrix, used to fill the matrix dimension, has no actual physical meaning, and only ensures the standardization of matrix operations; It is the original measured coordinate of the laser target along the X-axis in the station coordinate system established by the electronic total station. It is the original measured Y-axis coordinate of the laser target in the station coordinate system; These are the original measured coordinates of the laser target along the Z-axis in the station coordinate system; In S2.11, the coordinates of the known control point in the DTA coordinate system and the coordinates of the station coordinate system are obtained through static precision measurement, and the homogeneous transformation matrix is ​​used. The solution is achieved through least squares iteration, and the convergence condition of the iteration is that the coordinate transformation residuals satisfy the preset accuracy threshold. S2.12 Calculation of laser target attitude deviation: The formula for calculating the deviation between the actual attitude of the laser target and the DTA-designed attitude is as follows: ; ; ; ; ; ; in, It is the attitude deviation of the laser target in the X-axis direction. It is the attitude deviation of the laser target in the Y-axis direction. It is the attitude deviation of the laser target in the Z-axis direction. Rolling attitude deviation of the laser target Pitch attitude deviation of the laser target Yaw attitude deviation of laser target , , , respectively t Design coordinates of DTA at any time , , for t The angle of DTA at any given moment; This is the measured roll angle of the laser target in the DTA coordinate system. It is the measured value of the elevation angle of the laser target in the DTA coordinate system. It is the measured value of the yaw angle of the laser target in the DTA coordinate system; S2.13 Noise filtering: The moving average filtering algorithm is used to suppress the aiming jitter of the electronic total station and obtain the filtered laser target attitude deviation data.

[0022] In step S2, the visual data preprocessing specifically includes: S2.21. Image preprocessing: The visual perception data is subjected to noise suppression and contrast enhancement processes sequentially. Noise suppression uses a Gaussian filtering algorithm, the formula of which is: ; in, The dimension parameter corresponding to the filter kernel size. These are the original image pixel values. These are the filtered pixel values. m For the filter kernel in x The length of half a side along the axial direction, n For the filter kernel in y The length of half a side along the axial direction, The coordinates of the target pixel to be filtered in the image are integers. x For pixel column coordinates, y These are the pixel row coordinates; S2.22, Marker point detection: A target detection algorithm is used to identify photoelectric marker points in the image and output pixel coordinates. ; In S2.22, the target detection algorithm uses the YOLO series algorithm or an improved target detection network, and filters photoelectric markers by setting a pre-set confidence threshold and a non-maximum suppression algorithm to ensure detection accuracy.

[0023] S2.23. Camera calibration: Obtain camera intrinsic parameters using Zhang Zhengyou's calibration method. and extrinsic parameter matrix The expression for the intrinsic parameter matrix is: ; in, 、 For camera focal length, 、 Principal point coordinates; S2.24, World coordinate calculation: Based on the principle of perspective projection, the pixel coordinates are transformed to the DTA coordinate system. The formula is as follows: ; in, Let be the depth of the photoelectric marker point in the camera coordinate system. These are the pixel column coordinates of the photoelectric markers in the visual image. Let be the pixel row coordinates of the photoelectric marker in the visual image. It is the attitude rotation matrix of the camera coordinate system relative to the DTA coordinate system. It is the spatial translation vector of the camera coordinate system origin relative to the DTA coordinate system origin. 、 、 These are the world coordinates of the visual marker points in the DTA coordinate system; In S2.24, the depth of the visual marker point in the camera coordinate system It is obtained through binocular vision measurement, calculation of the spacing between preset marker points, or measurement assisted by a depth sensor.

[0024] S2.25. Attitude Deviation Calculation: Based on the world coordinates of the visual markers and the DTA design coordinates, calculate the attitude deviation measured visually. .

[0025] S3. Construct a deep learning heterogeneous fusion model, which includes a laser absolute feature branch, a visual spatiotemporal feature branch, an attention fusion layer, and an output layer, and train the model. Specifically, S3 includes: S3.1, Laser Absolute Feature Branch: Input the filtered 6-dimensional laser target attitude deviation data, perform feature encoding using multiple fully connected layers, and use ReLU as the activation function to output the laser feature vector. The formula is: ; in, This represents the total number of fully connected layers in the laser absolute feature branch. Here are the weight matrices for each layer. For each layer's bias vector, This is the filtered laser target attitude deviation feature vector; S3.2, Visual Spatiotemporal Feature Branch: The input consists of a temporal sequence of multiple consecutive frames of visual pose deviation data. First, a Convolutional Neural Network (CNN) extracts spatial features. Then, a Bidirectional Long Short-Term Memory (BiLSTM) network captures temporal dependencies. Finally, a fully connected layer outputs a visual feature vector. The formula is: ; ; ; in, It is a visual temporal feature sequence, composed of multiple consecutive frames of visual pose deviation data spliced ​​together in chronological order. It is the visual space feature vector extracted by the convolutional neural network (CNN). It is the output feature vector of the Bidirectional Long Short-Term Memory (BiLSTM) network. This is the weight matrix of the fully connected layer. This is the bias vector for the fully connected layer; In S3.2, the Convolutional Neural Network (CNN) adopts a structure of alternating convolutional and activation layers, and maintains the consistency of feature dimensions through zero padding or adaptive pooling; the input dimension of the Bidirectional Long Short-Term Memory Network (BiLSTM) matches the output feature dimension of the CNN, and captures temporal dependencies through forward and backward hidden state fusion. S3.3 Attention Fusion Layer, which integrates laser feature vectors With visual feature vectors The weights are concatenated and fused adaptively through a fully connected layer and a sigmoid function. The formula is: ; Feature fusion is achieved based on fusion weights, and the formula is as follows: ; in, This is the attention layer weight matrix. This is the attention layer bias vector. This indicates that the feature vectors are concatenated horizontally. In S3.3, the fusion weights The value range is [0,1]. The contribution of absolute laser features and spatiotemporal visual features is dynamically adjusted through the attention mechanism to achieve adaptive fusion under different working conditions. S3.4, Output Layer: Outputs the final 6-dimensional attitude deviation through a fully connected layer. The angular deviation is calculated by the geometric relationship between the positional deviation and the characteristic length of the shield body, using the following formula: ; ; ; in, Pre-set characteristic lengths for the tunnel boring machine; The X-axis position deviation in the final attitude deviation, The Y-axis position deviation in the final attitude deviation, The Z-axis position deviation in the final attitude deviation, The roll angle deviation is part of the final attitude deviation. The pitch angle deviation is part of the final attitude deviation. The yaw angle deviation in the final attitude deviation; In S3.4, the tunnel boring machine has a preset characteristic length. The distance from the center of the cutter head to the center of the laser target installation is obtained through actual precision measurement. S3.5 Model Training: Collect synchronous data from dual sensors of the tunnel boring machine under different working conditions to construct a dataset. Use the filtered laser target attitude deviation as a label, and train a deep learning heterogeneous fusion model using a gradient descent optimizer and a mean squared error loss function. The loss function formula is as follows: ; in, The number of training samples, To account for model prediction bias, For laser deviation labels.

[0026] In step S3.5, the dataset is divided into a training set, a validation set, and a test set according to a preset ratio. The training process uses an early stopping strategy or regularization method to avoid model overfitting and ensure the model's generalization ability.

[0027] S4. Input the preprocessed real-time data into the trained deep learning heterogeneous fusion model and dynamically calibrate the output results.

[0028] Specifically, S4 includes: S4.1 Input the preprocessed real-time laser data and visual temporal data into the trained deep learning heterogeneous fusion model, and output the attitude deviation prediction results. S4.2. The model prediction results are dynamically calibrated using the latest filtered laser target attitude deviation at a preset cycle to obtain the final attitude deviation. The calibration formula is as follows: ; in, This is the final 6-dimensional attitude deviation vector output after dynamic calibration, containing 3 position deviations ( ) and 3 angular deviations ( ), These are the weighting coefficients. This represents the vector of attitude deviation predictions output by the deep learning heterogeneous fusion model. This is a vector of tag values ​​based on the filtered laser target attitude deviation.

[0029] In S4.2, the calibration cycle is dynamically adjusted according to the working conditions of the tunnel boring machine. When the vibration intensity or environmental interference exceeds the preset threshold, the calibration cycle is shortened; when the working conditions are stable, the calibration cycle is extended.

[0030] The following is an explanation of some of the English terms and related technical terms involved in this embodiment: The Detached Temporal-Axis Coordinate System (DTA) is a spacetime coordinate system primarily used to describe motion in dynamic or non-inertial reference frames, especially in scenarios where time separation effects or relativistic spacetime transformations need to be considered.

[0031] The moving average filter algorithm is a classic time-domain signal processing algorithm that effectively suppresses random noise and preserves trend information by averaging local window data of a signal sequence.

[0032] This invention provides a method for measuring the attitude deviation of tunnel boring machines (TBMs) based on heterogeneous fusion of laser and vision deep learning. It involves real-time, high-precision measurement technology for attitude deviation during TBM tunneling, applicable to various TBM construction scenarios such as urban subways, highway tunnels, and railway tunnels. Addressing the shortcomings of existing single-sensor methods (limited laser range and weak visual interference resistance) and traditional fusion methods in handling heterogeneous data, this invention employs a dual-sensor approach: a laser target (in conjunction with an electronic total station) and a vision camera. This involves sensor deployment and synchronous data acquisition; laser data coordinate transformation, deviation calculation and filtering; visual data image preprocessing, marker point detection, camera calibration, and world coordinate calculation; and the construction of a deep learning network containing a laser absolute feature branch, a visual spatiotemporal feature branch (CNN extracting spatial features + BiLSTM capturing temporal dependencies), and an attention fusion layer. After training, the network outputs the attitude deviation; real-time inference is performed, and dynamic calibration is achieved by combining laser data. This invention dynamically adjusts sensor weights through an attention mechanism to adapt to different working conditions, providing guidance for the autonomous steering control of the TBM.

[0033] This invention solves the problem that fixed weights or linear filtering in traditional linear fusion schemes cannot dynamically adapt to working conditions, and realizes adaptive weight allocation based on working condition awareness, thereby improving the anti-interference capability in complex environments. Simultaneously, it addresses the issue of existing solutions failing to exploit the spatiotemporal characteristics of data, fully extracting and utilizing the spatiotemporal information of both types of data to improve dynamic measurement accuracy and real-time response speed. Real-time performance meets engineering requirements: the output frequency matches the visual camera frame rate, resolving the discontinuous measurement problem of the prism guidance system.

[0034] The core innovation of this invention lies in its innovative heterogeneous data fusion architecture: it proposes a dual-channel deep learning architecture of "laser absolute features + visual spatiotemporal features," and for the first time combines CNN (spatial feature extraction) and BiLSTM (temporal dependency capture) for shield machine attitude measurement, solving the problem of nonlinear heterogeneous fusion of laser and visual data; an adaptive weight allocation mechanism: dynamically learning fusion weights through an attention fusion layer. It can adaptively adjust the contribution of laser and vision according to environmental disturbances (such as dust) and working conditions (such as tunneling / assembly), breaking through the fixed weight limitation of traditional linear fusion; The core difference between existing technical solutions and this invention lies in the fusion mechanism. They employ traditional linear fusion algorithms such as fixed-weight weighted average or extended Kalman filter (EKF). For example, they assign a fixed dominant weight based on laser deviation and assign a secondary weight as an auxiliary weight based on visual deviation. Alternatively, they use EKF to linearly iterate and update the laser as the observation value and the visual as the prediction value. They do not introduce a deep learning architecture, cannot dynamically adapt the weights to the working conditions, and do not explore the temporal dependence of visual data and the spatial correlation features of laser data. This is the key difference between them and this invention in terms of technical essence.

[0035] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning, characterized in that, Includes the following steps: S1. A dual-sensor system, consisting of a laser target and a vision camera, is used to deploy the sensors and collect data, including laser data and vision data. S2. Preprocess the data, including laser data preprocessing and visual data preprocessing; laser data preprocessing includes laser data coordinate transformation, laser target attitude deviation calculation and noise filtering; visual data preprocessing includes image preprocessing, marker detection, camera calibration and world coordinate solution, and visual measurement attitude deviation calculation. S3. Construct a deep learning heterogeneous fusion model, which includes a laser absolute feature branch, a visual spatiotemporal feature branch, an attention fusion layer, and an output layer, and train the model. S4. Input the preprocessed real-time data into the trained deep learning heterogeneous fusion model and dynamically calibrate the output results.

2. The method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning according to claim 1, characterized in that, In step S1, the specific method for sensor deployment and data acquisition using a laser target in conjunction with an electronic total station and a dual-sensor visual camera includes: S1.1 Install the laser target at the center axis of the front end of the shield of the tunnel boring machine to ensure stable incident of the laser beam from the electronic total station; install the vision camera in an area of ​​the shield with little dust and no obstruction, facing the photoelectric marker point preset on the tunnel wall, with the photoelectric marker point and the laser target in the same designed tunnel axis DTA coordinate system; S1.2 The electronic total station collects the absolute attitude data of the laser target in the station coordinate system at a preset cycle, including position and angle; the visual camera collects the images of the photoelectric markers at a preset frame rate, which is the visual perception data. S1.

3. Using the electronic total station timestamp as a reference, the visual perception data is synchronized and aligned to ensure that the time synchronization error meets the measurement accuracy requirements.

3. The method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning according to claim 2, characterized in that, In step S1.3, time synchronization employs a linear interpolation algorithm or a timestamp alignment algorithm to ensure that the temporal correlation between laser data and visual data meets the input requirements of the deep learning heterogeneous fusion model. Specifically, this includes: Employing a timestamp alignment algorithm, both the electronic total station and the visual camera enable high-precision timestamp recording. Using the system time of the electronic total station as the reference clock, the visual camera timestamps undergo initial calibration via NTP protocol or hardware synchronization trigger. After data acquisition, the timestamp sequence of the visual perception data is traversed, and the visual image closest to each laser data timestamp is found through the nearest neighbor matching rule. If no direct match exists, the corresponding laser data or visual image is discarded to ensure the temporal correlation of the paired data. A linear interpolation algorithm is used to complete visual pose deviation data whose timestamps are not directly aligned when the laser data acquisition period does not match the visual camera frame rate. Let the timestamps of two adjacent valid visual data points be... 、 The corresponding attitude deviation is 、 The laser data timestamp is ,and The corresponding interpolation result is The same applies to the deviations on the Y-axis and Z-axis.

4. The method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning according to claim 2, characterized in that, In step S2, the laser data preprocessing specifically includes: S2.

11. Laser data coordinate transformation: Using at least three known control points, solve the 4×4 homogeneous transformation matrix using the least squares algorithm to transform the coordinates of the laser target from the station coordinate system to the DTA coordinate system. The formula is as follows: ； in, Let X be the X-axis coordinate of the laser target in the DTA coordinate system. Let Y be the Y-axis coordinate of the laser target in the DTA coordinate system. The Z-axis coordinate of the laser target in the DTA coordinate system It is the homogeneous transformation matrix from the station coordinate system to the DTA coordinate system. It is a 3×3 rotation matrix that describes the attitude rotation relationship of the station coordinate system relative to the DTA coordinate system, and satisfies orthogonality. , It is the identity matrix. It is a 3×1 translation vector, describing the spatial translation of the origin of the station coordinate system relative to the origin of the DTA coordinate system, including the offset distances in the X, Y, and Z directions. It is a 3-dimensional zero vector, a standard component of the homogeneous transformation matrix, used to fill the matrix dimension, has no actual physical meaning, and only ensures the standardization of matrix operations; It is the original measured coordinate of the laser target along the X-axis in the station coordinate system established by the electronic total station. It is the original measured Y-axis coordinate of the laser target in the station coordinate system; These are the original measured coordinates of the laser target along the Z-axis in the station coordinate system; S2.12 Calculation of laser target attitude deviation: The formula for calculating the deviation between the actual attitude of the laser target and the DTA-designed attitude is as follows: ； ； ； ； ； ； in, It is the attitude deviation of the laser target in the X-axis direction. It is the attitude deviation of the laser target in the Y-axis direction. It is the attitude deviation of the laser target in the Z-axis direction. Rolling attitude deviation of the laser target Pitch attitude deviation of the laser target Yaw attitude deviation of laser target , , , respectively t Design coordinates of DTA at any time , , for t The angle of DTA at any given moment; This is the measured roll angle of the laser target in the DTA coordinate system. It is the measured value of the elevation angle of the laser target in the DTA coordinate system. It is the measured value of the yaw angle of the laser target in the DTA coordinate system; S2.13 Noise filtering: The moving average filtering algorithm is used to suppress the aiming jitter of the electronic total station and obtain the filtered laser target attitude deviation data.

5. The method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning according to claim 4, characterized in that, In step S2, the visual data preprocessing specifically includes: S2.

21. Image preprocessing: The visual perception data is subjected to noise suppression and contrast enhancement processes sequentially. Noise suppression uses a Gaussian filtering algorithm, the formula of which is: ； in, The dimension parameter corresponding to the filter kernel size. These are the original image pixel values. These are the filtered pixel values. m For the filter kernel in x The length of half a side along the axial direction, n For the filter kernel in y The length of half a side along the axial direction, The coordinates of the target pixel to be filtered in the image are integers. x For pixel column coordinates, y These are the pixel row coordinates; S2.22, Marker point detection: A target detection algorithm is used to identify photoelectric marker points in the image and output pixel coordinates. ; S2.

23. Camera calibration: Obtain camera intrinsic parameters using Zhang Zhengyou's calibration method. and extrinsic parameter matrix The expression for the intrinsic parameter matrix is: ； in, 、 For camera focal length, 、 Principal point coordinates; S2.24, World coordinate calculation: Based on the principle of perspective projection, the pixel coordinates are transformed to the DTA coordinate system. The formula is as follows: ； in, Let be the depth of the photoelectric marker point in the camera coordinate system. These are the pixel column coordinates of the photoelectric markers in the visual image. Let be the pixel row coordinates of the photoelectric marker in the visual image. It is the attitude rotation matrix of the camera coordinate system relative to the DTA coordinate system. It is the spatial translation vector of the camera coordinate system origin relative to the DTA coordinate system origin. 、 、 These are the world coordinates of the visual marker points in the DTA coordinate system; S2.

25. Attitude Deviation Calculation: Based on the world coordinates of the visual markers and the DTA design coordinates, calculate the attitude deviation measured visually. .

6. The method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning according to claim 5, characterized in that, In S2.11, the coordinates of the known control point in the DTA coordinate system and the coordinates of the station coordinate system are obtained through static precision measurement, and the homogeneous transformation matrix is ​​used. The solution is achieved through least squares iteration, and the convergence condition of the iteration is that the coordinate transformation residuals satisfy the preset accuracy threshold. In S2.22, the target detection algorithm uses the YOLO series algorithm or an improved target detection network, and filters photoelectric markers by setting a pre-set confidence threshold and a non-maximum suppression algorithm to ensure detection accuracy. In S2.24, the depth of the visual marker point in the camera coordinate system It is obtained through binocular vision measurement, calculation of the spacing between preset marker points, or measurement assisted by a depth sensor.

7. The method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning according to claim 5, characterized in that, Specifically, S3 includes: S3.1, Laser Absolute Feature Branch: Input the filtered 6-dimensional laser target attitude deviation data, perform feature encoding using multiple fully connected layers, and use ReLU as the activation function to output the laser feature vector. The formula is: ； in, This represents the total number of fully connected layers in the laser absolute feature branch. Here are the weight matrices for each layer. For each layer's bias vector, This is the filtered laser target attitude deviation feature vector; S3.2, Visual Spatiotemporal Feature Branch: The input consists of a temporal sequence of multiple consecutive frames of visual pose deviation data. First, a Convolutional Neural Network (CNN) extracts spatial features. Then, a Bidirectional Long Short-Term Memory (BiLSTM) network captures temporal dependencies. Finally, a fully connected layer outputs a visual feature vector. The formula is: ； ； ； in, It is a visual temporal feature sequence, composed of multiple consecutive frames of visual pose deviation data spliced ​​together in chronological order. It is the visual space feature vector extracted by the convolutional neural network (CNN). It is the output feature vector of the Bidirectional Long Short-Term Memory (BiLSTM) network. This is the weight matrix of the fully connected layer. This is the bias vector for the fully connected layer; S3.3 Attention Fusion Layer, which integrates laser feature vectors With visual feature vectors The weights are concatenated and fused adaptively through a fully connected layer and a sigmoid function. The formula is: ； Feature fusion is achieved based on fusion weights, and the formula is as follows: ； in, This is the attention layer weight matrix. This is the attention layer bias vector. This indicates that the feature vectors are concatenated horizontally. S3.4, Output Layer: Outputs the final 6-dimensional attitude deviation through a fully connected layer. The angular deviation is calculated by the geometric relationship between the positional deviation and the characteristic length of the shield body, using the following formula: ； ； ； in, Pre-set characteristic lengths for the tunnel boring machine; The X-axis position deviation in the final attitude deviation, The Y-axis position deviation in the final attitude deviation, The Z-axis position deviation in the final attitude deviation, The roll angle deviation is part of the final attitude deviation. The pitch angle deviation is part of the final attitude deviation. The yaw angle deviation in the final attitude deviation; S3.5 Model Training: Collect synchronous data from dual sensors of the tunnel boring machine under different working conditions to construct a dataset. Use the filtered laser target attitude deviation as a label, and train a deep learning heterogeneous fusion model using a gradient descent optimizer and a mean squared error loss function. The loss function formula is as follows: ； in, The number of training samples, To account for model prediction bias, For laser deviation labels.

8. The method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning according to claim 7, characterized in that, In S3.2, the Convolutional Neural Network (CNN) adopts a structure of alternating convolutional and activation layers, and maintains the consistency of feature dimensions through zero padding or adaptive pooling; the input dimension of the Bidirectional Long Short-Term Memory Network (BiLSTM) matches the output feature dimension of the CNN, and captures temporal dependencies through forward and backward hidden state fusion. In S3.3, the fusion weights The value range is [0,1]. The contribution of absolute laser features and spatiotemporal visual features is dynamically adjusted through the attention mechanism to achieve adaptive fusion under different working conditions. In S3.4, the tunnel boring machine has a preset characteristic length. The distance from the center of the cutter head to the center of the laser target installation is obtained through actual precision measurement. In step S3.5, the dataset is divided into a training set, a validation set, and a test set according to a preset ratio. The training process uses an early stopping strategy or regularization method to avoid model overfitting and ensure the model's generalization ability.

9. The method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning according to claim 7, characterized in that, Specifically, S4 includes: S4.1 Input the preprocessed real-time laser data and visual temporal data into the trained deep learning heterogeneous fusion model, and output the attitude deviation prediction results. S4.

2. The model prediction results are dynamically calibrated using the latest filtered laser target attitude deviation at a preset cycle to obtain the final attitude deviation. The calibration formula is as follows: ； in, This is the final 6-dimensional attitude deviation vector output after dynamic calibration, containing 3 position deviations ( ) and 3 angular deviations ( ), These are the weighting coefficients. This represents the vector of attitude deviation predictions output by the deep learning heterogeneous fusion model. This is a vector of tag values ​​based on the filtered laser target attitude deviation.

10. The method for measuring the attitude deviation of a tunnel boring machine based on heterogeneous fusion of laser-vision deep learning according to claim 9, characterized in that, In S4.2, the calibration cycle is dynamically adjusted according to the working conditions of the tunnel boring machine. When the vibration intensity or environmental interference exceeds the preset threshold, the calibration cycle is shortened; when the working conditions are stable, the calibration cycle is extended.

Images