A tunnel boring machine real-time data outlier detection and correction method

Abstract

The application provides a kind of tunnel boring machine's measured data outlier detection and correction method, belongs to the field of outlier data detection, and the measured data of tunnel boring machine under different working conditions is detected by the method of sliding window, and the outlier point is corrected and filled up.The method first divides the original time series into multiple sub-time series by sliding window, and extracts the confidence interval radius of sub-time series slope by fast calculation and identifies abnormal sub-time series, then further determines the outliers using local outlier factor algorithm, and finally uses regression technique to reasonably fill the outliers removed.The application can effectively identify outliers in tunnel boring machine measured data, and reasonably correct and fill the outliers, ensuring the engineering usability of tunnel boring machine measured data and providing good conditions for further data analysis.

Description

Technical Field

[0001] This invention belongs to the field of abnormal data detection, and relates to a method for detecting and correcting outliers in measured data of tunnel boring machines. Background Technology

[0002] A tunnel boring machine (TBM) is a large-scale engineering machine specifically designed for full-face excavation in tunnel engineering. It is an intelligent tunnel construction equipment integrating mechanics, electrical systems, hydraulics, information systems, and control systems, playing a vital role in urban underground engineering construction. Its intelligent operation and maintenance system is driven by big data technology, performing data mining and analysis on data collected at the work site to predict necessary information, providing crucial references for operation and control, and ultimately ensuring the efficient, stable, safe, and scientific operation of the TBM.

[0003] During tunnel excavation, construction site data is collected by various sensors on the tunnel boring machine (TBM). This data is then digitized and stored to form the tunneling operation dataset. Machine learning and other artificial intelligence algorithms aim to discover connections and extract more useful information from this large and complex engineering data through data mining methods. However, such construction site data often suffers from unavoidable problems, such as data redundancy, outliers, and noise. These defects not only complicate data analysis and storage but also significantly hinder the development of data modeling algorithms. Meaningless outliers can cause problems in data modeling, such as difficulty in optimization convergence and reduced prediction accuracy. Since the generation of outliers is unavoidable, how to handle outliers and minimize their impact on data analysis becomes a key research concern. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a method for detecting and correcting outliers in measured data of tunnel boring machines. This method is used to detect measured operating data of tunnel boring machines under different working conditions, thereby determining whether there are outlier data in the operating data of tunnel boring machines, and correcting and filling the detected outlier data.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] An outlier detection and correction method for measured data of tunnel boring machines (TBMs) is presented, comprising two parts: outlier detection and correction. The outlier detection part has two stages: the first stage is outlier sub-time series detection, which uses a sliding window method to divide the unprocessed TBM measured data into several sub-time series, then extracts features from these sub-time series; if a feature exceeds a set threshold, the series is considered an outlier sub-time series. The second stage is outlier detection using the Local Outlier Factor (LOF) algorithm, which detects outliers in the outlier sub-time series. The outlier correction part uses a least squares method to establish a linear regression model for normal values ​​in the outlier sub-series to predict the values ​​of outlier points. The final outlier point value is the average of the predicted values ​​from multiple outlier sub-time series.

[0007] (I) Outlier Detection:

[0008] Input: A time series of length n, represented as X(t) = (x(t1), x(t2), ..., x(t3)). n )), where x(t) i (1≤i≤n) is t i Data recorded at any time, collection time t i It is strictly incremental. The sliding window length is w, and the threshold for judging abnormal sub-time series is γ.

[0009] Output: Outliers in the outlier sub-time series.

[0010] The steps are as follows:

[0011] The first step is to use a sliding window of length w to divide the time series X(t) of the tunnel boring machine's measured data into several sub-time series X. j (1≤j≤n-w+1).

[0012] The second step is to calculate the slope k(i) of two adjacent points in the first sub-time series using formula (1), and to calculate the mean slope of the first sub-time series using formulas (2) and (3) respectively. The mean square error σ1 is used to calculate the confidence interval radius d1 of the slope of the first sub-time series using formula (4).

[0013] The expressions for formulas (1), (2), (3), and (4) are as follows:

[0014]

[0015]

[0016]

[0017]

[0018] in, Let σ be the mean slope of the j-th sub-time series. j The mean squared error of the slope of the j-th sub-time series. θ is the upper confidence level. j The lower confidence limit can be calculated using formulas (5) and (6) respectively.

[0019]

[0020]

[0021] Where Z is a normally distributed random variable satisfying N(0,1), and α is the confidence level.

[0022] The third step is to use formula (7) to calculate the slope of the current time series from the previous sub-time series, and use formulas (8) and (9) to calculate the mean s of the slope of the previous sub-time series. j and mean square error σ j Calculate the mean slope of this time series. and mean square error σ j+1 Therefore, the confidence interval radius d of the slope of the original time series is calculated using equation (4). j+1 , where 1≤j <n-w+1。

[0023] The expressions for formulas (7), (8), and (9) are as follows:

[0024]

[0025]

[0026]

[0027] The fourth step is to compare the confidence interval radius of the sub-time series slope with the threshold γ to preliminarily identify abnormal sub-time series X containing outliers. l (1≤l≤n-w+1).

[0028] The fifth step involves analyzing the anomalous sub-time series X detected in the fourth step. l The LOF algorithm is used to calculate the outlier factor of each data point in order to identify outliers. The LOF algorithm first calculates the reachability distance of each data point using formula (10), then calculates the local reachability density of the data point using formula (11), and finally calculates the local outlier factor of the data point using formula (12).

[0029] The expressions for formulas (10), (11), and (12) are as follows:

[0030] RD k (p,o)=max{k-distance(o),dist(p,o)} (10)

[0031]

[0032]

[0033] Where, points p, o ∈ X l RD k (p,o) represents the k-th reachable distance from point p to point o. k-distance(o) is the K-nearest neighbor distance of point o, which is the distance between the k-th nearest neighbor point among the nearest samples to the test sample point o, and the test detection point o. dist(p,o) is the Euclidean distance from point p to point o. |N k-distance (p)| represents the k-th distance neighborhood of point p, referring to the set of points whose distance from point p is less than k-distance(p). LRD k (p) represents the local reachability density of point p, which is essentially the reciprocal of the average reachability distance from p to points in its k-th neighborhood. k (o) represents the local reachability density of point o. LOF(p) represents the local outlier factor of point p.

[0034] Step 6: Take half of the maximum value among the local outliers as the threshold ρ.

[0035] The seventh step is to compare the outlier factor of each data point with the threshold ρ. If it is greater than the threshold ρ, the data point is an outlier; otherwise, it is a normal value.

[0036] (II) Outlier Correction:

[0037] Input: Anomaly sub-time series X l Outliers in the anomalous sub-time series.

[0038] Output: Outlier imputation dataset.

[0039] The steps are as follows:

[0040] The first step is to use the least squares method to predict the values ​​of outlier points in the normal values ​​of the input outlier sub-time series.

[0041] The second step is to record the index of the outlier point in the tunnel boring machine's measured data. If the outlier point index is not present in the outlier filling dataset, the index is created and the predicted value is recorded. Otherwise, the predicted value of the indexed point is recorded directly.

[0042] The third step is to calculate the mean of the predicted values ​​of multiple sub-time series for each outlier point in the outlier imputation dataset if the sub-time series is the last sub-time series. This value is then used as the final imputation data for that outlier point. Otherwise, return to the fourth step in the outlier detection section.

[0043] The effective gain of this invention is as follows: This invention uses a sliding window method to perform layered outlier detection on the actual test dataset of tunnel boring machines, and fills and corrects the detected outlier points. However, due to the complex environment of construction sites, it is unavoidable that the actual collected data will contain outliers, severely reducing the data's usability. This invention can effectively identify outliers of different types of parameters in the actual test data of tunnel boring machines, and reasonably fills and corrects these outliers, ensuring the practicality of the tunnel boring machine test data. Attached Figure Description

[0044] Figure 1 This is a flowchart of the present invention;

[0045] Figure 2 This is a schematic diagram of the original measured thrust data of the original tunnel boring machine of the present invention;

[0046] Figure 3 This is a schematic diagram of the radius of the confidence interval for the slope of the sub-time series of thrust data of the tunnel boring machine according to the present invention;

[0047] Figure 4 This is a schematic diagram of tunnel boring machine thrust data with anomaly sub-time series marked according to the present invention;

[0048] Figure 5 This is a schematic diagram of the thrust data of a tunnel boring machine with outlier data marked according to the present invention;

[0049] Figure 6 This is a schematic diagram of the thrust data of the tunnel boring machine after filling and correcting outlier data according to the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to specific embodiments.

[0051] like Figure 1As shown, this invention proposes a method for outlier detection and correction in measured data of tunnel boring machines (TBMs). It employs a sliding window method to perform layered outlier detection on measured TBM data under different working conditions and then corrects and fills in outlier points. To verify the effectiveness and practicality of the proposed method, a collected measured dataset of TBM data is used for testing. This data originates from the thrust of TBMs in the tunnel construction section between Houting and Songgang on Shenzhen Metro Line 11. First, thrust data of the TBM under different working conditions is collected at the construction site using sensors, and the radius of the slope confidence interval of the sub-time series of thrust data is calculated. Then, a threshold γ is input to obtain the abnormal sub-time series of thrust data. The LOF algorithm is used to detect outlier points in the abnormal sub-time series. Finally, a multi-sub-time series-based outlier filling method is used to fill and correct the detected outliers.

[0052] The outlier detection section includes the following steps:

[0053] Input: Measured thrust data X(t) of a tunnel boring machine with a length of n = 1605, as shown in Figure 2. The sliding window length w is set to 7, and the threshold γ for judging abnormal sub-time series is 150.

[0054] Output: Outliers in the outlier sub-time series.

[0055] The first step is to divide the time series X(t) into equal-length sub-time series using a sliding window of length w with a step size of 1, resulting in several sub-time series X. j (1≤j≤n-w+1).

[0056] The second step is to calculate the slope k(i) of two adjacent points in the first sub-time series using formula (1), and to calculate the mean slope of the first sub-time series using formulas (2) and (3) respectively. The mean square error σ1 is used to calculate the confidence interval radius d1 of the slope of the first sub-time series using formula (4).

[0057] The expressions for formulas (1), (2), (3), and (4) are as follows:

[0058]

[0059]

[0060]

[0061]

[0062] in, Let σ be the mean slope of the j-th sub-time series. jThe mean squared error of the slope of the j-th sub-time series. θ is the upper confidence level. j The lower confidence limit can be calculated using formulas (5) and (6) respectively.

[0063]

[0064]

[0065] Where Z is a normally distributed random variable satisfying N(0,1), and α is the confidence level, which is taken as 0.5 in this implementation.

[0066] The third step is to use formula (7) to calculate the slope of the current time series from the previous sub-time series, and use formulas (8) and (9) to calculate the mean slope of the previous sub-time series. and mean square error σ j Calculate the mean slope of this time series. and mean square error σ j+1 Therefore, the confidence interval radius d of the slope of the original time series is calculated using equation (4). j+1 , where 1≤j <n-w+1。

[0067] The expressions for formulas (7), (8), and (9) are as follows:

[0068]

[0069]

[0070]

[0071] The fourth step is to compare the confidence interval radius of the sub-time series slope with the threshold γ, such as... Figure 3 As shown, the anomalous sub-time series X containing outliers has been preliminarily identified. l (1≤l≤n-w+1), such as Figure 4 As shown.

[0072] The fifth step involves analyzing the anomalous sub-time series X detected in the fourth step. l The LOF algorithm is used to calculate the outlier factor of each data point in order to identify outliers. The LOF algorithm first calculates the reachability distance of each data point using formula (10), then calculates the local reachability density of the data point using formula (11), and finally calculates the local outlier factor of the data point using formula (12).

[0073] The expressions for formulas (10), (11), and (12) are as follows:

[0074] RD k(p,o)=max{k-distance(o),dist(p,o)} (10)

[0075]

[0076]

[0077] Where, points p, o ∈ X l RD k (p,o) represents the k-th reachable distance from point p to point o. k-distance(o) is the K-nearest neighbor distance of point o, which is the distance between the k-th nearest neighbor point among the nearest samples to the test sample point o, and the test detection point o. dist(p,o) is the Euclidean distance from point p to point o. |N k-distance (p)| represents the k-th distance neighborhood of point p, referring to the set of points whose distance from point p is less than k-distance(p). LRD k (p) represents the local reachability density of point p, which is essentially the reciprocal of the average reachability distance from p to points in its k-th neighborhood. k (o) represents the local reachability density of point o. LOF(p) represents the local outlier factor of point p.

[0078] Step 6: Take half of the maximum value among the local outliers as the threshold ρ.

[0079] Step 7: Compare the outlier factor of each data point with a threshold ρ. If it is greater than the threshold, the data point is considered an outlier; otherwise, it is considered a normal value. Detected outlier points are listed below. Figure 5 As shown.

[0080] The outlier correction section includes the following steps:

[0081] Input: Anomaly sub-time series X l Outliers in the anomalous sub-time series.

[0082] Output: Outlier imputation dataset.

[0083] The first step is to use the least squares method to predict the values ​​of outlier points in the normal values ​​of the input outlier sub-time series.

[0084] The second step is to record the index of the outlier point in the tunnel boring machine's measured data. If the outlier point index is not present in the outlier filling dataset, the index is created and the predicted value is recorded. Otherwise, the predicted value of the indexed point is recorded directly.

[0085] The third step, if the sub-time series is the last sub-time series, calculates the mean of the predicted values ​​of multiple sub-time series for each outlier point in the outlier imputation dataset, and uses this as the final imputation data for that outlier point; otherwise, returns to step four in the outlier detection section, where the corrected thrust measured data is as follows: Figure 6 As shown, outlier data was removed and appropriately filled in.

[0086] The recall rate R and precision P are calculated using formulas (13) and (14) respectively to evaluate the outlier detection part of this embodiment. The closer the values ​​of recall rate R and precision P are to 100%, the better the detection performance of the method. The coefficient of determination R is calculated using formula (15). 2 The coefficient of determination is used to evaluate the outlier correction part in this embodiment. The closer the coefficient of determination is to 1, the better the correction and filling performance.

[0087] The expressions for formulas (13), (14), and (15) are as follows:

[0088]

[0089]

[0090]

[0091] Where m represents the number of outlier imputations. x represents the fill value obtained by the method. i This represents the actual source data value corresponding to the filled value. Let it be its mean.

[0092] Ten repeated experiments were conducted, and the average values ​​of each indicator were taken. The results show that the method proposed in this invention can accurately identify abnormal data information in the measured data of tunnel boring machines, with a recall rate of 95% and a precision rate of 90.48%. In addition, regression techniques were used to reasonably fill and correct the outliers after removal, and the determination coefficient of the filling result was greater than 0.9, which meets the usability requirements of engineering data.

Claims

1. A method for outlier detection and correction in measured data of tunnel boring machines, characterized in that, The method is divided into two parts: outlier detection and correction. The outlier detection part has two stages: the first stage is outlier sub-time series detection, which uses a sliding window method to divide the unprocessed tunnel boring machine measured data into several sub-time series, then extracts features from the sub-time series, and if a feature exceeds a set threshold, the series is considered an outlier sub-time series; the second stage is outlier detection using the Local Outlier Factor (LOF) algorithm, which uses the LOF algorithm to detect outliers in the outlier sub-time series; the outlier correction part uses the least squares method to establish a linear regression model for the normal values ​​in the outlier sub-series to predict the values ​​of outlier points, and finally, the outlier point values ​​are taken as the average of the predicted values ​​of multiple outlier sub-time series; the details are as follows: (a) Outlier Detection: Input: A time series of length n, represented as ,in ,for Data recorded in real time, collection time It is incremental; the sliding window length w is the threshold for judging abnormal sub-time series. ; Output: Outliers in the anomalous sub-time series; The steps are as follows: The first step is to use a sliding window of length w to analyze the time series of measured data from the tunnel boring machine. Divide the data into equal-length sub-time series to obtain several sub-time series. ; The second step is to use formula (1) to calculate the slope between two adjacent points in the first sub-time series. The mean slope of the first sub-time series is calculated using formulas (2) and (3) respectively. and mean square deviation Therefore, the confidence interval radius of the slope of the first sub-time series is calculated using formula (4). ; The expressions for formulas (1), (2), (3), and (4) are as follows: （1）； （2）； （3）； （4）； in, Let be the mean slope of the j-th sub-time series. The mean squared error of the slope of the j-th sub-time series; This is the upper confidence level. The lower confidence limit can be calculated using formulas (5) and (6) respectively; （5）； （6）； Where Z is a normally distributed random variable satisfying N(0,1). Confidence level; The third step is to use formula (7) to calculate the slope of the current time series from the previous sub-time series, and use formulas (8) and (9) to calculate the mean slope of the previous sub-time series. and mean square deviation Calculate the mean slope of this time series. and mean square deviation Therefore, the confidence interval radius of the slope of the original time series is calculated using equation (4). ,in ; The expressions for formulas (7), (8), and (9) are as follows: （7）； （8）； （9）； The fourth step is to compare the confidence interval radius of the sub-time series slope with the threshold. By comparing the data, we can initially identify anomalous sub-time series containing outliers. ; The fifth step involves analyzing the abnormal sub-time series detected in the fourth step. The LOF algorithm is used to calculate the outlier factor of each data point in order to identify outliers. The LOF algorithm first calculates the reachability distance of each data point using formula (10), then calculates the local reachability density of the data point using formula (11), and finally calculates the local outlier factor of the data point using formula (12). The expressions for formulas (10), (11), and (12) are as follows: （10）； （11）； （12）； Among them, point , Let represent the k-th reachable distance from point p to point o; k-distance(o) is the K-nearest neighbor distance of point o, which refers to the distance between the k-th nearest neighbor point among the nearest samples to the test sample point o and the test detection point o. It is the Euclidean distance from point p to point o; Let k represent the k-th distance neighborhood of point p, which is the set of points whose distance from point p is less than k-distance(p); Let represent the local reachability density of point p, which is essentially the reciprocal of the average reachability distance from p to points in its k-th neighborhood. Let represent the local reachability density of point o; LOF(p) represents the local outlier factor of point p. Step 6: Take half of the maximum value among the local outliers as the threshold. ; Step 7: Compare the outlier factor of each data point with the threshold. Compare, if it is greater than the threshold If the value is zero, then the data point is an outlier; otherwise, it is a normal value. (ii) Outlier Correction: Input: Anomalous sub-time series and outliers in anomalous sub-time series; Output: Outlier imputation dataset; The steps are as follows: The first step is to use the least squares method to predict the values ​​of outlier points in the normal values ​​of the input outlier sub-time series. The second step is to record the index of the outlier point in the actual measurement data of the tunnel boring machine. If there is no index for the outlier point in the outlier filling dataset, the index is created and the predicted value is recorded. Otherwise, the predicted value of the index location is recorded directly. The third step is to calculate the mean of the predicted values ​​of multiple sub-time series for each outlier point in the outlier imputation dataset if the sub-time series is the last sub-time series. This value is then used as the final imputation data for that outlier point. Otherwise, return to the fourth step in the outlier detection section.

Images