Sensor data validity discrimination method for RF construction with multiple correlation coefficient features

Abstract

The present application relates to a kind of RF construction sensor data validity discrimination methods of multiple correlation coefficient characteristics, comprising the following steps: step S1: obtaining the data set of all sensors of the same to be measured construction acquisition;Step S2: the data set obtained in step S1 is carried out data validity preliminary discrimination, and valid data set is obtained;Step S3: valid data set is carried out feature extraction, and validity quantization index is obtained;Step S4: based on validity quantization index, train random forest model, and obtain validity discrimination random forest model;Step S5: the sensor data to be measured is input into validity discrimination random forest model, and discrimination result is obtained.The present application effectively reduces the requirement to field knowledge, realizes the automation of sensor validity discrimination, and prediction accuracy is high.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of structure sensor data validity discrimination, and particularly relates to a random forest structure sensor data validity discrimination method based on multiple correlation coefficient features. BACKGROUND

[0002] The existing structure monitoring data validity discrimination system is not perfect, and the commonly used automatic identification method is to use principal component analysis method combined with other models to identify single point distortion data from the perspective of statistical analysis. This method requires a large amount of domain knowledge and has poor effect when directly applied to judge the validity of data. SUMMARY

[0003] Therefore, the present application aims to provide a random forest structure sensor data validity discrimination method based on multiple correlation coefficient features, which realizes the automation of sensor validity discrimination and has high prediction accuracy.

[0004] To achieve the above object, the present application adopts the following technical solutions:

[0005] A random forest structure sensor data validity discrimination method based on multiple correlation coefficient features comprises the following steps:

[0006] Step S1: obtaining data sets collected by all sensors of the same structure to be measured;

[0007] Step S2: performing preliminary data validity discrimination on the data sets obtained in step S1 to obtain valid data sets;

[0008] Step S3: performing feature extraction on the valid data sets to obtain validity quantization indexes;

[0009] Step S4: training a random forest model based on the validity quantization indexes to obtain a validity discrimination random forest model;

[0010] Step S5: inputting the sensor data to be measured into the validity discrimination random forest model to obtain a discrimination result.

[0011] Further, the step S2 specifically comprises:

[0012] Step S21: performing EDA data analysis on the data sets, and according to the basic situation of the data, the sensor collected data that meets the following conditions is judged as invalid data, i.e. it has no significance for analysis and research, and does not meet the physical discrimination, and the invalid reason is output, and the process is ended: the missing data reaches more than a preset value, it is a constant value, and the data that does not meet the sensor accuracy and range exceeds a preset value; otherwise, step S22 is executed.

[0013] Step S22: according to the physical connection between the sensors, the sensor data having physical connection with the sensor to be studied is retained.

[0014] Further, the step S3 uses an ARMA model to extract features from historical data of the sensor itself, and constructs a sensor correlation index, and sets data labeling.

[0015] Further, the ARMA model includes stationarity test, white noise test, ACF test, PACF test, specifically:

[0016] The stationarity test determines whether the sequence has a unit root: if the sequence is stationary, there is no unit root; otherwise, there will be a unit root, and the ADF test is performed on the data set, the t-statistic is solved, and the significance p-value is taken as a feature index;

[0017] The white noise test uses Q LB statistic to perform pure random test on the data set, solve Q LB statistic, and take the significance p-value as a feature index, and solve the formula of Q LB statistic as follows:

[0018]

[0019] In the formula, n is the number of observation periods of the sequence, and m is the specified delay period.

[0020] The ACF test is to solve the autocorrelation coefficient of the data set to be tested, and take the autocorrelation coefficient as a feature index input into the random forest model.

[0021] The PACF test is to solve the partial correlation coefficient of the data set to be tested, and take the autocorrelation coefficient as a feature index input into the random forest model.

[0022] Further, the formula for solving the t-statistic is as follows:

[0023]

[0024] Where R=(ζ1,…,ζ P-1 ,ρ)′;

[0025] H0: |ρ|≥1 vs H1: |ρ|<1, and the test statistic is The sample data set is Y=(y p+1 ,…,y T )′, Where e i =[0,…,0,1]′=(e) p×1 ,

[0026] Further, the C index function is as follows:

[0027]

[0028] wherein X t is the sensor that needs to be judged for data validity, σ(X t+1 ,X t+2 ,…,X n ) is the X t other sensors in the related field that meet the physical connection.

[0029] Further, a label is set for judging whether the sensor data is valid, wherein the valid label is 0 and the invalid label is 1, and the labeling method is as follows:

[0030]

[0031] wherein p ADF is a p value of stationarity test significance, is a p value of white noise test significance, L ACF , L PACF are autocorrelation coefficients and partial correlation coefficients, respectively.

[0032] Further, the step S4 is specifically: taking p ADF , L ACF , L PACF , C 指标 as independent variables, taking lable as dependent variable, inputting a random forest model, performing classification model training, and obtaining a validity judgment random forest model.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] The present application effectively reduces the requirement for domain knowledge, realizes automation of sensor validity judgment, has high prediction accuracy, and has strong engineering practicability. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a flow chart of the method of the present application;

[0036] Figure 2 is a C index construction flowchart in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The present application will be further described below in combination with the drawings and embodiments.

[0038] Please refer to Figure 1 , the present application provides a RF construction sensor data validity judgment method with multiple correlation coefficient characteristics, comprising the following steps:

[0039] Step S1: Obtain the data set collected by all sensors of the same structure to be tested;

[0040] Step S2: Perform preliminary data validity discrimination on the data set obtained in step S1 to obtain a valid data set;

[0041] Step S3: Perform feature extraction on the valid data set to obtain a validity quantization index;

[0042] Step S4: Train a random forest model based on the validity quantization index to obtain an effectiveness discrimination random forest model;

[0043] Step S5: Input the sensor data to be tested into the effectiveness discrimination random forest model to obtain a discrimination result.

[0044] In this embodiment, step S1 is specifically

[0045] The data set collected by all sensors of the same structure (bridge, reservoir) is used, which includes stress sensors and temperature sensors, seepage sensors, etc. The specific data format is as follows:

[0046]

[0047] The data set is a total data set, and step 2 filters the data set according to the set rules to determine the data set required for subsequent steps.

[0048] In this embodiment, step S2 is specifically

[0049] Step S21: Perform EDA data analysis on the data set to respectively explore the distribution, missing condition, extreme value, data trend, etc. of the data, so as to generally grasp the basic situation of the data set.

[0050] According to the basic situation of the data, the sensor collected data with more than 70% of missing data, a constant value, more than 70% of data not meeting the sensor accuracy and range, etc. is judged as invalid data, i.e. it has no research significance and does not meet the physical discrimination, and the invalid reason is output, and the process is ended; otherwise, step S22 is executed;

[0051] Step S22: Obtain the data set with research significance in step S21, and retain the data of the sensors having physical connection with the research sensors according to the physical connection between the sensors. For example, the research is for the reservoir displacement sensor, and the reservoir water level sensor is retained, and the reservoir rain gauge sensor is removed, because displacement and rainfall do not have direct physical connection, and displacement and reservoir water level have direct physical connection. The physical connection between sensors and sensors needs to be determined according to the physical structure connection of the bridge and the reservoir and the sensor installation standard. Step S3 is executed to perform data feature extraction.

[0052] In the present embodiment, step S3, specifically:

[0053] Feature extraction is performed on the historical data of the sensor itself using an ARMA model, which includes stationarity test, white noise test, ACF test, and PACF test, specifically:

[0054] Stationarity test is to determine whether the sequence has a unit root: if the sequence is stationary, there is no unit root; otherwise, there will be a unit root. ADF test is performed on the data set, the t-statistic is solved, and the significance p-value is taken as a feature index. The formula for solving the t-statistic is as follows:

[0055]

[0056] Where R=(ζ1,…,ζ P-1 ,ρ)′;

[0057] H0:|ρ|≥1 vs H1:|ρ|<1, the test statistic is The sample data set is Y=(y p+1 ,…,y T )′, Where e i =[0,…,0,1]′=(e) p×1 , The white noise test uses the Q LB statistic to perform a pure random test on the data set, solves the Q LB statistic, and takes the significance p-value as a feature index. The formula for solving the Q LB statistic is as follows:

[0058]

[0059] Where n is the number of observation periods of the sequence, and m is the specified delay period.

[0060] The white noise test uses the Q LB statistic to perform a pure random test on the data set, solves the Q LB statistic, and takes the significance p-value as a feature index. The formula for solving the Q LB statistic is as follows:

[0061]

[0062] Where n is the number of observation periods of the sequence, and m is the specified delay period.

[0063] ACF test, for the measured segment data set for the autocorrelation coefficient solution, the autocorrelation coefficient as a random forest model input feature index. Such as X1 displacement sensor, T1 to T n (n>1) time period of data collected for validity discrimination, that is, the segment data for the autocorrelation coefficient solution.

[0064] PACF test, for the measured segment data set for the partial correlation coefficient solution, the autocorrelation coefficient as a random forest model input feature index. Such as X1 displacement sensor, T1 to T n (n>1) time period of data collected for validity discrimination, that is, the segment data for the autocorrelation coefficient solution.

[0065] In this embodiment, the sensor and its same domain and physical contact sensor correlation analysis (Pearson correlation analysis, gray correlation analysis, Spearman correlation analysis, Kendall correlation analysis), so as to extract the sensor and its same domain and physical contact sensor characteristics, detailed C index construction process as shown in Figure 2

[0066] C index function as follows:

[0067]

[0068] Where X t is the sensor that needs to be data effective discrimination, σ(X t+1 ,X t+2 ,…,X n ) is X t about the field to meet the physical contact of other sensors.

[0069] In this embodiment, preferably, to determine whether the sensor data is valid set of labels, wherein the effective mark is 0, invalid mark is 1, the labeling method is:

[0070]

[0071] Where p ADF is the stationarity test significance p value, is the white noise test significance p value, L ACF , L PACF are the autocorrelation coefficient and partial correlation coefficient respectively.

[0072] Step 3 of the data of different sensors in different time period of feature extraction, after the extraction of five data validity quantization index, respectively, data stationarity test significance p value (p ADF ), white noise test significance p value autocorrelation coefficient (L​ACF ), partial correlation coefficient (L PACF ), correlation weighted average (C 指标 ). The extraction is stored in the database in the following table format.

[0073]

[0074] In this embodiment, step S4 is specifically: taking p ADF , L ACF , L PACF , C 指标 as the independent variable, and lable as the dependent variable, inputting the random forest model, performing classification model training, and obtaining the effectiveness discrimination random forest model. After training, it is stored in the cloud or locally for subsequent deployment.

[0075] In this embodiment, step S5 is specifically: reading the sensor effectiveness discrimination random forest model stored in the cloud or locally in step S4, and performing sensor data effectiveness discrimination, wherein the input data fields include sensor_name, p ADF , L ACF , L PACF , C 指标 , and the model will return the probability value of whether the sensor data is effective.

[0076] The above only describes the preferred embodiments of the present application, and any equivalent changes and modifications made within the scope of the present application should be included in the scope of the present application.

Claims

1. A method for determining the validity of RF structure sensor data based on multiple correlation coefficient features, characterized in that, Includes the following steps: Step S1: Obtain the dataset collected by all sensors on the same structure under test; Step S2: Perform a preliminary data validity assessment on the dataset obtained in Step S1 to obtain a valid dataset; Step S3: Extract features from the effective dataset to obtain a quantitative indicator of effectiveness; Step S4: Based on the effectiveness quantification index, train the random forest model to obtain the effectiveness discriminant random forest model; Step S5: Input the sensor data to be tested into the validity discrimination random forest model to obtain the discrimination result; Step S3 uses the ARMA model to extract features from the historical data of the sensor itself, constructs sensor correlation indicators, and sets data labels. The ARMA model includes stationarity test, white noise test, ACF test, and PACF test; The sensor correlation index uses the C index function as follows: Where X t For sensors that require effective data discrimination, σ(X) t+1 ,X t+2 ,…,X n ) is X t Other sensors that meet the physical connection requirements within the relevant field; The tags used to determine the validity of sensor data are 0, and 1, respectively. This tagging method is as follows: Where p ADF To determine the significance p-value for the stationarity test, For the white noise test, the p-value is L. ACF L PACF These are the autocorrelation coefficient and the partial correlation coefficient, respectively. Step S4 specifically involves: using p ADF , L ACF L PACF C 指标 Using label as the independent variable and label as the dependent variable, input the data into the random forest model to train the classification model and obtain the effective discriminant random forest model.

2. The method for determining the validity of RF structure sensor data based on multiple correlation coefficient features according to claim 1, characterized in that, Step S2 specifically involves: Step S21: Perform EDA data analysis on the dataset. Based on the basic characteristics of the data, sensor data that is missing more than a preset value, has a constant value, or does not meet the sensor accuracy and range requirements is judged as invalid data, i.e., it has no significance for analysis and research and does not meet the physical discrimination. Output the reason for its invalidity and the process ends; otherwise, proceed to step S22. Step S22: Based on the physical connections between the sensors, retain the sensor data that has a physical connection with the sensor under study.

3. The method for determining the validity of RF structure sensor data based on multiple correlation coefficient characteristics according to claim 1, characterized in that, The ARMA model includes stationarity tests, white noise tests, ACF tests, and PACF tests, specifically: The stationarity test determines whether a sequence has a unit root: if the sequence is stationary, there is no unit root; otherwise, there is a unit root. An ADF test is performed on the dataset, the t-statistic is calculated, and the significance p-value is used as a characteristic indicator. The white noise test uses Q. LB The statistic is used to perform a pure random test on the dataset, and the Q statistic is solved. LB The statistic is used, and the significance p-value is used as a characteristic index to solve Q. LB The formula for the statistic is as follows: In the formula, n is the number of observation periods in the sequence, and m is the specified lag period; The ACF test involves calculating the autocorrelation coefficient of the dataset to be tested and using this autocorrelation coefficient as a feature index input to the random forest model. The PACF test is used to calculate the partial correlation coefficient of the dataset to be tested, and the autocorrelation coefficient is used as a feature index input to the random forest model.

4. The method for determining the validity of RF structure sensor data based on multiple correlation coefficient characteristics according to claim 3, characterized in that, The formula for solving the t-statistic is as follows: where R = (ζ1,…,ζ P-1 ,ρ)′; H0:|ρ|≥1 vsH1:|ρ|<1, the test statistic is: The sample dataset is Where e i =[0,…,0,1]′=(e) p×1 ,

Images