A method for filling in missing data for distributed fiber optic sensors

Abstract

The application discloses a kind of filling method for the missing data of distributed optical fiber sensor, it includes: obtaining the data set consisting of distributed optical fiber sensor data, and the data in the data set is preprocessed to fill in the missing value therein, to obtain the data set after preprocessing;The preset autocorrelation neural network model is trained based on the data set after preprocessing, and the trained neural network model is used to dynamically fill in the missing data to be filled, and the data after filling is output;Wavelet denoising is carried out to the data output from the neural network model, to obtain the denoised data;Local error elimination is carried out to the denoised data, to obtain the final missing data filling result.The application has the characteristics of high accuracy and high processing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of missing data processing, and particularly relates to a method for filling in missing data of a distributed optical fiber sensor. BACKGROUND

[0002] The distributed optical fiber sensor is a sensor with promising application in the field of road detection, and can obtain the spatial distribution of road strain response due to its high spatial resolution. However, in the application process, a large amount of response data may be missing due to non-standard packaging, unreasonable prestress application and other reasons, and the data missing will have a great impact on subsequent road performance analysis.

[0003] At present, the commonly used data filling methods are mostly statistical, predictive or interpolation-based methods, which are not suitable for the case of a large amount of missing data, and therefore a method suitable for the case of a large amount of missing data and capable of adapting to the data characteristics of the distributed optical fiber sensor needs to be proposed. SUMMARY

[0004] The present application provides a method for filling in missing data of a distributed optical fiber sensor to solve the technical problem that the prior art is not suitable for the case of a large amount of missing data.

[0005] To solve the above technical problem, the present application provides the following technical scheme:

[0006] A method for filling in missing data of a distributed optical fiber sensor, comprising:

[0007] obtaining a data set composed of distributed optical fiber sensor data, and pre-processing the data in the data set to fill in the missing values therein, to obtain a pre-processed data set;

[0008] training a preset autocorrelation neural network model based on the pre-processed data set, using the trained neural network model to dynamically fill in the data to be filled, and outputting the filled data;

[0009] wavelet denoising the data output from the neural network model to obtain denoised data;

[0010] locally eliminating errors from the denoised data to obtain the final missing data filling result.

[0011] Further, the pre-processing of the data in the data set to fill in the missing values therein comprises:

[0012] filling in each missing value as the average value of the data in the column where the missing value is located.

[0013] Further, the preset autocorrelation neural network model is trained based on the preprocessed data set, the trained neural network model is used for dynamic filling of the to-be-filled data, and a filling result is output, including:

[0014] The training of the autocorrelation neural network model is represented as:

[0015]

[0016] y ij =ω (2) ·y h +b (2)

[0017] wherein, represents an input sample, y ij represents an output of the network model, ω (1) represents a connection weight of an input layer neuron and a hidden layer neuron, ω (2) represents a connection weight of a hidden layer neuron and an output layer neuron, b (1) represents a threshold value of a hidden layer neuron, b (2) represents a threshold value of an output layer neuron;

[0018] The cost function is defined as wherein, X c is a set of non-missing values of input data, x ij is a value of the i-th row and the j-th column of the input data, X m is a set of missing values of the input data, is a dynamic filling value, and a set is defined as The network model parameters and the dynamic filling variable W are updated by using a momentum-based stochastic gradient descent method to obtain W ′ , and a loop iteration is performed until a specified precision is met, and a filled data result Y no is output.

[0019] Further, the data output from the network model is wavelet denoised to obtain denoised data, including:

[0020] The data Y no output from the neural network model is subjected to N-level wavelet decomposition to obtain an approximate coefficient sequence a n of the N-level scale and a detail coefficient sequence b n of each scale, wherein n=1, 2, …, N, a n and b n are calculated according to the following formula:

[0021]

[0022]

[0023] wherein H represents a low-pass filter in wavelet decomposition, G represents a high-pass filter in wavelet decomposition, represents a convolution operation;

[0024] The soft threshold value is applied to each scale of the detail coefficient sequence, and when the detail coefficient is less than the threshold value, it is set to zero, and when the detail coefficient is greater than the threshold value, it is processed by using the denoising function to obtain the processed detail coefficient sequence b ′ n :

[0025]

[0026] wherein, The threshold value λ = σ 2 lg(M), M is the length of the data sequence, and σ is the standard deviation value of the data sequence, and the calculation method is

[0027] Using a n and the detail coefficient sequence b ′ n The data is reconstructed by wavelet, and finally the denoised data Y is obtained de .

[0028] Further, the value range of N is 3-8.

[0029] Further, the denoised data is subjected to local error elimination, including:

[0030] The data Y no output by the neural network model and the denoised data Y de are subtracted to obtain Y err ;

[0031] Y err is unitized to obtain The difference between each sample and other samples in the calculation is obtained to obtain the difference value sequence D ij , D ij represents the jth difference value between the ith sample in the calculation and other samples;

[0032] The maximum value D max and the minimum value D min in D ij are obtained, and the grey correlation coefficient of each difference value sequence is obtained by formula , and the grey correlation degree r is obtained from the grey correlation coefficient, and the entropy value of the target sample and the K nearest neighbor samples is obtained according to h = -log(K)·r·log(r), and the variation degree coefficient is calculated according to the relationship between the variation degree coefficient and the entropy value v = 1-h.

[0033] The weight of the neighboring sample is calculated by ω=1 / (K-1)·(1-v / sum(v)), and the error value at the missing value of the target sample is calculated according to the weight, and a selection threshold of the error value is set;

[0034] When the calculated error value is less than the selection threshold, no operation is taken, and when the error value is greater than the selection threshold, the newly calculated error value is used to correct the missing value.

[0035] The technical scheme provided by the present application has at least the following beneficial effects:

[0036] The present application provides a method for filling in missing data of a distributed optical fiber sensor, which fills in the missing data of the distributed optical fiber sensor based on a dynamic filling scheme of a de-tracking autoencoder combined with a wavelet denoising and local error elimination method, and has the characteristics of high accuracy and high processing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 is an execution flow diagram of the method for filling in missing data of a distributed optical fiber sensor provided by the embodiments of the present application;

[0039] Figure 2 is a filling result diagram of an autoencoder provided by the embodiments of the present application;

[0040] Figure 3 is a convergence curve diagram of an autoencoder filling algorithm provided by the embodiments of the present application;

[0041] Figure 4 is a filling effect diagram of an autoencoder+wavelet denoising provided by the embodiments of the present application;

[0042] Figure 5 is a filling effect diagram of an autoencoder+wavelet denoising+local error elimination provided by the embodiments of the present application;

[0043] Figure 6 is a filling effect diagram of different filling methods provided by the embodiments of the present application; (a) is a time series analysis filling result diagram, (b) is a linear regression filling result diagram, and (c) is an autoencoder+de-noisy data+local error elimination and original data comparison diagram. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0045] In order to solve the problem that the prior art is not applicable to the case of a large amount of missing data, the embodiment provides a filling method for missing data of a distributed optical fiber sensor based on machine learning and local error filling, so as to guarantee the integrity of the data of the distributed optical fiber sensor. The method can be implemented by an electronic device, which can be a terminal or a server. The execution flow of the method is as shown in Figure 1 The method comprises the following steps:

[0046] S1, acquiring a data set composed of data of a distributed optical fiber sensor, and pre-processing the data in the data set to fill the missing values in the data set, so as to obtain a pre-processed data set;

[0047] In the embodiment, the data in the data set acquired in S1 is two-dimensional data data in = x ij , wherein x ij represents the data value of the i th (i = 1, 2,..., n) row and the j th (i = 1, 2,..., m) column of data in . The missing values are found by scanning the data, and the missing value index idx miss and the non-missing value index idx unmiss are obtained; wherein idx miss = γ ij when idx unmiss = ∈ ij when The missing data is pre-filled to meet the needs of neural network training. Specifically, in the embodiment, the pre-filled missing data is filled with the average value of the column data, and the pre-filled data data pre is obtained.

[0048] S2, training a preset autocorrelation neural network model based on the pre-processed data set, using the trained neural network model to dynamically fill the data to be filled, and outputting the filled data;

[0049] Specifically, in the embodiment, the implementation process of S2 is as follows:

[0050] Let x i represent the input sample, and y i1 = [y i2 , y i3 , yi3 ,…,y in ] T Let the output sample be represented, then the trained model can be represented as:

[0051]

[0052] y ij =ω (2) ·y h +b (2)

[0053] in, Indicates the input sample, y ij ω represents the output of the network model. (1) ω represents the connection weights between input layer neurons and hidden layer neurons. (2) b represents the connection weights between hidden layer neurons and output layer neurons. (1) b represents the threshold of the hidden layer neurons. (2) This represents the threshold of the output layer neurons;

[0054] Define the cost function Among them, X c Let x be the set of non-missing values ​​of the input data. ij Let X be the value in the i-th row and j-th column of the input data. m The set of missing values ​​in the input data. Define a set for dynamically filled values. The network model parameters and dynamically filled variable W are updated using the momentum-based stochastic gradient descent method to obtain W. ′ The process is iterated and looped until the specified precision is met, at which point the filled-in data result Y is output. no .

[0055] S3, perform wavelet denoising on the data output from the neural network model to obtain the denoised data;

[0056] Specifically, in this embodiment, the implementation process of S3 is as follows:

[0057] For the data Y output from the neural network model no Performing N-level wavelet decomposition yields the approximate coefficient sequence a at the Nth scale. n and the sequence of detail coefficients b at each scale n Where n = 1, 2, ..., N, choosing a decomposition scale N that is too large will lead to severe loss of data information, while choosing a value that is too small will affect the denoising performance. Typically, N is taken as around 3-8, and here it is chosen, but not limited to, 5; a n With b n The calculation formula is as follows:

[0058]

[0059]

[0060] Here, H and G are the low-pass and high-pass filters in wavelet decomposition, respectively, and their values ​​depend on the chosen wavelet transform basis functions. Commonly used wavelet basis functions include dbN, symN, and harr wavelet functions. This embodiment selects, but is not limited to, db2. This represents the convolution operation.

[0061] A soft threshold is applied to the detail coefficient sequence at each scale. When the detail coefficient is less than the threshold, it is set to zero; when the detail coefficient is greater than the threshold, it is processed using a denoising function to obtain the processed detail coefficient sequence b. ′ n :

[0062]

[0063] in, Threshold λ = σ 2 lg(M), where M is the length of the data sequence and σ is the standard deviation of the data sequence, calculated as follows:

[0064] Using a n and detail coefficient sequence b ′ n Wavelet reconstruction is performed on the data to obtain the denoised data Y. de .

[0065] S4 performs local error elimination on the denoised data to obtain the final missing data filling result.

[0066] It should be noted that analysis of the denoised data revealed significant local errors in the data sequence. Therefore, it is necessary to eliminate these local errors. The specific implementation process is as follows:

[0067] The data Y output by the neural network model no And the denoised data Y de The difference is obtained by taking Y. err Local errors are eliminated based on the similarity of error sequences among different samples.

[0068] For Y err Unitize to obtain calculate The difference between each sample and other samples is used to obtain the difference sequence D. ij D ij express The differences between the i-th sample and the j-th sample are used to obtain D. ij The maximum value D inmax and minimum value D min , and then by the formula The grey correlation coefficients of each difference sequence are calculated, and the grey correlation degree r is obtained from the grey correlation coefficients. The entropy value of the target sample and the K nearest neighbor samples is obtained according to h = -log(K) r log(r). The variation degree coefficient is calculated according to the relationship between the variation degree coefficient and the entropy value v = 1-h. The weight of the nearest neighbor sample is calculated by ω = 1 / (K-1) (1-v / sum(v)). The error value at the missing value of the target sample is calculated according to the weight. The selection threshold of the error value is set. When the calculated error value is less than the selection threshold, no operation is taken, and when the error value is greater than the selection threshold, the newly calculated error value is used to correct the missing value. The final filling result is obtained.

[0069] The effect of the proposed data filling method is verified according to the simulation results. Test data sets with five different missing rates of 10%, 20%, 30%, 40% and 50% are constructed using the completed data set, and the filling effects of the neural network missing value filling method, the neural network + wavelet denoising missing value filling method and the neural network + wavelet denoising + local error elimination missing value filling method are compared longitudinally. Then, the filling effects of the neural network + wavelet denoising + local error elimination missing value filling method, the time series prediction data filling method and the linear regression filling method are compared horizontally.

[0070] The simulation conditions and contents are as follows:

[0071] Hardware condition: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz

[0072] Operating system: Windows 10

[0073] Running environment: Matlab2018a, Python2.7, TensorFlow

[0074] Data: The longitudinal strain data of the test road is collected by using 1m length of ODiSI-B distributed optical fiber sensor. 7 columns of attributes in the complete data part are selected for data filling method verification, which contains 12950 data. First, the selected data needs to be constructed with missing values. Because the analysis of the missing data in the collected data shows that the data missing mode belongs to random missing, it is necessary to construct random missing data for the selected complete data set. A 0 matrix Z is generated by using Matlab M×N , where M and N are 1850 and 7 for this example, and the random arrangement matrix Z is generated with values from 1 to MN 1×MN In order to meet the requirement of constructing different missing rates p, the matrix Z 1×MNThe first MN*p index values, and matrix Z. M×N The value at the corresponding index is set to 1, and the matrix Z is finally obtained. ′ M×N Used to construct a missing matrix for missing data.

[0075] The missing value imputation effects of the autocorrelation neural network missing value imputation method, the neural network + wavelet denoising missing value imputation method, and the neural network + wavelet denoising + local error elimination missing value imputation method are shown below. Figures 2 to 5 As shown. From Figure 3 It can be seen that the autoencoder filling method converges after 1000 iterations, and from Figure 2 It can be seen that this method can predict and impute missing data with relatively high accuracy, but there is a lot of detached data. To address this issue, wavelet denoising can be used to process the data imputed by the autoencoder method. The processed result is as follows: Figure 4 As shown. By Figure 4 It can be seen that the amount of free data has been reduced significantly, but there are still many data points in local data locations that differ greatly from the original data. This problem can be solved by using the designed local error elimination method.

[0076] from Figure 5 It can be seen that the missing data processed by the autoencoder + wavelet denoising + local error elimination method contains less free data than the above methods, and the improvement effect is more significant.

[0077] To study the imputation performance of the autoencoder + wavelet denoising + local error elimination imputation method, datasets with different missing values ​​were constructed. The imputation effects of the neural network + wavelet denoising + local error elimination imputation method and the time series prediction data imputation method under different missing values ​​are shown in Table 1.

[0078] Table 1. MAPE values ​​of different imputation methods at different missing rates.

[0079]

[0080] As shown in Table 1, among the three missing data imputation methods—autoencoder method, autoencoder + wavelet denoising method, and autoencoder + wavelet denoising + local error elimination method—the autoencoder + wavelet denoising + local error elimination method has the best imputation effect under different missing data rates. Moreover, the improvement in imputation effect becomes more and more obvious as the missing data rate increases.

[0081] A horizontal comparison was made of commonly used missing data imputation methods, including linear regression imputation, time series analysis imputation, and an autoencoder + wavelet denoising + local error elimination method. The missing data rate was set at 50%, and the MAPE values ​​for imputation are shown in Table 2. The imputation results are as follows:Figure 6 As shown.

[0082] Table 2 MAPE values of different filling methods

[0083]

[0084]

[0085] From Figure 6 It can be seen from the time series analysis filling method that there is a period difference between the filled data and the original complete data, which will cause a large filling error. The filling value of each data point of the linear regression filling method has a large difference with the original data, and the self-encoder + wavelet denoising + local error elimination filling method has a good filling effect. From the MAPE values of different methods in Table 2, it can be concluded that the self-encoder + wavelet denoising + local error elimination filling method is the best filling method compared with the other two methods.

[0086] In addition, it should be noted that the present application can be provided as a method, apparatus or computer program product. Therefore, the embodiments of the present application can adopt a completely hardware embodiment, a completely software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer usable storage media containing computer usable program code.

[0087] The embodiments of the present application are described with reference to flowcharts and / or block diagrams according to the method, terminal device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1 The apparatus for realizing the functions specified in one or more flows and / or blocks.

[0088] These computer program instructions can also be stored in a computer readable storage medium which can guide the computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction apparatus, which realizes the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one or more flows and / or blocks Figure 1The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0089] It is also noted that the aforementioned embodiments illustrate rather than limit the application, since various changes and modifications within the spirit and scope of the

[0090] Finally, it is to be noted that the above-mentioned description is of preferred embodiments of the application and a skilled person will appreciate that, although preferred embodiments of the application have been described, numerous modifications and enhancements can be made thereto without departing from the principles of the application, which are to be read in the light of the above description. Accordingly, the appended claims are intended to embrace all such alterations, permutations and modifications as fall within the scope of the embodiments of the application.

Claims

1. A method for filling in missing data in a distributed optical fiber sensor, characterized in that, include: A dataset consisting of distributed fiber optic sensor data is obtained, and the data in the dataset is preprocessed to fill in the missing values ​​to obtain a preprocessed dataset. The pre-processed dataset is used to train a pre-defined autocorrelation neural network model. The trained neural network model is then used to dynamically fill in the data to be filled, and the filled data is output. Wavelet denoising is performed on the data output from the neural network model to obtain the denoised data. Local error elimination is performed on the denoised data to obtain the final missing data imputation result; Local error elimination is performed on the denoised data, including: The data Y output by the neural network model no And the denoised data Y de The difference is obtained by taking Y. err ; For Y err Unitize to obtain calculate The difference between each sample and other samples is used to obtain the difference sequence D. ij D ij express The j-th difference between the i-th sample and other samples; D ij The maximum value D in max and minimum value D mim Then through formula Find the grey relational coefficient of each difference series, and obtain the grey relational degree r from the grey relational coefficient. Then, obtain the entropy value between the target sample and the K nearest neighbor samples according to h=-log(K)·r·log(r). Calculate the degree of variation coefficient according to the relationship between the degree of variation coefficient and the entropy value v=1-h. The weights of the nearest neighbor samples are calculated using ω = 1 / (K-1)·(1-v / sum(v)). The error value at the missing value of the target sample is calculated based on the weight, and a threshold for selecting the error value is set. If the calculated error value is less than the selected threshold, no action is taken. If the error value is greater than the selected threshold, the newly calculated error value is used to correct the missing value.

2. The method for filling missing data in a distributed optical fiber sensor as described in claim 1, characterized in that, The data in the dataset is preprocessed to impute missing values, including: Fill in each missing value with the average of the data in its column.

3. The method for filling missing data in a distributed optical fiber sensor as described in claim 1, characterized in that, A pre-defined autocorrelation neural network model is trained based on the preprocessed dataset. The trained neural network model is then used to dynamically fill in the data to be filled, outputting the filled data, including: The trained autocorrelation neural network model is represented as: y ij =ω (2) ·y h +b (2) in, Indicates the input sample, y ij ω represents the output of the network model. (1) ω represents the connection weights between input layer neurons and hidden layer neurons. (2) b represents the connection weights between hidden layer neurons and output layer neurons. (1) b represents the threshold of the hidden layer neurons. (2) This represents the threshold of the output layer neurons; Define the cost function Among them, X c Let x be the set of non-missing values ​​of the input data. ij Let X be the value in the i-th row and j-th column of the input data. m The set of missing values ​​in the input data. Define a set for dynamically filled values. The network model parameters and dynamically imputed variable W are updated using the momentum-based stochastic gradient descent method to obtain W′, and the process is iterated until the specified accuracy is met, at which point the imputed data result Y is output. no .

4. The method for filling missing data in a distributed optical fiber sensor as described in claim 1, characterized in that, Wavelet denoising is performed on the data output from the neural network model to obtain the denoised data, including: For the data Y output from the neural network model no Performing N-level wavelet decomposition yields the approximate coefficient sequence a at the Nth scale. n and the sequence of detail coefficients b at each scale n Where n = 1, 2, ..., N, a n With b n The calculation formula is as follows: Where H represents the low-pass filter in wavelet decomposition, and G represents the high-pass filter in wavelet decomposition. This represents the convolution operation; A soft threshold is applied to the detail coefficient sequence at each scale. When the detail coefficient is less than the threshold, it is set to zero; when the detail coefficient is greater than the threshold, it is processed using a denoising function to obtain the processed detail coefficient sequence b′. n : in, Threshold λ = σ 2 lg(M), where M is the length of the data sequence and σ is the standard deviation of the data sequence, calculated as follows: Using M n and detail coefficient sequence b′ n Wavelet reconstruction is performed on the data to obtain the denoised data Y. de .

5. The method for filling missing data in a distributed optical fiber sensor as described in claim 4, characterized in that, The value of N ranges from 3 to 8.

Images