Intelligent denoising and missing value filling method for distributed optical fiber strain data

Abstract

The invention provides an intelligent denoising and missing value filling method for distributed optical fiber strain data, and the method comprises the steps: coupling a rock and a distributed optical fiber under a triaxial test condition, and collecting optical fiber strain data through a distributed optical fiber monitoring system; representing and preprocessing the optical fiber strain data according to a time sequence to obtain noiseless optical fiber strain data; constructing a noisy point identification and data filling model, and training the noisy point identification and data filling model through the noiseless optical fiber strain data; and according to the trained noisy point identification and data filling model, carrying out de-noising processing and missing value filling on all the optical fiber strain data containing the spatial position information, and updating the parameters of the noisy point identification and data filling model in real time according to the change of a strain field in the monitored space along with time. The method provides high-precision data processing support for optical fiber monitoring, and has important engineering application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of distributed optical fiber monitoring, and in particular relates to an intelligent denoising and missing value filling method for distributed optical fiber strain data. Background Art

[0002] Large-scale monitoring of rocks is crucial for evaluating and predicting the stability of rock structures. The creep deformation failure of brittle rocks is considered to be the result of the slow accumulation of microcracks under static conditions. The failure process is sudden and difficult to predict based on traditional rock deformation monitoring methods. Because traditional rock deformation monitoring mostly conducts single-point monitoring on key parts of engineering structures, but rock failure has localized characteristics, local ruptures may expand rapidly and cause structural failures over a large area. Therefore, long-term distributed high-precision monitoring of rock surfaces under extreme conditions such as high pressure is of great significance for capturing local ruptures and predicting rock failure.

[0003] Distributed fiber technology, as an emerging monitoring method, can realize long-term distributed monitoring of rock surfaces. Among them, the monitoring method based on optical frequency domain reflectometry (OFDR) has the characteristics of high spatial resolution, high temporal resolution and high precision. Combined with the unsheathed distributed optical fiber with high strain transfer efficiency, it can capture the tiny deformation caused by the aggregation of microcracks. At present, distributed fiber strain monitoring technology has been used to obtain unconfined rock surface deformation, but protecting the optical fiber and obtaining rock surface deformation data in extreme environments such as triaxial high pressure are equally important for understanding the shear failure mechanism of rock. However, reasons such as fiber bending or large rock deformation will cause local light loss in the optical path, making the monitoring data mixed with complex noise, which interferes with the post-processing analysis of the data. Traditional data denoising methods mostly assume that the data is normally distributed or identify noise points based on statistical laws. These methods are widely applicable but the accuracy still needs to be improved. In recent years, denoising algorithms based on machine learning have been widely used, such as isolated forest trees. This tree-based algorithm requires setting the proportion of noise points, which often causes misjudgment or omission of noise points and takes a long time to calculate. At the same time, common missing value filling methods such as nearest neighbor interpolation and time series prediction methods based on machine learning are difficult to simultaneously consider the temporal continuity and spatial distribution of rock deformation.

[0004] The present invention provides a method for protecting a 250-micron diameter distributed optical fiber for long-term monitoring under triaxial high-pressure conditions, and combines a convolutional neural network and an autoencoder to consider the temporal continuity and spatial distribution of rock deformation to achieve intelligent identification of noise points and filling of missing values ​​in distributed optical fiber data under static monitoring conditions. Summary of the invention

[0005] The purpose of the present invention is to provide a method for protecting a 250-micron diameter distributed optical fiber and collecting strain data under triaxial high-pressure conditions, and to propose a noise point intelligent detection and missing data supplement method for the noise points in the strain monitoring data, so as to solve the problem of large amounts of noise and missing data in the optical fiber during high-pressure, long-term, static monitoring of rock deformation.

[0006] The technical solution of the present invention is as follows: A method for intelligent denoising and missing value filling of distributed optical fiber strain data, characterized in that the method comprises: Under triaxial test conditions, the rock and distributed optical fiber are coupled, and the optical fiber strain data is collected through the distributed optical fiber monitoring system; Representing the optical fiber strain data in time series and preprocessing the optical fiber strain data to obtain noise-free optical fiber strain data; Constructing a noise point recognition and data filling model, and training the noise point recognition and data filling model by using noise-free optical fiber strain data; All optical fiber strain data containing spatial position information are subjected to denoising and missing value filling according to the trained noise point identification and data filling model, and the noise point identification and data filling model parameters are updated in real time according to the change of the strain field in the monitored space over time.

[0007] Furthermore, the time-series representation and preprocessing of the optical fiber strain data includes: The optical fiber strain data obtained by the distributed optical fiber is represented as a time series; Standardize the optical fiber strain data containing position information obtained at each time point; The standardized data is transformed into two dimensions, and each set of standardized data is transformed from one dimension to two dimensions, and the strain at each spatial position occupies one dimension separately.

[0008] Furthermore, the noise recognition and data filling model is composed of an encoder and a decoder, the encoder includes multiple fully connected layers, convolutional layers and pooling layers, and the decoder includes multiple deconvolutional layers, upsampling layers and fully connected layers.

[0009] Furthermore, the noise point recognition and data filling model is trained by using noise-free optical fiber strain data as follows: Two or more groups of noise-free optical fiber strain data are selected and input into the encoder, and the fully connected layer multi-dimensionally expresses the single data and outputs multi-dimensional data; the multi-dimensional data is then extracted with a convolutional layer, and then the feature size is reduced with a pooling layer, and finally the high-dimensional feature data extracted from the encoder is input into the decoder; In the decoder, the deconvolution layer reduces the spatial dimension of the data by performing a reverse convolution operation on the high-dimensional feature data, and then further restores the details of the data through the upsampling layer, and finally reconstructs the data through the fully connected layer; The original data is compared with the reconstructed data, the loss function is calculated, the parameters of the model are adjusted according to the value of the loss function, and multiple trainings are performed at a certain learning rate to finally obtain a trained noise recognition and data filling model.

[0010] Furthermore, the noise removal and missing value filling of all optical fiber strain data containing spatial position information according to the trained noise point recognition and data filling model is specifically as follows: Predict the strain data at different spatial locations at the first time point through the trained noise point recognition and data filling model; calculate the mean square error between the predicted value and the actual measured value; When the error is greater than the set threshold, the strain data at this position is determined as a noise point, and the spatial position information of the noise point is recorded; Fill missing values ​​for identified noise points.

[0011] Furthermore, the formula for filling missing values ​​for the identified noise points is: , in, is the missing strain value at the current i-th position, is the average value of the spatial strain field at time point t where the missing value is located, is the mean and standard deviation of the spatial strain field at time point t where the missing value is located, is the standardized strain value at position i at the current time point t predicted by the model, i represents the position of the i-th noise point, and t represents the time point predicted by the current model.

[0012] Furthermore, the real-time updating of the noise point identification and data filling model parameters according to the change of the strain field in the monitored space over time is specifically: After completing the noise point identification and missing value filling of the optical fiber strain data at the t-1th time point, the data at the t-2th time point is used as the model input, and the reconstruction error between the model predicted data and the optical fiber strain data at the t-1th time point is calculated, so that the model can learn the spatial strain distribution at the t-1th time point and update the model parameters.

[0013] Furthermore, the method also includes removing Gaussian noise in the optical fiber strain data by weighted averaging after completing the missing value filling, specifically: First, calculate the one-dimensional convolution kernel according to the Gaussian function; Using the one-dimensional convolution kernel to perform a convolution operation on the data at each time point by means of equal-length convolution; After the convolution process is completed, the original data is replaced by the Gaussian filtered data to remove Gaussian noise in the optical fiber strain data.

[0014] Compared with the prior art, the present invention has the following advantages: The present invention proposes a method for long-term full-field deformation monitoring of rock surface by protecting unsheathed distributed optical fiber under triaxial high pressure conditions, thereby achieving high-precision strain data acquisition in a complex stress environment.

[0015] The present invention uses a deep learning method with an encoder-decoder structure to automatically learn data features, which can accelerate the convergence speed during training and reduce the consumption of computing resources. At the same time, the incremental training mechanism allows the model to update parameters in real time according to the time changes of the strain field, ensuring the effectiveness and stability under long-term monitoring. In data processing, the data distribution and time continuity are comprehensively considered, and noise points are effectively identified and processed, and missing values ​​are reasonably filled to ensure that the data is complete and accurate. For Gaussian noise, it can be effectively removed after the missing values ​​are filled by calculating the one-dimensional convolution kernel and the equal-length convolution operation to prevent misjudgment, providing high-precision data processing support for related monitoring, and has important engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings generally illustrate various embodiments by way of example and not limitation, and together with the description and claims, serve to illustrate the embodiments of the invention. Where appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present apparatus or method.

[0017] Figure 1 A diagram of the equipment and monitoring process in the method of the present invention; Figure 2 is a flow chart of the method of the present invention; Figure 3 Graphs of original data and denoised data using the method of the present invention. DETAILED DESCRIPTION

[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] refer to Figure 2 As shown, the present invention provides a method for protecting a distributed optical fiber with a diameter of 250 microns under harsh conditions to monitor rock deformation and perform noise removal and missing value filling on strain data, and the steps are as follows: Step 1: Under triaxial test conditions, the distributed optical fiber is effectively coupled to the rock, and the distributed optical fiber monitoring system is used to A G652b single-mode, sheathless distributed optical fiber with a diameter of 250 microns was selected. First, the optical fiber laying path was determined. The optical fiber track was drawn on the surface of the test sample with a waterproof marker. The optical fiber was laid on the rock surface according to the predetermined track to ensure that there was no excessive bend during the laying process. A certain prestress was maintained during the laying process to ensure that the distributed optical fiber was in full contact with the surface of the rock to be tested. A quick-drying glue with a high elastic modulus was used as a coupling agent to paste the distributed optical fiber on the rock surface. During the pasting process, it was ensured that the optical fiber was not damaged. The rock surface with the optical fiber is covered with a sealant with a thickness of more than 5 mm to prevent the hydraulic oil in the triaxial chamber from entering the rock during the test. At the same time, the sealant can protect the optical fiber from external environmental interference. The sealant is required to be waterproof, high temperature resistant, corrosion resistant, and able to withstand large deformation; Place the test sample with wound optical fiber into the triaxial chamber, and lead one end of the optical fiber out of the triaxial test machine as the inlet end to input the optical signal, and the other end of the optical fiber as the output end of the optical signal. Tie a knot at the output end of the optical fiber to reduce the influence of optical loss at the fiber break on the test results; Splice the optical fiber attached to the rock with the optical fiber connected to the demodulator to ensure that the optical loss is less than 0.01dB, and use a laser pen or demodulator to detect the optical fiber to ensure that the optical fiber can transmit signals normally; Step 2: Use rock distributed optical fiber monitoring system for data collection; The rock distributed optical fiber monitoring system comprises a strain sensing part and a data processing and control system, wherein the strain sensing part is a distributed optical fiber adhered to the rock to be measured, and the data processing and control system comprises a distributed optical fiber signal demodulator and a computer host; Before starting the test, check the optical loss in the optical fiber to ensure that the optical loss does not increase significantly with the extension of the measurement distance, and press the optical fiber to calibrate the position to determine the starting and ending points of data collection; Corresponding data parameters are set in the data processing and control system, including the strain transfer coefficient and temperature transfer coefficient of the selected optical fiber, the spatial resolution and time resolution of data acquisition. The equipment selected in the present invention is based on Optical Frequency Domain Reflectometry (OFDR), and the data sampling accuracy is 1με; Furthermore, a triaxial compression test of rock was carried out on a triaxial testing machine. The test loading and data acquisition were carried out simultaneously. The oil pressure in the triaxial chamber was first increased to 60MPa to ensure that the optical fiber could continue to work normally under high pressure. The triaxial compression creep test was started until the rock was destroyed.

[0020] Step 3: Represent and preprocess the data collected by distributed optical fiber in time series The strain distribution obtained by the distributed optical fiber is represented as a time series and preprocessed, including filtering noise-free data, standardizing input data, and dimensional conversion of the standardized data; Two or more groups of noise-free data are manually selected from the early monitoring data for training the model to learn the relationship between the initial strain and the spatial position. The requirements for selecting noise-free data are: there are no data points exceeding the test limit of the equipment, the data changes continuously without sudden changes, and the data collection time point is as close as possible to the data that needs to be denoised; The first step of the time series data preprocessing is to standardize the strain data containing position information obtained at each time point, remove the size characteristics of the data and only retain the data change trend, change the average value of the data sample at each time point to 0, and change the standard deviation to 1. The calculation formula is: , In the formula, is the strain at the ith position in the data at a time point, is the average strain of the data collected at that time point, is the standard deviation of the data, for Normalized data.

[0021] The standardized data is transformed into two dimensions: each set of standardized data is transformed from one dimension to two dimensions, and the strain at each spatial position occupies one dimension separately. The transformation process is as follows: , The shape of the converted data set changes from (m, n) to (m, n, 1), where the first dimension m represents different time points and the second dimension n represents the number of monitoring points. This is done to remove the spatial order features in the data and enable the model to simultaneously learn the strain features of all locations in order to reduce model parameters and training costs.

[0022] Step 4: Construct the encoder and decoder and train the parameters; The noise-free data after the standardization and dimension conversion in step 3 is input into the encoder and decoder for data reconstruction. The encoder includes multiple fully connected layers, convolutional layers and pooling layers, and the decoder includes multiple deconvolutional layers, upsampling layers and fully connected layers. 4.1 Input training data into encoder: The single strain data is expressed in multiple forms, and the data at time point t-1 is input into two fully connected layers to represent the strain at each spatial position with a set of data. The Relu activation function is used between the two fully connected layers to process the data. The process is as follows: , is the weight matrix, is the bias coefficient, k is the number of output nodes in the first layer, v is the number of output nodes in the second layer, is the input data and output data of the first layer, is the data after the fully connected layer; The multi-dimensional data output by the fully connected layer is input into the convolutional layer to extract features: First, the data at each time point is converted into a tensor of size (n, 1, v), where n is the number of monitoring points and v is the number of nodes in the last fully connected layer; The feature extraction is performed through the convolution kernel with an input channel of 1, an output channel of d, a size of 3, a sliding step of 1, and a padding of 1. The data is then processed through the Relu activation function. The dropout layer is used in the middle to prevent parameter overfitting. The pooling layer is then used to reduce the size of the feature map. The above convolution and pooling process is repeated to further increase the feature dimension and reduce the feature size. 4.2 Input the extracted data features into the decoder: The high-dimensional feature data extracted by the convolution layer is input into the decoder. The structure of the decoder is opposite to that of the encoder. The deconvolution layer corresponds to the convolution layer in the encoder, the upsampling layer corresponds to the pooling layer in the encoder, the number of input and output channels in the deconvolution layer corresponds to the number of output and input channels in the convolution layer, and other hyperparameters are the same as those in the encoder. The data dimension is continuously restored through the deconvolution and upsampling process. After the last layer of upsampling is completed, the data is restored from the shape of (n, 1, v) to (n, v), and the data is input into the fully connected layer for data reconstruction. The Relu activation function is used in the middle for processing. The shape of each group of reconstructed data is (n, 1), which is the same as the size of the original data, representing the strain value of each spatial position; 4.3 Calculating reconstruction error as loss function The mean square error between the original strain data at time t after standardization and the predicted data output by the model is calculated to evaluate the prediction effect of the model. The calculation formula is: , in, is the original data, To reconstruct the data, n is the number of monitoring points in a set of data; 4.4 Iterative training model According to the preset number of training times, the noise-free data selected in step 3 is input into the model in chronological order for training. Steps 4.1-4.3 are repeated in each iteration, and the learning rate is set to 0.001. The appropriate number of iterations is selected until the loss function is small enough and constant. After the iteration, the trained initial model is obtained.

[0023] Step 5: According to the initial model, noise points are identified, missing values ​​are filled, and Gaussian denoising is performed on all time-series strain data containing spatial position information, and the model parameters are updated in real time according to the changes of the strain field in the monitored space over time.

[0024] 5.1 Noise Identification and Missing Value Filling Use the initial model trained in step 4.4 to predict the strains at different spatial positions at the first time point, calculate the error according to the formula in step 4.3, regard the strain values ​​with errors greater than 1 as noise points, and record the spatial positions of the noise points; Furthermore, missing values ​​are filled for the noise strains according to the model prediction results and the strain data distribution at the current time point. The specific formula is as follows: , in, is the missing strain value at the current i-th position, is the average value of the spatial strain field at time point t where the missing value is located, is the mean and standard deviation of the spatial strain field at time point t where the missing value is located, is the normalized strain value at position i at the current time point t predicted by the model, i represents the position of the i-th noise point, and t represents the time point predicted by the current model; 5.2 Incremental training to dynamically update model parameters and Gaussian filtering During static monitoring, the spatial strain field changes slowly over time, so the model needs to learn the changing state of the strain field in real time and update the model parameters; Input the last time point data of the training data and repeat steps 4.1-4.4 to perform incremental training to update the model parameters. When calculating the loss function, the mean square error is calculated between the reconstructed data and the denoised strain data at the first time point. The updated model is used for noise point identification of the strain data at the second time point. Repeat step 5.1 to complete the noise point identification and missing value filling at the second time point. After completing the noise point identification and missing value filling of the strain data at the t-1th time point, take the data at the t-2th time point as input and calculate the reconstruction error between the model output data and the strain data at the t-1th time point, let the model learn the spatial strain distribution at the t-1th time point and update the model parameters to repeat steps 4.1-4.4 to perform denoising at the tth time point; Due to the limitation of instrument accuracy and external disturbances, slightly fluctuating Gaussian noise will appear during the monitoring process. This noise is not much different from the original data and may not be identified as noise points. However, in the process of parameter update and data filling, this noise may be learned as a feature and continuously strengthened, which may eventually mislead the model to identify normal data as noise, resulting in misjudgment and incorrect filling. Therefore, after each missing value filling, it is necessary to remove the Gaussian noise in the data by weighted averaging. First, the one-dimensional convolution kernel is calculated according to the Gaussian function. The formula is as follows: , in The range is an integer between [-n, n], where n is the size of the convolution kernel. is the corresponding weight value in the convolution kernel; The above convolution kernel is further convolved with the data at each time point. The size of the convolution kernel should not be too large, and the general value is 3. The one-dimensional convolution kernel obtained above is used to perform one-dimensional convolution on the denoised data. The marginal effect of the data is not considered in the calculation process, and the convolution method selects equal-length convolution, that is, the number of input data and output data of the Gaussian filter is kept consistent. The influence of marginal effects on the data within the effective monitoring range can be avoided by lengthening the number of input data. Finally, the original data is replaced by the data that has completed the Gaussian filter.

[0025] The present invention is based on distributed optical fiber monitoring technology and invents a method for protecting optical fiber under complex stress conditions such as triaxial high pressure, and for identifying noise points and filling missing values ​​in strain field data under static monitoring.

[0026] Figure 1 This is a diagram of the equipment and monitoring process during the test. The test environment is a laboratory environment with a high confining pressure of 60MPa, the test time is as long as 9 hours, the spatial resolution is 0.64-1mm, and the accuracy can reach 1με. Figure 3 This is a comparison chart before and after data processing. It can be found that the method proposed in the present invention can accurately remove noise and complete missing value filling, and the model can adapt to the changes in the spatial distribution of strain data over time during the monitoring process.

[0027] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An intelligent denoising and missing value filling method for distributed optical fiber strain data, characterized in that: The method comprises: Under triaxial test conditions, the rock and distributed optical fiber are coupled, and the optical fiber strain data is collected through the distributed optical fiber monitoring system; Representing the optical fiber strain data in time series and preprocessing the optical fiber strain data to obtain noise-free optical fiber strain data; Constructing a noise point recognition and data filling model, and training the noise point recognition and data filling model by using noise-free optical fiber strain data; All optical fiber strain data containing spatial position information are subjected to denoising and missing value filling according to the trained noise point identification and data filling model, and the noise point identification and data filling model parameters are updated in real time according to the change of the strain field in the monitored space over time.

2. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: Representing and preprocessing the optical fiber strain data in time series includes: The optical fiber strain data obtained by the distributed optical fiber is represented as a time series; Standardize the optical fiber strain data containing position information obtained at each time point; The standardized data is transformed into two dimensions, and each set of standardized data is transformed from one dimension to two dimensions, and the strain at each spatial position occupies one dimension separately.

3. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: The noise recognition and data filling model is composed of an encoder and a decoder, the encoder includes multiple fully connected layers, convolutional layers and pooling layers, and the decoder includes multiple deconvolutional layers, upsampling layers and fully connected layers.

4. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: The method of training the noise point recognition and data filling model by using noise-free optical fiber strain data is as follows: Two or more groups of noise-free optical fiber strain data are selected and input into the encoder, and the fully connected layer multi-dimensionally expresses the single data and outputs multi-dimensional data; the multi-dimensional data is then extracted with a convolutional layer, and then the feature size is reduced with a pooling layer, and finally the high-dimensional feature data extracted from the encoder is input into the decoder; In the decoder, the deconvolution layer reduces the spatial dimension of the data by performing a reverse convolution operation on the high-dimensional feature data, and then further restores the details of the data through the upsampling layer, and finally reconstructs the data through the fully connected layer; The original data is compared with the reconstructed data, the loss function is calculated, the parameters of the model are adjusted according to the value of the loss function, and multiple training is performed according to the learning rate to finally obtain a trained noise recognition and data filling model.

5. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: The method of performing denoising and missing value filling on all optical fiber strain data containing spatial position information according to the trained noise point recognition and data filling model is specifically as follows: Predict the strain data at different spatial locations at the first time point through the trained noise point recognition and data filling model; calculate the mean square error between the predicted value and the actual measured value; When the error is greater than the set threshold, the strain data at this position is determined as a noise point, and the spatial position information of the noise point is recorded; Fill missing values ​​for identified noise points.

6. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: The formula for filling missing values ​​for the identified noise points is: ， in, is the missing strain value at the current i-th position, is the average value of the spatial strain field at time point t where the missing value is located, is the mean and standard deviation of the spatial strain field at time point t where the missing value is located, is the normalized strain value at position i at the current time point t predicted by the model, i represents the position of the i-th noise point, and t represents the time point predicted by the current model.

7. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: The real-time updating of the noise point identification and data filling model parameters according to the change of the strain field in the monitored space over time is specifically: After completing the noise point identification and missing value filling of the optical fiber strain data at the t-1th time point, the data at the t-2th time point is used as the model input, and the reconstruction error between the model predicted data and the optical fiber strain data at the t-1th time point is calculated, so that the model can learn the spatial strain distribution at the t-1th time point and update the model parameters.

8. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: The method also includes removing Gaussian noise in the optical fiber strain data by weighted averaging after completing the missing value filling, specifically: First, calculate the one-dimensional convolution kernel according to the Gaussian function; Using the one-dimensional convolution kernel to perform a convolution operation on the data at each time point by means of equal-length convolution; After the convolution process is completed, the original data is replaced by the Gaussian filtered data to remove Gaussian noise in the optical fiber strain data.

Images