An intelligent denoising and missing value filling method for distributed optical fiber strain data

Abstract

This invention provides an intelligent denoising and missing value filling method for distributed optical fiber strain data. The method comprises: coupling rock with a distributed optical fiber under triaxial test conditions, and collecting optical fiber strain data via a distributed optical fiber monitoring system; representing the optical fiber strain data in a time series and preprocessing it to obtain noise-free optical fiber strain data; constructing a noise point identification and data filling model, and training the model using the noise-free optical fiber strain data; performing denoising and missing value filling on all optical fiber strain data containing spatial position information based on the trained noise point identification and data filling model, and updating the noise point identification and data filling model parameters in real time based on temporal changes in the strain field in the monitored space. This invention provides high-precision data processing support for optical fiber monitoring and has important engineering application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of distributed optical fiber monitoring, and in particular relates to an intelligent denoising and missing value filling method for distributed optical fiber strain data. Background Art

[0002] Large-scale monitoring of rock is crucial for evaluating and predicting the stability of rock structures. Creep deformation and failure in brittle rock is believed to be the result of the slow accumulation of microcracks under static conditions. This sudden failure process is difficult to predict using traditional rock deformation monitoring methods. Traditional rock deformation monitoring typically focuses on single-point monitoring of critical parts of engineering structures. However, rock failure is inherently localized, and localized fractures can rapidly expand and lead to widespread structural failure. Therefore, long-term, distributed, high-precision monitoring of the rock surface under extreme conditions, such as high pressure, is crucial for capturing localized fractures and predicting rock failure.

[0003] Distributed fiber optic technology, as an emerging monitoring method, enables long-term, distributed monitoring of rock surfaces. Optical frequency domain reflectometry (OFDR)-based monitoring offers high spatial and temporal resolution, as well as high precision. Combined with the high strain transmission efficiency of unsheathed distributed optical fibers, it can capture subtle deformations caused by microcrack aggregation. Currently, distributed fiber optic strain monitoring has been used to obtain unconfined rock surface deformation. Protecting the fiber and obtaining rock surface deformation data under extreme conditions, such as triaxial high pressure, is equally important for understanding rock shear failure mechanisms. However, factors such as fiber bending or large rock deformation can cause localized optical loss in the optical path, contaminating the monitoring data with complex noise, which interferes with post-processing analysis. Traditional data denoising methods often assume a normal data distribution or identify noise points based on statistical laws. While these methods have broad applicability, their accuracy still needs improvement. In recent years, machine learning-based denoising algorithms, such as the isolation forest tree algorithm, have gained widespread application. These tree-based algorithms require a specific noise ratio, which often leads to misidentification or omission of noise points and requires long computational time. At the same time, common missing value filling methods such as nearest neighbor interpolation and time series prediction methods based on machine learning are difficult to simultaneously consider the temporal continuity and spatial distribution of rock deformation processes.

[0004] The present invention provides a method for protecting a 250-micron diameter distributed optical fiber for long-term monitoring under triaxial high-pressure conditions. By combining a convolutional neural network and an autoencoder, the temporal continuity and spatial distribution of rock deformation are taken into account to achieve intelligent identification of noise points and filling of missing values ​​in distributed optical fiber data under static monitoring conditions. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for protecting a 250-micron diameter distributed optical fiber and collecting strain data under triaxial high-pressure conditions. Targeted noise points in strain monitoring data are targeted at intelligent noise detection and missing data supplementation methods, addressing the problems of large amounts of noise and missing data during high-pressure, long-term, and static monitoring of rock deformation using optical fibers.

[0006] The technical solutions of the present invention are as follows:

[0007] A method for intelligent denoising and missing value filling of distributed optical fiber strain data, characterized in that the method comprises:

[0008] Under triaxial test conditions, the rock is coupled to the distributed optical fiber, and the optical fiber strain data is collected through the distributed optical fiber monitoring system;

[0009] representing the optical fiber strain data in time series and preprocessing the data to obtain noise-free optical fiber strain data;

[0010] Constructing a noise point recognition and data filling model, and training the noise point recognition and data filling model using noise-free optical fiber strain data;

[0011] All optical fiber strain data containing spatial position information are denoised and missing values ​​are filled based on the trained noise point identification and data filling model, and the noise point identification and data filling model parameters are updated in real time according to the changes in the strain field in the monitored space over time.

[0012] Furthermore, the time-series representation and pre-processing of the optical fiber strain data includes:

[0013] The optical fiber strain data obtained by the distributed optical fiber is expressed as a time series;

[0014] Standardize the optical fiber strain data containing position information obtained at each time point;

[0015] The standardized data is transformed into two dimensions, and each set of standardized data is transformed from one dimension to two dimensions, with the strain at each spatial position occupying a separate dimension.

[0016] Furthermore, the noise recognition and data filling model is composed of an encoder and a decoder, the encoder includes multiple fully connected layers, convolutional layers and pooling layers, and the decoder includes multiple deconvolution layers, upsampling layers and fully connected layers.

[0017] Furthermore, the noise point identification and data filling model is trained using noise-free optical fiber strain data as follows:

[0018] Two or more sets of noise-free fiber strain data are selected and input into the encoder. The fully connected layer multi-represents the single data into multi-dimensional data. The multi-dimensional data is then passed through the convolution layer to extract features, and then the feature size is reduced through the pooling layer. Finally, the high-dimensional feature data extracted from the encoder is input into the decoder.

[0019] In the decoder, the deconvolution layer reduces the spatial dimension of the data by performing a reverse convolution operation on the high-dimensional feature data, and then further restores the details of the data through the upsampling layer, and finally reconstructs the data through the fully connected layer;

[0020] Compare the original data with the reconstructed data, calculate the loss function, adjust the model parameters according to the value of the loss function, and perform multiple training at a certain learning rate to finally obtain a trained noise recognition and data filling model.

[0021] Furthermore, the noise removal and missing value filling of all optical fiber strain data containing spatial position information are performed according to the trained noise point identification and data filling model as follows:

[0022] Predict the strain data at different spatial locations at the first time point using the trained noise point recognition and data filling model; calculate the mean square error between the predicted value and the actual measured value;

[0023] When the error is greater than the set threshold, the strain data at this position is determined to be a noise point, and the spatial position information of the noise point is recorded;

[0024] Fill missing values ​​for identified noise points.

[0025] Furthermore, the formula for filling missing values ​​for the identified noise points is:

[0026] ,

[0027] in, is the missing strain value at the current i-th position, is the average value of the spatial strain field at time point t where the missing value is located, is the mean and standard deviation of the spatial strain field at time point t where the missing value is located, is the normalized strain value at position i at the current time point t predicted by the model, where i represents the position of the i-th noise point and t represents the time point predicted by the current model.

[0028] Furthermore, the real-time updating of the noise point identification and data filling model parameters according to the change of the strain field in the monitored space over time is specifically as follows:

[0029] After completing the noise point identification and missing value filling of the optical fiber strain data at time point t-1, the data at time point t-2 is used as the model input, and the reconstruction error between the model predicted data and the optical fiber strain data at time point t-1 is calculated. The model is then taught to learn the spatial strain distribution at time point t-1 and update the model parameters.

[0030] Furthermore, the method further includes removing Gaussian noise in the optical fiber strain data by weighted averaging after completing the missing value filling, specifically:

[0031] First, calculate the one-dimensional convolution kernel according to the Gaussian function;

[0032] Using the one-dimensional convolution kernel to perform a convolution operation on the data at each time point by means of equal-length convolution;

[0033] After the convolution process is completed, the Gaussian filtered data is used to replace the original data to remove Gaussian noise in the optical fiber strain data.

[0034] Compared with the prior art, the present invention has the following advantages:

[0035] The present invention proposes a method for long-term full-field deformation monitoring of rock surfaces by protecting unsheathed distributed optical fibers under triaxial high-pressure conditions, thereby achieving high-precision strain data acquisition in complex stress environments.

[0036] This invention uses a deep learning method with an encoder-decoder structure to automatically learn data features, which can accelerate convergence during training and reduce computing resource consumption. At the same time, the incremental training mechanism allows the model to update parameters in real time based on the temporal changes of the strain field, ensuring effectiveness and stability under long-term monitoring. In data processing, a comprehensive consideration of data distribution and temporal continuity is taken into account to effectively identify and process noise points and properly fill missing values ​​to ensure data integrity and accuracy. Gaussian noise can be effectively removed after missing values ​​are filled by calculating a one-dimensional convolution kernel and performing equal-length convolution operations, preventing misjudgments. This provides high-precision data processing support for related monitoring and has important engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings illustrate various embodiments generally by way of example and not limitation, and together with the description and claims, serve to explain embodiments of the invention. Where appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive of the embodiments of the present apparatus or method.

[0038] Figure 1 A diagram of the equipment and monitoring process in the method of the present invention;

[0039] Figure 2is a flow chart of the method of the present invention;

[0040] Figure 3 Graphs of the original data and the data after denoising using the method of the present invention. DETAILED DESCRIPTION

[0041] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] refer to Figure 2 As shown, the present invention provides a method for protecting a distributed optical fiber with a diameter of 250 microns under harsh conditions to monitor rock deformation and perform noise removal and missing value filling on strain data, the steps of which are as follows:

[0043] Step 1: Under triaxial test conditions, the distributed optical fiber is effectively coupled with the rock, and the distributed optical fiber monitoring system is used to

[0044] A G652b single-mode, unjacketed distributed optical fiber with a diameter of 250 microns was selected. First, the fiber laying path was determined. The fiber trajectory was drawn on the surface of the test sample with a waterproof marker. The fiber was laid onto the rock surface along the predetermined trajectory, ensuring that there were no excessive bends during the laying process. A certain prestress was maintained during the laying process to ensure that the distributed optical fiber was in full contact with the rock surface. A quick-drying adhesive with a high elastic modulus was used as a coupling agent to adhere the distributed optical fiber to the rock surface. The fiber was ensured to be intact during the pasting process.

[0045] The rock surface where the optical fiber is attached is covered with a sealant with a thickness of more than 5 mm to prevent the hydraulic oil in the triaxial chamber from entering the rock during the test. The sealant also protects the optical fiber from external environmental interference. The sealant should be waterproof, high temperature resistant, corrosion resistant, and able to withstand large deformation.

[0046] Place the fiber-wound test sample into the triaxial chamber, and lead one end of the fiber out of the triaxial test machine as the input end to input the optical signal, and the other end of the fiber as the output end of the optical signal. Tie a knot at the output end to reduce the influence of optical loss at the fiber break on the test results.

[0047] Splice the optical fiber adhered to the rock with the optical fiber connected to the demodulator to ensure that the optical loss is less than 0.01dB. Use a laser pointer or demodulator to test the optical fiber to ensure that it can transmit signals normally.

[0048] Step 2: Use the rock distributed optical fiber monitoring system to collect data;

[0049] The rock distributed optical fiber monitoring system includes a strain sensing part and a data processing and control system, wherein the strain sensing part is a distributed optical fiber adhered to the rock to be measured, and the data processing and control system includes a distributed optical fiber signal demodulator and a computer host;

[0050] Before starting the test, check the optical loss in the optical fiber to ensure that the optical loss does not increase significantly with the extension of the measurement distance. Press the optical fiber to calibrate the position and determine the starting and ending points of data collection.

[0051] The corresponding data parameters are set in the data processing and control system, including the strain transfer coefficient and temperature transfer coefficient of the selected optical fiber, the spatial resolution and temporal resolution of the data acquisition. The equipment used in the present invention is based on Optical Frequency Domain Reflectometry (OFDR), and the data sampling accuracy is 1με.

[0052] Furthermore, a triaxial compression test of rock was carried out on a triaxial testing machine. The test loading and data acquisition were carried out simultaneously. The oil pressure in the triaxial chamber was first increased to 60 MPa to ensure that the optical fiber could continue to work normally under high pressure. The triaxial compression creep test was then started until the rock was destroyed.

[0053] Step 3: Represent and preprocess the data collected by distributed optical fiber in time series

[0054] The strain distribution obtained from the distributed optical fiber is represented as a time series and preprocessed, including filtering noise-free data, normalizing the input data, and performing dimensionality conversion on the normalized data.

[0055] Two or more groups of noise-free data are manually selected from the early monitoring data for model training to learn the relationship between the initial strain and spatial position. The requirements for selecting noise-free data are: no data points exceeding the equipment test limit, continuous data changes without sudden changes, and the data collection time point is as close as possible to the data to be denoised;

[0056] The first step in the time series data preprocessing is to standardize the strain data containing position information obtained at each time point, removing the size characteristics of the data and retaining only the data change trend. The mean value of the data sample at each time point is set to 0 and the standard deviation is set to 1. The calculation formula is:

[0057] ,

[0058] Where, is the strain at the i-th position in the data at a time point, is the average strain of the data collected at that time point, is the standard deviation of the data, for Normalized data.

[0059] The standardized data is transformed into two dimensions: each set of standardized data is transformed from one dimension to two dimensions, and the strain at each spatial position occupies one dimension. The transformation process is as follows:

[0060] ,

[0061] The shape of the converted dataset changes from (m, n) to (m, n, 1), where the first dimension m represents different time points and the second dimension n represents the number of monitoring points. This is done to remove the spatial order features in the data and enable the model to simultaneously learn the strain features of all locations to reduce model parameters and training costs.

[0062] Step 4: Build the encoder and decoder and train the parameters;

[0063] The noise-free data after standardization and dimension conversion in step 3 is input into the encoder and decoder for data reconstruction. The encoder contains multiple fully connected layers, convolutional layers, and pooling layers, and the decoder contains multiple deconvolution layers, upsampling layers, and fully connected layers.

[0064] 4.1 Input training data into encoder:

[0065] The single strain data is expressed in multiple ways, and the data at time point t-1 is input into two fully connected layers to represent the strain at each spatial position with a set of data. The ReLU activation function is used between the two fully connected layers to process the data. The process is as follows:

[0066] ,

[0067] is the weight matrix, is the bias coefficient, k is the number of output nodes in the first layer, v is the number of output nodes in the second layer, are the input data and output data of the first layer, is the data after the fully connected layer;

[0068] The multi-dimensional data output by the fully connected layer is input into the convolutional layer to extract features: First, the data at each time point is converted into a tensor of size (n, 1, v), where n is the number of monitoring points and v is the number of nodes in the last fully connected layer;

[0069] The feature extraction is performed through the convolution kernel with an input channel of 1, an output channel of d, a size of 3, a sliding step of 1, and a padding of 1. The data is then processed by the ReLU activation function. The dropout layer is used in the middle to prevent parameter overfitting. The pooling layer is then used to reduce the size of the feature map. The above convolution and pooling process is repeated to further increase the feature dimension and reduce the feature size.

[0070] 4.2 Input the extracted data features into the decoder:

[0071] The high-dimensional feature data extracted by the convolution layer is input into the decoder. The structure of the decoder is opposite to that of the encoder. The deconvolution layer corresponds to the convolution layer in the encoder, the upsampling layer corresponds to the pooling layer in the encoder, the number of input and output channels in the deconvolution layer corresponds to the number of output and input channels in the convolution layer, and other hyperparameters are the same as those in the encoder. The data dimension is continuously restored through the deconvolution and upsampling process;

[0072] After the last layer of upsampling is completed, the data is restored from the shape of (n, 1, v) to (n, v) and input into the fully connected layer for data reconstruction. The ReLU activation function is used in the middle for processing. The shape of each group of reconstructed data is (n, 1), the same size as the original data, representing the strain value of each spatial position;

[0073] 4.3 Calculating reconstruction error as loss function

[0074] The mean square error between the original strain data at time t after normalization and the predicted data output by the model is calculated to evaluate the prediction effect of the model. The calculation formula is:

[0075] ,

[0076] in, is the original data, To reconstruct the data, n is the number of monitoring points in a set of data;

[0077] 4.4 Iterative Model Training

[0078] According to the preset number of training times, the noise-free data selected in step 3 are input into the model in chronological order for training. Steps 4.1-4.3 are repeated for each iteration. The learning rate is set to 0.001. The appropriate number of iterations is selected until the loss function is small enough and constant. After the iteration, the trained initial model is obtained.

[0079] Step 5: Based on the initial model, noise points are identified, missing values ​​are filled, and Gaussian denoising is performed on all time-series strain data containing spatial position information, and the model parameters are updated in real time according to the changes in the strain field in the monitored space over time.

[0080] 5.1 Noise Identification and Missing Value Filling

[0081] Use the initial model trained in step 4.4 to predict the strain at different spatial locations at the first time point. Calculate the error using the formula in step 4.3. Treat strain values ​​with an error greater than 1 as noise points and record the spatial location of the noise points.

[0082] Furthermore, missing values ​​are filled for the noise strains according to the model prediction results and the strain data distribution at the current time point. The specific formula is as follows:

[0083] ,

[0084] in, is the missing strain value at the current i-th position, is the average value of the spatial strain field at time point t where the missing value is located, is the mean and standard deviation of the spatial strain field at time point t where the missing value is located, is the normalized strain value at position i at the current time point t predicted by the model, where i represents the position of the i-th noise point and t represents the time point predicted by the current model;

[0085] 5.2 Incremental Training Dynamically Updates Model Parameters and Gaussian Filtering

[0086] During static monitoring, the spatial strain field changes slowly over time, so the model needs to learn the changing state of the strain field in real time and update the model parameters;

[0087] Repeat steps 4.1-4.4 for incremental training to update the model parameters by inputting the last time point of the training data. When calculating the loss function, calculate the mean square error between the reconstructed data and the denoised strain data at the first time point. Use the updated model to identify noise points in the strain data at the second time point. Repeat step 5.1 to complete noise point identification and missing value filling at the second time point.

[0088] After completing the noise point identification and missing value filling of the strain data at time point t-1, use the data at time point t-2 as input and calculate the reconstruction error between the model output data and the strain data at time point t-1. Let the model learn the spatial strain distribution at time point t-1 and update the model parameters to repeat steps 4.1-4.4 to perform noise removal at time point t.

[0089] During the monitoring process, due to instrument accuracy limitations and external disturbances, slightly fluctuating Gaussian noise will appear. This noise is not much different from the original data and may not be identified as noise. However, during the parameter update and data filling process, this noise may be learned as a feature and continuously strengthened. It may eventually mislead the model to identify normal data as noise, resulting in misjudgment and incorrect filling. Therefore, after each missing value filling, it is necessary to remove the Gaussian noise in the data by weighted averaging. First, the one-dimensional convolution kernel is calculated according to the Gaussian function. The formula is as follows:

[0090] ,

[0091] in The range is an integer between [-n, n], where n is the size of the convolution kernel. is the corresponding weight value in the convolution kernel;

[0092] Further convolve the above convolution kernel with the data at each time point. The size of the convolution kernel should not be too large, and is generally set to 3.

[0093] The one-dimensional convolution kernel obtained above is used to perform one-dimensional convolution on the denoised data. The marginal effect of the data is not considered during the calculation process, and the convolution method selects equal-length convolution, that is, the number of input data and output data of the Gaussian filter is kept consistent. The influence of the marginal effect on the data within the effective monitoring range can be avoided by lengthening the number of input data. Finally, the original data is replaced by the data that has completed the Gaussian filter.

[0094] Based on distributed optical fiber monitoring technology, the present invention invents a method for protecting optical fibers under complex stress conditions such as triaxial high pressure, and for identifying noise points and filling missing values ​​in strain field data under static monitoring.

[0095] Figure 1 This diagram shows the equipment and monitoring process during the test. The test environment was a laboratory environment with a high confining pressure of 60MPa, lasting for 9 hours, with a spatial resolution of 0.64-1mm and an accuracy of up to 1με. Figure 3 The comparison diagram before and after data processing shows that the method proposed in the present invention can accurately remove noise and complete missing value filling, and the model can adapt to the changes in the spatial distribution of strain data over time during the monitoring process.

[0096] The above is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, can make equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. An intelligent denoising and missing value filling method for distributed optical fiber strain data, characterized in that: The method comprises: Under triaxial test conditions, the rock is coupled to the distributed optical fiber, and the optical fiber strain data is collected through the distributed optical fiber monitoring system; representing the optical fiber strain data in time series and preprocessing the data to obtain noise-free optical fiber strain data; Constructing a noise point recognition and data filling model, and training the noise point recognition and data filling model using noise-free optical fiber strain data; De-noising and filling missing values ​​are performed on all optical fiber strain data containing spatial position information according to the trained noise point identification and data filling model, and the noise point identification and data filling model parameters are updated in real time according to the change of the strain field in the monitored space over time; The noise point identification and data filling model is trained using noise-free optical fiber strain data as follows: Two or more sets of noise-free fiber strain data are selected and input into the encoder. The fully connected layer multi-represents the single data into multi-dimensional data. The multi-dimensional data is then passed through the convolution layer to extract features, and then the feature size is reduced through the pooling layer. Finally, the high-dimensional feature data extracted from the encoder is input into the decoder. In the decoder, the deconvolution layer reduces the spatial dimension of the data by performing a reverse convolution operation on the high-dimensional feature data, and then further restores the details of the data through the upsampling layer, and finally reconstructs the data through the fully connected layer; Compare the original data with the reconstructed data, calculate the loss function, adjust the model parameters according to the value of the loss function, and perform multiple training according to the learning rate to finally obtain a trained noise recognition and data filling model; The method further includes removing Gaussian noise in the optical fiber strain data by weighted averaging after completing the missing value filling, specifically: First, calculate the one-dimensional convolution kernel according to the Gaussian function; Using the one-dimensional convolution kernel to perform a convolution operation on the data at each time point by means of equal-length convolution; After the convolution process is completed, the Gaussian filtered data is used to replace the original data to remove Gaussian noise in the optical fiber strain data.

2. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: Representing and preprocessing the optical fiber strain data in time series includes: The optical fiber strain data obtained by the distributed optical fiber is expressed as a time series; Standardize the optical fiber strain data containing position information obtained at each time point; The standardized data is transformed into two dimensions, and each set of standardized data is transformed from one dimension to two dimensions, with the strain at each spatial position occupying a separate dimension.

3. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: The noise recognition and data filling model consists of an encoder and a decoder, wherein the encoder includes multiple fully connected layers, convolutional layers and pooling layers, and the decoder includes multiple deconvolution layers, upsampling layers and fully connected layers.

4. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: The noise removal and missing value filling of all optical fiber strain data containing spatial position information according to the trained noise point identification and data filling model are specifically performed as follows: Predict the strain data at different spatial locations at the first time point using the trained noise point recognition and data filling model; calculate the mean square error between the predicted value and the actual measured value; When the error is greater than the set threshold, the strain data at this position is determined to be a noise point, and the spatial position information of the noise point is recorded; Fill missing values ​​for identified noise points.

5. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 4, characterized in that: The formula for filling missing values ​​for the identified noise points is: ， in, is the missing strain value at the current i-th position, is the average value of the spatial strain field at time point t where the missing value is located, is the mean and standard deviation of the spatial strain field at time point t where the missing value is located, is the normalized strain value at position i at the current time point t predicted by the model, where i represents the position of the i-th noise point and t represents the time point predicted by the current model.

6. The method for intelligent denoising and missing value filling of distributed optical fiber strain data according to claim 1, characterized in that: The real-time updating of the noise point identification and data filling model parameters according to the change of the strain field in the monitored space over time is specifically as follows: After completing the noise point identification and missing value filling of the optical fiber strain data at time point t-1, the data at time point t-2 is used as the model input, and the reconstruction error between the model predicted data and the optical fiber strain data at time point t-1 is calculated. The model is then taught to learn the spatial strain distribution at time point t-1 and update the model parameters.

Images