A method and system for data analysis based on in-pipe detectors

Abstract

The application provides a pipeline detector-based data analysis method and system, relating to the technical field of data processing, the method comprising: obtaining historical pipeline signals; performing preprocessing operations on the pipeline signals; training a pipeline anomaly perception LSTM model using the historical pipeline signals to obtain a target pipeline anomaly perception LSTM model; obtaining real-time pipeline signals; inputting the real-time pipeline signals into the target pipeline anomaly perception LSTM model to obtain determination parameters of the real-time pipeline signals; the determination parameters comprising: predicted signals, residual sequences, and confidence levels; determining abnormal signals according to the determination parameters; the abnormal signals comprising: invalid signals, weak signals, and jump signals; and determining pipeline spatial positions corresponding to the abnormal signals based on mapping information, to solve the problems of low efficiency of current pipeline detection methods, and inability to automatically identify and process complex abnormal signals, resulting in low accuracy of detection results.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data analysis method and system based on an in-pipe detector. Background Technology

[0002] Pipelines, as vital carriers for energy transmission and chemical raw material transport, are highly susceptible to internal defects such as cracks and corrosion as their service life increases and their operating environment becomes increasingly complex and variable. If these defects are not detected and addressed promptly, they will seriously threaten the safe operation of pipelines and may even lead to major safety accidents. Therefore, regular and accurate safety inspections of pipelines are particularly urgent, and pipeline detectors, with their advantages of high efficiency and accuracy, have become a key means of ensuring safe pipeline operation. In actual inspection processes, it is not only necessary to accurately detect various defects within the pipeline, but also to precisely analyze the signals collected during the inspection to obtain reliable results.

[0003] Currently, there are various common pipeline inspection methods. Among them, the magnetic flux leakage method determines the presence and extent of defects by detecting the leakage of the magnetic field at the pipeline defect location; the eddy current method uses the principle of electromagnetic induction to detect defects based on changes in eddy currents within the pipeline; and odometer wheels are also used to measure the distance the detector travels within the pipeline, thus aiding in defect location. Furthermore, to obtain more comprehensive inspection information, multiple sensors are typically used to collect data simultaneously.

[0004] However, the aforementioned data lacks effective fusion methods, making it difficult to deeply integrate data collected from multiple sensors. This results in the data information not being fully explored and utilized, affecting the comprehensiveness and accuracy of the detection results. In actual detection signals, invalid signals, weak signals, and abrupt signals are mixed in, making it difficult to accurately identify these signals. This is especially true for abrupt signal detection, where a small number of subtle abrupt signals are currently difficult to detect. Moreover, existing abnormal signal detection methods are not only inefficient, unable to automatically identify and process complex abnormal signals, but are also susceptible to human interference, leading to inaccurate data analysis and severely impacting the accuracy and reliability of the detection results. Summary of the Invention

[0005] This application provides a data analysis method and system based on pipeline detectors to solve the technical problems of existing pipeline detection methods being not only inefficient but also unable to automatically identify and process complex abnormal signals.

[0006] The first aspect of this application provides a data analysis method based on an in-pipe detector, including:

[0007] Historical pipeline signals are acquired; the pipeline signals include: leakage magnetic field signals, eddy current signals, stress signals, and mileage wheel signals; the mileage wheel signals include mapping information between the pipeline detection time index and the corresponding mileage; the mapping information represents the correspondence between the pipeline signal acquisition time and the mileage of the pipeline under test;

[0008] The pipeline signal is preprocessed; the preprocessing operation includes time synchronization processing, improved normalization processing, adaptive signal de-drift processing, defect feature-protected noise filtering, and multi-threshold local peak preservation processing, which are executed sequentially.

[0009] The pipeline anomaly perception LSTM model is trained using the preprocessed historical pipeline signals to obtain the target pipeline anomaly perception LSTM model.

[0010] Acquire real-time pipeline signals and perform the preprocessing operation;

[0011] The preprocessed real-time pipeline signal is input into the target pipeline anomaly perception LSTM model to obtain the determination parameters of the real-time pipeline signal; the determination parameters include: predicted signal, residual sequence and confidence level;

[0012] Based on the determination parameters, abnormal signals are determined; the abnormal signals include: invalid signals, weak signals, and abrupt change signals.

[0013] Based on the mapping information, the spatial location of the pipeline corresponding to the abnormal signal is determined.

[0014] In some embodiments, the pipeline anomaly-aware LSTM model is an improvement on the LSTM network model; the pipeline anomaly-aware LSTM model includes: an input layer, a hidden layer, and an output layer;

[0015] The input layer of the pipeline anomaly sensing LSTM model integrates a preprocessing unit; the preprocessing unit is configured to separate the local peaks and trend components of the pipeline signal and extract the temporal features of the pipeline signal; the trend components include: the base interference signal generated by the material and wall thickness uniformity of the pipeline under test, the electromagnetic noise signal generated by the external magnetic field interference and the detector speed change, and the motion noise signal generated by the detector running speed change;

[0016] The hidden layer of the pipeline anomaly perception LSTM model introduces a gated confidence mechanism, and the hidden layer is configured to output a prediction signal and confidence based on the time series features.

[0017] The output layer of the pipeline anomaly perception LSTM model is equipped with a residual detection unit; the residual detection unit is configured to calculate the difference between the predicted signal and the corresponding pipeline signal to generate a residual sequence.

[0018] In some embodiments, the step of training a pipeline anomaly perception LSTM model using the preprocessed historical pipeline signal data to obtain a target pipeline anomaly perception LSTM model includes:

[0019] The historical pipeline signals after the preprocessing operation are labeled to obtain a normal signal sample set and an abnormal signal sample set, and then divided into a training set, a validation set and a test set according to a preset ratio.

[0020] The training set is input into the pipeline anomaly perception LSTM model for training, and the validation set is used to iteratively validate the pipeline anomaly perception LSTM model during the training process; the loss function of the model is composed of a weighted prediction error term and a confidence constraint term through adaptive weighted fusion;

[0021] When the signal recognition accuracy of the pipeline anomaly perception LSTM model on the test set reaches a preset accuracy threshold and the loss function value converges to a preset loss threshold, training is stopped, and the target pipeline anomaly perception LSTM model is obtained.

[0022] In some embodiments, the step of determining the abnormal signal based on the determination parameter includes:

[0023] Obtain the real-time signal value of the pipeline signal;

[0024] If the signal value of a preset number of adjacent signals in the real-time pipeline signal is 0 and the corresponding confidence level is less than a preset confidence threshold, then the corresponding signal segment of the real-time pipeline signal is an invalid signal.

[0025] In some embodiments, the step of determining the abnormal signal based on the determination parameter includes:

[0026] Calculate the prediction variance of the predicted signal within the sliding window; the prediction variance is:

[0027] ;

[0028] In the formula, N is the total number of predicted signals within the sliding window; x i Let be the signal value of the i-th predicted signal within the sliding window; The average value of the predicted signal within the sliding window;

[0029] Obtain the average signal energy of the real-time pipeline signal corresponding to the predicted signal within the sliding window;

[0030] If the prediction variance of the predicted signal is less than a preset variance threshold and the signal energy value is less than a preset energy threshold, then the real-time pipeline signal corresponding to the predicted signal within the sliding window is a weak signal.

[0031] In some embodiments, the step of determining the abnormal signal based on the determination parameter includes:

[0032] Calculate the absolute value of the difference between adjacent residual values ​​in the residual sequence;

[0033] The residual mutation characteristic value is obtained by weighting the absolute values ​​of the differences.

[0034] If the residual mutation feature value is greater than the preset jump threshold, then the real-time pipeline signal corresponding to the residual mutation feature value is a jump signal.

[0035] In some embodiments, the step of determining the spatial location of the pipeline corresponding to the abnormal signal based on the mapping information includes:

[0036] Based on the mapping information, determine the single-point mileage coordinates corresponding to the abnormal signal;

[0037] If the single-point mileage coordinates show a continuous distribution characteristic on the time axis, then the gated confidence mechanism of the target pipeline anomaly perception LSTM model is used to perform confidence-weighted smoothing fusion processing on the continuous abnormal signal segments to obtain the mileage range of the continuous abnormal signal segments.

[0038] Based on the single-point mileage coordinates and the mileage range of the abnormal signal section, the spatial location of the pipeline corresponding to the abnormal signal is determined.

[0039] In some embodiments, the method further includes:

[0040] Based on the type of the abnormal signal, the judgment parameters, and the spatial location of the pipeline corresponding to the abnormal signal, a pipeline inspection report is generated; the pipeline inspection report is used to display the mileage of the abnormal signal in the pipeline under test, the time index corresponding to the abnormal signal, and the confidence level corresponding to the abnormal signal.

[0041] A second aspect of this application provides a data analysis system based on an in-pipe detector, comprising:

[0042] The first acquisition module is configured to acquire historical pipeline signals; the pipeline signals include: leakage magnetic field signals, eddy current signals, stress signals, and mileage wheel signals, wherein the mileage wheel signals include mapping information between the pipeline detection time index and the corresponding mileage; the mapping information represents the correspondence between the pipeline signal acquisition time and the mileage of the pipeline to be tested;

[0043] The preprocessing module is configured to perform preprocessing operations on the pipeline signal; the preprocessing operations include time synchronization processing, improved normalization processing, adaptive signal de-drift processing, defect feature-protected noise filtering, and multi-threshold local peak preservation processing, which are executed sequentially.

[0044] The training module is configured to train the pipeline anomaly perception LSTM model using the preprocessed historical pipeline signals to obtain the target pipeline anomaly perception LSTM model.

[0045] The second acquisition module is configured to acquire real-time pipeline signals and perform the preprocessing operation.

[0046] The input module is configured to input the preprocessed real-time pipeline signal into the target pipeline anomaly perception LSTM model to obtain the determination parameters of the real-time pipeline signal; the determination parameters include: predicted signal, residual sequence and confidence level;

[0047] The first determining module is configured to determine an abnormal signal based on the determination parameters; the abnormal signal includes: invalid signal, weak signal, and transition signal;

[0048] The second determining module is configured to determine the spatial location of the pipeline corresponding to the abnormal signal based on the mapping information.

[0049] This application provides a data analysis method and system based on pipeline detectors. The method includes: acquiring historical pipeline signals; the pipeline signals include: leakage magnetic field signals, eddy current signals, stress signals, and mileage wheel signals; the mileage wheel signals include mapping information between the pipeline detection time index and the corresponding mileage; the mapping information represents the correspondence between the pipeline signal acquisition time and the mileage of the pipeline to be tested; performing preprocessing operations on the pipeline signals; the preprocessing operations include, sequentially, time synchronization processing, improved normalization processing, adaptive signal drift removal processing, defect feature-protected noise filtering, and multi-threshold local peak preservation processing; and using the preprocessed historical pipeline signals for pipeline anomaly sensing (LS). The LSTM model is trained to obtain a target pipeline anomaly perception model; real-time pipeline signals are acquired and the preprocessing operation is performed; the preprocessed real-time pipeline signals are input into the target pipeline anomaly perception LSTM model to obtain the judgment parameters of the real-time pipeline signals; the judgment parameters include: predicted signal, residual sequence, and confidence level; based on the judgment parameters, anomaly signals are determined; the anomaly signals include: invalid signals, weak signals, and jump signals; based on the mapping information, the spatial location of the pipeline corresponding to the anomaly signal is determined, realizing automated and high-precision identification of pipeline anomaly signals, while accurately locating the spatial location of the pipeline corresponding to the anomaly, greatly improving pipeline detection efficiency and the reliability of detection results. Attached Figure Description

[0050] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of the data analysis method based on an in-pipe detector in this application;

[0052] Figure 2 This is a schematic diagram of the structure of the pipeline anomaly perception LSTM model in this application. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0054] In many technologies, pipeline inspection methods rely heavily on manual analysis, which is not only inefficient but also unable to automatically identify and process complex abnormal signals. Specifically, sensor data collected by pipeline detectors (such as odometer wheel, temperature, inertial navigation, magnetic flux leakage, eddy current, etc.) is difficult to effectively integrate from multiple data sources and accurately identify anomalies (such as invalid signals, weak signals, and abrupt signals). The failure to effectively identify and locate invalid, weak, and abrupt signals leads to inaccurate detection results. To address this technical problem, this application provides a data analysis method and system based on pipeline detectors. The following describes the data analysis method and system based on pipeline detectors:

[0055] like Figure 1 The diagram shown is a flowchart of the data analysis method based on an in-pipe detector in this application.

[0056] The first aspect of this application provides a data analysis method based on an in-pipe detector, comprising the following steps:

[0057] S100: Acquire historical pipeline signals; the pipeline signals include: leakage magnetic field signals, eddy current signals, stress signals, and mileage wheel signals; the mileage wheel signals include the mapping information between the pipeline detection time index and the corresponding mileage; the mapping information represents the correspondence between the pipeline signal acquisition time and the mileage of the pipeline to be tested.

[0058] For example, the magnetic flux leakage signal is acquired by a magnetic flux leakage sensor, a non-destructive testing device that uses the principle of magnetic fields to detect defects on the metal surface. By detecting changes in the magnetic field, the magnetic flux leakage sensor can detect metal corrosion, abnormal circumferential welds, and other irregular defects. The eddy current signal is acquired by an eddy current sensor, which utilizes the principle of electromagnetic induction to detect defects such as cracks and corrosion in metal objects by detecting changes in eddy currents. By inducing changes in current in a conductor, the eddy current sensor can detect minute cracks on the metal surface. The odometer wheel signal is acquired by an odometer wheel, a device used to provide the position of a moving object, commonly used in pipeline inspection systems. By measuring position changes in real time, it provides markings and positioning of the pipeline at different locations. The signals output by each sensor constitute high-dimensional time-series data, with different sampling frequencies, amplitude characteristics, and noise characteristics.

[0059] S200: Perform preprocessing operations on the pipeline signal; the preprocessing operations include time synchronization processing, improved normalization processing, adaptive signal drift removal processing, defect feature-protected noise filtering, and multi-threshold local peak preservation processing, which are executed sequentially. The preprocessing operations preserve the defect response characteristics of the pipeline signal and suppress environmental interference.

[0060] Among them, the improved normalization process overcomes the problem of min-max normalization suppressing defect features by introducing a defect feature weighting factor, which enhances the amplitude difference of pipeline defect response features while normalizing. The specific formula is as follows:

[0061] ;

[0062] In the formula, The original pipeline signal value before preprocessing. This is the normalized pipe signal value; The mean of the original pipeline signal sequence is . The standard deviation of the original pipeline signal sequence; , These are the minimum and maximum values ​​of the original pipeline signal sequence, respectively. The weighting factor for enhancing defect features has a value range of 0.3 ≤ ≤0.6, calibrated through statistical analysis of historical defect signal samples (the higher the proportion of defect signals, the better). The larger the value.

[0063] Among them, the adaptive signal drift removal process addresses the slow time-varying drift of the pipeline signal over the detection time. It uses a sliding window weighted least squares method to fit the drift trend and achieve adaptive drift elimination. The specific formula is as follows:

[0064] ;

[0065] In the formula, The fitted value represents the drift trend within the sliding window. For time indexing, , These are the linear fitting coefficients; The length of the sliding window (preferably 50 ≤ ≤200 (set according to the pipeline signal sampling frequency). This is a time-weighted factor.

[0066] as well as, ( (As the time index of the middle of the sliding window), higher fitting weights are assigned to the signals in the middle region of the window to improve the drift fitting accuracy; This is the pipe signal value after drift removal.

[0067] Among them, the defect feature-protected noise filtering adopts a composite filtering algorithm of "wavelet threshold denoising + defect feature constraint", which suppresses environmental interference noise while avoiding the smooth suppression of pipeline defect features by the filter. The specific formula is as follows:

[0068] ;

[0069] In the formula, For wavelet transform operators, For the first The first layer of wavelet decomposition Wavelet coefficients; For the first The base threshold of the layer ( The standard deviation of noise. (for pipe signal length) This is a pipeline defect characteristic protection factor, with a value range of 0.6 ≤ ≤0.8, by reducing the wavelet coefficient threshold corresponding to the pipeline defect characteristics, the high-frequency components of the pipeline defect signal are preserved; These are the wavelet coefficients after thresholding. This is the pipe signal value after noise filtering. This is the signal timing index.

[0070] Specifically, the characteristic peaks corresponding to pipeline defects are screened using a dual constraint of "amplitude threshold + gradient threshold" to suppress noise spurious peaks. The specific formula is as follows:

[0071] ;

[0072] In the formula, , for The preceding and following adjacent signal values, for The local gradient magnitude of the signal at any given time characterizes the degree of signal abrupt change. The preset gradient threshold is a non-negative real number, which is calibrated by statistical analysis of historical pipeline defect signal gradients. The preset peak amplitude threshold is a non-negative real number that is greater than the maximum amplitude of the noise signal. The output signal is processed to maintain peak value. To smooth the operator, a 3-point moving average is preferred.

[0073] Specifically, in pipeline monitoring systems, multiple sensors may be used to collect signals at different locations or at different times. Time synchronization processing ensures that these signals are accurately aligned on the timeline, making the data collected by different sensors comparable and avoiding misjudgments of pipeline status due to time discrepancies. For example, when detecting pipeline leaks, signals collected by sensors at different locations must be synchronized in time to accurately analyze the location and time of the leak.

[0074] Since signals collected by different sensors may have different physical quantities and dimensions, such as pressure signals being measured in Pascals (Pa) and flow signals in cubic meters per second (m³ / s), normalization can convert these signals with different dimensions into a uniform numerical range, typically [0,1] or [-1,1], thus eliminating the influence of dimensional differences on the analysis results.

[0075] Pipeline signals may contain long-term trends or drift components caused by factors such as sensor zero-point drift and changes in ambient temperature. These drift components can mask the effective features of the signal, making signal analysis difficult. Drift removal processing can eliminate these drift components, causing the signal to fluctuate around zero mean or a fixed value, highlighting the features of the effective signal, and facilitating subsequent feature extraction and analysis.

[0076] Pipeline signals are susceptible to various noise interferences during acquisition and transmission, such as electromagnetic interference and mechanical vibration noise. These noises degrade signal quality, reduce the signal-to-noise ratio (SNR), and hinder the extraction and analysis of signal features. Noise filtering can suppress noise interference, improve the SNR, and make the signal clearer and more accurate.

[0077] Pipeline signals may contain important local peak features, such as pressure surge peaks and vibration peaks. These peak features are often closely related to abnormal pipeline conditions (such as metal corrosion defects and abnormal circumferential welds). Local peak preservation processing can retain these peak features, preventing them from being smoothed or lost in subsequent signal processing, and providing crucial information for fault diagnosis and condition assessment.

[0078] S300: The pipeline anomaly-aware LSTM model is trained using the preprocessed historical pipeline signals to obtain the target pipeline anomaly-aware LSTM model. The LSTM (Long Short-Term Memory) model is a special type of recurrent neural network (RNN) designed to solve the gradient vanishing and exploding problems encountered by RNNs in long sequence learning, and is particularly suitable for time series data processing. Specifically, the pipeline anomaly-aware LSTM model is a residual and confidence-based pipeline anomaly recognition LSTM model (PAC-LSTM (Pipeline Anomaly-aware Confidence LSTM)).

[0079] like Figure 2 The diagram shown is a schematic representation of the structure of the pipeline anomaly perception LSTM model in this application.

[0080] For example, the pipeline anomaly perception LSTM model is an improvement on the LSTM network model; the pipeline anomaly perception LSTM model includes: an input layer, a hidden layer, and an output layer.

[0081] The input layer of the pipeline anomaly perception LSTM model integrates a preprocessing unit; the preprocessing unit is configured to separate the local peaks and trend components of the pipeline signal and extract the temporal features of the pipeline signal; the trend components include: the base interference signal generated by the variation of the material and wall thickness uniformity of the pipeline under test, the electromagnetic noise signal generated by the external magnetic field interference and the change of detector speed, and the motion noise signal generated by the change of detector running speed.

[0082] The hidden layer of the pipeline anomaly perception LSTM model incorporates a gated confidence mechanism. This hidden layer is configured to output a predicted signal and a confidence score based on the temporal features. The gated confidence mechanism is a method that adds a layer to the network output, providing a confidence score rating of the model's trust in the prediction results. The confidence score refers to the model's degree of confidence in a prediction result, typically expressed as a probability value or the standard deviation of the signal; a higher confidence score indicates a more reliable prediction result.

[0083] The output layer of the pipeline anomaly perception LSTM model includes a residual detection unit. This unit is configured to calculate the difference between the predicted signal and the corresponding pipeline signal, generating a residual sequence. The difference between the predicted and actual values, i.e., the residual value, is calculated in the network output. By comparing the residual values, the degree of matching between the model's predicted signal and the actual signal is determined. The residual value is used to analyze the difference between the predicted and actual observed values, and the degree of signal anomaly is determined by calculating the difference.

[0084] The step of training the pipeline anomaly perception LSTM model using the preprocessed historical pipeline signal data to obtain the target pipeline anomaly perception LSTM model includes the following sub-steps:

[0085] S310: The historical pipeline signals after the preprocessing operation are labeled to obtain a normal signal sample set and an abnormal signal sample set, and are divided into a training set, a verification set and a test set according to a preset ratio.

[0086] S320: The training set is input into the pipeline anomaly perception LSTM model for training, and the validation set is used to iteratively validate the pipeline anomaly perception LSTM model during the training process; the loss function of the model is composed of a weighted prediction error term and a confidence constraint term through adaptive weighted fusion.

[0087] S330: When the signal recognition accuracy of the pipeline anomaly perception LSTM model on the test set reaches a preset accuracy threshold and the loss function value converges to a preset loss threshold, training is stopped, and the target pipeline anomaly perception LSTM model is obtained.

[0088] Specifically, by training with labeled normal and abnormal signal samples, the target pipeline anomaly perception LSTM model can learn the unique feature patterns of both normal and abnormal signals. The prediction error term prompts the model to predict various features of the signal as accurately as possible, such as the signal amplitude, frequency, and trend. This enables the model to accurately determine whether a new pipeline signal is normal or abnormal based on the learned feature patterns, reducing the probability of misjudgment and missed judgment.

[0089] Specifically, in the offline phase, the pipeline anomaly perception LSTM model is trained under supervision using normal and abnormal signal samples labeled in historical pipeline signal data. The loss function in the target pipeline anomaly perception LSTM model is a weighted fusion of a weighted prediction error term and a confidence constraint term, achieving dual-objective optimization of prediction accuracy and reliability to improve the model's sensitivity in anomaly scenarios. The model learns the temporal dependencies and defect response patterns of pipeline signals, forming hidden state parameters (such as predicted signal, residual sequence, and confidence) that characterize the dynamic features of the signal. After training, the obtained target pipeline anomaly perception LSTM model can predict and determine the validity of pipeline signals, providing a foundational model for the real-time detection phase.

[0090] The loss function in the target pipeline anomaly perception LSTM model is composed of a weighted fusion of a weighted prediction error term and a confidence constraint term, achieving dual-objective optimization of model prediction accuracy and prediction reliability; wherein:

[0091] The prediction error term represents the difference between the model's predicted value and the actual value. It is calculated using a mean square error weighted by residual amplitude, adapting to the non-stationarity of the pipeline signal's time series characteristics. Specifically:

[0092] ;

[0093] In the formula The weighted prediction error term represents the difference between the model's predicted value and the actual value, and its value is greater than or equal to 0. T is the total number of signal points in a single batch of training samples, and its value is a positive integer. Let be the predicted value of the t-th signal output by the model. This represents the true value of the t-th signal. This is the amplitude weighting factor, with values ​​[0,1], to achieve differentiated fitting of signal features.

[0094] The confidence constraint term is used to ensure that the confidence of the prediction is effectively constrained during training, while suppressing the interference of low-confidence samples on model training, and realizing the normalization constraint of confidence and gradient smoothing. The specific calculation method is as follows:

[0095] ;

[0096] In the formula This is the confidence constraint term, representing the constraint loss based on the model's confidence level; its value is greater than or equal to 0. Let be the confidence level of the t-th signal output by the model. denoted as the true confidence level of the t-th signal (obtained from historical sample annotations), both of which take values ​​of [0,1]. λ is the confidence level L2 regularization coefficient, preferably 0.01≤λ≤0.1.

[0097] The total loss function is used to adaptively weight and fuse the prediction error term and the confidence constraint term to achieve synergistic optimization of the two losses, specifically:

[0098] ;

[0099] In the formula, L is the total loss function of the model, which is a non-negative real number. The smaller the loss value, the better the model training effect. α is the adaptive weighting coefficient, which takes the value [0,1] to realize the dynamic ratio of prediction error term and confidence constraint term.

[0100] S400: Acquire real-time pipeline signals and perform the preprocessing operation. The preprocessing operation preserves the defect response characteristics of the real-time pipeline signals and suppresses environmental interference.

[0101] S500: Input the preprocessed real-time pipeline signal into the target pipeline anomaly perception LSTM model to obtain the determination parameters of the real-time pipeline signal; the determination parameters include: prediction signal, residual sequence and confidence level.

[0102] S600: Based on the determination parameters, identify abnormal signals; the abnormal signals include: invalid signals, weak signals, and abrupt change signals; the signal value of the invalid signal is continuously zero or close to zero, usually caused by factors such as sensor failure, signal loss, and environmental interference, and cannot provide effective information. The signal strength of the weak signal is low, possibly caused by factors such as reduced sensor sensitivity and signal attenuation. Weak signals are usually difficult to distinguish effectively from noise. The abrupt change signal is characterized by a sudden change in the signal in the time series, usually manifested as a sudden large change or a violent fluctuation in a short period of time.

[0103] Specifically, the step of determining the abnormal signal based on the determination parameters includes the following sub-steps:

[0104] S610: Acquire the signal value of the real-time pipeline signal; the signal value corresponds to the value collected by the sensor, such as the stress signal value being the stress value collected by the stress sensor.

[0105] S620: If the signal value of a preset number of adjacent signals in the real-time pipeline signal is 0 and the corresponding confidence level is less than the preset confidence level threshold, then the corresponding signal segment of the real-time pipeline signal is an invalid signal. The specific calculation formula is as follows.

[0106] ;

[0107] In the formula, The preset threshold for the number of adjacent signal points. This is the starting index of the signal segment to be determined in the real-time pipeline signal. For the first real-time pipeline signal The signal value of each signal point. For the first real-time pipeline signal The confidence level corresponding to each signal point To preset the reliability threshold, For this continuum The average confidence level of each signal point.

[0108] S630: Calculate the prediction variance of the predicted signal within the sliding window; the prediction variance is:

[0109] ;

[0110] In the formula, N is the total number of predicted signals within the sliding window; x iLet be the signal value of the i-th predicted signal within the sliding window; This is the average value of the predicted signal within the sliding window.

[0111] S640: Obtain the average signal energy of the real-time pipeline signal corresponding to the predicted signal within the sliding window; divide the predicted signal using the sliding window, and the average signal energy is:

[0112] ;

[0113] In the formula, X i Let be the signal energy value of the i-th predicted signal within the sliding window.

[0114] S650: If the prediction variance of the predicted signal is less than a preset variance threshold and the signal energy value is less than a preset energy threshold, then the real-time pipeline signal corresponding to the predicted signal within the sliding window is a weak signal.

[0115] S660: Calculate the absolute difference between adjacent residual values ​​in the residual sequence.

[0116] S670: Weighted calculations are performed using the absolute values ​​of the differences to obtain residual mutation characteristic values; the residual mutation characteristic values ​​are:

[0117] ;

[0118] In the formula, ζt is the residual abrupt change characteristic value of the t-th signal point. Let be the t-th residual value in the residual sequence. Let t be the (t-1)th adjacent residual value in the residual sequence, and ε be the preset transition threshold, where ε > 0.

[0119] S680: If the residual mutation feature value is greater than the preset jump threshold, then the real-time pipeline signal corresponding to the residual mutation feature value is a jump signal.

[0120] S700: Based on the mapping information, determine the spatial location of the pipeline corresponding to the abnormal signal.

[0121] Specifically, the step of determining the spatial location of the pipeline corresponding to the abnormal signal based on the mapping information includes the following sub-steps:

[0122] S710: Based on the mapping information, determine the single-point mileage coordinates corresponding to the abnormal signal. Through the mapping information, the single-point mileage coordinates of the signal point corresponding to each abnormal signal can be determined.

[0123] S720: If the single-point mileage coordinates exhibit a continuous distribution characteristic on the time axis, then the gated confidence mechanism of the target pipeline anomaly perception LSTM model is used to perform confidence-weighted smoothing fusion processing on the continuous abnormal signal segments to obtain the mileage range of the continuous abnormal signal segments. By fusing the continuous signal segments corresponding to the abnormal signals, users can more intuitively observe the signal segment to which the abnormal signal belongs.

[0124] S730: Based on the single-point mileage coordinates and the mileage range of the abnormal signal section, determine the spatial location of the pipeline corresponding to the abnormal signal.

[0125] The method further includes the following steps:

[0126] S800: Based on the type of the abnormal signal, the judgment parameters, and the spatial location of the pipeline corresponding to the abnormal signal, a pipeline inspection report is generated. The pipeline inspection report displays the mileage of the pipeline under test where the abnormal signal is located, the time index corresponding to the abnormal signal, and the confidence level corresponding to the abnormal signal. The pipeline inspection report is output in a structured data format for subsequent manual review, risk assessment, and pipeline maintenance decisions, realizing intelligent closed-loop processing of pipeline inspection data.

[0127] Specifically, the pipeline inspection report includes: abnormal signal types (invalid, weak, and abrupt signals); pipeline mileage and time index corresponding to abnormal sections; confidence, duration, and intensity statistics of various abnormal signals; and overall pipeline signal effectiveness analysis results and visualization charts.

[0128] This application provides a data analysis method based on an in-pipe detector, which has the following advantages:

[0129] 1. Intelligent and automated processing

[0130] Currently, the main approach relies on manual analysis or simple algorithms, which results in low processing efficiency and susceptibility to human factors. This is especially true for complex pipeline signals, where current anomaly detection methods struggle to achieve efficient and accurate anomaly identification.

[0131] This application achieves intelligent and automated signal analysis by introducing a pipeline anomaly sensing LSTM model (as shown in step S300 above). Multi-source data fusion: This application can effectively fuse data from various sensors, such as leakage magnetic field signals, eddy current signals, and stress signals, eliminating current shortcomings in signal fusion. Automatic anomaly identification: Through the pipeline anomaly sensing LSTM network and gated confidence mechanism, the system can automatically identify invalid signals, weak signals, and abrupt signal changes, reducing manual intervention and improving detection accuracy and efficiency.

[0132] 2. Targeted signal effectiveness analysis

[0133] Currently, it is difficult to effectively determine the validity of signals, especially for pipeline signals with characteristics such as high noise and local abrupt changes, leading to inaccurate data analysis. This application solves the problem of simultaneously processing the detailed and global features of signals by effectively extracting local peaks and trend components from leakage magnetic signals.

[0134] Signal Separation: Through multi-scale convolution, the pipeline anomaly perception LSTM model can process signals at different scales, ensuring the correct extraction of defect response features while suppressing unnecessary noise. Accurate Anomaly Detection: Combined with a gated confidence mechanism, this application can not only output predicted values ​​but also evaluate the validity of the signal, making the determination of anomaly signals more accurate.

[0135] 3. Accurate identification and location of abnormal signals

[0136] Currently, timely responses are often difficult when faced with subtle anomalies such as abrupt signal changes, especially for small and hidden anomalies in pipeline inspection. This application addresses this by using residual detection and training a pipeline anomaly perception LSTM model to calculate the difference between the predicted signal and the original signal (input signal) in real time, accurately identifying anomaly signals, particularly showing significant advantages in detecting abrupt and weak signals. Abrupt Signal Detection: When the residual of the signal suddenly increases, the system can quickly identify and locate the abrupt signal, ensuring the timeliness and accuracy of pipeline inspection. Location Accuracy: By using the time index and mileage mapping table output by the target pipeline anomaly perception LSTM model, this application can accurately locate the pipeline position corresponding to the anomaly signal.

[0137] 4. Improve the robustness and reliability of the model.

[0138] Current anomaly detection methods often rely on manually set thresholds or simple statistical methods, lacking flexibility and adaptability. This application utilizes a pipeline anomaly-aware LSTM training method, enabling the target pipeline anomaly-aware LSTM model to adaptively learn the dynamic characteristics of signals under different pipeline detection scenarios, thereby improving the system's robustness in various environments. Adaptive training: During the training phase, this application uses annotated normal and anomaly signal samples from historical pipeline signal data to learn the temporal dependencies and defect response patterns of the signals, thus improving the anomaly recognition capability in complex scenarios.

[0139] A second aspect of this application provides a data analysis system based on an in-pipe detector, comprising:

[0140] The first acquisition module is configured to acquire historical pipeline signals; the pipeline signals include: leakage magnetic field signals, eddy current signals, stress signals, and mileage wheel signals, wherein the mileage wheel signals include mapping information between the pipeline detection time index and the corresponding mileage; the mapping information represents the correspondence between the pipeline signal acquisition time and the mileage of the pipeline to be tested;

[0141] The preprocessing module is configured to perform preprocessing operations on the pipeline signal; the preprocessing operations include time synchronization processing, improved normalization processing, adaptive signal de-drift processing, defect feature-protected noise filtering, and multi-threshold local peak preservation processing, which are executed sequentially.

[0142] The training module is configured to train the pipeline anomaly perception LSTM model using the preprocessed historical pipeline signals to obtain the target pipeline anomaly perception LSTM model.

[0143] The second acquisition module is configured to acquire real-time pipeline signals and perform the preprocessing operation.

[0144] The input module is configured to input the preprocessed real-time pipeline signal into the target pipeline anomaly perception LSTM model to obtain the determination parameters of the real-time pipeline signal; the determination parameters include: predicted signal, residual sequence and confidence level;

[0145] The first determining module is configured to determine an abnormal signal based on the determination parameters; the abnormal signal includes: invalid signal, weak signal, and transition signal;

[0146] The second determining module is configured to determine the spatial location of the pipeline corresponding to the abnormal signal based on the mapping information.

[0147] It is worth noting that the effects of the above system embodiments can be found in the effects of the above method embodiments, and will not be repeated here.

[0148] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific embodiments of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A data analysis method based on an in-pipe detector, characterized in that, include: Historical pipeline signals are acquired; the pipeline signals include: leakage magnetic field signals, eddy current signals, stress signals, and mileage wheel signals; the mileage wheel signals include mapping information between the pipeline detection time index and the corresponding mileage; the mapping information represents the correspondence between the pipeline signal acquisition time and the mileage of the pipeline under test; The pipeline signal undergoes preprocessing operations, including time synchronization processing, improved normalization processing, adaptive signal drift removal processing, defect feature-protected noise filtering, and multi-threshold local peak preservation processing, performed sequentially. The improved normalization processing introduces a defect feature weighting factor, specifically formulated as follows: ； In the formula, The original pipeline signal value before preprocessing. This is the normalized pipe signal value; The mean of the original pipeline signal sequence is . The standard deviation of the original pipeline signal sequence; , These are the minimum and maximum values ​​of the original pipeline signal sequence, respectively. The weighting factor for enhancing defect features has a value range of 0.3 ≤ ≤0.6, calibrated through statistical analysis of historical defect signal samples; The pipeline anomaly perception LSTM model is trained using the preprocessed historical pipeline signals to obtain the target pipeline anomaly perception LSTM model; the pipeline anomaly perception LSTM model is an improvement on the LSTM network model; the pipeline anomaly perception LSTM model includes: an input layer, a hidden layer, and an output layer; The input layer of the pipeline anomaly sensing LSTM model integrates a preprocessing unit; the preprocessing unit is configured to separate the local peaks and trend components of the pipeline signal and extract the temporal features of the pipeline signal; the trend components include: the base interference signal generated by the material and wall thickness uniformity of the pipeline under test, the electromagnetic noise signal generated by the external magnetic field interference and the detector speed change, and the motion noise signal generated by the detector running speed change; The hidden layer of the pipeline anomaly perception LSTM model introduces a gated confidence mechanism, and the hidden layer is configured to output a prediction signal and confidence based on the time series features. The output layer of the pipeline anomaly perception LSTM model is equipped with a residual detection unit; the residual detection unit is configured to calculate the difference between the predicted signal and the corresponding pipeline signal to generate a residual sequence. Acquire real-time pipeline signals and perform the preprocessing operation; The preprocessed real-time pipeline signal is input into the target pipeline anomaly perception LSTM model to obtain the determination parameters of the real-time pipeline signal; the determination parameters include: predicted signal, residual sequence and confidence level; Based on the determination parameters, abnormal signals are determined; the abnormal signals include: invalid signals, weak signals, and abrupt change signals. Based on the mapping information, the spatial location of the pipeline corresponding to the abnormal signal is determined.

2. The data analysis method based on an in-pipe detector according to claim 1, characterized in that, The step of training the pipeline anomaly perception LSTM model using the preprocessed historical pipeline signal data to obtain the target pipeline anomaly perception LSTM model includes: The historical pipeline signals after the preprocessing operation are labeled to obtain a normal signal sample set and an abnormal signal sample set, and then divided into a training set, a validation set and a test set according to a preset ratio. The training set is input into the pipeline anomaly perception LSTM model for training, and the validation set is used to iteratively validate the pipeline anomaly perception LSTM model during the training process; the loss function of the model is composed of a weighted prediction error term and a confidence constraint term through adaptive weighted fusion; When the signal recognition accuracy of the pipeline anomaly perception LSTM model on the test set reaches a preset accuracy threshold and the loss function value converges to a preset loss threshold, training is stopped, and the target pipeline anomaly perception LSTM model is obtained.

3. The data analysis method based on an in-pipe detector according to claim 1, characterized in that, The step of determining the abnormal signal based on the determination parameters includes: Obtain the real-time signal value of the pipeline signal; If the signal value of a preset number of adjacent signals in the real-time pipeline signal is 0 and the corresponding confidence level is less than a preset confidence threshold, then the corresponding signal segment of the real-time pipeline signal is an invalid signal.

4. The data analysis method based on an in-pipe detector according to claim 1, characterized in that, The step of determining the abnormal signal based on the determination parameters includes: Calculate the prediction variance of the predicted signal within the sliding window; the prediction variance is: ； In the formula, N This represents the total number of predicted signals within the sliding window; x i For the first in the sliding window i The signal value of the predicted signal; The average value of the predicted signal within the sliding window; Obtain the average signal energy of the real-time pipeline signal corresponding to the predicted signal within the sliding window; If the prediction variance of the predicted signal is less than a preset variance threshold and the signal energy value is less than a preset energy threshold, then the real-time pipeline signal corresponding to the predicted signal within the sliding window is a weak signal.

5. The data analysis method based on an in-pipe detector according to claim 1, characterized in that, The step of determining the abnormal signal based on the determination parameters includes: Calculate the absolute value of the difference between adjacent residual values ​​in the residual sequence; The residual mutation characteristic value is obtained by weighting the absolute values ​​of the differences. If the residual mutation feature value is greater than the preset jump threshold, then the real-time pipeline signal corresponding to the residual mutation feature value is a jump signal.

6. The data analysis method based on an in-pipe detector according to claim 1, characterized in that, The step of determining the spatial location of the pipeline corresponding to the abnormal signal based on the mapping information includes: Based on the mapping information, determine the single-point mileage coordinates corresponding to the abnormal signal; If the single-point mileage coordinates show a continuous distribution characteristic on the time axis, then the gated confidence mechanism of the target pipeline anomaly perception LSTM model is used to perform confidence-weighted smoothing fusion processing on the continuous abnormal signal segments to obtain the mileage range of the continuous abnormal signal segments. Based on the single-point mileage coordinates and the mileage range of the abnormal signal section, the spatial location of the pipeline corresponding to the abnormal signal is determined.

7. The data analysis method based on an in-pipe detector according to claim 6, characterized in that, The method further includes: Based on the type of the abnormal signal, the judgment parameters, and the spatial location of the pipeline corresponding to the abnormal signal, a pipeline inspection report is generated; the pipeline inspection report is used to display the mileage of the abnormal signal in the pipeline under test, the time index corresponding to the abnormal signal, and the confidence level corresponding to the abnormal signal.

8. A data analysis system based on an in-pipe detector, characterized in that, include: The first acquisition module is configured to acquire historical pipeline signals; the pipeline signals include: leakage magnetic field signals, eddy current signals, stress signals, and mileage wheel signals, wherein the mileage wheel signals include mapping information between the pipeline detection time index and the corresponding mileage; the mapping information represents the correspondence between the pipeline signal acquisition time and the mileage of the pipeline to be tested; The preprocessing module is configured to perform preprocessing operations on the pipeline signal; the preprocessing operations include, sequentially, time synchronization processing, improved normalization processing, adaptive signal drift removal processing, defect feature-protected noise filtering, and multi-threshold local peak preservation processing; the improved normalization processing introduces a defect feature weighting factor, the specific formula of which is: ； In the formula, The original pipeline signal value before preprocessing. This is the normalized pipe signal value; The mean of the original pipeline signal sequence is . The standard deviation of the original pipeline signal sequence; , These are the minimum and maximum values ​​of the original pipeline signal sequence, respectively. The weighting factor for enhancing defect features has a value range of 0.3 ≤ ≤0.6, calibrated through statistical analysis of historical defect signal samples; The training module is configured to train the pipeline anomaly perception LSTM model using the preprocessed historical pipeline signals to obtain the target pipeline anomaly perception LSTM model; the pipeline anomaly perception LSTM model is an improvement on the LSTM network model; the pipeline anomaly perception LSTM model includes: an input layer, a hidden layer, and an output layer; The input layer of the pipeline anomaly sensing LSTM model integrates a preprocessing unit; the preprocessing unit is configured to separate the local peaks and trend components of the pipeline signal and extract the temporal features of the pipeline signal; the trend components include: the base interference signal generated by the material and wall thickness uniformity of the pipeline under test, the electromagnetic noise signal generated by the external magnetic field interference and the detector speed change, and the motion noise signal generated by the detector running speed change; The hidden layer of the pipeline anomaly perception LSTM model introduces a gated confidence mechanism, and the hidden layer is configured to output a prediction signal and confidence based on the time series features. The output layer of the pipeline anomaly perception LSTM model is equipped with a residual detection unit; the residual detection unit is configured to calculate the difference between the predicted signal and the corresponding pipeline signal to generate a residual sequence. The second acquisition module is configured to acquire real-time pipeline signals and perform the preprocessing operation. The input module is configured to input the preprocessed real-time pipeline signal into the target pipeline anomaly perception LSTM model to obtain the determination parameters of the real-time pipeline signal; the determination parameters include: predicted signal, residual sequence and confidence level; The first determining module is configured to determine an abnormal signal based on the determination parameters; the abnormal signal includes: invalid signal, weak signal, and transition signal; The second determining module is configured to determine the spatial location of the pipeline corresponding to the abnormal signal based on the mapping information.

Images