Pipeline leakage defect detection method based on attention enhancement converter

Abstract

The invention discloses a pipeline leakage defect detection method based on an attention enhancement converter, and aims to realize efficient and accurate detection of early-stage tiny leakage defects. Firstly, a high-sensitivity eddy current sensor is deployed in a pipeline monitoring area to collect original signals generated by defects of the inner wall; then, a frequency domain filtering algorithm is combined with a wavelet transform technology to remove environmental noise interference, and high-frequency weak characteristics caused by leakage defects are enhanced; and then, an optimized attention enhancement Transform model is utilized to process the preprocessed signal, a global dependency relationship in the preprocessed signal is captured, and extraction and modeling of key information are completed. And finally, constructing and training an off-line classification identification model containing normal state and different types of defect signals to realize accurate identification and classification of early-stage tiny leakage defects of the pipeline.

Description

Technical Field

[0001] This invention relates to the field of nondestructive testing technology, and in particular to a method for detecting pipeline leakage defects based on attention enhancement converters. Background Technology

[0002] Industrial pipelines are crucial infrastructure in industries such as petroleum, chemical, and natural gas. Their safety and reliability are paramount for preventing leaks and ensuring public safety. However, with increasing service life and the influence of the external environment, defects such as corrosion and cracks may appear on the inner walls of pipelines. If these defects are not detected and addressed in a timely manner, they can lead to serious leaks, causing economic losses and environmental pollution.

[0003] Traditional pipeline inspection methods mainly include non-destructive testing techniques such as ultrasonic testing, radiographic testing, and magnetic particle testing. These methods can identify defects inside or on the surface of pipelines to a certain extent, but they also have some limitations:

[0004] Ultrasonic testing: Although it can effectively test metal pipes, it is not effective for non-metallic materials or pipes with complex structures, and it requires the assistance of a coupling agent, making it inconvenient to operate.

[0005] X-ray inspection: It can provide high-resolution images, but the equipment is expensive, the operation is complicated, and it also poses potential risks to human health.

[0006] Magnetic particle testing: applicable to ferromagnetic materials, but not to non-ferromagnetic materials; and can only detect surface or near-surface defects, unable to meet the needs of deep inspection.

[0007] In recent years, eddy current testing, as an emerging non-destructive testing technology, has been widely used in industrial pipeline inspection due to its advantages such as not requiring direct contact with the object being tested and its sensitivity to surface and subsurface defects. However, the interpretation of eddy current test signals often relies on the experience of professionals, which limits its application scope and efficiency.

[0008] Meanwhile, the development of deep learning technology has provided new ideas for solving this problem. By training neural network models to automatically identify feature information in eddy current detection signals, detection accuracy and speed can be greatly improved, while reducing the influence of human factors.

[0009] Therefore, this invention aims to combine the advantages of eddy current detection technology and deep learning algorithms to propose an efficient and accurate method and system for identifying leakage defects in industrial pipelines, so as to make up for the shortcomings of existing technologies and improve the safety monitoring level of industrial pipelines. Summary of the Invention

[0010] To address the above problems, this invention proposes a pipeline leakage defect detection method based on an attention-enhanced converter. The specific steps are as follows:

[0011] Step 1: Construct an eddy current detection signal acquisition system, deploy eddy current detection sensors in the industrial pipeline monitoring area, and collect the original eddy current signals generated by defects on the inner wall of the pipeline.

[0012] Step 2: Use frequency domain filtering algorithm to remove environmental noise interference, and combine wavelet transform technology to enhance the high-frequency weak features caused by leakage defects.

[0013] Step 3: Build the core neural network model of the attention enhancement transformer. Based on the Transformer framework, optimize the weight allocation strategy of the multi-head self-attention layer, focus on strengthening the attention enhancement mechanism, and enable the model to more accurately capture the global dependencies in the eddy current time series signal, and complete the preliminary modeling and key information extraction of the pipeline defect signal.

[0014] Step 4: Establish an offline classification and identification model for pipeline leakage defects, construct a validation dataset containing normal signals and different types of defect signals, train the model to ensure that the model can effectively distinguish between normal pipeline conditions and early leakage defects; input the feature signals after model optimization into the trained classification decision module to achieve accurate offline identification of early minor pipeline leakage defects, providing a basis for defect diagnosis for pipeline safe operation and maintenance.

[0015] This invention relates to a pipeline leakage defect detection method based on an attention-enhanced converter, which has beneficial effects. The technical advantages of this invention are as follows:

[0016] 1. This invention uses deep learning algorithms to analyze and process eddy current detection signals, effectively identifying more subtle and complex defect features, and achieving higher accuracy and reliability compared to traditional methods. This helps in the early detection of potential safety hazards and prevents major accidents.

[0017] 2. This invention can be applied not only to the inspection of metallic pipes, but also to non-metallic or other composite material pipes, thus broadening its application scope. Furthermore, this technology is unaffected by the complexity of the pipe shape and is suitable for pipes of various types and sizes.

[0018] 3. This invention, by employing non-contact eddy current detection technology, avoids the potential health risks associated with traditional X-ray detection methods. Furthermore, the application of deep learning models makes the detection results more objective and impartial, improving the overall system's safety performance. Attached Figure Description

[0019] Figure 1 This is a flowchart of the present invention;

[0020] Figure 2 The flowchart shows the optimized weight allocation strategy of this invention. Detailed Implementation

[0021] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0022] This invention discloses a pipeline leakage defect detection method based on an attention-enhanced converter. The method first uses eddy current detection technology to acquire defect signals on the pipeline surface, and then uses a trained deep learning model to analyze and process these signals to accurately identify various types of defects. Compared with traditional detection methods, this invention not only improves detection accuracy and efficiency but also enhances the system's adaptability and safety, making it suitable for pipelines of various materials and sizes. The invention flowchart is shown below. Figure 1 As shown, the steps of the present invention will be described in detail below:

[0023] Step 1: Construct an eddy current detection signal acquisition system, deploy eddy current detection sensors in the industrial pipeline monitoring area, and collect the original eddy current signals generated by defects on the inner wall of the pipeline.

[0024] The eddy current detection signal acquisition system consists of an eddy current sensor, a signal exciter, a data acquisition card, and an industrial computer. The eddy current sensor detects eddy current changes caused by defects in the inner wall of the pipe. The signal exciter provides a stable high-frequency alternating current to the sensor. The data acquisition card converts the sensor's analog signal into a digital signal. The industrial computer controls the acquisition process, stores raw data, and processes the signal. The sensor is placed close to the outer wall of the pipe, with a gap of less than 0.1 mm. The exciter is activated, with the excitation signal set to a 5kHz excitation frequency and a 50mA current. The data acquisition card's sampling frequency is set to 20kHz, and the acquired signal is transmitted back to the industrial computer.

[0025] Step 2: Use frequency domain filtering algorithm to remove environmental noise interference, and combine wavelet transform technology to enhance the high-frequency weak features caused by leakage defects.

[0026] Step 2.1 Frequency Domain Filtering Algorithm

[0027] Step 2.1.1 Raw signals from each sensor Perform an N-point Discrete Fourier Transform to convert the time-domain signal into a frequency-domain signal. This reveals the frequency distribution of the signal, and the formula is:

[0028]

[0029] in, Let k be the voltage value of the i-th sensor at the n-th sampling time, k be the frequency index, j be the imaginary unit, and N be the number of sampling points. Let be the complex amplitude of the i-th sensor signal at the k-th frequency point. Let be the actual frequency at the k-th frequency point. for:

[0030]

[0031] in, This refers to the sensor sampling frequency.

[0032] Step 2.1.2 Design a bandpass filter

[0033] Based on the signal frequency characteristics, an ideal bandpass filter H(k) is designed to retain only the defective signal frequency band from 1 to 10 kHz. The filter coefficients are defined as follows:

[0034]

[0035] in, This is the upper limit for low-frequency noise. The upper limit of the defect signal; when A value of 1 indicates that the signal at that frequency point is retained, while a value of 0 indicates that noise at that frequency point is filtered out. The corresponding frequency point index range is:

[0036]

[0037] in, and The corresponding frequency point index range is N, where N is the number of sampling points.

[0038] Step 2.1.3 Frequency Domain Filtering and Time Domain Reconstruction

[0039] Frequency domain filtering: filtering frequency domain signals With filter coefficients Multiply to obtain the filtered frequency domain signal. :

[0040]

[0041] Temporal reconstruction: Perform an N-point inverse discrete Fourier transform to convert the frequency domain signal back to the time domain, obtaining the denoised signal. :

[0042]

[0043] Step 2.1.4 Wavelet transform to enhance high-frequency weak defect features

[0044] Denoising signal In the early stages, the signals of minute leak defects are still weak high-frequency signals, which are difficult to identify in the time domain. Wavelet transform can focus simultaneously in the time and frequency domains, enhancing the high-frequency defect characteristics. The steps are as follows:

[0045] For the denoised signal Perform J-level wavelet decomposition (J=5 levels) to obtain approximation coefficients for the low-frequency signal and detail coefficients for the high-frequency signal.

[0046] Approximation coefficient The formula for the low-frequency trend of the corresponding signal is:

[0047]

[0048] in, The scaling function coefficients of the db4 wavelet, j=1,2,...,J are the decomposition levels, and n and k are the coefficient indices;

[0049] Detail factor The high-frequency components of the corresponding signal, the defect signal is mainly in Layer, formula is:

[0050]

[0051] Where g(n) are the wavelet function coefficients of the db4 wavelet, derived from h(n):

[0052]

[0053] Step 2.2: High-frequency detail coefficient thresholding

[0054] Step 2.2.1 Calculate the threshold based on the noise standard deviation: The noise is mainly concentrated in the detail coefficients of the highest decomposition layer 5. In, its standard deviation The estimate is:

[0055]

[0056] Where median is the median, and 0.6745 is the conversion factor between the median and standard deviation under a normal distribution; final threshold , This represents the number of sampling points.

[0057] Step 2.2.2 Soft Threshold Function: For Layer detail factor Soft thresholding is applied to obtain the enhanced detail coefficients. :

[0058]

[0059] in, For a sign function, when When >0), it is 1; when When <0, it is -1; when hour, =0; when At the same time, defect signals are preserved and transitions are smoothed.

[0060] Step 2.3: Wavelet Reconstruction

[0061] Use the processed high-frequency detail coefficients , and unprocessed low-frequency approximation coefficients Wavelet reconstruction is performed with j=5 to obtain the final high-frequency feature enhancement signal. The formula is:

[0062]

[0063] in, Let J be the scaling function for layer J, where J is the highest decomposition layer and k is the coefficient index. Let be the wavelet function of layer j, corresponding to the enhanced high-frequency defect features, where j is the decomposition layer number and n is the time variable.

[0064] Step 3: Build the core neural network model of the attention enhancement transformer. Based on the Transformer framework, optimize the weight allocation strategy of the multi-head self-attention layer, focus on strengthening the attention enhancement mechanism, and enable the model to more accurately capture the global dependencies in the eddy current time series signal, and complete the preliminary modeling and key information extraction of the pipeline defect signal.

[0065] The attention-enhanced transformer is based on the standard Transformer encoder framework. It achieves the extraction of global features of time-series signals through a stacked structure of multi-head self-attention, feedforward network, residual connection, and layer normalization.

[0066] Step 3.1 Configure the Transformer encoder framework

[0067] Step 3.1.1 Transformer encoder parameters

[0068] For eddy current timing signals, the time step Multi-sensor channels M=10, set the adapted model parameters:

[0069] The encoder has 6 layers (L), with 1-2 shallow layers extracting local features and 3-6 deep layers capturing global dependencies; model dimensions are 6. Set to 256 dimensions; set the number of multi-head attention heads H to 8 heads, with each head having 32 dimensions. Feedforward network hidden layer dimensions. Set to 1024 dimensions; set the activation function to GELU; set the dropout probability to 0.1.

[0070] Step 3.1.2 Basic Encoder Single-Layer Structure

[0071] The standard Transformer encoder's single-layer structure follows the following process: layer normalization, multi-head self-attention, residual connection, layer normalization, feedforward network, residual connection.

[0072] Let the input of the l-th layer be l=1,2,...,L, T is the number of time steps. For model dimensions, then single-layer output The mathematical expression is:

[0073] Normalization formula for attention layer input:

[0074] Multi-head self-attention calculation formula:

[0075] Residual connectivity calculation formula:

[0076] Normalized calculation formula for feedforward network input:

[0077] Feedforward network calculation formula:

[0078] Residual connectivity calculation formula:

[0079] in, The high-frequency feature enhancement signal output from step 2 is the output of layer (l-1) and the input of layer l. Input for the first layer ; This represents the normalized input features of the c-th attention layer, and the subscript norm1 indicates the normalization of the first layer; This represents the multi-head self-attention output of layer l; This represents the output of the feedforward network at layer l; This represents the feature matrix after the multi-head self-attention output of the l-th layer is connected via residuals; This represents the normalized input features of the l-th layer feedforward network; This represents the final output feature matrix of the l-th layer encoder; This is a layer normalization function to eliminate feature distribution bias; For the self-attention of the bulls; It is a feedforward network that realizes nonlinear feature transformation.

[0080] Step 3.2: Multi-head self-attention layer optimization

[0081] Standard multi-head self-attention uses an average weighting of the eight attention heads, while introducing a learnable head weight allocation strategy to enhance the contribution of effective heads, suppress redundant heads, and improve the accuracy of global dependency capture.

[0082] Step 3.2.1 QKV decomposition and attention score of standard multi-head self-attention

[0083] QKV matrix generation: This involves generating the normalized input... Through 3 independent linear layers, the weight matrix , , Generate a matrix of query Q, key K, and value V:

[0084]

[0085] Multi-head partitioning: Q, K, and V are partitioned into H low-dimensional submatrices based on the number of heads H, with each head having the following dimension:

[0086]

[0087] Where h=1,2,...,H are the head indices.

[0088] Single-head attention score calculation: Calculate the attention score for each head, then combine it with... Weighted single-head attention output:

[0089]

[0090] in, , , This represents the query matrix Q, key K, and value V for the h-th head. As a scaling factor, avoid If the value is too large, the inner product will be too high, and the gradient will disappear after softmax. for The transpose of .

[0091] Step 3.2.2 Optimize the weight allocation strategy

[0092] The flowchart for optimizing the weight allocation strategy is as follows: Figure 2 As shown, based on standard multi-head self-attention, a head weight vector is introduced. Output for each head The weighted average is calculated using the following steps and formulas:

[0093] Step 3.2.2.1 Head weight normalization: The head weights are normalized using the softmax function. Normalized to weight coefficients :

[0094]

[0095] in, The weight coefficient for the h-th head is... ; The original weights of the h-th attention head are given.

[0096] Step 3.2.2.2 Weighted Multi-Head Attention Output: The output of each head... With the corresponding Multiply and then sum to get the weighted attention output:

[0097]

[0098] Step 3.2.2.3 Final attention layer output: By output linear layer This yields the optimized multi-head self-attention output, which is the output in step 3.1.2. :

[0099]

[0100] in, For the weighted bullish attention output, This is the output linear layer weight matrix of the attention layer.

[0101] Step 4: Establish an offline classification and identification model for pipeline leakage defects, construct a validation dataset containing normal signals and different types of defect signals, train the model to ensure that the model can effectively distinguish between normal pipeline conditions and early leakage defects; input the feature signals after model optimization into the trained classification decision module to achieve accurate offline identification of early minor pipeline leakage defects, providing a basis for defect diagnosis for pipeline safe operation and maintenance.

[0102] The module adopts an architecture of global average pooling, fully connected layers, Dropout, and Softmax, and is adapted in conjunction with the Transformer encoder.

[0103] Step 4.1 Feature Compression Layer: This layer compresses the temporal features output from the last layer of the Transformer, with dimensions T× Where T is the time step, global average pooling is performed to transform the temporal features into a fixed-dimensional global feature vector, as shown in the formula:

[0104]

[0105] in, The output feature of the Transformer at time step t has a dimension of 256. The compressed global features have a dimension of 256, eliminating the influence of temporal length on the classification layer.

[0106] Step 4.2 Fully Connected Layer and Regularization:

[0107] First layer fully connected: The 256-dimensional feature space is mapped to a 512-dimensional feature space; the activation function is GELU, consistent with step 3.1.1, to ensure gradient continuity, and the formula is:

[0108]

[0109] in, The weight matrix has dimensions 256×512; The bias vector has a dimension of 512. These are the features after the first fully connected layer and GELU activation.

[0110] Dropout layer: Add Dropout with a probability of 0.1 after the first fully connected layer to suppress overfitting. The formula is as follows:

[0111]

[0112] in, This is the first layer of features after Dropout processing.

[0113] Second-layer fully connected: The 512-dimensional map is mapped to the number of categories C, where C=4 corresponds to the number of categories. The formula is as follows:

[0114]

[0115] in, The weight matrix has dimensions 512×4; The bias vector has a dimension of 4. These are the category features after the second fully connected layer.

[0116] Step 4.3 Probability Output Layer: The Softmax function is used to... Transform the results into probability distributions for each category to quantify the classification results. The formula is:

[0117]

[0118] Where P(c) is the probability that a sample belongs to class c, c=1 is normal, c=2 is crack, c=3 is hole, c=4 is corrosion pit, and satisfies ∑P(c)=1. This is the original output of class c corresponding to the second fully connected layer.

[0119] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A pipeline leakage defect detection method based on attention enhancement converter, the specific steps of which are as follows, characterized in that: Step 1: Construct an eddy current detection signal acquisition system, deploy eddy current detection sensors in the industrial pipeline monitoring area, and collect the original eddy current signals generated by defects on the inner wall of the pipeline; Step 2: Use frequency domain filtering algorithm to remove environmental noise interference, and combine wavelet transform technology to enhance the weak high-frequency features caused by leakage defects; Step 3: Build the core neural network model of the attention enhancement transformer. Based on the Transformer framework, optimize the weight allocation strategy of the multi-head self-attention layer to enable the model to more accurately capture the global dependencies in the eddy current time series signal and complete the preliminary modeling and key information extraction of the pipeline defect signal. Step 4: Establish an offline classification and identification model for pipeline leakage defects, construct a validation dataset containing normal signals and different types of defect signals, train the model to ensure that the model can effectively distinguish between normal pipeline conditions and early leakage defects; input the feature signals after model optimization into the trained classification decision module to achieve accurate offline identification of early minor pipeline leakage defects, providing a basis for defect diagnosis for pipeline safe operation and maintenance.

2. The attention-augmented transformer-based pipe leak defect detection method of claim 1, wherein: The eddy current detection signal acquisition system constructed in step 1 can be represented as follows: The eddy current detection signal acquisition system consists of an eddy current sensor, a signal exciter, a data acquisition card, and an industrial computer. The eddy current sensor detects eddy current changes caused by defects in the inner wall of the pipe. The signal exciter provides a stable high-frequency alternating current to the sensor. The data acquisition card converts the sensor's analog signal into a digital signal. The industrial computer controls the acquisition process, stores the raw data, and processes the signal. The sensor is placed close to the outer wall of the pipe with a gap of less than 0.1 mm. The exciter is activated, and the excitation signal is set to a 5kHz excitation frequency and a 50mA current. The data acquisition card's sampling frequency is set to 20kHz, and the acquired signal is transmitted back to the industrial computer.

3. The attention-augmented transformer-based pipe leak defect detection method of claim 1, wherein: Step 2 is represented as follows: Step 2.1 Frequency Domain Filtering Algorithm; Step 2.1.1 Raw signal for each sensor Discrete Fourier Transform of N points to convert time domain signal to frequency domain signal Reveals the frequency distribution of the signal, formula: ； where k is the frequency point index, j is the imaginary unit, is the complex amplitude of the ith sensor signal at the kth frequency point, and the actual frequency of the kth frequency point is: ； Step 2.1.2: Design a bandpass filter; Based on the signal frequency characteristics, an ideal bandpass filter H(k) is designed to retain only the defective signal frequency band from 1 to 10 kHz. The filter coefficients are defined as follows: ； wherein, is the upper limit of low frequency noise, is the upper limit of defect signal; when is 1, it means that the signal of this frequency point is reserved, and is 0, it means that the noise of this frequency point is filtered out; the corresponding frequency point index range is: ； wherein, and is a corresponding frequency point index range, is a sampling frequency, and N is a sampling point number. Step 2.1.3 Frequency domain filtering and time domain reconstruction; Frequency domain filtering: filtering frequency domain signals With filter coefficients Multiply to obtain the filtered frequency domain signal. : ； Temporal reconstruction: Perform an N-point inverse discrete Fourier transform to convert the frequency domain signal back to the time domain, obtaining the denoised signal. : ； Step 2.1.4 Wavelet transform to enhance high-frequency weak defect features; Denoising signal In the early stages, the signals from minute leaks are still weak high-frequency signals, which are difficult to identify in the time domain; wavelet transform can focus simultaneously in the time and frequency domains, and the steps are as follows: For the denoised signal Perform J-level wavelet decomposition (J=5 levels) to obtain approximation coefficients for the low-frequency signal and detail coefficients for the high-frequency signal. Approximation coefficient The formula for the low-frequency trend of the corresponding signal is: ； in, The scaling function coefficients of the db4 wavelet, j=1,2,...,J are the decomposition levels, and k is the coefficient index; Detail factor The high-frequency components of the corresponding signal, the defect signal is mainly in Layer, formula is: ； Where g(n) are the wavelet function coefficients of the db4 wavelet, derived from h(n): ； Step 2.2: High-frequency detail coefficient thresholding; Step 2.2.1 Calculate the threshold based on the noise standard deviation: The noise is mainly concentrated in the detail coefficients of the highest decomposition layer 5. In, its standard deviation The estimate is: ； Where median is the median, and 0.6745 is the conversion factor between the median and standard deviation under a normal distribution; final threshold , This represents the number of sampling points; Step 2.2.2 Soft Threshold Function: For Layer detail factor Soft thresholding is applied to obtain the enhanced detail coefficients. : ； in, For a sign function, when When >0), it is 1; when When <0, it is -1; when hour, =0; when At the same time, retain defect signals and smooth transitions; Step 2.3: Wavelet reconstruction; Use the processed high-frequency detail coefficients , and unprocessed low-frequency approximation coefficients Wavelet reconstruction is performed with j=5 to obtain the final high-frequency feature enhancement signal. The formula is: ； in, This is the scaling function for layer J, corresponding to the low-frequency background. Let be the wavelet function of layer j, corresponding to the enhanced high-frequency defect features.

4. The pipeline leakage defect detection method based on attention enhancement converter according to claim 1, characterized in that: The core neural network model for building the attention-enhancing transformer in step 3 is represented as follows: Step 3.1 Set up the Transformer encoder framework; Step 3.1.1 Transformer encoder parameters; For eddy current timing signals, the time step Multi-sensor channels M=10, set the adapted model parameters: The encoder has 6 layers (L), with 1-2 shallow layers for extracting local features and 3-6 deep layers for capturing global dependencies. Model Dimension The dimensions are set to 256; the number of multi-head attention heads H is set to 8, and each head has 32 dimensions; the dimensions of the feedforward network hidden layers are... Set to 1024 dimensions; set the activation function to GELU; set the dropout probability to 0.

1. Step 3.1.2 Basic encoder single-layer structure; The standard Transformer encoder's single-layer structure follows this process: layer normalization, multi-head self-attention, residual connection, layer normalization, feedforward network, residual connection; Let the input of the l-th layer be l=1,2,...,L, T is the number of time steps. For model dimensions, then single-layer output The mathematical expression is: Normalization formula for attention layer input: ; Multi-head self-attention calculation formula: ; Residual connectivity calculation formula: ; Normalized calculation formula for feedforward network input: ; Feedforward network calculation formula: ; Residual connectivity calculation formula: ; in, This is a layer normalization function to eliminate feature distribution bias; For the self-attention of the bulls; It is a feedforward network that enables nonlinear feature transformation; Step 3.2: Multi-head self-attention layer optimization; Standard multi-head self-attention uses average weighting for the eight attention heads, while introducing a learnable head weight allocation strategy to enhance the contribution of effective heads, suppress redundant heads, and improve the accuracy of global dependency capture. Step 3.2.1 QKV decomposition and attention score of standard multi-head self-attention; QKV matrix generation: This involves generating the normalized input... Through 3 independent linear layers, the weight matrix , , Generate a matrix of query Q, key K, and value V: ； Multi-head partitioning: Q, K, and V are partitioned into H low-dimensional submatrices based on the number of heads H, with each head having the following dimension: ； Where h=1,2,...,H are the head indices; Single-head attention score calculation: Calculate the attention score for each head, then combine it with... Weighted single-head attention output: ； in, As a scale factor, avoid If the value is too large, the inner product will be too high, and the gradient will disappear after softmax. for Transpose of; Step 3.2.2 Optimize the weight allocation strategy; Based on standard multi-head self-attention, a head weight vector is introduced. Output for each head The weighted average is calculated using the following steps and formulas: Step 3.2.2.1 Head weight normalization: The head weights are normalized using the softmax function. Normalized to weight coefficients : ； in, The weight coefficient for the h-th head is... ; Step 3.2.2.2 Weighted Multi-Head Attention Output: The output of each head... With the corresponding Multiply and then sum to get the weighted attention output: ； Step 3.2.2.3 Final attention layer output: By output linear layer This yields the optimized multi-head self-attention output, which is the output in step 3.1.

2. : 。 5. The pipeline leakage defect detection method based on attention enhancement converter according to claim 1, characterized in that: The offline classification and identification model for pipeline leakage defects established in step 4 is represented as follows: The module adopts an architecture of global average pooling, fully connected layers, Dropout, and Softmax, and is adapted in conjunction with the Transformer encoder. Step 4.1 Feature Compression Layer: This layer compresses the temporal features output from the last layer of the Transformer, with dimensions T× Where T is the time step, global average pooling is performed to transform the temporal features into a fixed-dimensional global feature vector, as shown in the formula:

6. Among them, The output feature of the Transformer at time step t has a dimension of 256. The compressed global features have a dimension of 256, eliminating the influence of temporal length on the classification layer; Step 4.2 Fully Connected Layer and Regularization: First layer fully connected: The 256-dimensional feature space is mapped to a 512-dimensional feature space; the activation function is GELU, consistent with step 3.1.1, to ensure gradient continuity, and the formula is:

7. Among them, The weight matrix has dimensions 256×512; The bias vector dimension is 512; Dropout layer: Add Dropout with a probability of 0.1 after the first fully connected layer to suppress overfitting. The formula is as follows:

8. Second-layer fully connected: ... The 512-dimensional map is mapped to the number of categories C, where C=4 corresponds to the number of categories. The formula is as follows:

9. Among them, The weight matrix has dimensions 512×4; The bias vector has a dimension of 4. Step 4.3 Probability Output Layer: The Softmax function is used to... Transform the results into probability distributions for each category to quantify the classification results. The formula is:

10. Among them, P(c) is the probability that a sample belongs to class c, where c=1 is normal, c=2 is crack, c=3 is hole, and c=4 is corrosion pit, and satisfies ∑P(c)=1.

Images