A method for detecting leakage in ABR tubes based on multimodal data

By constructing a graph neural network model for multimodal data, the multimodal data of ABR pipelines is automatically collected and processed, solving the problem of low diagnostic accuracy caused by single-modal data in the existing technology, and achieving more accurate leakage detection.

CN117454309BActive Publication Date: 2026-11-17ZHENGZHOU UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311441144.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-11-01
Publication Date
2026-11-17
Estimated Expiration
2043-11-01

AI Technical Summary

Technical Problem

In existing technologies, ABR pipeline leakage detection methods mainly rely on single-modal data, resulting in low diagnostic accuracy and an inability to effectively combine multiple influencing factors. Therefore, there is an urgent need for a diagnostic method that integrates multimodal and multi-source data.

Method used

A leakage detection method for ABR pipes based on multimodal data is adopted. Multimodal data inside and outside the ABR pipe, including images, sound, stress, strain, flow, etc., are automatically collected and processed through a graph neural network model. A standardized dataset is constructed and trained using a graph attention mechanism to achieve intelligent judgment based on multimodal data.

Benefits of technology

It enables automatic acquisition and intelligent judgment of multimodal data of ABR pipelines, improves the accuracy of leakage detection, and can more accurately determine the probability of leakage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117454309B_ABST
    Figure CN117454309B_ABST
Patent Text Reader

Abstract

This invention discloses an ABR tube leakage detection method based on multimodal data, comprising: pre-acquiring multiple sets of multimodal data of ABR tubes and constructing multiple learning sample datasets and test datasets; constructing a graph neural network model with an attention mechanism, and training the graph neural network model using multiple learning sample datasets until convergence; acquiring multimodal data of the ABR tube to be tested and constructing a dataset as data input, using the graph neural network to calculate and output the leakage probability of the ABR tube. This invention enables automatic acquisition of multimodal data inside and outside the ABR tube and intelligent judgment of the leakage probability of the ABR tube using multimodal data, effectively improving the accuracy of the automatic identification results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of intelligent pipeline monitoring application technology, specifically relating to an ABR pipe leakage detection method based on multimodal data. Background Technology

[0002] Based on the blending of acrylic resin, polyvinyl chloride resin, and auxiliary materials such as heat stabilizers, ABR materials are generated through the strong intermolecular forces between the super-strong acrylic molecules and polyvinyl chloride molecules. They have the advantages of low temperature resistance, high impact resistance, high strength and toughness, and high external load capacity, and their application range is gradually becoming wider. However, the leakage problem they have is becoming increasingly difficult to ignore.

[0003] Existing methods for detecting pipeline leakage mainly employ acoustic methods, pressure methods, and hydraulic modeling, such as hydrophones, ground-penetrating radar, and infrared thermal imaging. These methods can only obtain data from one modality, and the hydraulic models and convolutional neural network models built upon them are all single-modal models. They consider only one factor and have low diagnostic accuracy. However, in actual engineering, pipeline leakage is often related to multiple influencing factors, requiring the combination of multimodal data such as images, sounds, and text to achieve a comprehensive diagnostic effect. Therefore, there is an urgent need to develop a diagnostic method for pipeline leakage using multimodal and multi-source data fusion, specifically for novel ABR (Automatic Boundary Theory) pipelines. Summary of the Invention

[0004] In view of this, the purpose of this invention is to propose an ABR tube leakage discrimination method based on multimodal data, so as to realize the automatic acquisition of multimodal data inside and outside the ABR tube and to make intelligent judgment on the leakage probability of the ABR tube using multimodal data.

[0005] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a method for ABR tube leakage detection based on multimodal data, including: Step S1: Pre-acquire multimodal data of multiple ABR tubes and construct multiple learning sample datasets and test datasets; Step S2: Construct a graph neural network model with an attention mechanism, and train the graph neural network model using multiple learning sample datasets until it converges; Step S3: Obtain multimodal data of the ABR tube under test and construct a dataset as data input. Use graph neural network to calculate and output the leakage probability of the ABR tube.

[0006] Preferably, step S1 includes: Multiple modal data of the ABR tube were obtained by pre-installed data acquisition devices inside and around the ABR tube. For any type of modal data, it is preprocessed according to preset processing rules based on its data format or the corresponding acquisition device type to obtain digital modal data; Data labels are constructed for each set of digital modal data, and standardized datasets are generated centrally. Determine the datasets corresponding to each of the multiple ABR tubes, and divide the multiple datasets according to the preset proportions to obtain multiple learning sample datasets and multiple test datasets.

[0007] Preferred, preset processing rules include: For image modal data, convolutional neural networks are used for modality transformation and feature extraction. In the convolutional neural network, the image is treated as a three-layer matrix and continuously convolutional operations are performed until the fully connected layer, so that the image is flattened from matrix form into a vector and digital modal data is obtained. For audio modal data, it is preprocessed by combining one or more of the following techniques: noise reduction, framing, and enhancement. Then, it is quantized and encoded into a digital signal, and then transformed into a frequency domain signal through Fourier transform to form a spectrum image. Finally, the spectrum image is transformed into digital modal data according to the processing rules of image modal data. For text modal data corresponding to stress, strain, and flow rate, after data cleaning, digital modal data is obtained by filtering according to preset filtering rules.

[0008] Preferably, constructing data labels for each set of digital modal data and centrally generating standardized datasets includes: Determine the original data type corresponding to any set of digital modal data, and construct data labels for the set of digital modal data based on the preset original data class-data label; Feature extraction is performed on each digital modality data according to the feature extraction method mapped by the data label, so as to obtain the feature data corresponding to each digital modality data and establish the mapping relationship between the feature data and the data label. The initial dataset is obtained by filling in each feature data according to the preset data table and its corresponding data labels; The feature data in the initial dataset are correlated to obtain a relational dataset, which is then output as a standardized dataset.

[0009] Preferably, step S2 includes: A graph neural network with a graph attention layer (GAT) is constructed, introducing the attention mechanism into the neighbor node aggregation process of the graph neural network. During neighbor node aggregation, the features of neighbor nodes are weighted and summed, where: The input to the graph attention layer (GAT) consists of the feature vector of each node. Composition, where N is the number of nodes and F is the number of features per node; The output of the graph attention layer GAT is set to the new feature vectors of these nodes. ; Using the graph attention layer GAT with weight matrix A linear transformation is applied to the features of each node, and the attention coefficients between the target node and its neighbors are calculated using an attention mechanism. in, The importance of the node features of neighboring node j of target node i to target node i is represented and used as the attention coefficient. For a feedforward neural network, Indicates to and The splicing process, Let be a non-linear activation function, where: In the formula, This represents the preset first type of correction parameter, which is set to 0.2 here; The attention coefficients of all neighboring nodes of any target node are normalized using the Softmax function to obtain the attention weights of each neighboring node. The normalization formula is as follows: in, Let represent the attention weight of the j-th neighbor node of the target node i. This represents the total number of neighboring nodes of the target node i; For a training sample dataset with multidimensional feature vectors, the attention weights of each neighbor node corresponding to the target node i are obtained by the following formula: in, This represents the multidimensional feature vectors in the learning sample dataset. This represents the weight matrix that performs a linear transformation on the features of each side; The graph attention layer (GAT) calculates a weighted sum of the features of neighboring nodes based on their attention weights, and uses the result to update the features of the target node. The calculation formula is as follows: In the formula, This represents a non-linear activation function, specifically expressed as: in, This represents the preset second type of correction parameter, which is set to 1 here; The graph neural network model is trained using multiple sample datasets and the model parameters are continuously adjusted. By continuously updating the weights until the model converges, the trained graph neural network model is obtained.

[0010] Preferably, multi-head attention computation is introduced to enhance the graph neural network model, specifically including: Set the number of attention heads to K, and calculate the attention weights and update the node features using K independently working attention mechanisms; The results from each attention mechanism are concatenated or averaged to obtain the final result.

[0011] The present invention has achieved at least the following beneficial effects: 1. To achieve automatic acquisition of multimodal data inside and outside the ABR tube and intelligent judgment of the leakage probability of the ABR tube using multimodal data; 2. Standardized processing of multimodal data has been implemented, enabling it to be directly used for training graph neural networks; 3. The generated dataset contains data that correlates the interrelationships between various data features under different modalities, effectively preserving the basis for judgment with synergy, and facilitating a more accurate determination of the leakage probability of ABR tubes using multimodal data.

[0012] Other advantages, objectives, and features of the invention will be set forth in the following description and will be apparent to those skilled in the art in some respects, or may be learned by practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0013] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following figures are provided for illustration: Figure 1 This is an overall flowchart of an ABR tube leakage detection method based on multimodal data in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the steps of an ABR tube leakage detection method based on multimodal data in an embodiment of the present invention. Detailed Implementation

[0014] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0015] This invention provides a method for ABR tube leakage detection based on multimodal data, referring to... Figure 1 ,include: Step S1: Pre-acquire multimodal data of multiple ABR tubes and construct multiple learning sample datasets and test datasets; Step S2: Construct a graph neural network model with an attention mechanism, and train the graph neural network model using multiple learning sample datasets until it converges; Step S3: Obtain multimodal data of the ABR tube under test and construct a dataset as data input. Use graph neural network to calculate and output the leakage probability of the ABR tube.

[0016] The working principle and beneficial effects of the above technical solution are as follows: Multiple sets of multimodal data of ABR pipes are pre-acquired, and multiple learning sample datasets and test datasets are constructed. The acquired multimodal data of ABR pipes includes a large number of ABR pipe samples with and without leaks. Various detection devices, such as hydrophones, ground-penetrating radar, pressure sensors, and various imaging devices, are used to collect data on various modes of ABR pipe, including images, stress, strain, flow rate, pressure, and sound wave propagation. This data is then processed into learning sample datasets and test sample datasets. A graph neural network model with an attention mechanism is constructed and trained using multiple learning sample datasets. The model is then validated using the test dataset, ultimately resulting in a trained graph neural network model. This invention is based on a pipeline robot equipped with a hydrophone and a high-definition camera to acquire sound and image data. Simultaneously, flow rate and stress / strain data are acquired through an electronic flow meter and pre-embedded optical fibers. Finally, a standardized dataset is constructed using the methods described above. Finally, the algorithm is used to train the dataset. After training, real-time monitoring and early warning effects can be achieved. By acquiring multimodal data of the ABR tube under test and constructing a dataset as input, a graph neural network is used to calculate and output the leakage probability of the ABR tube. This enables automatic acquisition of multimodal data inside and outside the ABR tube and intelligent judgment of the leakage probability of the ABR tube using multimodal data. The use of multimodal data can effectively improve the accuracy of automatic identification results.

[0017] In a preferred embodiment, step S1 includes: Multiple modal data of the ABR tube were obtained by data acquisition devices pre-installed inside and around the ABR tube. For any type of modal data, it is preprocessed according to preset processing rules based on its data format or the corresponding acquisition device type to obtain digital modal data; Data labels are constructed for each set of digital modal data, and standardized datasets are generated centrally. Determine the datasets corresponding to each of the multiple ABR tubes, and divide the multiple datasets according to the preset proportions to obtain multiple learning sample datasets and multiple test datasets.

[0018] The working principle and beneficial effects of the above technical solution are as follows: Multiple modal data of the ABR tube are obtained through data acquisition devices pre-installed inside and around the ABR tube; for any modal data, it is preprocessed according to its data format or the corresponding acquisition device type according to preset processing rules to obtain digital modal data, thus facilitating the training of graph neural network models using the acquired multimodal data; data labels are constructed for each group of digital modal data, and standardized datasets are generated centrally; the datasets corresponding to each of the multiple ABR tubes are determined, and the multiple datasets are divided according to preset proportions to obtain multiple learning sample datasets and multiple test datasets. This achieves standardized processing of multimodal data, enabling it to be directly used for training graph neural networks.

[0019] In a preferred embodiment, refer to Figure 2 The preset processing rules include: For image modal data, convolutional neural networks are used for modality transformation and feature extraction. In the convolutional neural network, the image is treated as a three-layer matrix and continuously convolutional operations are performed until the fully connected layer, so that the image is flattened from matrix form into a vector and digital modal data is obtained. For audio modal data, it is preprocessed by combining one or more of the following techniques: noise reduction, framing, and enhancement. Then, it is quantized and encoded into a digital signal, and then transformed into a frequency domain signal through Fourier transform to form a spectrum image. Finally, the spectrum image is transformed into digital modal data according to the processing rules of image modal data. For text modal data corresponding to stress, strain, and flow rate, after data cleaning, digital modal data is obtained by filtering according to preset filtering rules.

[0020] The working principle and beneficial effects of the above technical solution are as follows: In the process of processing multimodal data, for different modal data to be used for collaborative training of graph attention neural network models, modality transformation and feature extraction must be performed. Image and audio data are transformed and extracted into digital text modality using different methods. Specific modality transformation and feature extraction methods include: For image modality data, convolutional neural networks can be used for modality transformation and feature extraction. In the convolutional neural network, the image is first treated as a three-layer matrix and continuously convolved until the fully connected layer, where the image is... The matrix is ​​flattened into a vector to obtain the digital vector modal data required for subsequent training. For sound modal data, before modal conversion and feature extraction, a series of preprocessing operations (mainly including noise reduction, framing, enhancement, etc.) are performed. Then, it is quantized and encoded to convert it into a digital signal. After Fourier transform, it is converted into a frequency domain signal to form a spectrum image. Finally, it is converted into the digital modal data required for training by method (a). For text data such as stress, strain, and flow rate, no modal conversion is required. Only data cleaning is needed to select the required digital modal data.

[0021] In a preferred embodiment, constructing data labels for each set of digital modal data and centrally generating a standardized dataset includes: Determine the original data type corresponding to any set of digital modal data, and construct data labels for the set of digital modal data based on the preset original data class-data label; Feature extraction is performed on each digital modality data according to the feature extraction method mapped by the data label, so as to obtain the feature data corresponding to each digital modality data and establish the mapping relationship between the feature data and the data label. The initial dataset is obtained by filling in each feature data according to the preset data table and its corresponding data labels; The feature data in the initial dataset are correlated to obtain a relational dataset, which is then output as a standardized dataset.

[0022] The working principle and beneficial effects of the above technical solution are as follows: In the process of generating a standardized dataset, to preserve the coupling and interconnection relationships between multimodal data, the original data type corresponding to any set of digital modal data is determined, and data labels are constructed for this set of digital modal data based on preset original data classes and data labels. Features are extracted for each digital modal data according to the feature extraction method mapped by the data labels, obtaining the corresponding feature data for each digital modal data and establishing a mapping relationship between the feature data and the data labels. Based on a preset data table and its corresponding data labels, each feature data is filled in to obtain the initial dataset. Finally, the feature data in the initial dataset are correlated to obtain a relational dataset, which is then output as a standardized dataset. The dataset generated by this technical solution correlates the mutual influence relationships between various data features under different modalities, effectively preserving collaborative judgment criteria, and facilitating a more accurate determination of the leakage probability of ABR tubes using multimodal data.

[0023] In a preferred embodiment, refer to Figure 2 Step S2 includes: A graph neural network with a graph attention layer (GAT) is constructed, introducing the attention mechanism into the neighbor node aggregation process of the graph neural network. During neighbor node aggregation, the features of neighbor nodes are weighted and summed, where: The input to the graph attention layer (GAT) consists of the feature vector of each node. Composition, where N is the number of nodes and F is the number of features per node; The output of the graph attention layer GAT is set to the new feature vectors of these nodes. ; The graph attention layer (GAL) needs to transform the input features into higher-level features through a learnable linear transformation. Using the graph attention layer GAT with weight matrix A linear transformation is applied to the features of each node, and the attention coefficients between the target node and its neighbors are calculated using an attention mechanism. in, The importance of the node features of neighboring node j of target node i to target node i is represented and used as the attention coefficient. For a feedforward neural network, Indicates to and The splicing process, Let be a non-linear activation function, where: In the formula, This represents the preset first type of correction parameter, which is set to 0.2 here; To enable comparison of attention weights across different nodes, the Softmax function is used to normalize the attention coefficients of all neighboring nodes of any target node, yielding the attention weights for each neighboring node. The normalization formula is as follows: in, Let represent the attention weight of the j-th neighbor node of the target node i. This represents the total number of neighboring nodes of the target node i; For a training sample dataset with multidimensional feature vectors, the attention weights of each neighbor node corresponding to the target node i are obtained by the following formula: in, This represents the multidimensional feature vectors in the learning sample dataset. This represents the weight matrix that performs a linear transformation on the features of each side; The graph attention layer (GAT) calculates a weighted sum of the features of neighboring nodes based on their attention weights, and uses the result to update the features of the target node. The calculation formula is as follows: In the formula, This represents a non-linear activation function, specifically expressed as: in, This represents the preset second type of correction parameter, which is set to 1 here; The graph neural network model is trained using multiple sample datasets and the model parameters are continuously adjusted. By continuously updating the weights until the model converges, the trained graph neural network model is obtained.

[0024] The working principle and beneficial effects of the above technical solution are as follows: A graph attention neural network is selected, and the multimodal data features of the ABR pipeline are set as nodes in the graph neural model. Each node contains modality-transformed and standardized information such as sound, image, and flow rate (all stored in vector form), and a corresponding adjacency matrix is ​​constructed. By introducing the attention mechanism of the graph attention layer (GAT), important parts of the relational dataset, especially important parallel relationships, can be selectively focused on. The GAT introduces the attention mechanism into the neighbor node aggregation process of the graph neural network, which can visualize and mark the abnormal data locations corresponding to the image under the attention mechanism, including abnormally deformed pipelines in the image, abnormally fluctuating sound wave data in the sound data, and abnormal pipeline internal pressure data, etc. The weights of the features of each neighbor node are learned through the graph attention layer (GAL), so that the features of neighbor nodes are weighted and summed when aggregating neighbor nodes. Finally, training is performed to achieve a comprehensive prediction effect of leakage.

[0025] In a preferred embodiment, multi-head attention computation is introduced to enhance the graph neural network model, specifically including: Set the number of attention heads to K, and calculate the attention weights and update the node features using K independently working attention mechanisms; The results from each attention mechanism are concatenated or averaged to obtain the final result.

[0026] The working principle and beneficial effects of the above technical solution are as follows: The graph neural network model is enhanced by introducing multi-head attention computation. The number of attention heads is set to K, and attention weights are calculated and node features are updated by K independently operating attention mechanisms. The results obtained from each attention mechanism are concatenated or averaged to obtain the final result. The multi-head attention mechanism avoids the problem of excessive focus on one's own position when encoding information at the current position, thereby enhancing the expressive power of the graph neural network model.

[0027] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A method for detecting leakage in an ABR tube based on multimodal data, characterized in that, include: Step S1: Pre-acquire multimodal data of multiple ABR tubes and construct multiple learning sample datasets and test datasets; Step S2: Construct a graph neural network model with an attention mechanism, and train the graph neural network model using multiple learning sample datasets until it converges; Step S3: Obtain multimodal data of the ABR tube under test and construct a dataset as data input. Use graph neural network to calculate and output the leakage probability of the ABR tube. Step S1 includes: Multiple modal data of the ABR tube were obtained by data acquisition devices pre-installed inside and around the ABR tube. For any type of modal data, it is preprocessed according to preset processing rules based on its data format or the corresponding acquisition device type to obtain digital modal data; Data labels are constructed for each set of digital modal data, and standardized datasets are generated centrally. Determine the datasets corresponding to each of the multiple ABR tubes, and divide the multiple datasets according to the preset proportions to obtain multiple learning sample datasets and multiple test datasets. The preset processing rules include: For image modal data, a convolutional neural network is used to convert the image modal data into digital modal data and extract features. In the convolutional neural network, the image is treated as a three-channel matrix and continuously convolved until it reaches the fully connected layer, so that the image is flattened from matrix form into a vector and digital modal data is obtained. For audio modal data, it is preprocessed by combining one or more of the following techniques: noise reduction, framing, and enhancement. Then, it is quantized and encoded into a digital signal. The digital signal is then converted into a frequency domain signal to form a spectrum image through Fourier transform. Finally, the spectrum image is converted into digital modal data according to the processing rules of image modal data. For text modal data corresponding to stress, strain, and flow rate, after data cleaning, digital modal data is obtained by filtering according to preset filtering rules.

2. The ABR tube leakage detection method based on multimodal data according to claim 1, characterized in that, The process of constructing data labels for each set of digital modal data and centrally generating standardized datasets includes: Determine the original data type corresponding to any set of digital modal data, and construct data labels for the set of digital modal data based on the preset original data class-data label; Feature extraction is performed on each digital modality data according to the feature extraction method mapped by the data label, so as to obtain the feature data corresponding to each digital modality data and establish the mapping relationship between the feature data and the data label. The initial dataset is obtained by filling in each feature data according to the preset data table and its corresponding data labels; The feature data in the initial dataset are correlated to obtain a relational dataset, which is then output as a standardized dataset.

3. The ABR tube leakage detection method based on multimodal data according to claim 1, characterized in that, Step S2 includes: A graph neural network with a graph attention layer (GAT) is constructed, introducing the attention mechanism into the neighbor node aggregation process of the graph neural network. During neighbor node aggregation, the features of neighbor nodes are weighted and summed, where: The input to the graph attention layer (GAT) consists of the feature vector of each node. Composition, where N is the number of nodes and F is the number of features per node; The output of the graph attention layer (GAT) is defined as the new feature vectors of these nodes. ; Using the graph attention layer GAT with weight matrix A linear transformation is applied to the features of each node, and the attention coefficients between the target node and its neighbors are calculated using an attention mechanism. in, The importance of the node features of neighboring node j of target node i to target node i is represented and used as the attention coefficient. For a feedforward neural network, Indicates to and The splicing process, Let be a non-linear activation function, where: In the formula, This represents the preset first type of correction parameter, which is set to 0.2 here; The attention coefficients of all neighboring nodes of any target node are normalized using the Softmax function to obtain the attention weights of each neighboring node. The normalization formula is as follows: in, Let represent the attention weight of the j-th neighbor node of the target node i. This represents the total number of neighboring nodes of the target node i; For a training sample dataset with multidimensional feature vectors, the attention weights of each neighbor node corresponding to the target node i are obtained by the following formula: in, This represents the multidimensional feature vectors in the learning sample dataset. This represents the weight matrix that performs a linear transformation on the features of each side; The graph attention layer (GAT) calculates a weighted sum of the features of neighboring nodes based on their attention weights, and uses the result to update the features of the target node. The calculation formula is as follows: In the formula, This represents a non-linear activation function, specifically expressed as: in, This represents the preset second type of correction parameter, which is set to 1 here; The graph neural network model is trained using multiple sample datasets and the model parameters are continuously adjusted. By continuously updating the weights until the model converges, the trained graph neural network model is obtained.

4. The ABR tube leakage detection method based on multimodal data according to claim 3, characterized in that, It also includes introducing multi-head attention computation to enhance the graph neural network model, with specific steps including: The number of attention heads is pre-set to K, and attention weights are calculated and node features are updated through K independently working attention mechanisms; The results from each attention mechanism are concatenated or averaged to obtain the final result.

Citation Information

Patent Citations

  • Water supply pipe network leakage monitoring method

    CN109555979A

  • Water supply pipeline leakage detection method based on graph neural network

    CN115654381A