Pipeline multi-leakage-point detection method based on multi-modal fusion and Mamba architecture
By combining multimodal fusion and the Mamba architecture for pipeline multi-leak detection, and integrating acoustic and spatial features, this method uses cross-attention and the Mamba-SSM model to address the shortcomings in detection accuracy and real-time performance of traditional methods, achieving efficient leak detection and localization.
Patent Information
- Application Number
- CN202511637552.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-10
- Publication Date
- 2026-02-03
AI Technical Summary
Traditional pipeline leak detection methods suffer from low detection accuracy and poor real-time performance, making it difficult to meet the needs of complex environments. A single acoustic or vibration sensor cannot fully capture the multi-dimensional signal characteristics of the leak. Existing multi-sensor fusion methods have poor performance, and traditional RNNs suffer from the gradient vanishing problem. Transformers have high computational complexity, making it difficult to balance detection accuracy and real-time performance.
A multi-leak detection method for pipelines using multimodal fusion and Mamba architecture is proposed. By extracting MFCC features from dual-sensor acoustic signals and spatial topological features from CAD engineering drawings, and combining cross-attention multimodal fusion and Mamba-SSM model, deep fusion of acoustic and spatial features and long sequence modeling are achieved.
It significantly improves the accuracy and real-time performance of leak detection, enabling efficient identification of multiple leaks in complex environments and providing reliable support for smart water management.
Smart Images
Figure CN121456367A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the interdisciplinary field of intelligent sensing, deep learning, and industrial IoT, specifically a pipeline multi-leak detection method based on multimodal fusion and Mamba architecture. Background Technology
[0002] In the operation and maintenance of industrial pipelines, especially urban water supply networks, leak detection is a crucial link in ensuring stable network operation and reducing water waste. Traditional pipeline leak detection methods suffer from three major defects: low detection accuracy and poor real-time performance, making it difficult to meet the needs of complex environments; relying solely on a single acoustic or vibration sensor fails to comprehensively capture the multi-dimensional signal characteristics generated by leaks, making them susceptible to environmental noise interference and resulting in high false and false detection rates; existing multi-sensor fusion methods often employ simple feature splicing or weighted summation, failing to fully explore the correlation and complementarity between different modal data (such as acoustic signals and spatial structure information), leading to poor fusion results; and pipeline acoustic signals are typical long-term time-series data, where traditional RNNs (Recurrent Neural Networks) suffer from the vanishing gradient problem, and while Transformers can handle long dependencies, their computational complexity increases quadratically with the sequence length, making it difficult to balance detection accuracy and real-time performance.
[0003] Therefore, we propose a pipeline multi-leak detection method based on multimodal fusion and Mamba architecture. Summary of the Invention
[0004] The purpose of this invention is to provide a pipeline multi-leak detection method based on multimodal fusion and Mamba architecture, which solves the problems mentioned in the background art.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a pipeline multi-leakage detection method based on multimodal fusion and Mamba architecture, comprising the following method steps:
[0006] Step 1: Determine the feature fusion object: Extract the MFCC features of the dual-sensor acoustic signals and the spatial topology features of the CAD engineering drawing respectively. Perform Mel-frequency cepstral coefficient transformation on the dual-sensor acoustic signals to obtain MFCC features. Based on the ezdxf library, parse the CAD engineering drawing, and obtain the pipeline coordinates, sensor positions and network topology features through format recognition, geometric entity extraction, coordinate system establishment and topology and density feature construction.
[0007] Step 2, Dual-sensor acoustic feature fusion: The dual sensors collect acoustic signals at a preset sampling frequency. Abnormal events trigger high-frequency data reporting. The MFCC features are processed by sequence standardization. At time t, the dual-sensor feature fusion vector is constructed by feature splicing.
[0008] Step 3, Cross-Attention Multimodal Fusion: Using acoustic features as queries and spatial features as keys and values, heterogeneous modal interaction is achieved through multi-head parallel computation. The attention output is connected with the residual of the original acoustic features and layer-normalized. After splicing the outputs of multiple attention heads, the multimodal fusion features are projected through the output.
[0009] Step 4: Mamba-SSM model processing: Define and discretize the continuous-time state-space equation, dynamically generate model parameters based on the input, update the hidden state and the regression equation to generate prediction results, and send the results to the leak classification head, location regression head and confidence evaluation head to achieve multi-leak detection and location.
[0010] In a preferred embodiment of the present invention, the formula for the Mel-frequency cepstral coefficient transformation of the dual-sensor acoustic signal in step one is:
[0011] ;
[0012] in, Indicates the m-th frame. MFCC coefficient, This represents the output of the Mel filter bank, where K represents the number of Mel filters. This indicates the cepstral coefficient index.
[0013] In a preferred embodiment of the present invention, the preset sampling frequency in step two is 16kHz-44.1kHz. The acoustic signals acquired by the dual sensors include the time-domain waveform x_A(t) and sampling rate sr_A of sensor A, and the time-domain waveform x_B(t) and sampling rate sr_B of sensor B. The MFCC feature sequence normalization formula is:
[0014]
[0015] Where μMFCC is the characteristic mean of MFCC, σMFCC is the standard deviation, and ε is the numerical stability term.
[0016] In a preferred embodiment of the present invention, the formula for constructing the dual-sensor feature fusion vector in step two is as follows:
[0017] ,
[0018] FA_(t) and FB_(t) are the MFCC feature vectors of sensors A and B, respectively, and Concat is the feature concatenation operation.
[0019] In a preferred embodiment of the present invention, the key to the cross-attention multimodal fusion g in step three lies in calculating the cross-attention of acoustic and spatial features:
[0020]
[0021] in, For acoustic feature query matrix, Represents the spatial feature key value matrix. This represents the feature dimension. Here, the acoustic modality is used to query the spatial feature modality to generate a new feature representation that integrates the two types of information.
[0022] In a preferred embodiment of the present invention, the calculation method for residual connectivity and layer normalization in step three is as follows:
[0023]
[0024] in, This represents the fused multimodal features. These are the original acoustic features. This is the output of the cross-attention module;
[0025] The multi-attention head output splicing result is:
[0026]
[0027]
[0028] B is the batch size, L is the sequence length, and D is the total feature dimension;
[0029] The output projection formula is:
[0030] ;
[0031] To output the projection weight matrix, This is the output bias vector.
[0032] In a preferred embodiment of the present invention, the continuous-time state-space equation in step four is:
[0033]
[0034]
[0035] This indicates a hidden state. Indicates the fusion feature input, Indicates the output sequence. This represents the state-space parameter matrix.
[0036] In a preferred embodiment of the present invention, the formula for dynamically generating model parameters in step four is:
[0037]
[0038] The activation function ensures that the time step is positive, and the core formula for state updates is as follows:
[0039]
[0040]
[0041]
[0042] The model continuously receives updated and fused features x, and dynamically calculates the parameters through three linear layers. And based on dynamic Calculate the memory decay rate of the current step. ;
[0043] The formula for generating the prediction results is:
[0044] .
[0045] In a preferred embodiment of the present invention, when parsing the geometric entities of the CAD engineering drawing in step one, the line entity corresponding to the pipeline, the circle entity corresponding to the sensor position, and the point entity corresponding to the node are extracted; when constructing network topology features, the pipeline distribution density is determined based on the pipeline coordinates and node distribution, and the confidence factor of the leak point is adjusted according to the density.
[0046] In a preferred embodiment of the present invention, the Mamba-SSM model in step four processes long sequence dependencies with linear complexity, and the generated prediction results are used for collaborative training of leak detection, location regression and confidence prediction, supporting simultaneous detection and accurate localization of multiple leaks.
[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0048] This invention differs from traditional single-acoustic signal feature localization methods. This method incorporates engineering drawing features as input, providing auxiliary support for audio feature analysis from a spatial distribution perspective. It fuses multimodal information, enhancing the model's target localization capabilities in complex scenarios. Simultaneously, this method combines the Mamba model to analyze the fused multimodal features, significantly improving the accuracy of target detection and localization. This can greatly assist in leak detection in water conservancy systems under complex environments. This method achieves deep fusion of acoustic-spatial multimodal data and efficient processing of long sequence modeling, significantly improving the accuracy and real-time performance of multi-leak detection in water supply networks, providing reliable technical support for smart water management. Attached Figure Description
[0049] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0050] Figure 1 This is a system framework diagram of the pipeline multi-leak detection method based on multimodal fusion and Mamba architecture of the present invention. Detailed Implementation
[0051] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.
[0052] like Figure 1 As shown, the pipeline multi-leak detection method based on multimodal fusion and Mamba architecture processes the following hierarchical structure:
[0053] (1) Acoustic feature extraction layer: Perform MFCC (Mel frequency cepstral coefficients) transformation on the dual-sensor audio signal to extract time-frequency domain feature representation;
[0054] (2) Spatial feature coding layer: Extract spatial structural features such as pipe coordinates, sensor locations, and network topology from CAD engineering drawings;
[0055] (3) Cross-attention multimodal fusion layer: The adaptive weight allocation and deep information integration of acoustic features and spatial features are achieved through a multi-head cross-attention mechanism.
[0056] (4) The state-space model processing layer uses Mamba to process the output of multimodal fusion to obtain a high-confidence prediction vector, which is then fed into the leak prediction and localization head for further processing.
[0057] The specific implementation is as follows:
[0058] Step S1: Determine the feature fusion object
[0059] The audio signals transmitted from paired sensors are collected and MFCC features are extracted from them.
[0060] MFCC acoustic feature extraction and processing. Mel-frequency cepstral coefficient transform of dual-sensor acoustic signals:
[0061]
[0062] in, Indicates the m-th frame. MFCC coefficient, This represents the output of the Mel filter bank, where K represents the number of Mel filters and i represents the cepstral coefficient index. The output energy feature vector describes the energy distribution across different frequency bands of the sound, used to distinguish between leaks and normal water flow.
[0063] For CAD engineering drawings, relying on the ezdxf library, the system directly parses geometric entities to extract pipe information (line entities), sensor location information (circle entities), and node information (point entities). It also obtains normalized features and network topology features of the sensor and piping systems.
[0064] Detailed methods for feature processing of engineering drawings:
[0065] Design an engineering drawing feature extraction system. (1) First, perform format recognition on the input image to determine the feature processing method. (2) Perform geometric analysis on the image to obtain entity information of sensors, pipes, and nodes. (3) Establish a coordinate system based on the obtained entity information to convert the feature information on the engineering drawing into coordinate information, which facilitates subsequent fusion processing.
[0066] Considering the inconsistent pipeline distribution density in different regions, this method establishes network topology features, determines pipeline distribution density based on these features, and adjusts the confidence factor for leaks. Its key advantages are: 1. Multi-format support; 2. Geometric analysis, directly extracting precise coordinates from vector data; 3. Rich features, including multi-dimensional features of location, topology, density, and distance; 4. Data separation, strictly distinguishing between input features and training labels; 5. Standardization processing, normalizing to ensure consistency of feature scale.
[0067] Step S2: Fusion of acoustic features collected by the dual sensors.
[0068] (1) The dual sensors collect acoustic signals at a preset sampling frequency fs (e.g., 16kHz~44.1kHz). The data types include: Sensor A: time-domain waveform Sampling rate Sensor B: Time-domain waveform Sampling rate CAD coordinate mode: Pipeline coordinates Sensor position Network topology .
[0069] In response to abnormal events (such as sudden leaks or equipment failures), an event-triggered mechanism is used to immediately report high-frequency data.
[0070] (2) Perform sequence standardization on the MFCC features obtained in s1:
[0071]
[0072] Where μMFCC is the characteristic mean of MFCC, σMFCC is the standard deviation, and ε is the numerical stability term;
[0073] (3) Construct a dual-sensor feature fusion vector. For time t, the fused feature is represented as:
[0074]
[0075] in, Let A and B represent the MFCC feature vectors of sensors A and B respectively, and let Concat represent the feature concatenation operation.
[0076] Step s3: A cross-attention mechanism is used to perform multimodal fusion of the features obtained from the dual-sensor fusion and the extracted engineering drawing features. Cross-attention multimodal fusion is the core technological innovation of this invention, aiming to effectively integrate dual-sensor acoustic signals and CAD spatial topology information. This mechanism achieves adaptive weight allocation and depth information integration among heterogeneous modalities through multi-head attention, which, compared to traditional feature stitching or simple weighted fusion methods, can better capture the correlation and complementarity between modalities. This fusion process encompasses two main operations: cross-attention calculation and multimodal feature fusion, which are also the core of this method.
[0077] (1) The key to cross-attention multimodal fusion g lies in calculating the cross-attention of acoustic and spatial features:
[0078]
[0079] in, For acoustic feature query matrix, Represents the spatial feature key value matrix. This represents the feature dimension. Here, the acoustic modality is used to query the spatial feature modality to generate a new feature representation that integrates both types of information. However, the model does not allow the acoustic information to interact with the spatial information in only one way; instead, it constructs multiple parallel interaction heads.
[0080] The calculation process for one of the heads is as follows:
[0081]
[0082]
[0083] In this context, h obtains the information it needs to find in the spatial features from the acoustic features. In China, experts View spatial features It also generates an index label or keyword for each location or structure on the map. Chinese experts Re-examine the spatial features This involves extracting the actual information for each location or structure. This information will be used for the final feature synthesis. For multiple experts (…), Each of these elements will affect the sound characteristics. Generate a unique query Use this query to examine the map. Calculate an attention weight map for all positions on the graph. Finally, this weighted graph is used to selectively extract and fuse features from the map information, generating a highly contextualized and relevant output.
[0084] (2) Multimodal feature fusion output:
[0085] First, residual connections are performed on the original acoustic features and the model after cross-attention processing, i.e., they are summed and then normalized:
[0086]
[0087] in, This represents the fused multimodal features. These are the original acoustic features. This is the output of the cross-attention module. Presentation layer normalization operation;
[0088] Multi-head output splicing:
[0089]
[0090]
[0091] By concatenating the results of multiple expert head groups in the same dimension, the model obtains extremely rich feature information. When multiple expert heads give a high evaluation weight to a potential leak point, it can be determined with high accuracy whether it is a leak point.
[0092] Output projection:
[0093]
[0094] in:
[0095] The output projection weight matrix is used to remap the fused multi-head output back to the model's main feature space.
[0096] This provides additional degrees of freedom for the output bias vector, preventing the model from overfitting.
[0097] Step S4. Process the output of multimodal fusion using the Mamba-SSM state-space model.
[0098] (1) Mamba state-space modeling, defining the continuous-time state-space equations:
[0099]
[0100]
[0101] in, This indicates a hidden state. Indicates the fusion feature input, Indicates the output sequence. Represents the state-space parameter matrix. A zero-order hold is used to discretize the continuous system.
[0102]
[0103]
[0104] (2) The core innovation of the Mamba model is that it dynamically generates parameters based on the input, which can greatly optimize the weight allocation mechanism and reduce the memory allocation of the model on unnecessary feature information:
[0105]
[0106] That is, to dynamically update the parameters of the state-space equation based on the current input x.
[0107] (3) This method also uses The activation function ensures that the time step is positive, and the core formula for state updates is as follows:
[0108]
[0109]
[0110]
[0111] The model continuously receives updated and fused features x, and dynamically calculates the parameters through three linear layers. And based on dynamic Calculate the memory decay rate of the current step.
[0112] Final output prediction: The regression is performed to the state-space equation, and the prediction results are calculated accordingly.
[0113] Mamba transforms a fixed, linear state-space model into a dynamic, input-dependent, nonlinear system by introducing a selection mechanism. Instead of rigidly processing all information, it learns to selectively remember, forget, and output based on context. This allows it to achieve a level of processing long-range dependencies comparable to or even surpassing that of Transformers while maintaining the linear computational complexity of RNNs, making it ideal for handling time-series data such as acoustic signals.
[0114] The innovation of this invention, as demonstrated by the above process, lies in its differentiation from traditional single acoustic signal feature localization methods. This method introduces engineering drawing feature input, providing auxiliary support for audio feature analysis from a spatial distribution perspective, fusing multimodal information, and enhancing the model's target localization capabilities in complex scenarios. Simultaneously, this method combines the Mamba model to analyze the fused multimodal features, significantly improving the accuracy of target detection and localization, and providing substantial assistance for leak detection in water conservancy systems under complex environments. This method achieves deep fusion of acoustic-spatial multimodal data and efficient processing of long sequence modeling, significantly improving the accuracy and real-time performance of multi-leak detection in water supply networks, and providing reliable technical support for smart water management.
[0115] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or basic characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.
[0116] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. A pipeline multi-leak detection method based on multimodal fusion and Mamba architecture, characterized by: The methods and steps include the following: Step 1: Determine the feature fusion object: Extract the MFCC features of the dual-sensor acoustic signals and the spatial topology features of the CAD engineering drawing respectively. Perform Mel-frequency cepstral coefficient transformation on the dual-sensor acoustic signals to obtain MFCC features. Based on the ezdxf library, parse the CAD engineering drawing, and obtain the pipeline coordinates, sensor positions and network topology features through format recognition, geometric entity extraction, coordinate system establishment and topology and density feature construction. Step 2, Dual-sensor acoustic feature fusion: The dual sensors collect acoustic signals at a preset sampling frequency. Abnormal events trigger high-frequency data reporting. The MFCC features are processed by sequence standardization. At time t, the dual-sensor feature fusion vector is constructed by feature splicing. Step 3, Cross-Attention Multimodal Fusion: Using acoustic features as queries and spatial features as keys and values, heterogeneous modal interaction is achieved through multi-head parallel computation. The attention output is connected with the residual of the original acoustic features and layer-normalized. After splicing the outputs of multiple attention heads, the multimodal fusion features are projected through the output. Step 4: Mamba-SSM model processing: Define and discretize the continuous-time state-space equation, dynamically generate model parameters based on the input, update the hidden state and the regression equation to generate prediction results, and send the results to the leak classification head, location regression head and confidence evaluation head to achieve multi-leak detection and location.
2. The pipeline multi-leak detection method based on multimodal fusion and Mamba architecture according to claim 1, characterized in that: The formula for the Mel-frequency cepstral coefficient transformation of the acoustic signals from the dual sensors in step one is: ； in, Indicates the m-th frame. MFCC coefficient, This represents the output of the Mel filter bank, where K represents the number of Mel filters. This indicates the cepstral coefficient index.
3. The pipeline multi-leak detection method based on multimodal fusion and Mamba architecture according to claim 1, characterized in that: In step two, the preset sampling frequency is 16kHz-44.1kHz. The acoustic signals acquired by the dual sensors include the time-domain waveform x_A(t) and sampling rate sr_A of sensor A, and the time-domain waveform x_B(t) and sampling rate sr_B of sensor B. The MFCC feature sequence normalization formula is: Where μMFCC is the characteristic mean of MFCC, σMFCC is the standard deviation, and ε is the numerical stability term.
4. The pipeline multi-leak detection method based on multimodal fusion and Mamba architecture according to claim 1, characterized in that: The formula for constructing the dual-sensor feature fusion vector in step two is as follows: ， FA_(t) and FB_(t) are the MFCC feature vectors of sensors A and B, respectively, and Concat is the feature concatenation operation.
5. The pipeline multi-leak detection method based on multimodal fusion and Mamba architecture according to claim 1, characterized in that: In step three, the key to cross-attention multimodal fusion lies in calculating the cross-attention between acoustic and spatial features: in, For acoustic feature query matrix, Represents the spatial feature key value matrix. This represents the feature dimension. Here, the acoustic modality is used to query the spatial feature modality to generate a new feature representation that integrates the two types of information.
6. The pipeline multi-leak detection method based on multimodal fusion and Mamba architecture according to claim 1, characterized in that: The calculation method for residual connectivity and layer normalization in step three is as follows: in, This represents the fused multimodal features. These are the original acoustic features. This is the output of the cross-attention module; The multi-attention head output splicing result is: B is the batch size, L is the sequence length, and D is the total feature dimension; The output projection formula is: ； To output the projection weight matrix, This is the output bias vector.
7. The pipeline multi-leak detection method based on multimodal fusion and Mamba architecture according to claim 1, characterized in that: The continuous-time state-space equation in step four is: This indicates a hidden state. Indicates the fusion feature input, Indicates the output sequence. This represents the state-space parameter matrix.
8. The pipeline multi-leak detection method based on multimodal fusion and Mamba architecture according to claim 1, characterized in that: The formula for dynamically generating model parameters in step four is: The activation function ensures that the time step is positive, and the core formula for state updates is as follows: The model continuously receives updated and fused features x, and dynamically calculates the parameters through three linear layers. And based on dynamic Calculate the memory decay rate of the current step. ; The formula for generating the prediction results is: 。 9. The pipeline multi-leak detection method based on multimodal fusion and Mamba architecture according to claim 1, characterized in that: In step one, when parsing the geometric entities of the CAD engineering drawing, the line entity corresponding to the pipeline, the circle entity corresponding to the sensor location, and the point entity corresponding to the node are extracted. When constructing the network topology features, the pipeline distribution density is determined based on the pipeline coordinates and node distribution, and the confidence factor of the leak point is adjusted according to the density.
10. The pipeline multi-leak detection method based on multimodal fusion and Mamba architecture according to claim 1, characterized in that: In step four, the Mamba-SSM model handles long sequence dependencies with linear complexity. The generated prediction results are used for collaborative training of leak detection, location regression, and confidence prediction, supporting simultaneous detection and precise localization of multiple leaks.