Pipeline leakage detection method and device based on multi-modal feature fusion

Abstract

The application provides a pipeline leakage detection method based on multi-modal feature fusion, and relates to the field of pipeline leakage detection. The pipeline leakage detection method based on multi-modal feature fusion comprises the following steps: acquiring an acoustic signal in a pipeline, determining an acoustic fingerprint energy spectrum in the pipeline based on the acoustic signal; acquiring a thermal image in the pipeline, determining a thermal diffusion gradient in the pipeline based on the thermal image; acquiring a pressure signal in the pipeline, determining a fluctuation entropy in the pipeline based on the pressure signal; using a confidence function, respectively mapping the acoustic fingerprint energy spectrum, the thermal diffusion gradient and the fluctuation entropy into confidence values; fusing the confidence values corresponding to the acoustic fingerprint energy spectrum, the confidence values corresponding to the thermal diffusion gradient and the confidence values corresponding to the fluctuation entropy, and determining a detection result of whether the pipeline leaks. Through the above method, the false positive rate of pipeline leakage detection is reduced.

Description

Technical Field

[0001] This application relates to the field of pipeline leak detection, and in particular to a pipeline leak detection method and apparatus based on multimodal feature fusion. Background Technology

[0002] As the scale of various pipelines, such as heating pipelines, gas pipelines, and water supply pipelines, continues to expand and their service life increases year by year, pipeline leaks occur frequently, causing numerous losses to life and property.

[0003] Pipeline leak detection technologies are susceptible to environmental noise interference, cannot detect minute leaks, and have limited model feature dimensions. Furthermore, these technologies suffer from insufficient feature representation, deficiencies in confidence modeling, and low decision robustness. For example, the neglect of fluid physical properties in these technologies renders the models almost uninterpretable, and the fusion of conflicting decision data frequently fails, resulting in a high false alarm rate for pipeline leak detection. Summary of the Invention

[0004] Therefore, it is necessary to provide a pipeline leak detection method based on multimodal feature fusion that can reduce the false alarm rate of pipeline leak detection, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a pipeline leakage detection method based on multimodal feature fusion, which includes:

[0006] Acoustic signals within the pipeline are acquired, and the acoustic energy spectrum within the pipeline is determined based on these signals. A thermal image of the pipeline is acquired, and the thermal diffusion gradient within the pipeline is determined based on this image. Pressure signals within the pipeline are acquired, and the fluctuation entropy within the pipeline is determined based on these signals. A confidence function is used to map the acoustic energy spectrum, thermal diffusion gradient, and fluctuation entropy into confidence values ​​representing the probability of pipeline leakage. The confidence values ​​corresponding to the acoustic energy spectrum, thermal diffusion gradient, and fluctuation entropy are then fused to determine the detection result of whether the pipeline is leaking.

[0007] In some implementations, the acoustic signature energy spectrum within the pipe is determined based on the acoustic signals within the pipe, including:

[0008] A convolutional neural network is used to extract the acoustic temporal features of the acoustic signal inside the pipe; the convolutional neural network includes three convolutional layers and two pooling layers; the acoustic energy spectrum inside the pipe is determined based on the acoustic temporal features.

[0009] In some implementations, the acoustic time-domain features are multi-dimensional feature vectors; determining the acoustic energy spectrum within the pipe based on the acoustic time-domain features includes: summing the multi-dimensional feature vectors to obtain the acoustic energy spectrum within the pipe.

[0010] In some implementations, the thermal diffusion gradient within the pipe is determined based on thermal images within the pipe, including:

[0011] A residual neural network is used to extract image features of the thermal image inside the pipe; the image features are then input into a gradient decoder network to obtain the thermal diffusion gradient inside the pipe; the gradient decoder network is a fully connected neural network.

[0012] In some implementations, image features are input into a gradient decoder network to obtain the thermal diffusion gradient within the pipe, including:

[0013] The true gradient label is determined based on infrared image samples and operators; the predicted gradient is determined based on infrared image samples; the loss function is determined based on the true gradient label and the predicted gradient, and the gradient decoder network is trained using the loss function; the image features are input into the trained gradient decoder network to obtain the thermal diffusion gradient inside the pipe.

[0014] In some implementations, the fluctuation entropy within the pipeline is determined based on the pressure signal within the pipeline, including:

[0015] A predetermined number of pressure signal sampling points within the pipeline are input into a bidirectional long short-term memory network, and the sigmoid function in the bidirectional long short-term memory network is used to output the fluctuation entropy within the pipeline.

[0016] In some implementations, the confidence function is: ;

[0017] in, Indicates the confidence level value. This represents the acoustic signature energy spectrum, thermal diffusion gradient, or wave entropy. This represents the historical average of normal data. This represents the scale parameter, which adaptively adjusts to environmental noise. Indicates shape parameters.

[0018] In some implementations, the confidence values ​​corresponding to the acoustic signature energy spectrum, the thermal diffusion gradient, and the fluctuation entropy are fused to determine the detection results of whether a pipeline is leaking. These include:

[0019] The detection result is obtained by fusing the confidence values ​​corresponding to the acoustic energy spectrum, the thermal diffusion gradient, and the fluctuation entropy using the following fusion method:

[0020] ;

[0021] in, Indicates the test results. Indicates when This represents the confidence level when considering the voiceprint energy spectrum. Indicates when This represents the confidence level for the thermal diffusion gradient. Indicates when The confidence level value is the value for the fluctuation entropy.

[0022] Secondly, this application provides a pipeline leak detection device based on multimodal feature fusion. The pipeline leak detection device includes a processing module, a mapping module, and a fusion module. The processing module is used to acquire acoustic signals inside the pipeline and determine the acoustic energy spectrum inside the pipeline based on the acoustic signals inside the pipeline; it is also used to acquire thermal images inside the pipeline and determine the thermal diffusion gradient inside the pipeline based on the thermal images inside the pipeline; it is also used to acquire pressure signals inside the pipeline and determine the fluctuation entropy inside the pipeline based on the pressure signals inside the pipeline. The mapping module uses a confidence function to map the acoustic energy spectrum, thermal diffusion gradient, and fluctuation entropy into confidence values ​​representing the probability of pipeline leakage, respectively. The fusion module is used to fuse the confidence values ​​corresponding to the acoustic energy spectrum, the thermal diffusion gradient, and the fluctuation entropy to determine the judgment result of pipeline leakage.

[0023] Thirdly, this application provides an electronic device including a memory and a processor. The memory is used to store a computer program, and the processor is used to execute the computer program. When the computer program is executed by the processor, it implements the steps of any of the above-mentioned pipeline leakage detection methods based on multimodal feature fusion.

[0024] The pipeline leak detection method based on multimodal feature fusion provided in this application determines the acoustic energy spectrum of the pipeline by acquiring acoustic signals inside the pipeline; determines the thermal diffusion gradient of the pipeline by acquiring thermal images inside the pipeline; and determines the fluctuation entropy of the pipeline by acquiring pressure signals inside the pipeline. The confidence values ​​of the acoustic energy spectrum, thermal diffusion gradient, and fluctuation entropy mapping are obtained respectively, and the confidence values ​​of the three are fused. Based on multimodal data, pipeline leaks are detected, thereby reducing the false alarm rate of pipeline leak detection. Attached Figure Description

[0025] Figure 1 Flowcharts for pipeline leak detection methods based on multimodal feature fusion in some implementations;

[0026] Figure 2 A flowchart for determining the acoustic signature energy spectrum within a pipe in some implementation methods;

[0027] Figure 3A first flowchart for determining the thermal diffusion gradient within a pipe in some implementation methods;

[0028] Figure 4 A second flowchart for determining the thermal diffusion gradient within a pipe in some implementation methods;

[0029] Figure 5 A flowchart for determining the fluctuation entropy within a pipeline in some implementation methods;

[0030] Figure 6 This is a schematic diagram of a pipeline leak detection device based on multimodal feature fusion in some implementation methods;

[0031] Figure 7 This is a schematic diagram of an electronic device in some implementation methods. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] It should be noted that the terms "first," "second," and similar terms used in this application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. "A plurality" or "several" indicates at least two. "Comprising" or "including" and similar terms mean that the elements or objects preceding "comprising" or "including" encompass the elements or objects listed following "comprising" or "including" and their equivalents, and do not exclude other elements or objects. "Connected" or "linked" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0034] like Figure 1 As shown, this application provides a pipeline leak detection method based on multimodal feature fusion, which is applied to pipeline leak detection. The pipeline leak detection method based on multimodal feature fusion includes the following steps:

[0035] Step 101: Acquire the acoustic signal inside the pipe, and determine the acoustic energy spectrum inside the pipe based on the acoustic signal inside the pipe.

[0036] Among them, the acoustic energy spectrum is the energy distribution map of each frequency component of the acoustic signal in the frequency domain, which can characterize the characteristics of the acoustic signal in the pipe.

[0037] In some implementations, a sound pressure detection module is installed inside the pipe, which can acquire acoustic signals inside the pipe.

[0038] For example, the sound pressure detection module is a sound pressure sensor, such as a fiber optic acoustic wave sensor or a capacitive sound pressure sensor. The acoustic signal acquired by the sound pressure detection module is a mechanical wave.

[0039] Step 102: Obtain a thermal image inside the pipe and determine the thermal diffusion gradient inside the pipe based on the thermal image.

[0040] Among them, the thermal image is an image of the temperature distribution inside the reaction pipe.

[0041] In some implementations, an image acquisition module is installed inside the pipe, which can acquire thermal images of the pipe.

[0042] For example, the image acquisition module is an infrared thermal imager, which obtains thermal images inside the pipe by acquiring infrared radiation signals.

[0043] Step 103: Obtain the pressure signal inside the pipeline, and determine the fluctuation entropy inside the pipeline based on the pressure signal inside the pipeline.

[0044] Among them, fluctuation entropy is a characteristic index used to quantify the fluctuation complexity or uncertainty of time series data such as pressure / flow rate in pipelines, and its value ranges from 0 to 1.

[0045] In some implementations, a pressure detection module is installed inside the pipeline, which can acquire the pressure signal inside the pipeline.

[0046] For example, the pressure detection module is a pressure sensor, such as a strain gauge pressure sensor or a piezoresistive pressure sensor, which can acquire pressure signals inside the pipeline.

[0047] Step 104: Using the confidence function, map the acoustic energy spectrum, thermal diffusion gradient, and fluctuation entropy into confidence values ​​that represent the probability of pipeline leakage.

[0048] It should be noted that the prediction results of pipeline leakage represented by acoustic energy spectrum, thermal diffusion gradient and fluctuation entropy may conflict. For example, the pipeline leakage probability predicted by acoustic energy spectrum is higher than that predicted by thermal diffusion gradient. In this case, the pipeline leakage probability cannot be determined. Therefore, a confidence value is introduced. The confidence value is a numerical indicator to quantify the reliability of the pipeline leakage probability prediction results.

[0049] In some implementations, the confidence function satisfies the following relationship:

[0050] ;

[0051] in, Indicates the confidence level value. This represents the acoustic signature energy spectrum, thermal diffusion gradient, or wave entropy. This represents the historical average of normal data. This represents the scale parameter, which adaptively adjusts to environmental noise. This represents the shape parameter, which has high sensitivity to both sudden and slow leaks.

[0052] The scale parameters satisfy the following relationship:

[0053] ;

[0054] in, This is the environmental noise weighting coefficient. The environmental noise standard deviation is the value when there is no pipe leakage. standard deviation The maximum historical deviation of the feature satisfies the following relationship:

[0055] ;

[0056] In the implementation of this application, =0.6.

[0057] The shape parameters satisfy the following relationship:

[0058] ;

[0059] in, These are the basic shape parameters of the medium inside the pipeline. If the medium inside the pipeline is water or oil, then... 1.5, If the medium in the pipeline is gas, then 2.0; To adjust the amplitude, when the pressure P in the pipeline is less than 1 MPa, =0.5, when the pressure P in the pipeline is greater than or equal to 1 MPa. =1; As the response time constant, when the viscosity of the medium inside the pipe, η, is less than 10 cP, =0.1, when the viscosity of the medium in the pipeline η≥10cP, =0.05.

[0060] Step 105: Combine the confidence values ​​corresponding to the acoustic energy spectrum, the thermal diffusion gradient, and the fluctuation entropy to determine the detection result of whether the pipeline is leaking.

[0061] By fusing the confidence values ​​corresponding to acoustic energy spectrum, thermal diffusion gradient, and fluctuation entropy, the influence of multimodal data can be comprehensively considered, thereby improving the accuracy of pipeline leak detection results.

[0062] The pipeline leak detection method based on multimodal feature fusion provided in this application determines the acoustic energy spectrum of the pipeline by acquiring acoustic signals inside the pipeline; determines the thermal diffusion gradient of the pipeline by acquiring thermal images inside the pipeline; and determines the fluctuation entropy of the pipeline by acquiring pressure signals inside the pipeline. The confidence values ​​of the acoustic energy spectrum, thermal diffusion gradient, and fluctuation entropy mapping are obtained respectively, and the corresponding confidence values ​​of the three are fused. The pipeline leak is detected by multimodal means, thereby reducing the false alarm rate of pipeline leak detection.

[0063] like Figure 2 As shown, in some implementations, step 101, determining the acoustic energy spectrum within the pipe based on the acoustic signals within the pipe, specifically includes the following steps:

[0064] Step 201: Use a convolutional neural network to extract the acoustic temporal features of the acoustic signal inside the pipe.

[0065] Among them, the acoustic time-domain features are multi-dimensional feature vectors.

[0066] In the implementation of this application, the convolutional neural network adopts a 1D-CNN network, that is, a one-dimensional convolutional neural network, which includes an input layer, three convolutional layers, two pooling layers, a flattening layer and an output layer.

[0067] Specifically, as shown in Table 1, the structural parameters of the 1D-CNN network in the implementation of this application are as follows: Conv1D (One-Dimensional Convolutional Layer) can process one-dimensional sequence data to extract local temporal features; MaxPool (Maximum Pooling Layer) reduces the dimensionality of features and retains the most significant features by taking the maximum value of a local region; Flatten can convert multi-dimensional feature maps into one-dimensional feature vectors; the output layer can map the flattened one-dimensional feature vectors to the category space and output the final feature values; the ReLU activation function can introduce nonlinearity, enabling the convolutional neural network to fit complex nonlinear relationships; and the Linear activation function can retain the original information of the input data, ensuring that the input data and output data of the layer are consistent.

[0068] Table 1 1D-CNN Network Structure

[0069]

[0070] It should be noted that the input data of the 1D-CNN network is the acoustic signal acquired inside the pipe, and the output data of the 1D-CNN network is the acoustic temporal features (such as the pulse waveform unique to a pipe leak). In the implementation of this application, the acoustic temporal features are 256-dimensional vectors.

[0071] Through the above steps, three convolutional layers are used to progressively abstract features, ensuring that the convolutional neural network can capture the essence of the data. Two pooling layers are used to optimize computational efficiency, ensuring that the convolutional neural network can run in real time, thereby achieving a balance between the feature abstraction capability and real-time performance of the convolutional neural network.

[0072] Step 202: Determine the acoustic energy spectrum within the pipe based on acoustic time-domain characteristics.

[0073] In some implementations, the acoustic energy spectrum within the pipe is obtained by summing up the multi-dimensional feature vectors.

[0074] The voiceprint energy spectrum satisfies the following relationship:

[0075] ;

[0076] in, Voiceprint energy spectrum, Let be the i-th feature vector output by the 1D-CNN network.

[0077] It should be noted that there is always environmental noise inside the pipeline (such as the sound of the medium flowing inside the pipeline). When the pipeline leaks, it has a unique leakage acoustic signal, which has extremely low energy compared to the environmental noise.

[0078] Through the above steps, the 1D-CNN network obtains acoustic time-domain features based on the acoustic signals inside the pipeline, and calculates the acoustic energy spectrum inside the pipeline based on the acoustic time-domain features. It can capture relatively weak leakage acoustic signals unique to leaks in the ever-present environmental noise, improve the ability to capture acoustic features of pipeline leaks, and reduce the false alarm rate of pipeline leak detection.

[0079] like Figure 3 As shown, in some implementations, step 102, determining the thermal diffusion gradient within the pipe based on the thermal image inside the pipe, specifically includes the following steps:

[0080] Step 301: Use a residual neural network to extract image features of the thermal image inside the pipe.

[0081] In some implementations, a ResNet-18 residual neural network is used to extract image features of thermal images inside the pipe. ResNet-18 contains 18 weighted layers (such as convolutional layers, pooling layers, etc.).

[0082] Specifically, the thermal image inside the pipe is input into ResNet-18, and ResNet-18 ultimately outputs the following 1024-dimensional image features:

[0083] F t =[ f 1 ,f 2 ,...,f 1024 ] ;

[0084] in, F t For image features, f i This is the i-th feature vector output by ResNet-18.

[0085] Step 302: Input the image features into the gradient decoder network to obtain the thermal diffusion gradient inside the pipe.

[0086] The gradient decoder network is a fully connected neural network.

[0087] In some implementations, the gradient decoder network takes 1024-dimensional image features as input and outputs 2-dimensional feature values ​​without activation functions.

[0088] like Figure 4 As shown, in some implementations, step 302 involves inputting image features into a gradient decoder network to obtain the thermal diffusion gradient within the pipe, specifically including the following steps:

[0089] Step 401: Determine the true gradient label based on infrared image samples and using operators.

[0090] In some implementations, infrared image sample I∈ * (H=240, W=320), the Sobel operator is used to determine the true gradient labels.

[0091] The Sobel operator includes the Sobel gradient operator in the x-direction and the Sobel gradient operator in the y-direction. The horizontal and vertical gradients are calculated using the Sobel gradient operators in the x-direction and y-direction, and the edge strength is determined by the gradient magnitude.

[0092] Specifically, the horizontal gradient satisfies the following relationship:

[0093] G x true =I S x ;

[0094] Among them, G x true For the horizontal gradient, S x The Sobel gradient operator in the x-direction can highlight vertical edge features in infrared image samples (such as the edges of pipe valves). x The following relationship must be satisfied:

[0095] S x = [[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]],

[0096] The vertical gradient satisfies the following relationship:

[0097] G y true =I S y ;

[0098] Among them, G y true For the vertical gradient, S y The Sobel gradient operator in the y-direction highlights horizontal edge features in infrared image samples (such as transverse welds on pipes). y The following relationship must be satisfied:

[0099] S y = S x = [[-1, -2, -1], [0, 0, 0], [1, 2, 1]],

[0100] The gradient magnitude satisfies the following relationship:

[0101] || T‖ true =sqrt(S x 2 +S y 2 );

[0102] in,‖ T‖ true Let S be the gradient magnitude, sqrt(S) x 2 +S y 2 ) is for (S x 2 +S y 2 Square root operation.

[0103] Step 402: Determine the prediction gradient based on infrared image samples.

[0104] The prediction gradient includes the prediction gradient G in the x-direction. x pred The predicted gradient G in the y-direction y pred And predict gradient magnitude‖ T‖ pred .

[0105] Step 403: Determine the loss function based on the true gradient label and the predicted gradient, and train the gradient decoder network using the loss function.

[0106] In some implementations, the loss function determined based on the true gradient label and the predicted gradient satisfies the following relationship:

[0107] L = a·|G x pred -G x true |+b·|G y pred -G y true |+c·|‖ T‖ pred -‖ T‖ true |;

[0108] Where L is the loss value, a is the horizontal gradient weight, b is the vertical gradient weight, and c is the gradient magnitude weight. By minimizing the loss value L, the gradient decoder network can be trained, thereby improving its predictive ability.

[0109] In the implementation of this application, the horizontal gradient weight a is 0.4, the vertical gradient weight b is 0.4, and the gradient magnitude weight c is 0.2.

[0110] Step 404: Input the image features into the trained gradient decoder network to obtain the thermal diffusion gradient inside the pipe.

[0111] The thermal diffusion gradient satisfies the following relationship:

[0112] || T‖=sqrt(G x 2 +G y 2 );

[0113] in,‖ T‖ is the thermal diffusion gradient inside the pipe, G x and G y G represents the output value of the gradient decoder network. x G represents the thermal diffusion gradient component in the x-direction. yThis represents the thermal diffusion gradient component in the y-direction.

[0114] Through the above steps, a gradient decoder network is designed and trained using the Sobel operator. This allows the image features input to the gradient decoder network to be mapped into a thermal diffusion gradient that conforms to thermodynamic laws, eliminating interference from environmental heat sources (such as sunlight) and reducing the false alarm rate of pipeline leak detection.

[0115] like Figure 5 As shown, in some implementations, step 103, determining the fluctuation entropy within the pipeline based on the pressure signal within the pipeline, specifically includes the following steps:

[0116] Step 501: Input a preset number of pressure signal sampling points continuously within the pipeline into a bidirectional long and short memory network.

[0117] The bidirectional Long Short-Term Memory (LSS) network comprises a forward LSTM and a backward LSTM. The forward LSTM processes the input sequence in chronological order to capture historical dependencies, while the backward LSTM processes the input sequence in reverse chronological order to capture future dependencies. Through this bidirectional processing, both chronologically and in reverse chronologically, the bidirectional LSSM network can capture the causal relationships among the data in the input sequence and avoid data loss, thus improving its ability to handle long-range dependencies.

[0118] Step 502: Output the fluctuation entropy within the pipeline using the sigmoid function in the bidirectional long short-term memory network.

[0119] The sigmoid function can map the pressure characteristic values ​​of pressure signal sampling points to the (0,1) interval, that is, map them to the fluctuation entropy.

[0120] In the implementation of this application, when training the bidirectional long short-term memory network, the fluctuation entropy within the pipe satisfies the following relationship:

[0121] F p = ∑p i logp i ;

[0122] Among them, F p Let p be the entropy of the fluctuation within the pipe. i Let be the probability density of the pressure characteristic value of the pressure signal sampling point in the interval i.

[0123] By employing the above steps, a bidirectional long short-term memory network is used to quantify the nonlinear complexity of the pressure characteristic values ​​at the pressure signal sampling points, thereby improving the sensitivity to pressure changes during minor pipeline leaks and reducing the false alarm rate of pipeline leak detection.

[0124] In some implementations, in step 105, when fusing the confidence values ​​corresponding to the acoustic signature energy spectrum, the thermal diffusion gradient, and the fluctuation entropy to determine the detection result of whether the pipeline is leaking, the following fusion method is used to fuse the confidence values ​​corresponding to the acoustic signature energy spectrum, the thermal diffusion gradient, and the fluctuation entropy to obtain the detection result:

[0125] ;

[0126] in, Indicates the test results. Indicates when The confidence value is the acoustic energy spectrum, which represents the probability of determining pipeline leakage based on the acoustic energy spectrum. Indicates when is the confidence level value for the thermal diffusion gradient, i.e., the probability of judging pipeline leakage based on the thermal diffusion gradient; Indicates when is the confidence level value when the fluctuation entropy is reached, that is, the probability of judging pipeline leakage based on the fluctuation entropy.

[0127] Specifically, The value range of is [0,1], which can characterize the probability of pipeline leakage. The closer the value is to 1, the higher the probability of a pipeline leak. The closer the value is to 0, the lower the possibility of pipeline leakage.

[0128] In the implementation of this application, when When the value is ≥0.85, the fusion module 13 determines that the pipeline is in a high-confidence leakage alarm state, and it is recommended to inspect the pipeline immediately; when the value is ≤0.65, the pipeline is in a high-confidence leakage alarm state. When <0.85, the fusion module 13 determines that the pipeline is in a medium confidence state. In this case, it is recommended to combine multiple data points or manually verify to determine if the pipeline is leaking. When the value is less than 0.65, the fusion module 13 determines that the pipeline is in a low confidence state. At this time, it can be regarded as the pipeline has no leakage, and it is recommended to continuously monitor the leakage status of the pipeline.

[0129] Through the above steps, the confidence values ​​of the acoustic, thermodynamic and pressure modes are transformed into probability intervals with a range of [0, 1]. This preserves the uncertainty information in the detection, avoids ignoring uncertainty information when the prediction results of different modes conflict, thus preventing false alarms in pipeline leak detection, improving the robustness of data fusion, and reducing the false alarm rate of pipeline leak detection.

[0130] like Figure 6As shown, in some embodiments, this application provides a pipeline leak detection device 100 based on multimodal feature fusion, which includes a processing module 11, a mapping module 12, and a fusion module 13. The processing module 11 can acquire acoustic signals inside the pipeline and determine the acoustic energy spectrum inside the pipeline based on the acoustic signals; it can also acquire thermal images inside the pipeline and determine the thermal diffusion gradient inside the pipeline based on the thermal images; it can also acquire pressure signals inside the pipeline and determine the fluctuation entropy inside the pipeline based on the pressure signals. The mapping module 12 can use a confidence function to map the acoustic energy spectrum, thermal diffusion gradient, and fluctuation entropy into confidence values ​​representing the probability of pipeline leakage, respectively. The fusion module 13 is used to fuse the confidence values ​​corresponding to the acoustic energy spectrum, the thermal diffusion gradient, and the fluctuation entropy to determine the judgment result of pipeline leakage.

[0131] In some implementations, the processing module 11 can use a convolutional neural network to extract the acoustic temporal features of the acoustic signal within the pipe, and determine the acoustic energy spectrum within the pipe based on the acoustic temporal features. The convolutional neural network includes three convolutional layers and two pooling layers.

[0132] In some implementations, the acoustic time-domain features are multi-dimensional feature vectors, and the processing module 11 can sum the multi-dimensional feature vectors to obtain the acoustic energy spectrum within the pipe.

[0133] In some implementations, the processing module 11 can use a residual neural network to extract image features of the thermal image inside the pipe, and input the image features into a gradient decoder network to obtain the thermal diffusion gradient inside the pipe. The gradient decoder network is a fully connected neural network.

[0134] In some implementations, the processing module 11 can determine the true gradient label and predicted gradient based on infrared image samples and using operators, determine the loss function based on the true gradient label and predicted gradient, and train the gradient decoder network using the loss function. Inputting image features into the trained gradient decoder network allows the acquisition of the thermal diffusion gradient within the pipe.

[0135] In some implementations, the processing module 11 can input a preset number of pressure signal sampling points in the pipeline into a bidirectional long short-term memory network, and output the fluctuation entropy in the pipeline through the sigmoid function in the bidirectional long short-term memory network.

[0136] In some implementations, the mapping module 12 can obtain the confidence value based on the following confidence function:

[0137] ;

[0138] In some implementations, the fusion module can fuse the confidence values ​​corresponding to the acoustic signature energy spectrum, the thermal diffusion gradient, and the fluctuation entropy using the following fusion methods to obtain the detection results:

[0139] .

[0140] In some implementations, the processing module 11 includes a signal receiving unit (not shown) and a processing unit (not shown). The signal receiving unit is communicatively connected to the detection module to receive detection signals sent by the detection module that characterize relevant parameters within the pipeline. The processing unit is able to process the relevant parameters based on the detection signals.

[0141] For example, the signal receiving unit is a data acquisition card, and the processing unit is a big data system.

[0142] In some implementations, the pipeline leak detection system based on multimodal feature fusion also includes an interaction module (not shown in the figure), which can receive the detection results obtained by the fusion module 13 and present the detection results in real time. When the pipeline is in a high-confidence leak alarm state, the interaction module can sound an alarm to remind staff to check whether the pipeline is leaking.

[0143] In some implementations, the interaction module is a fixed terminal that is communicatively connected to the fusion module 13.

[0144] In other implementations, the interaction module is a mobile terminal, which is convenient for staff to carry with them.

[0145] The pipeline leakage detection device based on multimodal feature fusion provided in this application has the following steps: the processing module 11 determines the acoustic energy spectrum of the pipeline by acquiring acoustic signals inside the pipeline; determines the thermal diffusion gradient of the pipeline by acquiring thermal images inside the pipeline; and determines the fluctuation entropy of the pipeline by acquiring pressure signals inside the pipeline. The mapping module 12 obtains the confidence values ​​of the acoustic energy spectrum, thermal diffusion gradient, and fluctuation entropy mapping. The fusion module 13 fuses the corresponding confidence values ​​of the three, thereby detecting whether the pipeline is leaking through multimodal means and reducing the false alarm rate of pipeline leakage detection.

[0146] For example, a detection module is installed inside a gas pipeline with a pressure of 0.8 MPa. Several sound pressure detection modules are spaced 10 m apart. The image acquisition module is aligned with the welds and / or valves of the pipeline, and the pressure detection modules are located at both ends and / or the midpoint of the pipeline. A 0.8 mm orifice leak is injected, and the following test results are obtained: The pipeline leak detection system based on multimodal feature fusion alarms within 120 seconds, and the detection result is [m]. fused =0.93, with a false alarm rate of 3.2%.

[0147] As can be seen from the above examples, the pipeline leakage detection method and device based on multimodal feature fusion provided in this application can detect pipeline leaks in a timely and accurate manner, and issue an alarm in a short period of time, thereby avoiding prolonged pipeline leakage and potential property and safety risks.

[0148] like Figure 7 As shown, this application also provides an electronic device 200, which includes a memory 21 and a processor 22. The memory 21 stores a computer program. When the computer program is executed by the processor 22, the processor 22 performs the steps of the above-mentioned pipeline leakage detection method based on multimodal feature fusion.

[0149] Specifically, processor 22 may include a central processing unit, or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement embodiments of the present invention.

[0150] In some implementations, memory 21 may include a large-capacity memory for data or instructions.

[0151] For example, the memory includes a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, a universal serial bus (USB) drive, or any combination of the above-mentioned memory.

[0152] For example, the memory 21 may be located inside or outside the electronic device 200.

[0153] In some possible implementations, the electronic device 200 also includes a communication interface 23 and a bus 24. The processor 22, memory 21, and communication interface 23 are connected via the bus 24 and communicate with each other.

[0154] The communication interface 23 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0155] Bus 24 includes hardware and / or software that couples components of electronic device 200 together. For example, and not to limit, bus 24 may include an accelerated graphics port or other graphics bus, an enhanced industry standard architecture bus, a front-side bus, a low pin count bus, a memory bus, or other suitable bus or any combination of the above buses.

[0156] It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A pipeline leakage detection method based on multimodal feature fusion, characterized in that, The method includes: Acoustic signals within the pipe are acquired, and acoustic energy spectrum within the pipe is determined based on the acoustic signals within the pipe. Acquire a thermal image inside the pipe, and determine the thermal diffusion gradient inside the pipe based on the thermal image inside the pipe; Acquire the pressure signal inside the pipeline, and determine the fluctuation entropy inside the pipeline based on the pressure signal inside the pipeline; Using a confidence function, the acoustic energy spectrum, the thermal diffusion gradient, and the fluctuation entropy are respectively mapped to confidence values ​​representing the probability of pipeline leakage; By integrating the confidence values ​​corresponding to the acoustic energy spectrum, the thermal diffusion gradient, and the fluctuation entropy, the detection result of whether the pipeline is leaking is determined. Determining the thermal diffusion gradient within the pipe based on the thermal image within the pipe includes: The residual neural network is used to extract image features of the thermal image inside the pipe; The image features are input into a gradient decoder network to obtain the thermal diffusion gradient within the pipe; wherein, the gradient decoder network is a fully connected neural network.

2. The pipeline leakage detection method according to claim 1, characterized in that, Determining the acoustic energy spectrum within the pipe based on the acoustic signals within the pipe includes: A convolutional neural network is used to extract the acoustic temporal features of the acoustic signal inside the pipe; wherein the convolutional neural network includes three convolutional layers and two pooling layers; The acoustic energy spectrum within the pipe is determined based on the acoustic time-domain characteristics.

3. The pipeline leakage detection method according to claim 2, characterized in that, The acoustic time-domain features are multi-dimensional feature vectors; Determining the acoustic energy spectrum within the pipe based on the acoustic time-domain features includes: summing the multi-dimensional feature vectors to obtain the acoustic energy spectrum within the pipe.

4. The pipeline leakage detection method according to claim 1, characterized in that, The image features are input into a gradient decoder network to obtain the thermal diffusion gradient within the pipe, including: Based on infrared image samples and using operators to determine the true gradient labels; Determine the prediction gradient based on infrared image samples; The loss function is determined based on the true gradient label and the predicted gradient, and the gradient decoder network is trained using the loss function. The image features are input into the trained gradient decoder network to obtain the thermal diffusion gradient within the pipe.

5. The pipeline leakage detection method according to claim 1, characterized in that, Determining the fluctuation entropy within the pipeline based on the pressure signal within the pipeline includes: A predetermined number of pressure signal sampling points within the pipeline are input into a bidirectional long short-term memory network, and the fluctuation entropy within the pipeline is output through the sigmoid function in the bidirectional long short-term memory network.

6. The pipeline leakage detection method according to claim 1, characterized in that, The confidence function is: ; in, Indicates the confidence level value. This refers to the acoustic signature energy spectrum, the thermal diffusion gradient, or the wave entropy. This represents the historical average of normal data. This represents a scale parameter that adaptively adjusts with environmental noise. Indicates shape parameters.

7. The pipeline leakage detection method according to claim 6, characterized in that, The detection result for determining whether the pipeline is leaking is determined by integrating the confidence values ​​corresponding to the acoustic energy spectrum, the thermal diffusion gradient, and the fluctuation entropy. The detection result is obtained by fusing the confidence values ​​corresponding to the acoustic signature energy spectrum, the thermal diffusion gradient, and the fluctuation entropy using the following fusion method: ； in, Indicates the test results. Indicates when The confidence level value for the aforementioned voiceprint energy spectrum. Indicates when The confidence level value for the thermal diffusion gradient is given. Indicates when The confidence level value is the value of the fluctuation entropy.

8. A pipeline leak detection device based on multimodal feature fusion, characterized in that, The pipeline leak detection device implements the steps of the pipeline leak detection method based on multimodal feature fusion as described in any one of claims 1 to 7, and the pipeline leak detection device includes: The processing module is used to acquire acoustic signals inside the pipe and determine the acoustic energy spectrum inside the pipe based on the acoustic signals inside the pipe; it is also used to acquire thermal images inside the pipe and determine the thermal diffusion gradient inside the pipe based on the thermal images inside the pipe; it is also used to acquire pressure signals inside the pipe and determine the wave entropy inside the pipe based on the pressure signals inside the pipe. The mapping module uses a confidence function to map the acoustic energy spectrum, the thermal diffusion gradient, and the fluctuation entropy into confidence values ​​representing the probability of pipeline leakage, respectively. The fusion module is used to fuse the confidence values ​​corresponding to the acoustic energy spectrum, the thermal diffusion gradient, and the fluctuation entropy to determine the judgment result of the pipeline leakage.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program, which, when executed by the processor, implements the steps of the pipeline leak detection method based on multimodal feature fusion as described in any one of claims 1 to 7.

Images