Method and apparatus for detecting leaks in a pipeline

Abstract

The application discloses a pipeline leakage detection method and device, the method comprises the following steps: acquiring video images and pressure data of multiple continuous time steps in a pipeline, fusing the video images and the pressure data to obtain fusion information; performing space-time feature extraction on the fusion information by using a space-time feature extraction network to obtain a feature vector; performing classification prediction on the feature vector by using each classification head in a trained multi-heterogeneous structure classification model to obtain pipeline state categories output by each classification head; and determining a final pipeline state category by using an entropy balance decision function according to the pipeline state categories output by each classification head and weights of each classification head, so that the accuracy of pipeline leakage detection is improved.

Description

Technical Field

[0001] This application relates to the field of pipeline inspection technology, and in particular to a pipeline leakage detection method and apparatus. Background Technology

[0002] In the oil and gas sector, pipeline safety has become a key constraint on public safety, resource conservation, and the achievement of dual-carbon goals.

[0003] Among relevant technical solutions, pipeline leak detection technologies include acoustic detection, pressure detection, liquid monitoring, and machine learning. Acoustic detection is a sound-based method for detecting pipeline leaks. When leaking gas is ejected from the damaged area, it forms a sound wave of a specific frequency. This acoustic detection method is highly sensitive and typically uses acoustic sensors installed on the pipe wall to collect and analyze the data. Pressure detection is a relatively simple and low-cost measurement method suitable for real-time monitoring of long-distance pipelines. Liquid monitoring is an indirect method for detecting leaks in transport pipelines, detecting leaks by monitoring the flow state of liquids such as water or oil. Machine learning technology learns normal pipeline operating patterns from historical data to autonomously identify anomalies.

[0004] The above technical solutions have the following drawbacks: 1. The fragmented single-modal features lead to both false negatives and false positives. For example, sound wave, pressure, or visual detection are all single-modal perceptions. Sound wave / pressure sensors cannot directly observe the visible jets, water mist, or oil films formed by leaks, and pure video algorithms lack the ability to perceive transient changes in pipe pressure. Therefore, when the leakage is small or the environmental noise is high, single-modal detection is prone to false negatives; when there are similar frequency vibrations or light and shadow interference in the background, a large number of false positives are generated. 2. Using pre-trained convolutional neural networks for fine-tuning or directly training with general artificial intelligence algorithms results in a lack of coupling between the model weight updates and physical weights such as the elastic modulus of the pipe and the compressibility of the fluid. This leads to drastic performance fluctuations in different material scenarios such as steel pipes, PVC pipes, and composite pipes, making it impossible to adaptively adjust the gradient or learning rate. Gradient conflicts and convergence oscillations occur, ultimately resulting in poor robustness across different scenarios. 3. Simple fusion strategies such as "feature-level stitching" or "decision-level voting" are often used for multi-source heterogeneous data, leading to misalignment between video frames and pressure data at the temporal granularity. Spatially, pressure point sensors can only provide single-point scalars, while videos are two-dimensional matrices. The mismatch in information granularity during fusion results in feature redundancy or loss. 4. Neural networks typically use parallel networks with identical structures and consistent initialization. During training, sub-networks are prone to homogenization, resulting in highly correlated outputs and an inability to form complementary opinions. When a sub-network overfits, the overall decision is "hijacked," missing high-risk micro-leakage. Furthermore, traditional voting or averaging fusion assigns fixed weights to all classification heads, ignoring the differences in fluid motion entropy at the moment of leakage in pipes of different materials. For highly elastic PVC pipes, micro-cracks can lead to significant entropy increases; while for steel pipes, the entropy increase from the same crack is relatively small. Fixed weights cannot adaptively switch between the two materials, resulting in a high false alarm rate in PVC scenarios and insufficient sensitivity in steel scenarios. The aforementioned shortcomings cannot meet the requirements for high reliability, low latency, cross-material compatibility, and full-scenario pipeline leak detection. Summary of the Invention

[0005] The purpose of this application is to provide a pipeline leak detection method that can improve the accuracy of pipeline leak detection.

[0006] In a first aspect, this application provides a pipeline leakage detection method, the method comprising:

[0007] The video images and pressure data of multiple consecutive time steps within the pipeline are acquired, and the video images and pressure data are fused to obtain fused information.

[0008] The fused information is subjected to spatiotemporal feature extraction using a spatiotemporal feature extraction network to obtain a feature vector;

[0009] The feature vector is classified and predicted using each classifier head in the trained multi-heterogeneous structure classification model to obtain the pipeline state category output by each classifier head. The network structure, activation function and weight initialization strategy used by each classifier head are different. The pipeline state category includes normal state and leakage state.

[0010] Based on the pipeline state category output by each classification head and the weights of each classification head, the final pipeline state category is determined through an entropy balance decision function.

[0011] Optionally, the multi-heterogeneous structure classification model is trained using the following steps:

[0012] Acquire multiple historical video images and multiple historical pressure data of at least one pipeline, fuse the multiple historical video images and multiple historical pressure data, and output multiple historical feature vectors by a spatiotemporal feature extraction network through the multiple fused historical information. Use each historical feature vector as a training sample to construct a training sample set.

[0013] Based on the training sample set, perform iterative training on the multi-heterogeneous structure classification model to be trained, and obtain the trained multi-heterogeneous structure classification model upon completion of training. During one iterative training cycle, the following operations are performed:

[0014] Obtain a batch of training samples for the current iteration from the training sample set, wherein the batch of training samples includes multiple training samples;

[0015] For each classifier head in the multi-heterogeneous structure classification model to be trained, output the pipeline state category and the category prediction probability distribution corresponding to each training sample;

[0016] By combining the acquired pipeline state categories and category prediction probability distributions, the total loss function is determined using the cross-entropy loss function, diversity constraint loss function, and balancing loss function of each classifier head. The weights of the multi-heterogeneous structure classification model to be trained are then adjusted based on the total loss function.

[0017] Optionally, combining the acquired pipeline state categories and category prediction probability distributions, the total loss function is determined using the cross-entropy loss function, diversity constraint loss function, and balancing loss function for each classifier head, including:

[0018] For each classifier, the cross-entropy loss function is determined based on the deviation between the true class label of each training sample and the corresponding pipeline state class.

[0019] The cosine similarity between the logits vectors output by any two classifier heads in the multi-heterogeneous structure classification model to be trained is used to determine the corresponding diversity constraint loss function.

[0020] For each classifier, the corresponding fluid motion entropy is calculated based on the category prediction probability distribution output by the classifier, and the current weight of the classifier is calculated based on the temperature coefficient and the fluid motion entropy; the balancing loss function is determined based on the current weight of each classifier and its respective cross-entropy loss function.

[0021] The total loss function is determined by summing the cross-entropy loss function, diversity constraint loss function, and balancing loss function for each classifier.

[0022] Optional, diversity-constrained loss function The calculation formula is:

[0023] ;

[0024] in, Represents the canonical strength coefficient. This represents the logits vector output by the i-th classification head. This represents the logits vector output by the j-th classification head;

[0025] Fluid motion entropy of the i-th classifier The calculation formula is:

[0026] ;

[0027] ;

[0028] in, Represents the probability distribution of category predictions. This represents the logits vector output by the classification head, where c represents the pipeline status category, c∈{1,2}, where 1 represents the leak category and 2 represents the normal category;

[0029] The current weight of the i-th category head The calculation formula is:

[0030] ;

[0031] Where n represents the number of classification heads, Indicates the temperature coefficient;

[0032] Balanced loss function The calculation formula is:

[0033] ;

[0034] in, Let represent the cross-entropy loss function of the i-th classifier head.

[0035] Optionally, the weights of the multi-heterogeneous structure classification model to be trained are adjusted based on the total loss function, including:

[0036] For the i-th category head,

[0037] The gradient g of the total loss function for the weights of the classifier head is calculated using the backpropagation algorithm. i ;

[0038] Calculate gradient g i The gradient g with the j-th class head j The gradient difference between them;

[0039] If the gradient difference exceeds a preset threshold, the gradient g is adjusted based on the physical weights of the pipe material and the gradient difference. i After adjustment, the adjusted gradient g is obtained. i ’ Otherwise, do not adjust gradient g. i The threshold is determined based on the physical weight of the pipe material.

[0040] The adjusted adaptive learning rate is calculated based on the gradient direction consistency coefficient, the physical weights of the pipeline material, and the base learning rate. This adaptive learning rate is then used to update and optimize the weights of the i-th classifier head. The gradient direction consistency coefficient is based on gradient g. i The gradient mean of each classifier head is determined.

[0041] Optional, gradient g i The gradient g with the j-th class head j The formula for calculating the gradient difference between them is expressed as:

[0042] ;

[0043] Adjusted gradient g i ’ Represented as;

[0044] ;

[0045] in, Indicates the threshold. Indicates the elastic coefficient of the pipe material. Indicates the elastic modulus of a material;

[0046] Adaptive learning rate The calculation formula is expressed as:

[0047] ;

[0048] ;

[0049] in, Indicates the basic learning rate. This represents the gradient direction consistency coefficient, and n represents the number of classification heads. This represents the average gradient of each classification head.

[0050] Optionally, based on the pipeline state category output by each classification head and the weights of each classification head, the final pipeline state category is determined through an entropy balance decision function, including:

[0051] The entropy balance decision function is expressed as follows:

[0052] ;

[0053] in, Indicates the final pipeline state category. This represents the sign function, where n represents the number of classification heads. This represents the weight of the i-th classifier head after training. This represents the pipeline state category output by the i-th category header. This indicates the leakage threshold.

[0054] Optionally, the multi-heterogeneous structure classification model includes three classification heads, referred to as the first classification head, the second classification head, and the third classification head. Each classification head includes an input layer, multiple fully connected layers, an activation layer, and an output layer. The dimensions and activation functions of the fully connected layers in each classification head are different.

[0055] The fully connected layer of the first classifier has a dimension of 256, the activation function of the activation layer is the GELU activation function, and the weights of the first classifier are initialized using the He normal method.

[0056] The fully connected layer of the second classification head has a dimension of 128, the activation function of the activation layer is the ReLU activation function, and the weights of the second classification head are initialized using the Xavier uniform method.

[0057] The fully connected layer of the third classifier has a dimension of 512, and the activation function of the activation layer is the Swish activation function. The weights of the third classifier are initialized in an orthogonal manner.

[0058] Optionally, video images and pressure data from multiple consecutive time steps within the pipeline are acquired, and the video images and pressure data are fused to obtain fused information, including:

[0059] The acquired video images from multiple consecutive time steps are processed into a video image tensor of shape (B, T, C, H, W), where B is the batch size, T is the time step, C is the number of channels, H is the height, and W is the width.

[0060] For each time step, the pressure data of that time step is converted into a pseudo-image of H×W. The pseudo-images of multiple consecutive time steps are processed into a pseudo-image tensor of shape (B, T, H, W), where the pixel values ​​in the pseudo-image are the pressure data of that time step.

[0061] Using pseudo-image tensors as additional channel data in video image tensors, the pseudo-image tensors and video image tensors are fused to obtain fused information.

[0062] Secondly, this application provides a pipeline leak detection device, the device comprising:

[0063] The camera assembly is used to capture video images of multiple consecutive time steps within the pipe;

[0064] Sensor components are used to acquire pressure data at multiple consecutive time steps within the pipeline;

[0065] The data processing module is used to fuse the video images and pressure data to obtain fused information;

[0066] The feature extraction module is used to extract spatiotemporal features from the fused information using a spatiotemporal feature extraction network to obtain feature vectors;

[0067] The category prediction module is used to perform classification prediction on the feature vector using each classification head in the trained multi-heterogeneous structure classification model, and to obtain the pipeline state category output by each classification head. The network structure, activation function and weight initialization strategy used by each classification head are different. The pipeline state category includes normal state and leakage state.

[0068] The decision module is used to determine the final pipeline state category based on the pipeline state category output by each classification head and the weights of each classification head, using an entropy balance decision function.

[0069] This application fuses video images and pressure data from multiple consecutive time steps. A spatiotemporal feature extraction network extracts spatiotemporal features from the fused information. The extracted feature vectors are input into each classifier in a multi-heterogeneous structure classification model to predict the classification of the feature vectors, obtaining the pipe state category output by each classifier. An entropy balance decision mechanism is used to fuse the pipe state categories output by multiple classifiers to obtain the final pipe state category, which is used to determine whether the pipe is leaking. Video images and pressure data are aligned at the temporal granularity within the spatiotemporal feature extraction network architecture, solving the problems of information granularity mismatch and insufficient feature decoupling during fusion in existing technologies, greatly improving feature extraction efficiency and achieving an inference latency of less than 80ms. The use of a multi-heterogeneous structure classification model for category prediction ensures the heterogeneity of the classifiers, enabling the fluid motion trajectory, pressure data, and video data to be co-optimized, improving the model's predictive ability, and thus improving the accuracy of pipe leak detection. Attached Figure Description

[0070] Figure 1 This is a first flowchart of a pipeline leakage detection method provided in an embodiment of this application;

[0071] Figure 2 This is a second flowchart of the pipeline leakage detection method provided in the embodiments of this application;

[0072] Figure 3 A third flowchart of the pipeline leakage detection method provided in the embodiments of this application;

[0073] Figure 4 This is a system block diagram of the pipeline leak detection device provided in an embodiment of this application. Detailed Implementation

[0074] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0075] Please refer to Figure 1 This application provides a pipeline leakage detection method, which includes steps S101-S104.

[0076] S101: Acquire video images and pressure data from multiple consecutive time steps within the pipeline, and fuse the video images and pressure data to obtain fused information.

[0077] The time step is a time sequence point obtained at a fixed sampling frequency.

[0078] For example, the pipeline can be an oil and gas pipeline, a water supply pipeline, a drainage pipeline, a chemical liquid pipeline, or a heating pipeline, etc. The materials of the pipeline include, but are not limited to, steel, PVC, composite materials, etc.

[0079] For example, an image acquisition device is used to acquire video images at multiple consecutive time steps within the pipeline, i.e., to acquire multiple consecutive frames of video images. The total duration of the multiple consecutive time steps can be flexibly set based on the actual situation and can be determined according to the acquisition frequency of the image acquisition device. For example, the image acquisition device is an explosion-proof high-definition camera with a frame rate of 30 fps.

[0080] For example, a pressure sensor is used to acquire pressure data at multiple consecutive time steps within the pipe. For instance, a high-frequency pressure sensor with a sampling rate of 100 Hz.

[0081] For example, the PTP (Precise Network Time Protocol) protocol can be used to synchronize the acquisition time of the image acquisition device with the acquisition time of the pressure sensor, so as to ensure that video images and pressure data at the same time step can be acquired.

[0082] Video images and pressure data from multiple consecutive time steps are fused to obtain fused information.

[0083] S102, a spatiotemporal feature extraction network is used to extract spatiotemporal features from the fused information to obtain feature vectors.

[0084] The fused information is input into a spatiotemporal feature extraction network, which extracts the spatiotemporal features of the fused information and outputs the corresponding feature vectors. The spatiotemporal feature extraction network is a pre-trained network model. Spatiotemporal feature extraction networks include, but are not limited to, at least one of the following networks: 3D-ResNet, 3D-CNN, I3D, and SlowFast Network. For example, the 3D-ResNet network can be a 3D-ResNet50 network.

[0085] For example, the 3D-ResNet network takes a five-dimensional tensor as input and processes the fused information into a five-dimensional tensor of shape (B, T, C, H, W), where B is the batch size, T is the time step, C is the number of channels, H is the height, and W is the width. Convolution operations are performed in three dimensions (time dimension T, spatial dimensions H and W) through 3D convolutional layers. After processing by the residual structure in the network, the feature vector is output.

[0086] Leakage is a continuous event in time and space. The minute deformation caused by pipeline vibration requires deep feature extraction capabilities. Therefore, by learning and fusing deep spatiotemporal features from information through a spatiotemporal feature extraction network, it is possible to capture the micro-vibration features of pipeline leaks, the fluid movement trajectory inside the pipeline, and the correlation between pressure information and visual information.

[0087] S103: Use each classifier head in the trained multi-heterogeneous structure classification model to classify and predict the feature vector, and obtain the pipeline state category output by each classifier head.

[0088] Among them, the multi-heterogeneous structure classification model includes multiple classification heads, and each classification head uses a different network structure, activation function and weight initialization strategy.

[0089] Due to the complexity of fluid movement and the differences in the physical properties of pipe materials in pipeline leak detection, the leakage signals of pipelines are more complex. This embodiment adopts a multi-heterogeneous structure classification model, in which the network structure, activation function and weight initialization strategy used by each classification head are different, so as to ensure the heterogeneity of the network structure and solve the problem of identifying complex leakage signals.

[0090] The feature vector is input into each classifier head. Each classifier head performs classification prediction on the feature vector and outputs the corresponding pipeline state category. Thus, the pipeline state category output by each classifier head can be obtained. The pipeline state categories include normal state and leaking state.

[0091] S104. Based on the pipeline state category output by each classification head and the weights of each classification head, the final pipeline state category is determined through the entropy balance decision function.

[0092] By employing an entropy balance decision mechanism to fuse the pipe state categories output by multiple classification heads, the final pipe state category is obtained to determine whether the pipe is leaking.

[0093] For example, the entropy balance decision function is expressed as follows:

[0094] ;

[0095] in, Indicates the final pipeline state category. This represents the sign function, where n represents the number of classification heads. This represents the weight of the i-th classifier head after training. This represents the pipeline state category output by the i-th category header. This indicates the leakage threshold. For example, Set it to 0.65. A value of -1 indicates that the pipeline is in a normal state. A value of 1 indicates that the pipeline is in a leaking state.

[0096] The entropy balance decision function described above can be used to determine the final pipeline state category, thereby enabling us to judge whether the pipeline is leaking.

[0097] In this embodiment, video images and pressure data from multiple consecutive time steps are fused. A spatiotemporal feature extraction network extracts spatiotemporal features from the fused information. The extracted feature vectors are input into each classifier in a multi-heterogeneous structure classification model to predict the classification of the feature vectors, obtaining the pipe state category output by each classifier. An entropy balance decision mechanism is used to fuse the pipe state categories output by multiple classifiers to obtain the final pipe state category, which is used to determine whether the pipe is leaking. Video images and pressure data are aligned at the temporal granularity within the spatiotemporal feature extraction network architecture, solving the problems of information granularity mismatch and insufficient feature decoupling during fusion in existing technologies. This greatly improves feature extraction efficiency and can achieve an inference latency of less than 80ms. The use of a multi-heterogeneous structure classification model for category prediction ensures the heterogeneity of the classifiers, enabling the fluid motion trajectory, pressure data, and video data to be optimized collaboratively, improving the model's predictive ability and thus improving the accuracy of pipe leak detection.

[0098] One embodiment of this application, such as Figure 2 As shown, video images and pressure data from multiple consecutive time steps within the pipeline are acquired, and the video images and pressure data are fused to obtain fused information, including:

[0099] S201, the acquired video images from multiple consecutive time steps are processed into video image tensors of shape (B, T, C, H, W);

[0100] S202, for each time step, the pressure data of that time step is converted into a pseudo-image of H×W, and the pseudo-images of multiple consecutive time steps are processed into a pseudo-image tensor of shape (B, T, H, W), where the pixel values ​​in the pseudo-image are all the pressure data of that time step.

[0101] S203, using the pseudo-image tensor as the newly added channel data of the video image tensor, the pseudo-image tensor and the video image tensor are fused to obtain fused information.

[0102] For example, video images acquired at multiple consecutive time steps are processed into a five-dimensional video image tensor of shape (B, T, C, H, W), where B is the batch size, T is the time step, C is the number of channels, H is the height, and W is the width. For instance, the batch size B is 32, the time step T is 16 (i.e., 16 consecutive video frames), the number of channels C is 3 (i.e., R, G, and B channels), and H×W is 112×112.

[0103] For the acquired stress data, at each time step, the stress data for that time step is converted into an H×W matrix. Each element of this matrix is ​​then assigned the stress data for that time step, resulting in an H×W pseudo-image. The pixel values ​​in this pseudo-image are all the stress data for that time step. Based on the acquired pseudo-images from multiple consecutive time steps, a four-dimensional pseudo-image tensor of shape (B, T, H, W) is generated. For example, the batch size B is 32, the time step T is 16, and H×W is 112×112.

[0104] For example, a pseudo-image tensor is used as the additional channel data of the video image tensor, increasing the number of channels in the video image tensor from 3 to 4, namely R, G, B, and pressure data channels, thereby obtaining fusion information. The fusion information is then input into a spatiotemporal feature extraction network, which outputs the corresponding feature vector.

[0105] In this embodiment, video images and stress data from multiple consecutive time steps are aligned within a spatiotemporal feature extraction network architecture. This achieves temporal granularity alignment between video data and high-frequency stress data, resolving the issues of information granularity mismatch and insufficient feature decoupling during fusion in existing technologies, and significantly improving feature extraction efficiency. During model inference, an inference latency of less than 80ms can be achieved.

[0106] In one embodiment of this application, a multi-heterogeneous structure classification model includes three classification heads, referred to as the first classification head, the second classification head, and the third classification head. Each classification head includes an input layer, multiple fully connected layers, an activation layer, and an output layer. The dimensions and activation functions of the fully connected layers in each classification head are different. The fully connected layer of the first classification head has a dimension of 256, and the activation function of the activation layer is the GELU activation function. The weights of the first classification head are initialized using the He normal distribution, which enhances the nonlinear expressive power of the model. The fully connected layer of the second classification head has a dimension of 128, and the activation function of the activation layer is the ReLU activation function. The weights of the second classification head are initialized using the Xavier uniform distribution, which simplifies and improves the computational efficiency of the model. The fully connected layer of the third classification head has a dimension of 512, and the activation function of the activation layer is the Swish activation function. The weights of the third classification head are initialized using an orthogonal distribution, which smooths the nonlinearity of the model and maintains the gradient stability of the model.

[0107] In one embodiment of this application, the multi-heterogeneous structure classification model is trained using the following steps:

[0108] Acquire multiple historical video images and multiple historical pressure data of at least one pipeline, fuse the multiple historical video images and multiple historical pressure data, and output multiple historical feature vectors by a spatiotemporal feature extraction network through the multiple fused historical information. Use each historical feature vector as a training sample to construct a training sample set.

[0109] Based on the training sample set, perform iterative training on the multi-heterogeneous structure classification model to be trained, and obtain the trained multi-heterogeneous structure classification model upon completion of training. The following operations are performed during one iterative training cycle:

[0110] Obtain a batch of training samples for the current iteration from the training sample set, wherein a batch of training samples includes multiple training samples;

[0111] For each classifier head in the multi-heterogeneous structure classification model to be trained, output the pipeline state category and the category prediction probability distribution corresponding to each training sample;

[0112] By combining the obtained pipeline state categories and category prediction probability distributions, the total loss function is determined using the cross-entropy loss function, diversity constraint loss function, and balancing loss function of each classifier head. The weights of the multi-heterogeneous structure classification model to be trained are then adjusted based on the total loss function.

[0113] The current batch of training samples is input into each classification head, and each classification head outputs the pipeline state category corresponding to each training sample and the corresponding category prediction probability distribution. The pipeline state categories include normal and leaky.

[0114] The multi-heterogeneous structure classification model to be trained is subjected to iterative training based on the training sample set until the model training conditions are met, at which point training stops, and the trained multi-heterogeneous structure classification model is obtained.

[0115] The heterogeneous classification model employed in this application includes multiple classification heads. Simply using multiple classification heads presents the following problems: homogenization of classification heads (each head may learn the same features, reducing decision diversity); adversarial nature of classification heads (gradient conflicts may occur, leading to convergence oscillations); and resource imbalance of classification heads (during the training process, a single head may excessively dominate training, causing other classification heads to malfunction). Therefore, this embodiment introduces a diversity constraint loss function and a balancing loss function into the model's total loss function during training. This addresses the aforementioned problems of homogenization, adversarial nature, and resource imbalance, enabling the synergistic optimization of fluid motion trajectory, pressure data, and video data, improving the model's predictive ability, and ultimately enhancing the accuracy of pipeline leak detection.

[0116] One embodiment of this application, such as Figure 3 As shown, combining the acquired pipeline state categories and category prediction probability distributions, the total loss function is determined using the cross-entropy loss function, diversity constraint loss function, and balancing loss function for each classifier head, including:

[0117] S301, For each classification head, determine the cross-entropy loss function of the classification head based on the deviation between the true class label of each training sample and the corresponding pipeline state class;

[0118] S302, determine the corresponding diversity constraint loss function based on the cosine similarity between the logits vectors output by any two classifier heads in the multi-heterogeneous structure classification model to be trained;

[0119] S303, for each classification head, calculate the corresponding fluid motion entropy based on the category prediction probability distribution output by the classification head, calculate the current weight of the classification head based on the temperature coefficient and fluid motion entropy; determine the balancing loss function based on the current weight of each classification head and its respective cross-entropy loss function;

[0120] S304. The total loss function is determined by summing the cross-entropy loss function, diversity constraint loss function, and balancing loss function for each classifier.

[0121] The cross-entropy loss function measures the deviation between the pipeline state class of a training sample and the true class label. For example, the cross-entropy loss function L... task Represented as:

[0122] ;

[0123] Where N is the number of training samples, y i P represents the true class label of the i-th training sample. i This represents the pipeline state category of the i-th training sample.

[0124] For example, the diversity-constrained loss function The calculation formula is:

[0125] ;

[0126] in, Represents the canonical strength coefficient. This represents the logits vector output by the i-th classification head. This represents the logits vector output by the j-th classifier head. The logits vector can be understood as the raw values ​​output by the model's output layer, which are then transformed into a class prediction probability distribution after softmax processing. For example, the regularization strength coefficient can be set to 0.7.

[0127] In this implementation, the cosine similarity between the logits vectors output by any two classifiers is calculated to determine the corresponding diversity constraint loss function. This forces each classifier to focus on different physical features, making the outputs of the three classifiers as different as possible and reducing the similarity of the outputs of each classifier, thereby solving the technical problem of classifier homogenization.

[0128] Fluid motion entropy of the i-th classifier The calculation formula is:

[0129] ;

[0130] ;

[0131] in, Represents the probability distribution of category predictions. This represents the logits vector output by the classification head, where c represents the pipeline status category, c∈{1,2}, where 1 represents the leak category and 2 represents the normal category.

[0132] Using the above formula, the corresponding fluid motion entropy can be calculated based on the category prediction probability distribution output by the classification head. It can be understood that one training sample corresponds to one category prediction probability distribution. During one iteration of training, the classification head inputs multiple training samples, thus obtaining multiple category prediction probability distributions corresponding to multiple training samples in that iteration. In the above formula for calculating fluid motion entropy, the category prediction probability distribution... The mean of the prediction probability distributions for multiple categories can be used in this iteration. The maximum value among the probability distributions of multiple categories can be used in this iteration, and no limitation is made here.

[0133] Based on fluid motion entropy The current weight of the i-th class head The calculation formula is:

[0134] ;

[0135] Where n represents the number of classification heads, This represents the temperature coefficient, which can be set to 1.

[0136] The balanced loss function is determined based on the current weights and cross-entropy loss functions of each classifier head. The calculation formula is:

[0137] ;

[0138] in, Let represent the cross-entropy loss function of the i-th classifier head.

[0139] In this implementation, the weights of each classification head are dynamically adjusted based on the fluid motion entropy output by each classification head, reducing the resource imbalance rate of the classification heads and ensuring matching of resource allocation among the classification heads, thus solving the technical problem of resource imbalance of the classification heads.

[0140] The total loss function is determined by summing the cross-entropy loss function, diversity constraint loss function, and balancing loss function for each classifier head. The weights of the multi-heterogeneous structure classification model to be trained are then adjusted based on the total loss function.

[0141] One embodiment of this application involves adjusting the weights of a multi-heterogeneous structure classification model to be trained based on the total loss function, including:

[0142] For the i-th category head,

[0143] The gradient g of the total loss function for the weights of the classifier head is calculated using the backpropagation algorithm. i ;

[0144] Calculate gradient g i The gradient g with the j-th class head j The gradient difference between them;

[0145] If the gradient difference exceeds a preset threshold, the gradient g is adjusted based on the physical weights of the pipe material and the gradient difference. i After adjustment, the adjusted gradient g is obtained. i ’ Otherwise, do not adjust gradient g. i The threshold is determined based on the physical weight of the pipe material;

[0146] The adjusted adaptive learning rate is calculated based on the gradient direction consistency coefficient, the physical weights of the pipeline material, and the base learning rate. This adaptive learning rate is then used to update and optimize the weights of the i-th classifier head. The gradient direction consistency coefficient is based on gradient g. i The gradient mean of each classifier head is determined.

[0147] For example, the physical weights of pipe materials include the pipe's elastic coefficient and the material's elastic modulus. Pipe material information is obtained through pre-configuration or database queries. During system deployment, the pipe's elastic coefficient and elastic modulus are retrieved from a predefined weight table based on the pipe type (e.g., steel, PVC, PE, etc.). The material's elastic modulus can be directly retrieved from the online database MatWeb. A preset threshold is the product of the pipe's elastic coefficient and the material's elastic modulus.

[0148] In the gradient difference calculation in the above embodiments, the gradient difference between the i-th classifier and any other classifier is calculated, and subsequent judgment and processing are performed based on the calculated gradient difference.

[0149] For example, gradient g i The gradient g with the j-th class head j The formula for calculating the gradient difference between them is expressed as:

[0150] ;

[0151] Adjusted gradient g i ’ Represented as;

[0152] ;

[0153] in, Indicates the threshold. Indicates the elastic coefficient of the pipe material. Indicates the elastic modulus of a material;

[0154] Adaptive learning rate The calculation formula is expressed as:

[0155] ;

[0156] ;

[0157] in, Indicates the basic learning rate. This represents the gradient direction consistency coefficient, and n represents the number of classification heads. This represents the average gradient of each classification head.

[0158] In this embodiment, the gradient of each classifier head is calculated in real time during each training iteration. The gradient of each classifier head is adjusted by the gradient differences between them and the elastic constraints (based on the physical weights of the pipe material), which reduces gradient conflicts and improves the convergence speed of the model. Simultaneously, a gradient coupling mechanism driven by the elastic modulus of the pipe material is introduced into the gradient adjustment and adaptive learning rate adjustment, allowing the model to incorporate the physical laws of fluid motion during the training phase. This ensures a reduced false alarm rate for pipe leaks in scenarios involving pipes made of different materials such as steel, PVC, and PE.

[0159] In one specific embodiment of this application, a DN300 steel gas pipeline with a pipe elastic coefficient of 0.3, a material elastic modulus of 210 GPa, and a gradient shear threshold of 0.3 is used. An explosion-proof high-definition camera (30 fps) and a MEMS high-frequency pressure sensor (100 Hz sampling rate) are co-located 0.5 m above the top of the pipeline, collecting 220 hours of video images and pressure data. If a convolutional neural network weight fine-tuning technique is used, the measured model's inference latency is greater than 200 ms, cross-scene transfer capability is less than 80%, the false negative rate is 15%, and the false positive rate is 22%. If the technical solution of this application is used, the measured model's inference latency is 68 ms, the cross-scene transfer capability is 93.1%, the false negative rate is 1.8%, and the false positive rate is 4.2%. The above false positive rates are based on continuous 72-hour testing conditions.

[0160] In one specific embodiment of this application, a Φ110 PVC water supply pipeline is used underground. The pipeline's elastic coefficient is 0.3, its elastic modulus is 210 GPa, and the adaptive learning rate is amplified by 2.3 times. An explosion-proof high-definition camera (30fps) and a MEMS high-frequency pressure sensor (100Hz sampling rate) are co-located 0.5 m above the top of the pipeline. The camera and high-frequency pressure sensor are encapsulated in an IP68 stainless steel housing. 220 hours of video images and pressure data are collected. If a convolutional neural network weight fine-tuning technique is used, the measured model's inference latency is greater than 200ms, its cross-scene transfer capability is less than 80%, its false negative rate is 15%, and its false positive rate is 22%. If the technical solution of this application is used, the measured model's inference latency is 75ms, its cross-scene transfer capability is 92.4%, its false negative rate is 2.1%, and its false positive rate is 6%. The above false positive rates are based on a continuous 72-hour test condition.

[0161] like Figure 4 As shown, this application provides a pipeline leak detection device, which includes:

[0162] Camera component 401 is used to acquire video images of multiple consecutive time steps within the pipe;

[0163] Sensor assembly 402 is used to acquire pressure data at multiple consecutive time steps within the pipeline;

[0164] Data processing module 403 is used to fuse video images and pressure data to obtain fused information;

[0165] Feature extraction module 404 is used to extract spatiotemporal features from the fused information using a spatiotemporal feature extraction network to obtain feature vectors;

[0166] The category prediction module 405 is used to perform classification prediction on the feature vector using each classification head in the trained multi-heterogeneous structure classification model, and to obtain the pipeline state category output by each classification head. The network structure, activation function and weight initialization strategy used by each classification head are different. The pipeline state categories include normal state and leaking state.

[0167] The decision module 406 is used to determine the final pipeline state category based on the pipeline state category output by each classification head and the weight of each classification head through an entropy balance decision function.

[0168] Optionally, the data processing module 403 is specifically used for:

[0169] The acquired video images from multiple consecutive time steps are processed into video image tensors of shape (B, T, C, H, W);

[0170] For each time step, the pressure data of that time step is converted into a pseudo-image of H×W. The pseudo-images of multiple consecutive time steps are processed into a pseudo-image tensor of shape (B, T, H, W), where the pixel values ​​in the pseudo-image are the pressure data of that time step, B is the batch size, T is the time step, C is the number of channels, H is the height, and W is the width.

[0171] Using pseudo-image tensors as additional channel data in video image tensors, the pseudo-image tensors and video image tensors are fused to obtain fused information.

[0172] Optionally, the multi-heterogeneous structure classification model in the category prediction module 405 includes three classification heads, referred to as the first classification head, the second classification head, and the third classification head. Each classification head includes an input layer, multiple fully connected layers, an activation layer, and an output layer. The dimensions and activation functions of the fully connected layers in each classification head are different.

[0173] The fully connected layer of the first classifier has a dimension of 256, the activation function of the activation layer is the GELU activation function, and the weights of the first classifier are initialized using the He normal method.

[0174] The fully connected layer of the second classification head has a dimension of 128, the activation function of the activation layer is the ReLU activation function, and the weights of the second classification head are initialized using the Xavier uniform method.

[0175] The fully connected layer of the third classifier has a dimension of 512, and the activation function of the activation layer is the Swish activation function. The weights of the third classifier are initialized in an orthogonal manner.

[0176] Optionally, the category prediction module 405 is specifically used to train the multi-heterogeneous structure classification model using the following steps:

[0177] Acquire multiple historical video images and multiple historical pressure data of at least one pipeline, fuse the multiple historical video images and multiple historical pressure data, and output multiple historical feature vectors by a spatiotemporal feature extraction network through the multiple fused historical information. Use each historical feature vector as a training sample to construct a training sample set.

[0178] Based on the training sample set, perform iterative training on the multi-heterogeneous structure classification model to be trained, and obtain the trained multi-heterogeneous structure classification model upon completion of training. The following operations are performed during one iterative training cycle:

[0179] Obtain a batch of training samples for the current iteration from the training sample set, wherein a batch of training samples includes multiple training samples;

[0180] For each classifier head in the multi-heterogeneous structure classification model to be trained, output the pipeline state category and the category prediction probability distribution corresponding to each training sample;

[0181] By combining the obtained pipeline state categories and category prediction probability distributions, the total loss function is determined using the cross-entropy loss function, diversity constraint loss function, and balancing loss function of each classifier head. The weights of the multi-heterogeneous structure classification model to be trained are then adjusted based on the total loss function.

[0182] Optionally, the category prediction module 405 is specifically used for:

[0183] For each classifier, the cross-entropy loss function is determined based on the deviation between the true class label of each training sample and the corresponding pipeline state class.

[0184] The cosine similarity between the logits vectors output by any two classifier heads in the multi-heterogeneous structure classification model to be trained is used to determine the corresponding diversity constraint loss function.

[0185] For each classifier, the corresponding fluid motion entropy is calculated based on the category prediction probability distribution output by the classifier, and the current weight of the classifier is calculated based on the temperature coefficient and the fluid motion entropy; the balancing loss function is determined based on the current weight of each classifier and its respective cross-entropy loss function.

[0186] The total loss function is determined by summing the cross-entropy loss function, diversity constraint loss function, and balancing loss function for each classifier.

[0187] Optionally, the category prediction module 405 is specifically used for:

[0188] For the i-th category head,

[0189] The gradient g of the total loss function for the weights of the classifier head is calculated using the backpropagation algorithm. i ;

[0190] Calculate gradient g i The gradient g with the j-th class head j The gradient difference between them;

[0191] If the gradient difference exceeds a preset threshold, the gradient g is adjusted based on the physical weights of the pipe material and the gradient difference. i After adjustment, the adjusted gradient g is obtained. i ’ Otherwise, do not adjust gradient g. i The threshold is determined based on the physical weight of the pipe material.

[0192] The adjusted adaptive learning rate is calculated based on the gradient direction consistency coefficient, the physical weights of the pipeline material, and the base learning rate. This adaptive learning rate is then used to update and optimize the weights of the i-th classifier head. The gradient direction consistency coefficient is based on gradient g. i The gradient mean of each classifier head is determined.

[0193] Optionally, decision module 406 is specifically used for:

[0194] The entropy balance decision function is expressed as follows:

[0195] ;

[0196] in, Indicates the final pipeline state category. This represents the sign function, where n represents the number of classification heads. This represents the weight of the i-th classifier head after training. This represents the pipeline state category output by the i-th category header. This indicates the leakage threshold.

[0197] This application provides an electronic device comprising a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. In implementation, the processor, communication interface, and memory may also communicate with each other using other connection methods besides the communication bus.

[0198] The memory can be used to store computer programs, which may include instructions and data to implement the steps of any of the pipeline leak detection methods described above. In the embodiments of this application, the memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, optical memory, and registers. The memory may include a hard disk and / or RAM.

[0199] The processor can be a general-purpose processor, which performs specific steps and / or operations by reading and executing a computer program stored in memory (e.g., a computer program). The general-purpose processor may use data stored in memory (e.g., data in memory) during the execution of said steps and / or operations. The general-purpose processor can be, for example, but not limited to, a central processing unit (CPU). Furthermore, the processor can also be a special-purpose processor, which is a processor specifically designed to perform specific steps and / or operations. Special-purpose processors can be, for example, but not limited to, ASICs and FPGAs. Additionally, the processor can be a combination of multiple processors, such as a multi-core processor.

[0200] A communication interface can include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within a network device, as well as interfaces used for interconnecting the network device with other devices (e.g., network devices). The communication network can be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface can be a module, circuit, transceiver, or any device capable of enabling communication.

[0201] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (ROM), electrically erasable programmable memory (EPR), or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0202] Although preferred embodiments of the present application have been disclosed for illustrative purposes, those skilled in the art will recognize that various modifications, additions, and substitutions are possible without departing from the scope and spirit of the present application as disclosed in the appended claims.

Claims

1. A method for detecting pipeline leaks, characterized in that, The method includes: The video images and pressure data of multiple consecutive time steps within the pipeline are acquired, and the video images and pressure data are fused to obtain fused information. The fused information is subjected to spatiotemporal feature extraction using a spatiotemporal feature extraction network to obtain a feature vector; The feature vector is classified and predicted using each classifier head in the trained multi-heterogeneous structure classification model to obtain the pipeline state category output by each classifier head. The network structure, activation function and weight initialization strategy used by each classifier head are different. The pipeline state category includes normal state and leakage state. Based on the pipeline state category output by each classification head and the weights of each classification head, the final pipeline state category is determined through the entropy balance decision function. This involves acquiring video images and pressure data from multiple consecutive time steps within the pipeline, fusing the video images and pressure data to obtain fused information, including: The acquired video images from multiple consecutive time steps are processed into a video image tensor of shape (B, T, C, H, W), where B is the batch size, T is the time step, C is the number of channels, H is the height, and W is the width. For each time step, the pressure data of that time step is converted into a pseudo-image of H×W. The pseudo-images of multiple consecutive time steps are processed into a pseudo-image tensor of shape (B, T, H, W), where the pixel values ​​in the pseudo-image are the pressure data of that time step. Using pseudo-image tensors as newly added channel data in video image tensors, the pseudo-image tensors and video image tensors are fused to obtain fused information; The multi-heterogeneous structure classification model is trained using the following steps: For each classifier, the cross-entropy loss function is determined based on the deviation between the true class label of each training sample and the corresponding pipeline state class. The cosine similarity between the logits vectors output by any two classifier heads in the multi-heterogeneous structure classification model to be trained is used to determine the corresponding diversity constraint loss function. For each classifier, the corresponding fluid motion entropy is calculated based on the category prediction probability distribution output by the classifier, and the current weight of the classifier is calculated based on the temperature coefficient and the fluid motion entropy; the balancing loss function is determined based on the current weight of each classifier and its respective cross-entropy loss function. The total loss function is determined by summing the cross-entropy loss function, diversity constraint loss function, and balancing loss function of each classifier head. Based on the total loss function, the weights of the multi-heterogeneous structure classification model to be trained are adjusted to obtain the trained multi-heterogeneous structure classification model. Based on the pipeline state category output by each classification head and the weights of each classification head, the final pipeline state category is determined through an entropy balance decision function, including: The entropy balance decision function is expressed as follows: ； in, Indicates the final pipeline state category. This represents the sign function, where n represents the number of classification heads. This represents the weight of the i-th classifier head after training. This represents the pipeline state category output by the i-th category header. This indicates the leakage threshold.

2. The pipeline leakage detection method according to claim 1, characterized in that, The multi-heterogeneous structure classification model is trained using the following steps: Acquire multiple historical video images and multiple historical pressure data of at least one pipeline, fuse the multiple historical video images and multiple historical pressure data, and output multiple historical feature vectors by a spatiotemporal feature extraction network through the multiple fused historical information. Use each historical feature vector as a training sample to construct a training sample set. Based on the training sample set, perform iterative training on the multi-heterogeneous structure classification model to be trained, and obtain the trained multi-heterogeneous structure classification model upon completion of training. During one iterative training cycle, the following operations are performed: Obtain a batch of training samples for the current iteration from the training sample set, wherein the batch of training samples includes multiple training samples; For each classifier head in the multi-heterogeneous structure classification model to be trained, output the pipeline state category and the predicted probability distribution of each training sample.

3. The pipeline leakage detection method according to claim 2, characterized in that, Diversity-constrained loss function The calculation formula is: ； in, Represents the canonical strength coefficient. This represents the logits vector output by the i-th classification head. This represents the logits vector output by the j-th classification head; Fluid motion entropy of the i-th classifier The calculation formula is: ； ； in, Represents the probability distribution of category predictions. This represents the logits vector output by the classification head, where c represents the pipeline status category, c∈{1,2}, where 1 represents the leak category and 2 represents the normal category; The current weight of the i-th category head The calculation formula is: ； Where n represents the number of classification heads, Indicates the temperature coefficient; Balanced loss function The calculation formula is: ； in, Let represent the cross-entropy loss function of the i-th classifier head.

4. The pipeline leakage detection method according to claim 1, characterized in that, Based on the total loss function, the weights of the multi-heterogeneous structure classification model to be trained are adjusted, including: For the i-th category head, The gradient g of the total loss function for the weights of the classifier head is calculated using the backpropagation algorithm. i ; Calculate gradient g i The gradient g with the j-th class head j The gradient difference between them; If the gradient difference exceeds a preset threshold, the gradient g is adjusted based on the physical weights of the pipe material and the gradient difference. i After adjustment, the adjusted gradient g is obtained. i ’ Otherwise, do not adjust gradient g. i The threshold is determined based on the physical weight of the pipe material. The adjusted adaptive learning rate is calculated based on the gradient direction consistency coefficient, the physical weights of the pipeline material, and the base learning rate. This adaptive learning rate is then used to update and optimize the weights of the i-th classifier head. The gradient direction consistency coefficient is based on gradient g. i The gradient mean of each classifier head is determined.

5. The pipeline leakage detection method according to claim 4, characterized in that, gradient g i The gradient g with the j-th class head j The formula for calculating the gradient difference between them is expressed as: ； Adjusted gradient g i ’ Represented as; ； in, Indicates the threshold. Indicates the elastic coefficient of the pipe. Indicates the elastic modulus of a material; Adaptive learning rate The calculation formula is expressed as: ； ； in, Indicates the basic learning rate. This represents the gradient direction consistency coefficient, and n represents the number of classification heads. This represents the average gradient of each classification head.

6. The pipeline leakage detection method according to claim 1, characterized in that, The heterogeneous classification model includes three classification heads, denoted as the first, second, and third classification heads. Each classification head comprises an input layer, multiple fully connected layers, an activation layer, and an output layer. The dimensions and activation functions of the fully connected layers in each classification head are different. The fully connected layer of the first classifier has a dimension of 256, the activation function of the activation layer is the GELU activation function, and the weights of the first classifier are initialized using the He normal method. The fully connected layer of the second classification head has a dimension of 128, the activation function of the activation layer is the ReLU activation function, and the weights of the second classification head are initialized using the Xavier uniform method. The fully connected layer of the third classifier has a dimension of 512, and the activation function of the activation layer is the Swish activation function. The weights of the third classifier are initialized in an orthogonal manner.

7. A pipeline leak detection device, characterized in that, The device includes: The camera assembly is used to capture video images of multiple consecutive time steps within the pipe; Sensor components are used to acquire pressure data at multiple consecutive time steps within the pipeline; The data processing module is used to fuse the video images and pressure data to obtain fused information; The feature extraction module is used to extract spatiotemporal features from the fused information using a spatiotemporal feature extraction network to obtain feature vectors; The category prediction module is used to perform classification prediction on the feature vector using each classification head in the trained multi-heterogeneous structure classification model, and to obtain the pipeline state category output by each classification head. The network structure, activation function and weight initialization strategy used by each classification head are different. The pipeline state category includes normal state and leakage state. The decision module is used to determine the final pipeline state category based on the pipeline state category output by each classification head and the weight of each classification head through an entropy balance decision function. Specifically, the data processing module is used for: The acquired video images from multiple consecutive time steps are processed into a video image tensor of shape (B, T, C, H, W), where B is the batch size, T is the time step, C is the number of channels, H is the height, and W is the width. For each time step, the pressure data of that time step is converted into a pseudo-image of H×W. The pseudo-images of multiple consecutive time steps are processed into a pseudo-image tensor of shape (B, T, H, W), where the pixel values ​​in the pseudo-image are the pressure data of that time step. Using pseudo-image tensors as newly added channel data in video image tensors, the pseudo-image tensors and video image tensors are fused to obtain fused information; The category prediction module is specifically used for: For each classifier, the cross-entropy loss function is determined based on the deviation between the true class label of each training sample and the corresponding pipeline state class. The cosine similarity between the logits vectors output by any two classifier heads in the multi-heterogeneous structure classification model to be trained is used to determine the corresponding diversity constraint loss function. For each classifier, the corresponding fluid motion entropy is calculated based on the category prediction probability distribution output by the classifier, and the current weight of the classifier is calculated based on the temperature coefficient and the fluid motion entropy; the balancing loss function is determined based on the current weight of each classifier and its respective cross-entropy loss function. The total loss function is determined by summing the cross-entropy loss function, diversity constraint loss function, and balancing loss function of each classifier head. Based on the total loss function, the weights of the multi-heterogeneous structure classification model to be trained are adjusted to obtain the trained multi-heterogeneous structure classification model. The decision-making module is specifically used for: the entropy balance decision function is expressed as follows: ； in, Indicates the final pipeline state category. This represents the sign function, where n represents the number of classification heads. This represents the weight of the i-th classifier head after training. This represents the pipeline state category output by the i-th category header. This indicates the leakage threshold.

Images