A multi-sensor fusion pipeline condition monitoring system and method

Abstract

This invention relates to the field of pipeline monitoring technology and discloses a multi-sensor fusion pipeline condition monitoring system and method. This invention uses machine learning-based technology to fuse and analyze data from multiple sensors, including sound, visible light, and thermal imaging, to monitor the condition of water pipelines. It can detect leaks and other anomalies in water pipelines in real time. Multi-sensor fusion verifies anomalies from multiple dimensions and angles, ensuring the accuracy of anomaly detection. Leakage points are quickly located through sensor and detection point coordinate tracking. A more advanced deep learning feature extraction network and high-precision machine learning algorithms process data from various sensors in real time, thereby better understanding the global structure of the image and enhancing the network's feature extraction capabilities.

Description

Technical Field

[0001] This invention relates to the field of pipeline monitoring technology, and in particular to a pipeline condition monitoring system and method with multi-sensor fusion. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] Pipelines are common equipment in various engineering projects. To ensure the normal operation of the project, pipelines often need to be inspected and monitored to prevent leakage of media caused by local damage or performance failure. Due to the long length, concealed layout, and complex environment of pipelines, pipeline inspection is particularly inconvenient. Currently, manual inspection or 24-hour monitoring is commonly used. Manual inspection requires a large workforce, resulting in high labor costs. 24-hour monitoring can lead to inspectors becoming fatigued, frequently resulting in missed detections, inaccurate reflection of the status of each pipeline, high management costs, and a lack of traceability. Although existing technologies have improved manual inspection methods, such as Chinese patent CN118654239A, which discloses a method of installing a special leak detection device on the pipeline, where water flows into a storage box and triggers a leak alarm when a leak occurs, this method requires the installation of numerous devices at intervals along each pipeline, resulting in high installation costs and requiring modifications to the existing pipelines, leading to high modification costs.

[0004] Therefore, existing technologies still need improvement. Summary of the Invention

[0005] This invention addresses the problems of high labor costs, low accuracy, low scalability, and limited versatility in current inspection methods by proposing a pipeline status monitoring system and method based on multi-sensor fusion. This system can monitor the pipeline status in real time and accurately detect and locate abnormalities such as leaks, forming a real-time pipeline status management backtracking system that greatly saves costs.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] Firstly, this invention provides a multi-sensor fusion pipeline condition monitoring system, including a visible light sensor, a thermal imaging sensor, a sound sensor, and a computing module. The visible light sensor is used to acquire image information of pipeline inspection points; the thermal imaging sensor is used to perform thermal imaging temperature measurement on the pipeline inspection points; the sound sensor is used to acquire the sound flow of the pipeline inspection points; and the computing module is equipped with a pipeline anomaly detection model, a sound feature classification model, and a judgment module for judging the pipeline condition. The pipeline anomaly detection model and the sound feature classification model are respectively used to judge pipeline leakage based on the image information and the sound flow.

[0008] This invention uses a combination of sound, visible light, and thermal imaging sensors to monitor abnormal pipe leaks. This prevents high false detection rates caused by external interference from a single sensor, and the data from the three sensors is cross-validated from three dimensions, resulting in higher accuracy.

[0009] In some implementations, the pipeline anomaly detection model employs an optimized target detection network, MHSA-YOLOv8, which incorporates a DW-MHSA structural module consisting of a multi-head attention mechanism and depthwise separable convolutions. The addition of the DW-MHSA module enables the model to effectively capture local features and model long-range dependencies.

[0010] In some implementations, the MHSA-YOLOv8 network consists of a backbone network and a neck network. The 1st, 2nd, 4th, 6th, and 8th layers of the backbone network are convolutional layers, and the 3rd, 5th, 7th, and 9th layers are multi-head self-attention modules. The convolutional layers include 3×3 depthwise convolutional layers and 1×1 pointwise convolutional layers. The calculation formula for the DW-MHSA structure is as follows: F(.)=(DW_{Conv})(MHSA(.)).

[0011] The novel network incorporating the DW-MHSA structure has lower computational and parameter requirements than the original YOLOv8 network. DW-Conv can capture more local features and better understand local details. Multi-head attention (MHSA) can capture long-distance dependencies between different locations in the feature map, enabling the model to better understand the global structure of the image. Furthermore, MHSA can dynamically learn attention weights based on different input image features, thus better adapting to different image tasks. The combination of DW-Conv and MHSA not only allows the model to better capture local image features but also models long-distance relationships, ensuring a lightweight model and significantly improving its ability to learn image features. This embodiment employs a more advanced deep learning feature extraction network, fusing multi-head attention (MHSA) into a convolutional neural network. This allows the network to capture long-distance dependencies between different locations in the feature map, thereby better understanding the global structure of the image and enhancing the network's feature extraction capabilities.

[0012] Furthermore, the visible light sensor, thermal imaging sensor, and / or the sound sensor can detect multiple detection points. This avoids the waste of resources caused by setting up a sensor at each detection point, as well as the problem of mutual interference due to overly dense sensor installation.

[0013] In some embodiments of the present invention, the device further includes a control module for controlling the activation of the thermal imaging sensor and / or the sound sensor. By activating a sensor only when it is needed, the control module avoids the waste of resources caused by activating all sensors simultaneously and extends the sensor's lifespan.

[0014] In some embodiments, the system also stores address information of the visible light sensor, the thermal imaging sensor, and / or the sound sensor, as well as coordinate information of each detection point. By storing the coordinates and address information of each detection point and sensor, the system can quickly locate the target sensor or target detection point when an anomaly is detected, reducing computational load and time, and improving detection efficiency.

[0015] On the other hand, the present invention also provides a pipeline condition monitoring method based on multi-sensor fusion, comprising: a visible light sensor acquiring image information of pipeline detection points and transmitting it to a computing module;

[0016] The calculation module uses a pipeline anomaly detection model to infer and detect pipeline leaks from the image information. If the pipeline anomaly detection model detects a leak, it uses a thermal imaging sensor to measure the temperature of the pipeline detection point and transmits the result to the calculation module. If the pipeline anomaly detection model result is empty, it uses a sound sensor to acquire the sound stream from the pipeline detection point and transmits the sound stream to the calculation module. The calculation module uses a sound feature classification model to determine dripping conditions. The calculation module combines the image information to identify pipeline leaks, thermal imaging results, and / or dripping conditions to determine the pipeline status.

[0017] Furthermore, the calculation module combines the image information to identify pipe leakage, thermal imaging temperature measurement results, and / or dripping conditions to determine the pipe status, including: if the thermal imaging temperature measurement result indicates leakage, and the image information or sound flow judgment result indicates pipe leakage, then the pipe status is determined to be abnormal; if the image information and the sound flow judgment result both indicate that the pipe is normal, then the pipe is determined to be normal.

[0018] Furthermore, before performing thermal imaging temperature measurement on the pipeline detection point using a thermal imaging sensor, and / or before acquiring the sound flow of the pipeline detection point using a sound sensor, the coordinate information of the detection point is acquired, the address information of the thermal imaging sensor and / or sound sensor at the detection point is acquired based on the coordinate information, and the thermal imaging sensor and / or sound sensor at the detection point is controlled to be turned on.

[0019] Finally, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method. Attached Figure Description

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 This is a schematic diagram of the pipeline condition monitoring system with multi-sensor fusion according to Embodiment 1 of the present invention;

[0022] Figure 2 This is a schematic diagram of the training process of the pipeline abnormal state detection model in Embodiment 1 of the present invention;

[0023] Figure 3 This is a structural diagram of the pipeline abnormal state detection model DW-MHSA of Embodiment 1 of the present invention;

[0024] Figure 4This is a flowchart of the training process for the sound feature classification model in Embodiment 1 of the present invention;

[0025] Figure 5 This is a schematic diagram of the pipeline condition monitoring method using multi-sensor fusion according to Embodiment 1 of the present invention. Detailed Implementation

[0026] In various engineering projects, pipeline inspections and monitoring are frequently required. Currently, manual inspections or 24-hour monitoring are commonly used. However, manual inspections are prone to fatigue and often result in missed detections. While some technologies can detect leaks by installing leak detection devices, this requires installing numerous devices at intervals along each pipeline, leading to high installation costs and the need to modify existing pipelines, which also incurs high costs. Other technologies detect leaks by sound, converting the leak audio into a Mel spectrogram and then using deep learning-based image recognition technology to classify the Mel spectrogram to determine if a leak has occurred. This method, relying solely on sound to identify leaks, is easily affected by external noise, resulting in numerous false positives. Furthermore, its implementation is very complex and it cannot pinpoint the location of the leak.

[0027] This invention addresses the problems of high labor costs, low accuracy, low scalability, and limited versatility in current inspection methods by proposing a pipeline status monitoring system and method based on multi-sensor fusion. This system can monitor the pipeline status in real time and accurately detect and locate abnormalities such as leaks, forming a real-time pipeline status management backtracking system that greatly saves costs.

[0028] This invention uses machine learning-based technology to fuse and analyze data from multiple sensors to monitor the status of water pipelines. It can detect abnormalities such as leaks in water pipelines in the first instance. Multi-sensor fusion verifies the pipeline with abnormalities from multiple dimensions and angles to ensure the accuracy of anomaly detection. High-precision machine learning algorithms process the data from each sensor in real time.

[0029] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this invention are only for explaining this invention and are not intended to limit this invention.

[0030] It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments, which also fall within the scope of protection of this invention.

[0031] In the description of this invention, the following terms need to be explained:

[0032] For directional terms, the terms "center," "lateral," "longitudinal," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise" used in the specification and claims of this application indicate the orientation and positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. They should not be construed as limiting the specific protection scope of the present invention.

[0033] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0034] The terms “comprising” and “having”, and any variations thereof, in the specification and claims of this application are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0035] When an element is referred to as being "fixed to" or "set on" another element, it can be directly on the other element or may have an intervening element present. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or may have an intervening element present. When an element is referred to as being "provided with" another element, it can be located on the surface or inside the element.

[0036] Unless otherwise expressly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "comprising" shall be understood to include the stated elements or components without excluding other elements or other components.

[0037] In the specification and claims of this application, the term "electrical connection" can refer to a physical contact circuit connection, a communication connection, a wired communication connection, or a wireless communication connection.

[0038] Example 1

[0039] First, this embodiment provides a multi-sensor fusion pipeline condition monitoring system, such as... Figure 1As shown, the system includes a visible light sensor, a thermal imaging sensor, a sound sensor, and a computing module. The visible light sensor acquires image information of the pipeline inspection point; the thermal imaging sensor performs thermal imaging temperature measurement on the pipeline inspection point; the sound sensor acquires the sound stream of the pipeline inspection point; and the computing module deploys a pipeline anomaly detection model, a sound feature classification model, and a judgment module to determine the pipeline status. The pipeline anomaly detection model and the sound feature classification model are used to determine pipeline leakage based on the image information and the sound stream, respectively. By fusing sound, visible light, and thermal imaging sensors to monitor pipeline leakage anomalies, the high false detection rate caused by external interference from a single sensor can be prevented. Furthermore, the data from the three sensors cross-validates from three dimensions, resulting in higher accuracy.

[0040] It is worth noting that the solution in this embodiment has strong generalization ability, high scalability, and is combinable. This invention can be a mobile device for monitoring abnormal pipe leaks, integrating sensors, a computing module, and a control module into a single mobile device to achieve dynamic monitoring. It can be composed of the computing module and three types of sensors in any combination. For example, it could be a smart camera device consisting of a computing module and a visible light sensor camera, or a smart sound device consisting of a sound sensor and a computing module. These combinations can be freely adapted to different scenario requirements. Alternatively, the modules can be deployed separately in different locations. For instance, the sensors could be deployed at the pipe, while the computing module and / or control module could be deployed on a robot or mobile device, or on a remote processing system, allowing for flexible design based on different usage requirements and environmental conditions.

[0041] Visible light sensors and thermal imaging sensors can be separate sensors or integrated into a single device, such as a thermal imaging dual-spectrum PTZ camera. By pre-setting inspection points on the PTZ camera, inspection points can be set at flanges in pipelines where leaks are likely to occur. If a leak is detected, the location of the leak can be accurately pinpointed, resulting in more real-time monitoring, more accurate positioning, and lower cost.

[0042] Specifically, visible light sensors, thermal imaging sensors, and / or the aforementioned sound sensors are strategically positioned at locations on the pipeline that are prone to leaks or are critical points. Depending on the specific site conditions and sensor performance, a single sensor can monitor multiple consecutive detection points, avoiding the cost waste of setting up a sensor at each detection point and the problem of mutual interference caused by densely packed sensor installations. At points where sensor performance is limited or critical, a dedicated sensor can monitor only one detection point, recording the coordinate information of each detection point, as well as the information of the detection points to which each sensor is set, forming the sensor's address information. In this way, when an anomaly is detected, the target sensor or target detection point can be quickly located using the coordinate and address information, reducing computational load and time, and improving detection efficiency.

[0043] Optionally, these sensors do not all need to be activated simultaneously. A control module can be configured to control which sensor is activated based on actual detection needs or the detection results of each sensor, thereby saving resources and extending the sensor's lifespan. For example, if the visible light sensor acquires image information of a detection point and sends it to the computing module, the computing module uses a pipeline anomaly detection model to determine whether there is an abnormal water spraying state in the pipeline. If the model detects an abnormal water spraying state, it returns the coordinates of the spraying area and then activates the thermal imaging sensor in that area to perform thermal imaging temperature measurement at the current inspection point. It compares the temperature of the spraying area coordinates with the temperature of the surrounding area and the average temperature. If a difference is found, it outputs a pipeline anomaly status; otherwise, if no difference is found, it outputs a normal pipeline status. This process does not require the participation of the sound sensor, which can be put into sleep mode to save energy. However, if the above pipeline anomaly detection model detects nothing, the sound sensor is activated to collect the sound flow around the pipeline and transmit it to the computing module. The computing module's sound feature classification model then determines whether there is a leak. As can be seen, the sound sensor only activates when the visible light sensor detects no leakage. The sound sensor double-checks whether there is truly no leakage, preventing false detections due to small leaks, insufficient ambient light, or a malfunctioning visible light sensor. If the visible light sensor detects a leak, the thermal imaging sensor is activated directly to confirm the leak without needing to activate the sound sensor. Furthermore, the combined use of the light and sound sensors can also indicate the severity of the leak. The visible light and sound sensors are responsible for identifying severe and minor leaks, respectively, catering to different leakage levels. For example, some leaks cannot be detected by the visible light sensor; they can only be detected in cases of large leaks or spraying water, while the sound sensor can detect minor leaks, such as occasional drips or very low-frequency drips.

[0044] This embodiment utilizes machine learning technology to fuse and analyze data from multiple sensors for status analysis and monitoring of water pipelines. It can detect anomalies such as leaks in the pipeline in real time. Multi-sensor fusion verifies anomalies from multiple dimensions and angles, ensuring the accuracy of anomaly detection. A high-precision machine learning algorithm processes data from each sensor in real time. The technical implementation of this embodiment consists of two parts: first, an optimized machine learning algorithm; and second, a multi-sensor data fusion processing part.

[0045] First, the pipeline anomaly detection model in the computation module adopts the optimized target detection network MHSA-YOLOv8. The MHSA-YOLOv8 network adds a DW-MHSA structure module composed of a multi-head attention mechanism and depthwise separable convolutions. The addition of the DW-MHSA module enables the model to effectively capture local features and model long-range dependencies.

[0046] Specifically, firstly, images of normal pipeline conditions and water spraying are collected. Then, data augmentation is used to increase the number of original images. Next, the images spraying water are manually labeled. Normal pipeline data is also added as background data to the detection model for training. After multiple rounds of iterative optimization, the optimal pipeline anomaly detection model is obtained and finally deployed on the computing module of our device. In this embodiment, the detection model used is the optimized target detection network MHSA-YOLOv8. The training process of the optimal pipeline anomaly detection model is as follows: Figure 2As shown, the MHSA-YOLOv8 network adds a DW-MHSA structural module composed of multi-head attention (MHSA) and depthwise separable convolution (DW-Conv). The addition of DW-MHSA enables the model to effectively capture local features and model long-range dependencies. The new YOLOv8 network consists of a backbone network and a neck network. The backbone network consists of 10 bl ock modules, of which layers 1, 2, 4, 6, and 8 are bl ock modules. `ock` represents the Conv layer, and layers 3, 5, 7, and 9 are DW-MHSA modules. DW-Conv consists of two different convolutional layers: a 3×3 layer and a 1×1 layer. The 3×3 convolution is a depthwise convolution, which performs convolution operations independently on each feature channel without cross-channel feature interaction. The 1×1 convolution is a pointwise convolution, which performs 1×1 convolution operations on all feature channels and performs cross-channel feature fusion. Multi-head attention (MHSA) decomposes the input feature map into multiple heads, performing attention calculations independently on each head. Each head contains three linear projection matrices, representing the query (Q), key (K), and value (V), respectively. Attention weights are obtained by calculating the similarity between the query (Q) and key (K), and the values ​​(V) are weighted and summed to obtain the final output feature map. The calculation formula for the DW-MHSA structure is as follows:

[0047] F(.)=(DW_{Conv})(MHSA(.))

[0048] The structure of DW-MHSA is as follows: Figure 3 As shown, the 10th layer of the backbone network is an SPPF structure. The main function of the SPPF structure is to perform multi-scale fusion of features extracted by the backbone network, reduce the computational cost of the backbone network, and expand the receptive field.

[0049] Through the above model optimization, a new network with the DW-MHSA structure is obtained, which has lower computational and parameter requirements than the original YOLOv8 network. DW-Conv can capture more local features and better understand the local details of features. Multi-head attention (MHSA) can capture long-distance dependencies between different locations in the feature map, thus enabling the model to better understand the global structure of the image. Furthermore, MHSA can dynamically learn attention weights based on different image features of the input, thus better adapting to different image tasks. The combination of DW-Conv and MHSA not only allows the model to better capture local image features but also to model long distances, ensuring the model's lightweight nature and greatly improving the model's ability to learn image features.

[0050] In the Neck part of the model, layers 11 and 14 are upsampling layers (using the nearest neighbor difference method to enlarge the feature map), layers 12, 15, 18, and 20 are concat layers, used to stitch feature maps from different dimensions, mainly by adding them on the channels, layers 17 and 20 are CBL structures (convolutional layer-batch layer-activation function layer), and layers 13, 16, 19, and 22 are C2f structures.

[0051] Secondly, the sound feature classification model in the computing module first collects a large amount of dripping water sounds and other non-dripping water sounds as training data using sound sensors. Then, the obtained sound data is uniformly cropped to a fixed 3-second length, followed by preprocessing such as noise reduction (removing environmental noise) and enhancement. Then, feature extraction is performed. The key steps in feature extraction are: first, a Fast Fourier Transform (FFT) is performed to convert the time domain to the frequency domain to obtain spectral information; then, a Mel-Filter Bank is used to filter the frequency domain; then, a logarithmic transform is performed; finally, a Discrete Cosine Transform (DCT) is applied to the filtered logarithmic energy vector to obtain the MFCCs feature vector values. MFCCs features can effectively represent the spectral characteristics of sound signals and have a certain robustness to noise and pitch changes. After obtaining the sound's MFCCs features, an SVM classification model is trained. After training iterations, the optimal SVM classification model is obtained and then deployed on the computing module of our device. The optimal sound feature classification model training process is as follows: Figure 4 As shown.

[0052] The pipeline anomaly detection model and sound feature classification model obtained from the above scheme are deployed into the computing module, and then multi-sensor data fusion is performed. The pipeline condition monitoring method based on multi-sensor fusion is used for monitoring, as detailed below.

[0053] A multi-sensor fusion method for pipeline condition monitoring, such as Figure 5 As shown. The visible light sensor acquires image information of pipeline inspection points and transmits it to the computing module. Taking the above thermal imaging dual-spectrum PTZ camera as an example, firstly, multiple inspection points are set at pipeline flanges on the thermal imaging dual-spectrum PTZ camera, and the location information of each inspection point is recorded. Then, the visible light sensor of the thermal imaging dual-spectrum PTZ camera is used to acquire visible light images of each inspection point. The obtained image stream data is then transmitted via HTTP to the optimal pipeline abnormality detection model deployed on the computing module.

[0054] The calculation module uses a pipeline anomaly detection model to infer and detect pipeline leaks from the image information.

[0055] If the pipeline anomaly detection model detects abnormal water spraying in the pipeline, it obtains the coordinate information of the detection point of the abnormal water spraying in the pipeline, obtains the address information of the thermal imaging sensor at the detection point based on the coordinate information, controls the thermal imaging sensor to be turned on, and performs thermal imaging temperature measurement on the pipeline detection point through the thermal imaging sensor. It compares the temperature of the water spraying area coordinates with the temperature of the surrounding area and the average temperature. If a difference is found, it outputs that the pipeline is in an abnormal state. If no difference is found in the temperature comparison, it outputs that the pipeline is in a normal state.

[0056] If the pipeline anomaly detection model result is empty, obtain the coordinate information of the detection point of the above-mentioned abnormal water spraying in the pipeline, obtain the address information of the sound sensor at the detection point according to the coordinate information, control the sound sensor to turn on, obtain the sound flow around the pipeline detection point through the sound sensor, and transmit the sound flow to the calculation module via HTTP.

[0057] The calculation module determines the dripping condition using a sound feature classification model. The calculation module first needs to cut the sound stream to a fixed length of 3 seconds, then preprocess it and extract features to feed it into the optimal sound feature classification model to obtain the classification result. If the classification result indicates a dripping condition, the thermal imaging sensor needs to be activated to perform thermal imaging temperature measurement on the current sound acquisition point, just like the steps mentioned above. If the temperature of the dripping area is abnormal compared to the temperature of other areas, the abnormal pipe status is directly output; otherwise, the normal pipe status is output.

[0058] Finally, this embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0059] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pipeline condition monitoring system based on multi-sensor fusion, characterized in that, The system includes a visible light sensor, a thermal imaging sensor, a sound sensor, and a computing module. The visible light sensor acquires image information of the pipeline inspection point; the thermal imaging sensor performs thermal imaging temperature measurement on the pipeline inspection point; the sound sensor acquires the sound stream of the pipeline inspection point; and the computing module deploys a pipeline anomaly detection model, a sound feature classification model, and a judgment module to determine the pipeline status. The pipeline anomaly detection model and the sound feature classification model are used to determine pipeline leakage based on the image information and the sound stream, respectively. The pipeline anomaly detection model employs an optimized target detection network, MHSA-Y. The object detection network MHSA-YOLOv8 adds a DW-MHSA structure module composed of a multi-head attention mechanism and depthwise separable convolution. The object detection network MHSA-YOLOv8 consists of a backbone network and a neck network. The blocks of layers 1, 2, 4, 6, and 8 of the backbone network are convolutional layers, and layers 3, 5, 7, and 9 are multi-head self-attention modules. The convolutional layers include 3×3 depthwise convolutional layers and 1×1 pointwise convolutions. The calculation formula of the DW-MHSA structure is as follows: F(.)=(DW_{Conv})(MHSA(.)); The visible light sensor, thermal imaging sensor, and / or the sound sensor can detect multiple detection points; the system also includes a control module for controlling the activation of the thermal imaging sensor and / or the sound sensor; the system also stores the coordinate information of the visible light sensor, the thermal imaging sensor, and / or the sound sensor, as well as the coordinate information of each detection point.

2. A pipeline condition monitoring method for a multi-sensor fusion pipeline monitoring system according to claim 1, characterized in that, include: The visible light sensor acquires image information of the pipeline inspection points and transmits it to the computing module; The calculation module uses a pipeline anomaly detection model to infer and detect pipeline leaks from the image information. If the pipeline anomaly detection model detects abnormal water leakage in the pipeline, it uses a thermal imaging sensor to perform thermal imaging temperature measurement on the pipeline detection point and transmits the thermal imaging temperature measurement result to the calculation module; if the pipeline anomaly detection model result is empty, it uses a sound sensor to obtain the sound stream of the pipeline detection point and transmits the sound stream to the calculation module. The calculation module determines the dripping situation using a sound feature classification model. The calculation module combines the image information to identify pipe leakage, thermal imaging temperature measurement results, and / or dripping conditions to determine the pipe status.

3. The pipeline condition monitoring method according to claim 2, characterized in that, The calculation module combines the image information to identify pipe leakage, thermal imaging temperature measurement results, and / or dripping conditions to determine the pipe status, including: if the thermal imaging temperature measurement result indicates leakage, and the image information or sound flow result indicates pipe leakage, then the pipe status is determined to be abnormal; if both the image information and the sound flow result indicate that the pipe is normal, then the pipe is determined to be normal.

4. The pipeline condition monitoring method according to claim 3, characterized in that, Before performing thermal imaging temperature measurement on the pipeline inspection point using a thermal imaging sensor, and / or before acquiring the sound flow of the pipeline inspection point using a sound sensor, the coordinate information of the inspection point is obtained, and the address information of the thermal imaging sensor and / or sound sensor at the inspection point is obtained based on the coordinate information, and the thermal imaging sensor and / or sound sensor at the inspection point is controlled to be turned on.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the pipeline condition monitoring method according to any one of claims 2 to 4.

Images