Multi-sensor fusion pipeline state monitoring system and method

Abstract

The embodiment of the invention relates to the technical field of pipeline monitoring, and discloses a multi-sensor fusion pipeline state monitoring system and method. According to the invention, state analysis monitoring is carried out on the water delivery pipeline by fusing and analyzing data of various sensors such as sound, visible light and thermal imaging based on a machine learning technology, so that abnormal conditions such as water leakage of the water delivery pipeline can be detected at the first time; multi-sensor fusion performs multi-dimensional and multi-angle verification on a pipeline with an abnormal condition, the accuracy of anomaly detection is ensured, a water leakage point is quickly positioned through a sensor and detection point coordinate tracking mode, a more advanced deep learning feature extraction network is adopted, and the detection accuracy is improved. The high-precision machine learning algorithm processes the data of each sensor in real time, so that the global structure of the image is better understood, and the feature extraction capability of the network is stronger.

Description

Technical Field

[0001] The present invention relates to the technical field of pipeline monitoring, and in particular to a pipeline status monitoring system and method using multi-sensor fusion. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] In various engineering occasions, pipelines are very common equipment. In order to ensure the normal operation of the project, it is often necessary to patrol or monitor the pipeline to prevent the medium in the pipeline from leaking out due to local damage or performance failure of the pipeline. Due to the long length of the pipeline, the hidden layout and complex environment, the inspection of the pipeline is more inconvenient. At present, manual inspection or 24-hour monitoring inspection is mostly adopted. The manual inspection method requires a large amount of manpower, and the manpower cost is high. In the 24-hour monitoring inspection method, the inspectors are prone to fatigue, which often leads to missed inspections, and the status of each pipeline cannot be accurately reflected. The management cost is high and it is impossible to trace the source. Although the prior art has improved the manual inspection method, such as Chinese patent CN118654239A, a special water leakage detection device is installed on the pipeline. When the pipeline leaks, it flows into the water storage box and triggers a water leakage alarm. Although this method can detect the occurrence of water leakage, it is necessary to install a lot of devices on each pipeline at intervals, the installation cost is high, and the original pipeline needs to be modified, and the modification cost is high.

[0004] Therefore, the prior art needs to be improved. Summary of the invention

[0005] In order to solve the problems of high labor cost, low precision, low scalability and weak versatility in the current inspection method, the present invention proposes a pipeline status monitoring system and method based on multi-sensor fusion, which can accurately monitor the status of the pipeline in real time, accurately detect and locate abnormal conditions such as water leakage, form a management backtracking of the real-time status of the pipeline, and greatly save costs.

[0006] In order to achieve the above object, the present invention adopts the following technical solution.

[0007] First, on the one hand, the present invention provides a multi-sensor fusion pipeline state monitoring system, including a visible light sensor, a thermal imaging sensor, a sound sensor and a computing module, the visible light sensor is used to obtain image information of a pipeline detection point; the thermal imaging sensor is used to perform thermal imaging temperature measurement on the pipeline detection point, the sound sensor is used to obtain the sound stream of the pipeline detection point, the computing module is deployed with a pipeline anomaly detection model, a sound feature classification model and a judgment module for judging the pipeline state, the pipeline anomaly detection model and the sound feature classification model are used to judge the pipeline leakage according to the image information and the sound stream, respectively.

[0008] The present invention monitors abnormal water leakage in pipelines by integrating three sensors: sound, visible light and thermal imaging. This can prevent a single sensor from having a high false detection rate due to interference from external factors. The three sensors can verify each other through data from three dimensions, which has a higher accuracy rate.

[0009] In some embodiments, the pipeline anomaly detection model uses an optimized target detection network MHSA-YOLOv8, and the MHSA-YOLOv8 network adds a DW-MHSA structure module consisting of a multi-head attention mechanism and a deep separable convolution. The addition of the DW-MHSA module enables the model to effectively capture local features and model long-distance dependencies.

[0010] In some embodiments, the MHSA-YOLOv8 network is composed of a backbone network and a neck network, the 1st, 2nd, 4th, 6th, and 8th layers of the backbone network are block convolutional layers, the 3rd, 5th, 7th, and 9th layers are multi-head self-attention modules, the convolutional layers include 3×3 depth convolutional layers and 1×1 point-by-point convolutions, and the calculation formula of the DW-MHSA structure is as follows: F(.)=(DW_{Conv})(MHSA(.)).

[0011] The new network with the DW-MHSA structure has lower computational complexity and parameter count than the original YOLOv8 network, and DW-Conv can capture more local features and better understand local details of features. Multi-head attention (MHSA) can capture long-distance dependencies between different positions of feature maps, so that the model can better understand the global structure of the image, and MHSA can dynamically learn attention weights according to different input image features, so that it can better adapt to different image tasks. The combination of DW-Conv and MHSA not only allows the model to better capture local features of the image, but also models long distances, ensuring the lightweight of the model and greatly improving the model's ability to learn image features. This embodiment uses a more advanced deep learning feature extraction network. By integrating multi-head attention (MHSA) in a convolutional neural network, the network can capture long-distance dependencies between different positions of the feature map, so as to better understand the global structure of the image and make the network's feature extraction capability stronger.

[0012] Furthermore, the visible light sensor, thermal imaging sensor and / or sound sensor can detect multiple detection points, thus avoiding the problem of setting a sensor for each detection point, resulting in cost waste and the problem of sensors being installed too closely and interfering with each other.

[0013] In some embodiments of the present invention, the device further includes a control module for controlling the activation of the thermal imaging sensor and / or the sound sensor. The control module activates a certain sensor only when it is needed, thereby avoiding resource waste caused by activating all sensors at the same time and extending the service life of the sensors.

[0014] In some embodiments, the system also stores the address information of the visible light sensor, the thermal imaging sensor and / or the sound sensor, and the coordinate information of each detection point. The system stores the coordinate and address information of each detection point and sensor, and can quickly find the target sensor or target detection point through the coordinate information and address information when an abnormality is found, thereby reducing the amount of calculation and calculation time and improving the detection efficiency.

[0015] On the other hand, the present invention also provides a pipeline status monitoring method using multi-sensor fusion, comprising: a visible light sensor acquires image information of a pipeline detection point and transmits the image information to a computing module;

[0016] The computing module performs inference detection on the image information through the pipeline anomaly detection model to identify the pipeline leakage; if the pipeline anomaly detection model detects abnormal pipeline leakage, thermal imaging temperature measurement is performed on the pipeline detection point through the thermal imaging sensor, and the thermal imaging temperature measurement result is transmitted to the computing module; if the result of the pipeline anomaly detection model is empty, the sound stream of the pipeline detection point is obtained through the sound sensor, and the sound stream is transmitted to the computing module; the computing module determines the dripping situation through the sound feature classification model, and the computing module determines the pipeline state by combining the image information to identify the pipeline leakage situation, the thermal imaging temperature measurement result and / or the dripping situation.

[0017] Furthermore, the calculation module identifies pipeline leakage, thermal imaging temperature measurement results and / or dripping conditions in combination with the image information to determine the pipeline status, including: if the thermal imaging temperature measurement result is leakage, and the image information or the sound stream judgment result is pipeline leakage, then the pipeline status is judged to be abnormal; if the image information and the sound stream judgment results are both that the pipeline is normal, then the pipeline is judged to be normal.

[0018] Furthermore, before performing thermal imaging temperature measurement on the pipeline detection point by the thermal imaging sensor and / or obtaining the sound flow of the pipeline detection point by the sound sensor, the coordinate information of the detection point is obtained, and the address information of the thermal imaging sensor and / or the sound sensor at the detection point is obtained according to the coordinate information, and the thermal imaging sensor and / or the sound sensor at the detection point is controlled to turn on.

[0019] Finally, the present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 Schematic diagram of a pipeline state monitoring system using multi-sensor fusion according to Embodiment 1 of the present invention;

[0022] Figure 2 This is a schematic diagram of the pipeline abnormal state detection model training process of Example 1 of the present invention;

[0023] Figure 3 1 is a structural diagram of a pipeline abnormal state detection model DW-MHSA according to Example 1 of the present invention;

[0024] Figure 4This is a flow chart of the sound feature classification model training in Example 1 of the present invention;

[0025] Figure 5 It is a schematic diagram of a pipeline status monitoring method using multi-sensor fusion according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0026] In various engineering situations, it is often necessary to inspect or monitor pipelines. Currently, manual inspections or 24-hour monitoring inspections are mostly used. Manual inspections are prone to fatigue and often lead to missed detections. Although some technologies can detect water leaks by installing leak detection devices, it is necessary to install a large number of devices at intervals on each pipeline, which has high installation costs and requires the original pipelines to be modified, which has high modification costs. Some technologies detect water leaks by the sound of the water leak, convert the water leak audio into a Mel language spectrogram, and then use deep learning-based image recognition technology to classify the Mel language spectrogram of the water leak audio to determine whether it is a water leak. This method only identifies water leaks by sound, is easily interfered by external sounds, has many misidentifications, is very complicated to implement, and cannot locate the leaking area.

[0027] In order to solve the problems of high labor cost, low precision, low scalability and weak versatility in the current inspection method, the present invention proposes a pipeline status monitoring system and method based on multi-sensor fusion, which can accurately monitor the status of the pipeline in real time, accurately detect and locate abnormal conditions such as water leakage, form a management backtracking of the real-time status of the pipeline, and greatly save costs.

[0028] The present invention uses machine learning-based technology to fuse and analyze data from multiple sensors to perform status analysis and monitoring of water pipelines. It can detect abnormal conditions such as water leakage in the water pipeline in the first place. Multi-sensor fusion performs multi-dimensional and multi-angle verification of pipelines with abnormal conditions to ensure the accuracy of abnormality detection. High-precision machine learning algorithms process the data of each sensor in real time.

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0030] It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments, which also fall within the scope of protection of the present invention.

[0031] In the description of the present invention, the following terms need to be explained:

[0032] As for directional words, the terms "center", "lateral", "longitudinal", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicating directions and positional relationships in the specification and claims of this application are based on the directions or positional relationships shown in the drawings and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and should not be understood as limiting the specific scope of protection of the present invention.

[0033] The terms "first", "second", etc. in the description and claims of the present application are used to distinguish similar objects and are not necessarily used to describe a particular sequence or order.

[0034] The terms "including" and "having" and any variations thereof in the specification and claims of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0035] When an element is referred to as being "fixed to" or "disposed on" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. When an element is referred to as being "provided with" another element, it may be provided on the surface or inside of the element.

[0036] Unless explicitly stated otherwise, throughout the specification and claims, the term "comprise" or its variations such as "include" or "comprising", etc. will be understood to include the stated elements or components but not to exclude other elements or other components.

[0037] The term "electrical connection" in the specification and claims of this application may refer to a circuit connection with physical contact or a communication connection, which may be a wired communication connection or a wireless communication connection.

[0038] Example 1

[0039] First, this embodiment provides a pipeline status monitoring system with multi-sensor fusion, such as Figure 1As shown, it includes a visible light sensor, a thermal imaging sensor, a sound sensor and a computing module. The visible light sensor is used to obtain image information of pipeline detection points; the thermal imaging sensor is used to perform thermal imaging temperature measurement on pipeline detection points, and the sound sensor is used to obtain the sound flow of pipeline detection points. The computing module is deployed with a pipeline anomaly detection model, a sound feature classification model and a judgment module for judging the pipeline state. The pipeline anomaly detection model and the sound feature classification model are used to judge the pipeline leakage according to the image information and the sound flow respectively. By integrating the three sensors of sound, visible light and thermal imaging to monitor the abnormal state of pipeline leakage, it can prevent a single sensor from having a high false detection rate due to interference from external factors, and the three sensors can verify each other from data in three dimensions, with a higher accuracy rate.

[0040] It is worth noting that the solution of this embodiment has strong generalization ability, high expansibility, and can be combined. The present invention can be a mobile device for monitoring abnormal pipeline leakage. The sensor, computing module, and control module are all integrated in a mobile device to achieve dynamic monitoring. It can be composed of a computing module plus any combination of three sensors, such as a smart camera device composed of a computing module + a visible light sensor camera, or a smart sound device composed of a sound sensor + a computing module. It can be combined at will according to different scene requirements. It is also possible to deploy the above modules separately in different places, such as deploying sensors in pipelines, and deploying computing modules and / or control modules in robots or mobile devices, or in remote processing systems, and flexibly design according to different usage requirements and environmental conditions.

[0041] Visible light sensors and thermal imaging sensors can be separate sensors or integrated into one device, such as a thermal imaging dual-spectrum dome camera. By presetting the inspection points of the dome camera, inspection points can be set at the flanges where abnormal leakage is likely to occur in the pipeline. If the abnormal leakage is identified, the leakage position can be accurately located, achieving more real-time monitoring, more accurate positioning, and lower cost.

[0042] Specifically, the visible light sensor, thermal imaging sensor and / or the sound sensor are arranged at the leak-prone or important points of the pipeline. According to the specific site conditions and the performance of the sensor, one sensor can monitor multiple continuous detection points, avoiding the problem of setting a sensor for each detection point, resulting in cost waste and the problem of sensors being installed too densely and interfering with each other. For points with limited sensor performance or very important points, one sensor can also be used to monitor one detection point, record the coordinate information of each detection point, and the information of the detection point set by each sensor, to form the address information of the sensor. In this way, when an abnormality is found, the target sensor or target detection point can be quickly found through the coordinate information and address information, reducing the amount of calculation and calculation time, and improving the detection efficiency.

[0043] Optionally, these sensors may not all be turned on at the same time, and a control module may be configured to control which sensor to turn on according to the actual detection requirements or the detection results of each sensor, thereby saving resources and extending the service life of the sensor. For example, if the visible light sensor obtains the image information of a certain detection point and sends it to the calculation module, the calculation module uses the pipeline anomaly detection model to judge based on the image information whether the pipeline is in an abnormal water spraying state: if the model detection result is an abnormal water spraying state of the pipeline, the coordinates of the water spraying area are returned, and then the thermal imaging sensor in the area is started to perform thermal imaging temperature measurement on the current inspection point, and the temperature of the water spraying area coordinates is compared with the difference between the temperature of the nearby area and the average temperature. If there is a difference, the pipeline abnormal state is output, otherwise the temperature comparison does not show a difference, and the pipeline normal state is output. This process does not require the participation of the sound sensor, and the sound sensor can be put into sleep to save energy. However, if the detection result of the above pipeline anomaly detection model is empty, the sound sensor is started to collect the sound flow around the pipeline and transmit it to the calculation module, and the sound feature classification model of the calculation module determines whether there is a water leak. It can be seen that the sound sensor is activated only when the visible light sensor detects that there is no water leakage. The sound sensor is used to confirm whether there is really no water leakage again to prevent false detection due to small water leakage, insufficient ambient light or visible light sensor failure. If the visible light sensor detects a water leakage, the thermal imaging sensor is directly activated to directly determine whether there is a water leakage through the thermal imaging sensor without activating the sound sensor. In addition, the cooperation of the light sensor and the sound sensor can also reflect the degree of abnormal leakage. The visible light sensor and the sound sensor are responsible for identifying the two abnormal states of severe leakage and mild leakage respectively, which can meet the needs of different degrees of leakage. For example, some water leakage cannot be detected by the visible light sensor. Only when there is a large amount of water leakage or water spraying, the visible light sensor can detect it, while the sound sensor can detect lighter water leakage, such as occasional dripping, and dripping with very low frequency can be detected.

[0044] This embodiment uses machine learning technology to fuse and analyze the data of multiple sensors to perform state analysis and monitoring of water pipelines, and can detect abnormal conditions such as water leakage in the water pipeline in the first time. Multi-sensor fusion verifies the pipeline with abnormal conditions in multiple dimensions and angles to ensure the accuracy of abnormality detection. High-precision machine learning algorithms process the data of each sensor in real time. The technical implementation scheme of this embodiment consists of two parts: first, the optimized machine learning algorithm part; second, the multi-sensor data fusion processing part.

[0045] First, the pipeline anomaly detection model in the calculation module uses the optimized target detection network MHSA-YOLOv8, and the MHSA-YOLOv8 network adds a DW-MHSA structure module consisting of a multi-head attention mechanism and deep separable convolution. The addition of the DW-MHSA module enables the model to effectively capture local features and model long-distance dependencies.

[0046] Specifically, first, it is necessary to collect images of normal pipelines and water spraying, and then enhance the number of original images through data enhancement. Then, the water spraying images are annotated by manual annotation. The normal pipeline data is also added to the detection model as background data for training. After multiple rounds of iterative optimization, the optimal pipeline abnormal state detection model is obtained, and finally deployed on the computing module of our device. The detection model used in this embodiment is the optimized target detection network MHSA-YOLOv8. The optimal pipeline abnormal state detection model training process is as follows: Figure 2As shown in the figure. The MHSA-YOLOv8 network adds a DW-MHSA structure module consisting of multi-head attention (MHSA) and deep separable convolution (DW-Conv). The addition of DW-MHSA enables the model to effectively capture local features and model long-distance dependencies. The new YOLOv8 network consists of a backbone network and a neck network. The backbone network consists of 10 black modules, of which the bl modules of the 1st, 2nd, 4th, 6th, and 8th layers are ock is the Conv layer, and the 3rd, 5th, 7th, and 9th are DW-MHSA modules: DW-Conv consists of two different convolutional layers, namely 3×3 and 1×1 convolutions. The 3×3 convolution is a deep convolution, which performs convolution operations on each feature channel independently and does not perform cross-channel feature interaction. The 1×1 convolution is a point-by-point convolution, which performs 1×1 convolution operations on all feature channels and performs cross-channel feature fusion; Multi-head attention (MHSA) decomposes the input feature map into multiple heads, and performs attention calculations on each head independently. Each head contains three linear projection matrices, which are used for query (Q), key (K), and value (V), respectively. Then, by calculating the similarity between the query (Q) and the key (K), the attention weight is obtained, and the value (V) is weighted summed to obtain the final output feature map; The calculation formula of the DW-MHSA structure is as follows:

[0047] F(.)=(DW_{Conv})(MHSA(.))

[0048] The structure of DW-MHSA is as follows Figure 3 The 10th layer of the backbone network is the SPPF structure. The main function of the SPPF structure is to perform multi-scale fusion of the features extracted by the backbone network, reduce the computational complexity of the backbone network, and expand the receptive field.

[0049] Through the above model optimization, a new network with DW-MHSA structure is obtained, which has lower computational complexity and parameter quantity than the original YOLOv8 network. DW-Conv can capture more local features and better understand the local details of features. Multi-head attention (MHSA) can capture the long-distance dependencies between different positions of the feature map, so that the model can better understand the global structure of the image. MHSA can dynamically learn attention weights according to different image features of the input, so that it can better adapt to different image tasks. The combination of DW-Conv and MHSA can not only allow the model to better capture the local features of the image, but also model the long distance, ensuring the lightweight of the model and greatly improving the model's ability to learn image features.

[0050] In the Neck part of the model, the 11th and 14th layers are upsampling layers (using the nearest neighbor difference method to enlarge the feature map), the 12th, 15th, 18th, and 20th layers are concat layers, which are used to splice feature maps of different dimensions, mainly by adding them on the channels, the 17th and 20th layers are CBL structures (convolutional layer-batch layer-activation function layer), and the 13th, 16th, 19th, and 22nd layers are C2f structures.

[0051] Secondly, the sound feature classification model in the computing module first collects a large amount of dripping sounds and other non-dripping sounds as training data through sound sensors, and then uniformly trims the obtained sound data to obtain sounds of a fixed length of 3 seconds, and then performs preprocessing, such as denoising (removing environmental noise) and enhancement, and then performs feature extraction. The key steps of feature extraction are: first, a fast Fourier transform is required to convert the time domain into the frequency domain to obtain spectrum information, and then a Mel filter group is used to filter the frequency domain, and then a logarithmic transform is performed. Finally, a discrete cosine transform is performed on the filtered logarithmic energy vector to obtain the MFCCs feature vector value. The MFCCs feature can effectively represent the spectral characteristics of the sound signal, and at the same time has a certain robustness to noise and pitch changes. After obtaining the MFCCs features of the sound, it is then trained through the SVM classification model. After training iterations, the optimal SVM classification model is obtained and then deployed on the computing module of our device. The optimal sound feature classification model training process is as follows: Figure 4 shown.

[0052] The pipeline anomaly detection model and sound feature classification model obtained by the above scheme are deployed in the computing module, and then multi-sensor data fusion is performed, and monitoring is performed through the pipeline status monitoring method of multi-sensor fusion, as follows.

[0053] A pipeline condition monitoring method based on multi-sensor fusion, such as Figure 5 As shown. The visible light sensor obtains the image information of the pipeline inspection point and transmits it to the computing module; taking the above-mentioned thermal imaging dual-spectrum ball camera as an example, firstly, the thermal imaging dual-spectrum ball camera is set with multiple inspection points at the pipeline flange, and the location information of each inspection point is recorded. Then, the visible light sensor of the thermal imaging dual-spectrum ball camera is used to collect the visible light image of each inspection point, and then the obtained image stream data is transmitted to the optimal pipeline abnormal state detection model deployed on the computing module through http.

[0054] The computing module performs reasoning detection on the image information through a pipeline anomaly detection model to identify pipeline leakage;

[0055] If the pipeline abnormality detection model detects abnormal water spraying in the pipeline, the coordinate information of the detection point of the abnormal water spraying in the pipeline is obtained, and the address information of the thermal imaging sensor at the detection point is obtained according to the coordinate information, and the thermal imaging sensor is controlled to be turned on. The thermal imaging temperature measurement is performed on the pipeline detection point by the thermal imaging sensor, and the difference between the temperature of the water spraying area coordinates and the nearby area and the average temperature is compared. If there is a difference, the pipeline abnormal state is output, and if there is no difference in the temperature comparison, the pipeline normal state is output;

[0056] If the result of the pipeline abnormality detection model is empty, obtain the coordinate information of the detection point of the abnormal water spray in the pipeline, obtain the address information of the sound sensor at the detection point according to the coordinate information, control the sound sensor to turn on, obtain the sound stream around the pipeline detection point through the sound sensor, and transmit the sound stream to the calculation module through http;

[0057] The calculation module determines the dripping situation through the sound feature classification model. The calculation module needs to first fix the sound stream into a length of 3 seconds, and then preprocess it and extract the features to give it to the optimal sound feature classification model to obtain the classification result. If the classification result is dripping, it is necessary to start the thermal imaging sensor to perform thermal imaging temperature measurement on the current sound collection point as in the above steps. If the temperature in the dripping area is abnormal with the temperature in other areas, the pipeline abnormal state is directly output, otherwise the pipeline normal state is output.

[0058] Finally, this embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0059] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents, and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor fusion pipeline status monitoring system, characterized in that: The system comprises a visible light sensor, a thermal imaging sensor, a sound sensor and a computing module. The visible light sensor is used to obtain image information of pipeline detection points. The thermal imaging sensor is used to perform thermal imaging temperature measurement on pipeline detection points. The sound sensor is used to obtain sound streams at pipeline detection points. The computing module is provided with a pipeline anomaly detection model, a sound feature classification model and a judgment module for judging pipeline status. The pipeline anomaly detection model and the sound feature classification model are used to judge pipeline leakage according to the image information and the sound stream, respectively.

2. The pipeline status monitoring system according to claim 1, characterized in that: The pipeline anomaly detection model adopts an optimized target detection network MHSA-YOLOv8, and the MHSA-YOLOv8 network adds a DW-MHSA structure module consisting of a multi-head attention mechanism and a deep separable convolution.

3. The pipeline status monitoring system according to claim 2, characterized in that: The MHSA-YOLOv8 network consists of a backbone network and a neck network. The blocks of the 1st, 2nd, 4th, 6th, and 8th layers of the backbone network are convolutional layers, and the 3rd, 5th, 7th, and 9th layers are multi-head self-attention modules. The convolutional layers include 3×3 depth convolutional layers and 1×1 point-by-point convolutions. The calculation formula of the DW-MHSA structure is as follows: F(.)=(DW_{Conv})(MHSA(.)).

4. The pipeline status monitoring system according to claim 1, characterized in that: The visible light sensor, the thermal imaging sensor and / or the sound sensor may detect a plurality of detection points.

5. The pipeline status monitoring system according to claim 1, characterized in that: The system further comprises a control module for controlling the activation of the thermal imaging sensor and / or the sound sensor.

6. The pipeline status monitoring system according to claim 5, characterized in that: The system also stores coordinate information of the visible light sensor, the thermal imaging sensor and / or the sound sensor, and coordinate information of each detection point.

7. A pipeline status monitoring method based on multi-sensor fusion, characterized in that: include: The visible light sensor obtains the image information of the pipeline inspection point and transmits it to the computing module; The computing module performs reasoning detection on the image information through a pipeline anomaly detection model to identify pipeline leakage; If the pipeline anomaly detection model detects abnormal water leakage in the pipeline, the thermal imaging sensor is used to perform thermal imaging temperature measurement on the pipeline detection point, and the thermal imaging temperature measurement result is transmitted to the calculation module; If the result of the pipeline anomaly detection model is empty, the sound stream of the pipeline detection point is obtained through the sound sensor, and the sound stream is transmitted to the calculation module; The calculation module determines the dripping situation through the sound feature classification model. The computing module identifies the water leakage of the pipeline in combination with the image information, the temperature measurement result of thermal imaging and / or the water dripping situation to determine the state of the pipeline.

8. The pipeline status monitoring method according to claim 7, characterized in that: The calculation module identifies pipeline leakage, thermal imaging temperature measurement results and / or dripping conditions in combination with the image information to determine the pipeline status, including: if the thermal imaging temperature measurement result is leakage, and the image information or the sound stream judgment result is pipeline leakage, then it is determined that the pipeline status is abnormal; if the image information and the sound stream judgment results are both that the pipeline is normal, then it is determined that the pipeline is normal.

9. The pipeline status monitoring method according to claim 8, characterized in that: Before performing thermal imaging temperature measurement on the pipeline detection point by the thermal imaging sensor, and / or obtaining the sound flow of the pipeline detection point by the sound sensor, the coordinate information of the detection point is obtained, and the address information of the thermal imaging sensor and / or the sound sensor at the detection point is obtained according to the coordinate information, and the thermal imaging sensor and / or the sound sensor at the detection point is controlled to turn on.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the pipeline status monitoring method described in any one of claims 7 to 9 is implemented.

Images