Intelligent detection method for defects in pipeline based on multi-sensor fusion and deep learning

Abstract

The invention discloses an intelligent detection method for defects in a pipeline based on multi-sensor fusion and deep learning, and relates to the field of industrial defect detection. According to the technical scheme, three-dimensional point cloud data and two-dimensional image data in a pipeline are collected; performing synchronous positioning and map construction processing on the three-dimensional point cloud data and the two-dimensional image data in the pipeline to obtain inspection path data; performing independent feature extraction processing on the inspection path data to obtain visual, magnetic flux leakage and ultrasonic feature vector data; performing uncertainty coefficient quantification processing on the visual, magnetic flux leakage and ultrasonic feature vector data to obtain corrected joint feature data; performing graph neural network multi-level Bayesian iterative calculation processing on the corrected joint feature data to obtain global defect judgment probability data; and processing the defect judgment probability data associated with the pose data to obtain three-dimensional coordinate mapping to generate defect thermodynamic diagram data. According to the invention, the problems of single detection, insufficient precision and low path planning capability in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial defect detection, and particularly relates to an intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning. Background Art

[0002] In the industrial field, various pipeline systems (such as urban gas transmission pipelines, petrochemical production pipelines, urban water supply networks, etc.) serve as key infrastructure and undertake important tasks of material transmission and energy supply. However, with the increase in service life and the influence of complex environmental factors, cracks, corrosion, sediment accumulation and other structural defects inevitably occur inside the pipelines. These defects not only weaken the overall strength and sealing performance of the pipelines, but may also trigger serious production accidents such as leakage and explosion, posing potential threats to public safety and the ecological environment.

[0003] Traditional detection technologies mainly rely on external detection equipment or single sensors (such as visual imaging, magnetic flux leakage detection, ultrasonic flaw detection, etc.), and have great limitations: First, detection blind spots are inevitable. External equipment cannot directly obtain the real-time state of the inner wall of the pipeline, and single sensors are limited by the detection angle and are prone to missing key defect information; Second, the data island phenomenon is prominent. Each sensor independently collects data and lacks a collaborative analysis mechanism, resulting in fragmented detection results; Third, the internal environment of the pipeline is complex (such as narrow spaces, multiple bends, low illuminance, etc.), and traditional robots are difficult to move flexibly, and both the detection efficiency and the positioning accuracy are restricted. These technical bottlenecks seriously restrict the accurate identification and timely repair of pipeline defects. Summary of the Invention

[0004] The purpose of the present invention is to provide an intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning to solve the problem that the existing detection technologies cannot directly obtain the real-time state of the inner wall of the pipeline.

[0005] The technical solution of the present invention to solve the above technical problems is as follows: An intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning, comprising the following steps: S1. Collect three-dimensional point cloud data and two-dimensional image data inside the pipeline; S2. Perform simultaneous localization and mapping processing on the three-dimensional point cloud data and two-dimensional image data inside the pipeline to obtain inspection path data; S3. Perform independent feature extraction processing on the inspection path data to obtain visual feature vector data, magnetic flux leakage feature vector data, and ultrasonic feature vector data; S4. Perform uncertainty coefficient quantization processing on the visual, magnetic flux leakage, and ultrasonic feature vector data to obtain corrected joint feature data; S5. Perform multi-level Bayesian iterative calculation processing on the corrected combined feature data to obtain global defect determination probability data; S6. Process the defect determination probability data associated with the pose data to obtain three-dimensional coordinate mapping to generate defect heat map data.

[0006] Furthermore, step S2 includes using a vision sensor to collect internal pipeline image signals , and the original data obtained by the sensor is processed by a convolutional neural network to extract a vision feature vector, denoted as .

[0007] Furthermore, step S2 includes using a magnetic flux leakage sensor to collect magnetic induction signals by detecting magnetic field changes in the metal pipeline wall, denoted as , and the original data obtained by the sensor is processed by a wavelet transform algorithm to extract a magnetic flux leakage feature vector, denoted as .

[0008] Furthermore, step S2 includes using the ultrasonic echo signal of an ultrasonic sensor to detect internal pipeline detail signals ; the original data obtained by the sensor is processed by a recurrent neural network to extract an ultrasonic feature vector, denoted as: .

[0009] Furthermore, in step S4, after performing uncertainty coefficient quantization processing on the vision, magnetic flux leakage, and ultrasonic feature vector data, dynamic weighting is performed on the features of each sensor and then fusion processing is carried out to obtain the corrected combined feature data; Perform dynamic adjustment processing on the corrected combined feature data to obtain data on the contribution of the dynamic adjustment detection signal in the combined feature, where the detection signals include image data, magnetic flux leakage signals, and ultrasonic signals.

[0010] Furthermore, in step S4, the fusion form of the detection signal features is:

[0011] where, is the final fused feature representation, which combines the features of both the Transformer and CNN models; is the feature representation extracted by the Transformer model; is the feature representation extracted by the convolutional neural network CNN; is the adaptive fusion coefficient, with a value range between 0 and 1, controlling the contribution degree of the Transformer features in the final feature representation; The uncertainty coefficient quantization processing is denoted as , the dynamic weighting expression:

[0012] Among them, is the weight of feature , reflecting the importance of this feature in the overall determination, satisfying ; is the feature index, representing different feature categories; is the adjustment parameter, controlling the uncertainty sensitivity.

[0013] Furthermore, the multi-layer Bayesian iterative calculation of the graph neural network in step S5 includes inputting the corrected joint feature into the graph neural network GNN, constructing the interaction relationship between the feature nodes of each sensor, and iteratively updating the local defect probability ; After the multi-layer Bayesian iterative calculation of the graph neural network, the expression for realizing the global defect determination probability data through the automatically learned fusion function is:

[0014] Among them, represents the defect category or state, that is, the target variable to be predicted or classified; represents the conditional probability distribution of the defect category given the fusion feature ; represents the conditional probability estimation of the defect category by the -th layer model or base classifier given the fusion feature ; represents the total number of layers or the number of the model or base classifier; represents the fusion function used to integrate the outputs of each layer of the model, combining the probability estimations of all base classifiers into the final conditional probability distribution.

[0015] Furthermore, the expression of the Transformer model is:

[0016] Among them, , and are the query, key, and value respectively, is the dimension of the key; At the same time, to fuse local features, a local convolutional branch is introduced, and the fusion formula after combining the two is:

[0017] Among them, It is the feature representation for final fusion, which combines the features of both the Transformer and CNN models; It is the feature representation extracted by the Transformer model; It is the feature representation extracted by the convolutional neural network CNN.

[0018] The present invention has the following beneficial effects: Utilize multi-modal data such as vision, magnetic flux leakage, and ultrasonic to comprehensively cover the internal environment of the pipeline, effectively making up for the detection blind spots of a single sensor; introduce an uncertainty evaluation and dynamic weighting mechanism to improve the robustness and interpretability of the detection results in an environment with a narrow pipeline and high noise; achieve deep fusion of global and local features through cross-modal Transformer and graph neural network, significantly improving the accuracy of defect judgment; combine robot motion data to generate a three-dimensional heat map of defects, providing an intuitive and scientific basis for pipeline maintenance. Brief Description of the Drawings

[0019] Figure 1 It is an exemplary flowchart of a method for intelligent detection of internal pipeline defects based on multi-sensor fusion and deep learning according to some embodiments of this specification.

[0020] Figure 2 It is the generation result of the defect heat map for the internal pipeline defect detection of the present invention based on multi-sensor fusion and deep learning, where Figure 2 the Class Activation Map (CAM) in is a visualization technique used to achieve interpretability in a deep convolutional neural network; its function is to back-project the classification result of the network into the input image space, thereby highlighting the image regions that play a major role in classification; the colors in the class activation map use pseudo-color mapping to represent the intensity of the network attention region, and different colors represent the contribution degree of this region to the final classification decision. Detailed Embodiments

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0022] Please refer to Figure 1-2 For a specific implementation manner of a method for intelligent detection of internal pipeline defects based on multi-sensor fusion and deep learning, it is as follows: First, collect the three-dimensional point cloud data and two-dimensional image data inside the pipeline: Obtain the three-dimensional point cloud data inside the pipeline in real time and acquire the two-dimensional image data inside the pipeline. Conduct three-dimensional mapping of the pipeline interior and determine the real-time position within the pipeline. Specifically, construct a three-dimensional model of the pipeline through the point cloud data, and use the image data for feature matching and loop closure detection to improve the accuracy and robustness of mapping. After the map construction is completed, adopt a path planning algorithm (such as the A* algorithm or Dijkstra algorithm) to plan the optimal inspection path, which needs to consider factors such as the geometric shape of the pipeline, obstacle distribution, and detection requirements to ensure that the interior area of the pipeline can be efficiently and comprehensively covered.

[0023] Next, perform simultaneous localization and mapping processing on the three-dimensional point cloud data and two-dimensional image data inside the pipeline to obtain the inspection path data. The specific steps include: Conduct pipeline inspection based on the planned optimal inspection path. During the inspection process, collect the data inside the pipeline. After the collected image signals are preprocessed (such as denoising, enhancement, etc.), use a convolutional neural network (CNN) for feature extraction to obtain visual feature vectors. The CNN model can be pre-trained on a large amount of pipeline image data to learn the feature representation of pipeline defects. Collect magnetic induction signals by detecting the magnetic field changes in the metal pipeline wall. After filtering, amplification, etc., use methods such as wavelet transform to extract magnetic flux leakage feature vectors. Wavelet transform can effectively extract the time-frequency features in the signal and reflect the defect information of the pipeline wall. Use ultrasonic echo signals to detect the internal detail signals of the pipeline. After signal processing (such as pulse compression, beamforming, etc.), use deep learning models (such as recurrent neural network RNN or long short-term memory network LSTM) to extract ultrasonic feature vectors. These deep learning models can learn the temporal features in the ultrasonic signals and improve the accuracy of defect detection.

[0024] Perform independent feature extraction processing on the inspection path data to obtain visual, magnetic flux leakage, and ultrasonic feature vector data. After the feature extraction is completed, conduct uncertainty evaluation and feature fusion. The specific steps are as follows: Based on the extracted feature vectors, quantify the uncertainty coefficients of the detection signal (image data, magnetic flux leakage signal, and ultrasonic signal) features of the sensor measurement results. The uncertainty coefficient can be obtained by calculating statistical quantities such as the variance and entropy of the feature vectors, which reflects the reliability of the sensor measurement results. According to the uncertainty coefficient, dynamically weight and fuse the features of each sensor. The dynamic weighting formula is: where, is the weight of the feature, which reflects the importance of this feature in the overall determination and satisfies ; is the feature index, indicating different feature categories; is the uncertainty coefficient of the th feature; is a tuning parameter that controls the uncertainty sensitivity. By adjusting the value, the contribution degree of different features in the fusion process can be changed. The fusion form of the detected signal (image data, magnetic flux leakage signal, and ultrasonic signal) features is: where, is the finally fused feature representation, which combines the features of both the Transformer and CNN models; is the feature representation extracted by the Transformer model; is the feature representation extracted by the convolutional neural network CNN; is the fusion weight coefficient, whose value range is between [0,1], and it controls the contribution degree of the Transformer features in the final feature representation. The Transformer model can capture the long-range dependencies between features, while the CNN model can extract local features. By combining the advantages of both models, the richness and accuracy of the feature representation can be improved.

[0025] After the feature fusion is completed, defect probability determination and depth recognition are carried out. The specific steps are as follows: Regarding the uncertainty of the multi-sensor data of the in-pipe robot, a multi-level Bayesian iterative calculation is realized by using the graph neural network (GNN). The corrected joint features are input into the graph neural network GNN to construct the interaction relationship between the feature nodes of each sensor, and the local defect probability is iteratively updated. In each layer of iteration, the defect probability of the current node is updated according to the features of the neighbor nodes and the features of the current node. After multiple layers of iteration, the global defect probability aggregation is realized through an automatically learned fusion function. The fusion function can be realized by methods such as weighted average and neural network, and the defect probabilities of each node are combined into the final global defect probability.

[0026] At the same time, a global and local information interaction is carried out by combining the cross-modal Transformer module. The basic calculation formula of the Transformer is:

[0027] where Q, K, and V are the query, key, and value respectively, is the dimension of the key. By calculating the similarity between the query and the key, the attention weight is obtained, and then the attention weight is multiplied by the value to obtain the weighted feature representation. In order to fuse the local features, a local convolution branch is introduced, and the combined fusion formula is: where, is the adaptive fusion coefficient; is the finally fused feature representation; is the feature representation extracted by the Transformer model; is the local feature representation extracted by the convolutional neural network CNN. By combining global features and local features, the accuracy and robustness of defect detection can be improved.

[0028] Finally, result post-processing and 3D heat map generation are performed. The specific steps are as follows: Associate the depth recognition result with the real-time pose data of the robot, and generate a defect heat map through 3D coordinate mapping. When generating the heat map, the position information of the defect needs to be converted into 3D coordinates, and different color values are set according to the severity of the defect. For example, severe defects are represented by red, and minor defects are represented by green. By intuitively showing the spatial distribution of internal defects in the pipeline, it can provide a decision-making basis for pipeline maintenance personnel and help them take timely measures for repair and maintenance.

[0029] In the specific implementation process, some practical factors also need to be considered, such as the accuracy and stability of the sensor, the motion control accuracy of the robot, and the limitation of computing resources. To improve the reliability and stability of the system, methods such as redundant design, fault diagnosis, and fault-tolerant control can be adopted. At the same time, to improve the accuracy of defect detection, more pipeline data can be collected to continuously train and optimize the neural network model. By continuously improving and perfecting this intelligent detection method, efficient and accurate detection of internal defects in the pipeline can be achieved, providing a strong guarantee for the safe operation of the pipeline.

[0030] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning, characterized in that, Including the following steps: S1. Collect three-dimensional point cloud data and two-dimensional image data inside the pipeline; S2. Perform simultaneous localization and mapping processing on the three-dimensional point cloud data and two-dimensional image data inside the pipeline to obtain inspection path data; S3. Perform independent feature extraction processing on the inspection path data to obtain visual feature vector data, magnetic flux leakage feature vector data, and ultrasonic feature vector data; S4. Perform uncertainty coefficient quantization processing on the visual, magnetic flux leakage, and ultrasonic feature vector data to obtain corrected combined feature data; S5. Perform graph neural network multi-level Bayesian iterative calculation processing on the corrected combined feature data to obtain global defect determination probability data; S6. Process the defect determination probability data associated with the pose data to obtain three-dimensional coordinate mapping to generate defect heat map data.

2. The intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning according to claim 1, wherein Step S2 includes collecting the internal image signals of the pipeline by using a vision sensor , the raw data obtained by the sensor is processed by a convolutional neural network to extract a visual feature vector, denoted as .

3. The intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning according to claim 1, characterized in that, Step S2 includes collecting a magnetic induction signal by using a magnetic flux leakage sensor to detect the magnetic field change in the metal pipeline wall, expressed as , the original data obtained by the sensor is processed by the wavelet transform algorithm, and the magnetic flux leakage feature vector is extracted, expressed as .

4. The intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning according to claim 1, characterized in that, Step S2 includes detecting the internal detail signals of the pipeline using the ultrasonic echo signals of the ultrasonic sensor ; the original data acquired by the sensor is processed by a recurrent neural network to extract ultrasonic feature vectors, expressed as: .

5. The intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning according to claim 1, characterized in that In step S4, after performing uncertainty coefficient quantization processing on the visual, magnetic flux leakage, and ultrasonic feature vector data, dynamic weighting fusion processing is performed on each sensor feature to obtain the corrected joint feature data; For the corrected combined features Perform dynamic adjustment processing on the data to obtain the data contributed by the dynamic adjustment detection signal in the combined features, where the detection signals include image data, magnetic flux leakage signals, and ultrasonic signals.

6. The intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning according to claim 5, wherein In step S4, The fusion form of the detection signal features is: Among them, is the finally fused feature representation, which combines the features of both the Transformer and CNN models; is the feature representation extracted by the Transformer model; is the feature representation extracted by the convolutional neural network CNN; is the adaptive fusion coefficient, with a value range between 0 and 1, controlling the contribution degree of the Transformer features in the final feature representation; The quantification process of the uncertainty coefficient is denoted as , and the dynamic weighting expression is: Among them, is the weight of the feature , reflecting the importance of the feature in the overall determination, satisfying ; is the feature index, representing different feature categories; is the adjustment parameter, controlling the uncertainty sensitivity.

7. The intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning according to claim 5, characterized in that, The multi-level Bayesian iterative calculation of the graph neural network in step S5 includes using the corrected joint features as the input of the graph neural network GNN, constructing the interaction relationships between the feature nodes of each sensor, and iteratively updating the local defect probability ; After graph neural network multi-level Bayesian iterative calculation, the expression of the global defect determination probability data realized by the automatically learned fusion function is: Among them, represents the defect category or status, that is, the target variable to be predicted or classified; represents the conditional probability distribution of the defect category under the condition of the given fused feature ; represents the conditional probability estimation of the defect category by the -th layer model or base classifier when the fused feature is given; represents the total number of layers or the number of the model or base classifier; represents the fusion function used to integrate the outputs of each layer of the model, which combines the probability estimations of all base classifiers into the final conditional probability distribution.

8. The intelligent detection method for in-pipe defects based on multi-sensor fusion and deep learning according to claim 6, wherein, The expression of the Transformer model is: Among them, , and are query, key, and value respectively, is the dimension of the key; Meanwhile, to fuse local features, a local convolution branch is introduced, and the fusion formula after the combination of the two is: Among them, is the finally fused feature representation, which combines the features of both the Transformer and CNN models; is the feature representation extracted by the Transformer model; is the feature representation extracted by the convolutional neural network CNN.

Images