Infrared vision-assisted pipeline outer wall crack recognition and early warning method and system

Abstract

The present invention discloses an infrared vision-assisted pipeline outer wall crack identification and early warning method and system, uses infrared monitoring equipment to build a pipeline crack infrared monitoring system, monitors the outer wall of a sample pipeline to obtain infrared image sample data, uses a feature fusion pyramid network to perform in-depth feature extraction, and uses a multi-level feature optimization network to further optimize and fuse features through bottom-up and top-down information transmission paths. The bottom-up path enhances the capture of local details, while the top-down path transmits higher-level semantic information; pipeline crack identification data is obtained by sharing a dual-label detection head, and pipeline cracks are quickly and accurately identified. The present invention realizes efficient, accurate identification and timely early warning of pipeline cracks, which can not only improve the accuracy of crack detection, but also greatly improve the efficiency of detection, reduce detection costs, and significantly improve the technical level of pipeline safety detection.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional pipeline crack recognition and deep learning, and in particular to an infrared vision-assisted pipeline outer wall crack recognition and early warning method and system. Background Art

[0002] In the process of industrial development and urbanization, the safety and reliability of pipeline facilities are particularly important. Pipelines (including oil and gas pipelines, water pipes and chemical pipelines, etc.) are an important part of modern infrastructure; these pipelines (including oil and gas pipelines, water pipes and chemical pipelines, etc.) are in underground or other harsh environments for a long time and are susceptible to corrosion, mechanical damage, temperature changes and other external factors, leading to safety problems such as cracks and leaks. Therefore, timely and accurate identification and early warning of cracks and other defects in pipelines are of great significance to ensuring public safety and reducing economic losses.

[0003] Traditional pipeline inspection methods mainly include manual visual inspection, ultrasonic inspection, magnetic particle inspection, etc. These methods have their own advantages and disadvantages. For example, manual visual inspection is easy to operate but inefficient and easily affected by subjective factors; ultrasonic inspection can see through the pipe wall, but it requires high skills of the operator and the equipment is relatively expensive; magnetic particle inspection is suitable for finding tiny surface cracks, but it cannot be applied to pipelines made of non-magnetic materials. These traditional methods often fail to meet the high efficiency and high precision requirements of modern pipeline safety inspection. With the rapid development of sensing technology and image processing technology, non-contact detection technology has gradually become a research hotspot; deep learning, as an advanced artificial intelligence technology, has rich application results in the fields of image recognition and data analysis; based on this, how to use non-contact detection technology combined with deep learning to realize pipeline outer wall crack recognition and early warning, and realize rapid detection and early warning of pipeline outer wall cracks is an important direction for pipeline outer wall crack monitoring, and is also of great significance to ensuring public safety. Summary of the invention

[0004] The purpose of the present invention is to provide an infrared vision-assisted pipeline outer wall crack identification and early warning method, which combines infrared thermal imaging technology and a deep learning algorithm including a variety of network models to achieve efficient and accurate identification of pipeline cracks and timely early warning. It can not only improve the accuracy of crack detection, but also greatly improve the detection efficiency and reduce the detection cost, which significantly improves the technical level and application effect of pipeline safety detection, and is of great significance to ensuring public safety.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] An infrared vision-assisted pipeline outer wall crack recognition and early warning method, the method comprising:

[0007] S1. Use infrared monitoring equipment to build a pipeline crack infrared monitoring system, monitor the outer wall of the sample pipeline to obtain infrared image sample data, perform preliminary processing on the infrared image sample data, and use image annotation tools to annotate the infrared image in the infrared image sample data to annotate cracks;

[0008] S2, construct a feature fusion pyramid network including a PCV layer, a four-layer convolution feature extraction module and a fast spatial pyramid pooling module, input the infrared image sample data into the feature fusion pyramid network, and the PCV layer identifies the valid infrared image data that is not blocked; then the first part of the four-layer convolution feature extraction module sequentially extracts features and inputs the feature map A into the fast spatial pyramid pooling module, the fast spatial pyramid pooling module includes a feature analysis module PCBS and three MaxPool2d pooling layers, the feature analysis module PCBS extracts features and inputs them into the three MaxPool2d pooling layers for sequential processing and fusion with the features extracted by the feature analysis module PCBS to output feature map B; then the feature map B and the second part of the four-layer convolution feature extraction module are sequentially downsampled and data fused to obtain a feature map C, and the second part of the convolution feature extraction module uses three bottleneck residual modules Bottleneck to perform feature extraction and channel splicing fusion;

[0009] S3, construct a multi-level feature optimization network including a convolution feature extraction module and four PCV layers, input the feature map C into the multi-level feature optimization network, the convolution feature extraction module optimizes the crack features, and then performs downsampling and data fusion in sequence to obtain a feature map of four scales;

[0010] S4, constructing a shared dual-label detection head including a one-to-many allocation module and / or a one-to-one allocation module, wherein the one-to-many allocation module and the one-to-one allocation module of the shared dual-label detection head respectively process to obtain corresponding crack data, wherein the crack data includes a position frame and a mask image;

[0011] S5. Perform model iterative training on the infrared image sample data of step S1 according to steps S2 to S4; use the pipeline crack infrared monitoring system to monitor the outer wall of the pipeline to obtain infrared monitoring video frame data, perform preliminary processing on the infrared monitoring video frame data, and then process the trained feature fusion pyramid network, multi-level feature optimization network, and shared dual label detection head in sequence to obtain pipeline crack recognition data, and the pipeline crack recognition data includes a position frame and a mask image of the predicted pipeline crack.

[0012] In order to better implement the pipeline outer wall crack identification and early warning method of the present invention, in step S1 and step S5, the pipeline crack infrared monitoring system includes a plurality of infrared monitoring devices and covers the pipeline area with 360 degrees for shooting, and the pipeline crack infrared monitoring system is set to a frame extraction frequency of 30 frames per second; the preliminary processing includes denoising, image enhancement and contrast adjustment processing, the denoising includes using a Gaussian blur method to remove background noise, and the image enhancement includes using an adaptive histogram equalization processing; the crack annotation of the infrared image includes position, shape, and classification type.

[0013] Preferably, in step S2, the four-layer convolution feature extraction modules are sequentially a first layer convolution feature extraction module, a second layer convolution feature extraction module, a third layer convolution feature extraction module and a fourth layer convolution feature extraction module, the PCV layer removes redundant information and distinguishes unobstructed and obstructed pixels in the image through a mask; the first part of the first layer convolution feature extraction module performs feature extraction and inputs the second layer convolution feature extraction module, the first part of the second layer convolution feature extraction module performs feature extraction and inputs the third layer convolution feature extraction module, the first part of the third layer convolution feature extraction module performs feature extraction and inputs the fourth layer convolution feature extraction module, the fourth part performs feature extraction to obtain the feature map A and inputs it into a fast spatial pyramid pooling module, the feature analysis module PCBS of the fast spatial pyramid pooling module further extracts features to obtain a feature map A1, and then inputs three MaxPool2d pooling layers for sequential processing to obtain feature maps of three scales, the feature maps of the three scales are data fused with the feature map A1 and passed through a feature analysis module PCBS to obtain the feature map B.

[0014] Preferably, in step S2, the feature map B output by the fast spatial pyramid pooling module is fused with the feature output by the second part of the fourth-layer convolutional feature extraction module to output the feature map B1, the feature map B1 is downsampled with the feature output by the second part of the third-layer convolutional feature extraction module, the data is fused, and the feature map B2 is output, the feature map B2 is downsampled with the feature output by the second part of the second-layer convolutional feature extraction module, the data is fused, and the feature map B3 is output, the feature map B3 is downsampled with the feature output by the second part of the first-layer convolutional feature extraction module, the data is fused, and the feature map B4 is output, and the feature map B4 is downsampled with the feature output by the PCV layer in the feature fusion pyramid network, the data is fused, and the feature map C is output.

[0015] Preferably, in step S3, the four PCV layers are the first PCV layer, the second PCV layer, the third PCV layer and the fourth PCV layer in sequence, and the first PCV layer, the second PCV layer, the third PCV layer and the fourth PCV layer obtain feature maps C1, feature map C2, feature map C3 and feature map C4 of four hierarchical scales in sequence, and the feature map C is input into the first PCV layer after being optimized for crack feature processing by the convolution feature extraction module of the multi-level feature optimization network, and the feature map C4 of the fourth PCV layer is input into a convolution feature extraction module. The module is processed to obtain feature map C41, and feature map C41 and feature map C3 are downsampled and data fused to obtain feature map C31, feature map C31 and feature map C2 are downsampled and data fused to obtain feature map C21, feature map C21 and feature map C1 are downsampled and data fused to obtain feature map C11, feature map C11 and the output features of the initial convolutional feature extraction module are downsampled and data fused to obtain feature map C10, and feature map C10, feature map C11, feature map C21, and feature map C31 constitute feature maps of four scales.

[0016] Preferably, in step S4, in the one-to-many assignment module, each real position box is assigned multiple positive samples; in the one-to-one assignment module, each real position box is assigned one positive sample, so that the best predicted bounding box is selected to correspond to the positive sample.

[0017] Preferably, in step S5, the model iterative training uses the EIoULoss loss function for loss constraint, and the expected intersection-over-union loss L of the EIoULoss loss function is EIoU The loss constraint formula is as follows:

[0018]

[0019] The predicted box represents the predicted location box or bounding box, denoted by b; the true box represents the true location box or bounding box, denoted by b gt represents; ρ represents the weight for adjusting the distance loss and aspect ratio loss, w represents the width of the prediction box, and w gt represents the width of the real box, h represents the height of the predicted bounding box, and h gt represents the height of the true bounding box, c represents the scale parameter of the scaled distance loss, C w represents the scale parameter of the scaling width loss, C h represents the scale parameter of the scaling height loss, IoU is the intersection over union ratio;

[0020] Set the number of iterative training times and perform multiple iterative trainings. In each iterative training, use the gradient descent algorithm to reduce the model loss value and optimize and update the model parameters.

[0021] An infrared vision-assisted pipeline outer wall crack recognition and early warning system comprises a pipeline crack infrared monitoring system, an infrared image sample database, a feature fusion pyramid network, a multi-level feature optimization network and a shared dual label detection head. The pipeline crack infrared monitoring system is used to monitor the outer wall of the pipeline to obtain infrared image sample data and infrared monitoring video frame data. The infrared image sample data in the infrared image sample database comprises an infrared image and a crack annotation file corresponding to the infrared image. The feature fusion pyramid network comprises a PCV layer, a four-layer convolution feature extraction module and a fast spatial pyramid pooling module. The PCV layer receives the infrared image sample data and identifies the effective infrared image data that is not blocked. Then, the first part of the four-layer convolution feature extraction module sequentially extracts features and inputs a feature map A into a fast spatial pyramid pooling module. The fast spatial pyramid pooling module comprises a feature analysis module PCBS and three MaxPool2d pooling layers. The feature analysis module PCBS extracts features and inputs them into three MaxPool2d pooling layers, which are processed in sequence and fused with the features extracted by the feature analysis module PCBS to output a feature map B. Then, the feature map B is combined with the second part of the four-layer convolution feature extraction module according to the feature extraction method. Downsampling and data fusion are performed to obtain a feature map C. In the second part of the convolution feature extraction module, three bottleneck residual modules Bottleneck are used to perform feature extraction and channel splicing fusion. The multi-level feature optimization network includes a convolution feature extraction module and four PCV layers. The feature map C is input into the multi-level feature optimization network. The convolution feature extraction module optimizes the crack features, and then downsampling and data fusion are performed in sequence to obtain a feature map of four scales. The shared dual label detection head includes a one-to-many allocation module or / and a one-to-one allocation module. The one-to-many allocation module and the one-to-one allocation module of the shared dual label detection head are processed to obtain corresponding crack data respectively, and the crack data includes a position frame and a mask image. The infrared image sample data in the infrared image sample database is preliminarily processed, and then model iteration training is performed in sequence through a feature fusion pyramid network, a multi-level feature optimization network and a shared dual label detection head. The infrared monitoring video frame data after preliminary processing is processed in sequence through a trained feature fusion pyramid network, a multi-level feature optimization network and a shared dual label detection head to obtain pipeline crack recognition data, and the pipeline crack recognition data includes a position frame and a mask image of a predicted pipeline crack.

[0022] In order to better realize the pipeline outer wall crack recognition and early warning system of the present invention, the infrared vision-assisted pipeline outer wall crack recognition and early warning system composed of feature fusion pyramid network, multi-level feature optimization network and shared dual label detection head adopts EIoU Loss loss function for loss constraint. The expected intersection-over-union loss L of EIoU Loss function is 1.18×10. EIoU The loss constraint formula is as follows:

[0023]

[0024] The predicted box represents the predicted location box or bounding box, denoted by b; the true box represents the true location box or bounding box, denoted by b gt Represents; ρ represents the weight for adjusting the distance loss and aspect ratio loss, W represents the width of the prediction box, w gt represents the width of the real box, h represents the height of the predicted bounding box, and h gt represents the height of the true bounding box, c represents the scale parameter of the scaled distance loss, C w represents the scale parameter of the scaling width loss, C h represents the scale parameter of the scaling height loss, IoU is the intersection over union ratio;

[0025] Set the number of iterative training times and perform multiple iterative trainings. In each iterative training, use the gradient descent algorithm to reduce the model loss value and optimize and update the model parameters.

[0026] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0027] (1) The present invention combines infrared thermal imaging technology and deep learning algorithms including multiple network models to achieve efficient and accurate identification and timely warning of pipeline cracks. It can not only improve the accuracy of crack detection, but also greatly improve the efficiency of detection and reduce the cost of detection. It has significantly improved the technical level and application effect of pipeline safety detection, which is of great significance to ensuring public safety.

[0028] (2) The present invention innovatively proposes an infrared vision-assisted pipeline outer wall crack recognition and early warning method and system by integrating infrared vision-assisted technology and deep learning framework. It can not only realize effective monitoring under low-light conditions, but also effectively extract and process the multi-scale information of pipeline cracks through feature fusion pyramid network (PCFFPN) and multi-level feature optimization network (MLFON), thereby enhancing the adaptability and generalization ability of the model. The application of shared dual label detection head (SDLD hand) improves the efficiency of model training and inference speed, significantly improves the practicality and reliability of the method, and provides strong technical support for pipeline safety monitoring and maintenance. The present invention not only optimizes the crack recognition process, but also reduces the cost of complex data processing, making it have broad prospects and important practical value in practical applications.

[0029] (3) The present invention adopts infrared auxiliary technology, which utilizes its thermal sensitivity to perform real-time monitoring around the clock. In particular, under low light or no light conditions, it can still effectively identify tiny cracks and damage caused by internal stress, material fatigue or environmental factors, greatly improving the reliability and practicality of monitoring data, so that potential cracks and damage can be discovered and processed in time at an early stage; the present invention creatively develops a feature fusion pyramid network (PCFFPN) and uses partial convolutional layers. This layer structure is particularly suitable for reducing the consumption of computing resources and speeding up model training. Through the pyramid principle, the network can extract information at multiple scales, ensuring that features from fine to coarse can be effectively captured; in addition, the application of the spatial pyramid pooling module ensures the consistency and stability of the output, reduces the dependence on image preprocessing, and enhances the adaptability and generalization ability of the model in a changing environment.

[0030] (4) The present invention constructs a multi-level feature optimization network (MLF0N), which strengthens the connection between features through bottom-up and top-down information transmission paths. This design effectively balances the capture of local details and global semantic information; the bottom-up path strengthens the recognition of subtle features, while the top-down path ensures that these features are utilized in a higher-level semantic framework, thereby optimizing the detection capability of small cracks.

[0031] (5) The present invention constructs a multi-level feature optimization network (MLFON), which strengthens the connection between features through bottom-up and top-down information transmission paths. This design effectively balances the capture of local details and global semantic information; the bottom-up path strengthens the recognition of subtle features, while the top-down path ensures that these features are utilized in a higher-level semantic framework, thereby optimizing the detection capability of small cracks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A method flow chart of the pipeline outer wall crack identification and early warning method of the present invention;

[0033] Figure 2 This is a principle structure block diagram of the pipeline outer wall crack identification and early warning system of the present invention;

[0034] Figure 3 It is a schematic diagram of the principle of the method and system of the present invention;

[0035] Figure 4 Schematic diagram of the principle of the partial convolution module PCV of the PCV layer in the embodiment;

[0036] Figure 5 Schematic diagram of the convolution feature extraction module PC2F in the embodiment;

[0037] Figure 6 Schematic diagram of the principle of the bottleneck residual module Bottleneck in the embodiment;

[0038] Figure 7 Schematic diagram of the principle of the feature analysis module PCSS in the embodiment;

[0039] Figure 8 Schematic diagram of the principle of the fast spatial pyramid pooling module SPPF in the embodiment. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with embodiments:

[0041] Example

[0042] like Figure 1 , Figure 3 to Figure 8 As shown, an infrared vision-assisted pipeline outer wall crack recognition and early warning method comprises:

[0043] S1. Use infrared monitoring equipment to build a pipeline crack infrared monitoring system, monitor the outer wall of the sample pipeline to obtain infrared image sample data, and perform preliminary processing on the infrared image sample data. The pipeline crack infrared monitoring system includes a plurality of infrared monitoring devices (in order to improve the monitoring shooting quality, the infrared monitoring device can be selected from the aspects of device sensitivity, resolution, stability and the ability to adapt to different environmental conditions; the infrared monitoring device should ensure that it can clearly capture the infrared image of the pipeline crack at night or in an environment with insufficient light, such as capturing a clear infrared image in a low light environment of 0.005 Lux, the camera resolution is 1920x1080, which can ensure that the crack details are clearly visible, and can work stably in the temperature range of -20°C to 60°C) and is composed to cover the pipeline area for shooting at 360 degrees (for example, double-sided infrared monitoring devices are arranged accordingly, and the shooting field of view of the infrared monitoring device on each side is 180 degrees, ensuring that there is no blind spot monitoring), and the frame extraction frequency of the pipeline crack infrared monitoring system is set to 30 frames per second (continuously recording the development process of the crack, which is convenient for evolution analysis and research under experimental conditions); the present invention can be applied to a certain area of ​​the outer wall of the pipeline, and can also be applied to the entire area of ​​the outer wall of the pipeline (when the application area is wide, pipeline modeling can be performed and infrared images can be packaged and numbered).

[0044] The preliminary processing may adopt one of denoising, image enhancement or contrast adjustment processing or a combination of each other, mainly removing noise, improving image quality and contrast (adjusting contrast to 80% to optimize the visualization effect of cracks) through preliminary processing, and the preliminary processing may also include other processing methods. The purpose of preliminary processing is to improve the recognizability of crack features; in some embodiments, the preliminary processing includes denoising, image enhancement and contrast adjustment processing, denoising processing includes using Gaussian blur method to remove background noise, and image enhancement processing includes using adaptive histogram equalization processing. The infrared image in the infrared image sample data is annotated with an image annotation tool to annotate cracks. The crack annotation can use an image annotation tool (such as LabelMe or VGG Image Annotator) to outline the position (specific position in the image) and shape (including the length and width of the crack), and annotate the type of crack. The crack annotation of the infrared image includes position, shape, and classification type. When using infrared image sample data to train the pipeline outer wall crack recognition and early warning system (including feature fusion pyramid network, multi-level feature optimization network, shared dual label detection head), the data set can be divided into corresponding subsets according to the needs of training, verification and testing. For example, the data set is divided according to the ratio of 70% training set, 20% verification set and 10% test set to ensure the representativeness and adequacy of each part of the data.

[0045] S2, build including PCV layer (corresponding to Figure 3 PCV in PCFPN), four-layer convolution feature extraction module and fast spatial pyramid pooling module PC2F (such as Figure 3 The feature fusion pyramid network PCFFPN (abbreviated as PC2F) Figure 3 The infrared image sample data is input into the feature fusion pyramid network, and the PCV layer identifies the valid infrared image data that is not blocked. Then the first part of the four-layer convolution feature extraction module (used to extract multi-scale feature information) extracts features in turn and inputs the feature map A into the fast spatial pyramid pooling module. The fast spatial pyramid pooling module includes the feature analysis module PCBS and three MaxPool2d pooling layers. The feature analysis module PCBS extracts features and inputs them into the three MaxPool2d pooling layers for processing in turn and fusion with the features extracted by the feature analysis module PCBS to output the feature map B. Then the feature map B and the second part of the four-layer convolution feature extraction module are downsampled and data fused in turn to obtain the feature map C. The second part of the convolution feature extraction module uses three bottleneck residual modules Bott leneck to perform feature extraction and channel splicing fusion; see Figure 6Preferably, the Bottleneck module continuously passes the input data through two feature analysis modules PCBS based on spatiotemporal statistics, and obtains the final output after data fusion of the output data and the input data.

[0046] In some embodiments, Figure 1 As shown, the four-layer convolution feature extraction module (used to extract multi-scale feature information) in step S2 is sequentially a first-layer convolution feature extraction module, a second-layer convolution feature extraction module, a third-layer convolution feature extraction module and a fourth-layer convolution feature extraction module, and the PCV layer removes redundant information and distinguishes between unobstructed and obstructed pixels in the image through a mask; Figure 3 As shown, in this embodiment, when the infrared image enters the PCV layer, the mask is used to distinguish between valid (unobstructed) and invalid (obstructed) pixels in the image, and the convolution kernel is only calculated with the pixels marked as valid by the mask, while the obstructed part does not participate in the calculation, and then the convolution kernel (convolution kernel size 3x3, step size 1) performs standard convolution processing on these valid pixels. The first part of the first-layer convolution feature extraction module performs feature extraction and inputs it into the second-layer convolution feature extraction module. The first part of the second-layer convolution feature extraction module performs feature extraction and inputs it into the third-layer convolution feature extraction module. The first part of the third-layer convolution feature extraction module performs feature extraction and inputs it into the fourth-layer convolution feature extraction module. The fourth part performs feature extraction to obtain the feature map A and inputs it into the fast spatial pyramid pooling module. The feature analysis module PCBS of the fast spatial pyramid pooling module further extracts features to obtain feature map A1, which is then input into three MaxPool2d pooling layers for sequential processing to obtain feature maps of three scales. The feature maps of the three scales are data fused with the feature map A1 and passed through a feature analysis module PCBS to obtain the feature map B (the feature map B obtained by processing the fast spatial pyramid pooling module is a deep feature map); Figure 7 As shown, the feature analysis module PCBS includes a PCV layer, a normalization layer (BN layer) and a silu activation layer. The feature analysis module PCBS outputs the input data after passing the input data through the PCV layer, the normalization layer and the silu activation layer in sequence.

[0047] In some embodiments, in step S2, Figure 1 As shown, the fast spatial pyramid pooling module SPPF (corresponding to Figure 1 The feature map B output by the SPPF in the feature fusion pyramid network is fused with the features output by the second part of the fourth layer convolution feature extraction module and a feature map B1 is output; preferably, Figure 8As shown, the fast spatial pyramid pooling module SPPF includes a first feature analysis module PCBS, three MaxPoo12d pooling layers and a second feature analysis module PCBS. The first feature analysis module PCBS first processes the feature map data to obtain a feature map after further feature extraction, and then the feature map is input into the MaxPool2d pooling layer with three different pooling kernels (1x1, 2x2, 4x4, each pooling kernel performs global pooling, maximum pooling and average pooling on the feature map respectively) for processing, and then the output feature maps from the pooling layers of different scales are fused to form a comprehensive, fixed-length feature map, which is then processed and output by the second feature analysis module PCBS. The fast spatial pyramid pooling module SPPF can effectively extract useful features from input images of any size, can be more flexible and powerful, and performs well when processing inputs of different sizes. The feature map B1 and the features output from the second part of the third-layer convolution feature extraction module are downsampled, data fused, and feature map B2 is output. The feature map B2 and the features output from the second part of the second-layer convolution feature extraction module are downsampled, data fused, and feature map B3 is output. The feature map B3 and the features output from the second part of the first-layer convolution feature extraction module are downsampled, data fused, and feature map B4 is output. The feature map B4 and the features output from the PCV layer in the feature fusion pyramid network are downsampled (downsampling can gradually collect detail features and ensure that the size of the feature map is consistent), data fused, and feature map C is output. The feature map B (deep feature map) obtained by the fast spatial pyramid pooling module is fused with the output of the second part of the four-layer convolution feature extraction module layer by layer. The feature maps B1 to B4 will gradually collect detail features. Finally, the feature map C contains a fusion feature map of overall features without losing detail features. Feature map C is a pipeline crack feature map rich in multi-scale feature information.

[0048] S3. Construct a multi-level feature optimization network MLFOM including a convolutional feature extraction module PC2F and four PCV layers. Input the feature map C into the multi-level feature optimization network. The convolutional feature extraction module PC2F optimizes the crack features (reduces the amount of data for subsequent crack identification while retaining the main feature information, speeding up model recognition efficiency). Then the four PCV layers are downsampled and data fused in turn to obtain feature maps of four scales. The subsequent shared dual-label detection head can perform detection on feature maps of corresponding scales when dealing with cracks of different sizes, greatly improving the crack recognition speed.

[0049] In some embodiments, in step S3, four PCV layers (corresponding to Figure 1The four PCV layers in the multi-level feature optimization network MLFOM are the first PCV layer, the second PCV layer, the third PCV layer and the fourth PCV layer. The first PCV layer, the second PCV layer, the third PCV layer and the fourth PCV layer obtain the feature maps C1, C2, C3 and C4 of four hierarchical scales respectively. The feature map C is input into the first PCV layer after the crack feature is optimized by the convolution feature extraction module of the multi-level feature optimization network. The feature map C4 of the fourth PCV layer is input into a convolution feature extraction module. The extraction module processes the feature map C41, and the feature map C41 and the feature map C3 are downsampled and data fused to obtain the feature map C31. The feature map C31 and the feature map C2 are downsampled and data fused to obtain the feature map C21. The feature map C21 and the feature map C1 are downsampled and data fused to obtain the feature map C11. The feature map C11 and the output features of the initial convolution feature extraction module are downsampled and data fused to obtain the feature map C10. The feature maps C10, C11, C21 and C31 constitute the feature maps of four scales. The feature maps of the four scales have different scales, so that more high-level overall feature information is integrated into the low-level detail feature map, making the multi-level feature optimization network MLFOM more sensitive to the recognition of small cracks.

[0050] S4, construct a shared dual label detection head (SDLD hand for short) including a one-to-many assignment module or / and a one-to-one assignment module, the detection head shares parameters, and its main parts are Figure 1 In the STM hand, preferably, the one-to-many allocation module and the one-to-one allocation module (the one-to-many allocation module and the one-to-one allocation module are processed in parallel) of the shared dual label detection head respectively process the corresponding crack data, and the crack data includes the position box and the mask image. The one-to-many allocation module uses the one-to-many allocation strategy in the training stage to allocate multiple positive samples for each true bounding box, providing rich supervision signals to help the model learn better; the one-to-one allocation module uses the one-to-one allocation strategy to allocate one positive sample for each true bounding box to ensure that the model can learn to select the best prediction bounding box during training; in the prediction stage, only the one-to-one head is used for prediction (the one-to-many allocation module assists in training and improves the selection of the best prediction bounding box, i.e., the position box, during the training process), and the one-to-one allocation ensures that each target has only one positive sample, avoiding redundant predictions and NMS post-processing.

[0051] In step S4, in a one-to-many assignment module, each true position box is assigned multiple positive samples. In a one-to-one assignment module, each true position box is assigned one positive sample, so that the best predicted bounding box is selected to correspond to the positive sample.

[0052] S5. Perform iterative model training on the infrared image sample data of step S1 according to steps S2 to S4, input the test data set into the network model trained with the training data set, and obtain the recognition result of pipeline cracks. The recognition result is evaluated for accuracy, and the specific evaluation indicators include: confusion matrix; precision P (Precision): the proportion of the number of cracks that can be correctly identified as cracks in the model to the number of cracks predicted; recall R (Recall): the proportion of the number of cracks that can be correctly identified as cracks by the model to the number of real cracks; F1 score (F1-score): the sum and average of precision and recall, an indicator for evaluating model recognition performance; determination coefficient R2: a statistic for model goodness of fit. The pipeline crack infrared monitoring system is used to monitor the outer wall of the pipeline to obtain infrared monitoring video frame data (the pipeline crack infrared monitoring system includes a number of infrared monitoring devices and covers the pipeline area with 360 degrees. The frame extraction frequency of the pipeline crack infrared monitoring system is set to 30 frames per second.), and the infrared monitoring video frame data is preliminarily processed (the preliminary processing includes denoising, image enhancement and contrast adjustment processing. The denoising includes removing background noise by using the Gaussian blur method, and the image enhancement includes using adaptive histogram equalization processing. The crack annotation of the infrared image includes position, shape, and classification type.), and then the pipeline crack recognition data is obtained by sequentially processing through the trained feature fusion pyramid network, the multi-level feature optimization network, and the shared dual label detection head. The pipeline crack recognition data includes the position frame and mask image of the predicted pipeline crack.

[0053] In some embodiments, the model iteration training uses the EIoU Loss loss function for loss constraint, and the expected intersection-over-union loss L of the EIoULoss loss function is EIoU The loss constraint formula is as follows:

[0054]

[0055] The predicted box represents the predicted location box or bounding box, denoted by b. The true box represents the true location box or bounding box, denoted by b gt ρ represents the weight for adjusting the distance loss and aspect ratio loss, w represents the width of the prediction box, and w gt represents the width of the real box, h represents the height of the predicted bounding box, and h gt represents the height of the true bounding box, c represents the scale parameter of the scaled distance loss, C w represents the scale parameter of the scaling width loss, C h represents the scale parameter of the scaling height loss, IoU is the intersection over union ratio;

[0056]

[0057] L EIoUrepresents the expected intersection-over-union loss, L IoU represents the intersection-over-union loss, L dis Represents the distance loss, L asp Indicates aspect ratio loss.

[0058] Set the number of iterations and perform multiple iterations. In each iteration, the gradient descent algorithm is used to reduce the model loss value, while optimizing and updating the model parameters (the model parameters refer to the weight values ​​of the connections between layers in the neural network).

[0059] Table 1 Network parameter settings

[0060]

[0061] Table 2 Server configuration

[0062]

[0063] like Figure 2As shown, an infrared vision-assisted pipeline outer wall crack recognition and early warning system includes a pipeline crack infrared monitoring system, an infrared image sample database, a feature fusion pyramid network, a multi-level feature optimization network and a shared dual label detection head. The pipeline crack infrared monitoring system is used to monitor the outer wall of the pipeline to obtain infrared image sample data and infrared monitoring video frame data. The infrared image sample data in the infrared image sample database includes an infrared image and a crack annotation file corresponding to the infrared image. The feature fusion pyramid network includes a PCV layer, a four-layer convolution feature extraction module and a fast spatial pyramid pooling module. The PCV layer receives the infrared image sample data and identifies the valid infrared image data that is not blocked. Then the first part of the four-layer convolution feature extraction module sequentially extracts features and inputs the feature map A into the fast spatial pyramid pooling module. The fast spatial pyramid pooling module includes a feature analysis module PCBS and three MaxPool2d pooling layers. The feature analysis module PCBS extracts features and inputs the three MaxPool2d pooling layers for sequential processing and outputs the feature map B after fusion with the features extracted by the feature analysis module PCBS. Then, the feature map B and the second part of the four-layer convolution feature extraction module are sequentially downsampled and data fused to obtain the feature map C. The second part of the convolution feature extraction module is subjected to feature extraction and channel splicing fusion through three bottleneck residual modules Bott leneck. The multi-level feature optimization network includes a convolution feature extraction module and a four-layer PCV layer. The feature map C is input into the multi-level feature optimization network. The convolution feature extraction module optimizes the crack features, and then sequentially downsamples and data fuses to obtain a feature map of four scales. The shared dual label detection head includes a one-to-many allocation module and / or a one-to-one allocation module. The one-to-many allocation module and the one-to-one allocation module of the shared dual label detection head are processed to obtain the corresponding crack data respectively. The crack data includes a position box and a mask image. The infrared image sample data in the infrared image sample database is preliminarily processed, and then the model is iteratively trained in sequence through the feature fusion pyramid network, the multi-level feature optimization network and the shared dual label detection head. The infrared surveillance video frame data after preliminary processing is processed in sequence through the trained feature fusion pyramid network, multi-level feature optimization network and shared dual label detection head to obtain pipeline crack recognition data. The pipeline crack recognition data includes the location frame and mask image of the predicted pipeline crack.

[0064] The infrared vision-assisted pipeline outer wall crack recognition and early warning system composed of feature fusion pyramid network, multi-level feature optimization network and shared dual label detection head adopts EIoU Loss loss function for loss constraint. The expected intersection-over-union loss L of EIoU Loss function is EIoU The loss constraint formula is as follows:

[0065]

[0066] The predicted box represents the predicted location box or bounding box, denoted by b. The true box represents the true location box or bounding box, denoted by b gt ρ represents the weight for adjusting the distance loss and aspect ratio loss, W represents the width of the prediction box, and w gt represents the width of the real box, h represents the height of the predicted bounding box, and h gt represents the height of the true bounding box, c represents the scale parameter of the scaled distance loss, C w represents the scale parameter of the scaling width loss, C h represents the scale parameter of the scaled height loss, and IoU is the intersection over union ratio.

[0067]

[0068] L EIoU represents the expected intersection-over-union loss, L IoU represents the intersection-over-union loss, L dis Represents the distance loss, L asp Indicates aspect ratio loss.

[0069] Set the number of iterations and perform multiple iterations. In each iteration, the gradient descent algorithm is used to reduce the model loss value, while optimizing and updating the model parameters (the model parameters refer to the weight values ​​of the connections between layers in the neural network).

[0070] The present invention uses infrared monitoring equipment to monitor and shoot pipelines with cracks, obtains a certain number of pipeline crack infrared images through frame extraction technology, annotates these infrared images, and constructs an infrared image sample data set containing precise crack labels, which provides the necessary data basis for model training; then, uses a feature fusion pyramid network (PCFFPN) based on partial convolution to perform in-depth feature extraction on these data sets. The feature fusion pyramid network PCFFPN reduces computational redundancy and improves processing speed through partial convolution layers, and uses the pyramid principle to effectively capture different scale features from fine-grained to coarse-grained, thereby enhancing the feature representation capability of the model. Then, a multi-level feature optimization network (MLFON) is used to further optimize and fuse these features through bottom-up and top-down information transmission paths. The bottom-up path enhances the capture of local details, while the top-down path transmits higher-level semantic information. Finally, through the strategy of shared dual label detection head (SDLD hand), one-to-one (corresponding to one-to-one assignment module) and one-to-many (corresponding to one-to-many assignment module) are integrated for supervision, which not only optimizes the training process of the model, but also reduces the computational burden of the inference stage through the shared detection head technology, improves the inference speed, and ensures the fast and accurate identification of pipeline cracks in practical applications.

[0071] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An infrared vision-assisted pipeline outer wall crack recognition and early warning method, characterized in that: The methods include: S1. Use infrared monitoring equipment to build a pipeline crack infrared monitoring system, monitor the outer wall of the sample pipeline to obtain infrared image sample data, perform preliminary processing on the infrared image sample data, and use image annotation tools to annotate the infrared image in the infrared image sample data to annotate cracks; S2, construct a feature fusion pyramid network including a PCV layer, a four-layer convolution feature extraction module and a fast spatial pyramid pooling module, input the infrared image sample data into the feature fusion pyramid network, and the PCV layer identifies the valid infrared image data that is not blocked; then the first part of the four-layer convolution feature extraction module sequentially extracts features and inputs the feature map A into the fast spatial pyramid pooling module, the fast spatial pyramid pooling module includes a feature analysis module PCBS and three MaxPool2d pooling layers, the feature analysis module PCBS of the fast spatial pyramid pooling module includes a first feature analysis module PCBS and a second feature analysis module PCBS, the feature analysis module PCBS extracts features and inputs the three MaxPool2d pooling layers of the first feature analysis module PCBS for sequential processing and feature fusion, and then processes the output feature map B through the second feature analysis module PCBS; then the feature map B and the second part of the four-layer convolution feature extraction module are sequentially downsampled and data fused to obtain a feature map C, and the second part of the convolution feature extraction module performs feature extraction and channel splicing fusion through three bottleneck residual modules Bottleneck; S3. Construct a multi-level feature optimization network including a convolutional feature extraction module and four PCV layers, input the feature map C into the multi-level feature optimization network, optimize the crack features with the convolutional feature extraction module, and then perform downsampling and data fusion in sequence to obtain a feature map of four scales; the four PCV layers are the first PCV layer, the second PCV layer, the third PCV layer and the fourth PCV layer in sequence, and the first PCV layer, the second PCV layer, the third PCV layer and the fourth PCV layer obtain feature maps C1, feature map C2, feature map C3 and feature map C4 of four hierarchical scales in sequence, and the feature map C is optimized by the convolutional feature extraction module of the multi-level feature optimization network. After feature processing, the feature map C4 of the fourth PCV layer is input into a convolutional feature extraction module to obtain a feature map C41. The feature map C41 and the feature map C3 are downsampled and data fused to obtain a feature map C31. The feature map C31 and the feature map C2 are downsampled and data fused to obtain a feature map C21. The feature map C21 and the feature map C1 are downsampled and data fused to obtain a feature map C11. The feature map C11 and the output features of the initial convolutional feature extraction module are downsampled and data fused to obtain a feature map C10. The feature maps C10, C11, C21, and C31 constitute feature maps of four scales. S4, constructing a shared dual-label detection head including a one-to-many allocation module and / or a one-to-one allocation module, wherein the one-to-many allocation module and the one-to-one allocation module of the shared dual-label detection head respectively process to obtain corresponding crack data, wherein the crack data includes a position frame and a mask image; S5. Perform model iterative training on the infrared image sample data of step S1 according to steps S2 to S4; use the pipeline crack infrared monitoring system to monitor the outer wall of the pipeline to obtain infrared monitoring video frame data, perform preliminary processing on the infrared monitoring video frame data, and then process the trained feature fusion pyramid network, multi-level feature optimization network, and shared dual label detection head in sequence to obtain pipeline crack recognition data, and the pipeline crack recognition data includes a position frame and a mask image of the predicted pipeline crack.

2. The infrared vision-assisted pipeline outer wall crack identification and early warning method according to claim 1 is characterized in that: In step S1 and step S5, the pipeline crack infrared monitoring system includes a plurality of infrared monitoring devices and covers the pipeline area with 360 degrees for shooting. The frame extraction frequency of the pipeline crack infrared monitoring system is set to 30 frames per second; the preliminary processing includes denoising, image enhancement and contrast adjustment processing, the denoising includes removing background noise by using Gaussian blur method, and the image enhancement includes adaptive histogram equalization processing; the crack annotation of the infrared image includes position, shape, and classification type.

3. The infrared vision-assisted pipeline outer wall crack recognition and early warning method according to claim 1 is characterized in that: In step S2, the four-layer convolution feature extraction modules are sequentially a first layer convolution feature extraction module, a second layer convolution feature extraction module, a third layer convolution feature extraction module and a fourth layer convolution feature extraction module, the PCV layer removes redundant information and distinguishes unobstructed and obstructed pixels in the image through a mask; the first part of the first layer convolution feature extraction module performs feature extraction and inputs the second layer convolution feature extraction module, the first part of the second layer convolution feature extraction module performs feature extraction and inputs the third layer convolution feature extraction module, the first part of the third layer convolution feature extraction module performs feature extraction and inputs the fourth layer convolution feature extraction module, the fourth part performs feature extraction to obtain the feature map A and inputs the fast spatial pyramid pooling module, the feature analysis module PCBS of the fast spatial pyramid pooling module further extracts features to obtain a feature map A1, and then inputs three MaxPool2d pooling layers to process in sequence to obtain feature maps of three scales, the feature maps of the three scales are data fused with the feature map A1 and passed through a feature analysis module PCBS to obtain the feature map B.

4. The infrared vision-assisted pipeline outer wall crack recognition and early warning method according to claim 3 is characterized in that: In step S2, the feature map B output by the fast spatial pyramid pooling module is fused with the features output by the second part of the fourth-layer convolutional feature extraction module to output the feature map B1; the feature map B1 is downsampled with the features output by the second part of the third-layer convolutional feature extraction module, the data is fused, and the feature map B2 is output; the feature map B2 is downsampled with the features output by the second part of the second-layer convolutional feature extraction module, the data is fused, and the feature map B3 is output; the feature map B3 is downsampled with the features output by the second part of the first-layer convolutional feature extraction module, the data is fused, and the feature map B4 is output; the feature map B4 is downsampled with the features output by the PCV layer in the feature fusion pyramid network, the data is fused, and the feature map C is output.

5. The infrared vision-assisted pipeline outer wall crack recognition and early warning method according to claim 1 is characterized in that: In step S4, in the one-to-many assignment module, each true position box is assigned multiple positive samples; in the one-to-one assignment module, each true position box is assigned one positive sample, so that the best predicted bounding box is selected to match the positive sample.

6. The infrared vision-assisted pipeline outer wall crack recognition and early warning method according to claim 1 is characterized in that: In step S5, the model iterative training uses the EIoU Loss function for loss constraint. The expected intersection-over-union loss L of the EIoU Loss function is EIoU The loss constraint formula is as follows: The predicted box represents the predicted location box or bounding box, denoted by b; the true box represents the true location box or bounding box, denoted by bgt; ρ represents the weight for adjusting the distance loss and aspect ratio loss, w represents the width of the predicted box, w gt represents the width of the real box, h represents the height of the predicted bounding box, and h gt represents the height of the true bounding box, c represents the scale parameter of the scaled distance loss, C w represents the scale parameter of the scaling width loss, C h represents the scale parameter of the scaling height loss, IoU is the intersection over union ratio; Set the number of iterative training times and perform multiple iterative trainings. In each iterative training, use the gradient descent algorithm to reduce the model loss value and optimize and update the model parameters.

7. An infrared vision-assisted pipeline outer wall crack identification and early warning system, characterized in that: The invention comprises an infrared monitoring system for pipeline cracks, an infrared image sample database, a feature fusion pyramid network, a multi-level feature optimization network and a shared dual label detection head. The infrared monitoring system for pipeline cracks is used to monitor the outer wall of the pipeline to obtain infrared image sample data and infrared monitoring video frame data. The infrared image sample data in the infrared image sample database includes an infrared image and a crack annotation file corresponding to the infrared image. The feature fusion pyramid network comprises a PCV layer, a four-layer convolution feature extraction module and a fast spatial pyramid pooling module. The PCV layer receives the infrared image sample data and identifies the valid infrared image data that is not blocked. Then the first part of the four-layer convolution feature extraction module extracts features in turn and identifies the features. Figure A inputs the fast spatial pyramid pooling module, the fast spatial pyramid pooling module includes a feature analysis module PCBS and three MaxPool2d pooling layers, the feature analysis module PCBS of the fast spatial pyramid pooling module includes a first feature analysis module PCBS and a second feature analysis module PCBS, the feature analysis module PCBS extracts features, inputs the three MaxPool2d pooling layers of the first feature analysis module PCBS, processes and fuses the features in sequence, and then processes the feature map B through the second feature analysis module PCBS; then, the feature map B and the second part of the four-layer convolution feature extraction module are downsampled and data fused in sequence to obtain the feature map C, and the second part of the convolution feature extraction module is processed through three A bottleneck residual module Bottleneck is used for feature extraction and channel splicing and fusion; the multi-level feature optimization network includes a convolutional feature extraction module and four PCV layers, the feature map C is input into the multi-level feature optimization network, the convolutional feature extraction module optimizes the crack features, and then downsampling and data fusion are performed in sequence to obtain a feature map of four scales; the four PCV layers are the first PCV layer, the second PCV layer, the third PCV layer and the fourth PCV layer in sequence, the first PCV layer, the second PCV layer, the third PCV layer and the fourth PCV layer in sequence obtain feature maps C1, feature map C2, feature map C3 and feature map C4 of four hierarchical scales, and the feature map C is passed through the multi-level feature optimization network The convolution feature extraction module optimizes the crack feature processing and inputs it into the first PCV layer. The feature map C4 of the fourth PCV layer is input into a convolution feature extraction module to obtain a feature map C41. The feature map C41 and the feature map C3 are downsampled and data fused to obtain a feature map C31. The feature map C31 and the feature map C2 are downsampled and data fused to obtain a feature map C21. The feature map C21 and the feature map C1 are downsampled and data fused to obtain a feature map C11. The feature map C11 and the output features of the initial convolution feature extraction module are downsampled and data fused to obtain a feature map C10. The feature maps C10, C11, C21 and C31 constitute feature maps of four scales.The shared dual label detection head includes a one-to-many allocation module or / and a one-to-one allocation module. The one-to-many allocation module and the one-to-one allocation module of the shared dual label detection head respectively process to obtain corresponding crack data, and the crack data includes a position frame and a mask image; the infrared image sample data in the infrared image sample database is preliminarily processed, and then the model is iteratively trained in sequence through the feature fusion pyramid network, the multi-level feature optimization network and the shared dual label detection head; the infrared monitoring video frame data after the preliminarily processing is processed in sequence through the trained feature fusion pyramid network, the multi-level feature optimization network and the shared dual label detection head to obtain pipeline crack recognition data, and the pipeline crack recognition data includes the position frame and the mask image of the predicted pipeline crack. ; 8. The infrared vision-assisted pipeline outer wall crack identification and early warning system according to claim 7 is characterized in that: The infrared vision-assisted pipeline outer wall crack recognition and early warning system composed of feature fusion pyramid network, multi-level feature optimization network and shared dual label detection head adopts EIoU Loss loss function for loss constraint. The expected intersection-over-union loss L of EIoU Loss function is EIoU The loss constraint formula is as follows: The predicted box represents the predicted location box or bounding box, denoted by b; the true box represents the true location box or bounding box, denoted by bgt; ρ represents the weight for adjusting the distance loss and aspect ratio loss, w represents the width of the predicted box, w gt represents the width of the real box, h represents the height of the predicted bounding box, and h gt represents the height of the true bounding box, c represents the scale parameter of the scaled distance loss, C w represents the scale parameter of the scaling width loss, C h Represents the scale parameter of the scaling height loss, IoU is the intersection over union ratio; set the number of iterative training times and perform multiple iterative training; in each iterative training, use the gradient descent algorithm to reduce the model loss value, and optimize and update the model parameters at the same time.

Images