Vision-based pipeline surface damage rapid identification method

Abstract

The invention belongs to the field of computer vision, and particularly relates to a vision-based pipeline surface damage rapid identification method, which comprises the following steps: S1, acquiring an outer wall image of a nuclear power plant pipeline, marking damage in the acquired image, and constructing a pipeline damage image data set; s2, an improved RT-DETR model is adopted, image data in the pipeline damage image data set serves as input of a backbone network, and a corrosion detection model is constructed through the backbone network, a BiFPN module and a hybrid encoder; s3, training the constructed corrosion detection model by adopting the constructed pipeline damage image data set, and detecting a corrosion area by using the constructed corrosion detection model after training is completed; and S4, evaluating a detection result of the model. According to the method, the delay in the detection process can be effectively reduced, the inference speed of the detector is not delayed, the detector can stably operate, the accuracy of defect detection is remarkably improved, and the detection performance is greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a vision-based method for rapid identification of surface damage in pipelines. Background Technology

[0002] Nuclear power plant pipelines are often subjected to harsh conditions such as high temperature and high pressure. Welds and bends within these pipelines may be subject to corrosion, erosion, and abrasion, leading to gradual thinning of the pipe wall and non-uniform thinning. This affects the pipeline's strength and, in severe cases, threatens its safe operation. Therefore, it is necessary to select appropriate testing methods for accurate measurement of thinned pipelines.

[0003] Real-time object detection is an important research area with wide applications, such as object tracking, video surveillance, and autonomous driving. Existing real-time detectors typically employ CNN-based architectures, achieving a reasonable trade-off between detection speed and accuracy. However, these real-time detectors usually require Non-Maximum Sampling (NMS) for post-processing, which is often difficult to optimize and not robust enough, leading to delays in the detector's inference speed.

[0004] Currently, deep learning-based image defect detection technology has been widely applied in industrial fields. The Transformer-based Object Detector (DETR), due to its elimination of various hand-designed components such as Non-maximum Suppression (NMS), has received widespread attention from the academic community since its inception. This architecture greatly simplifies the object detection pipeline, enabling end-to-end object detection.

[0005] However, DETR still has some drawbacks, such as high computational cost: while introducing multi-scale features helps accelerate training convergence, it significantly increases the sequence length of the input encoder. The high computational cost generated by the interaction of multi-scale features makes the Transformer encoder a computational bottleneck, limiting its practicality; difficulty in query optimization: the difficulty in optimizing target queries hinders DETR performance. Current query selection schemes directly use classification scores for selection, ignoring the fact that the detector needs to model both object category and location simultaneously, resulting in significant uncertainty in the selected encoder features and affecting detector performance; and inability to perform real-time detection: DETR is still computationally intensive and has not been designed for real-time detection, thus failing to meet the requirements of real-time detection. Summary of the Invention

[0006] The purpose of this invention is to provide a vision-based method for rapid identification of pipe surface damage. This method effectively reduces the delay in the detection process, ensures that the inference speed of the detector is not delayed and can operate stably, significantly improves the accuracy of defect detection, and greatly enhances the detection performance.

[0007] Technical solution to achieve the purpose of this invention:

[0008] A vision-based method for rapid identification of pipe surface damage, the method comprising:

[0009] Step S1: Collect images of the outer wall of pipelines in nuclear power plants, annotate the damage in the collected images, and construct a pipeline damage image dataset;

[0010] Step S2: Using the improved RT-DETR model, the image data in the pipeline damage image dataset is used as the input of the backbone network. The output of the backbone network is first used as the input of the BiFPN module, and then the output of the BiFPN module is used as the input of the hybrid encoder to construct a corrosion detection model.

[0011] Step S3: Train the constructed corrosion detection model using the constructed pipeline damage image dataset. After training, use the constructed corrosion detection model to infer the corrosion area using the test set or actual pipeline outer wall images.

[0012] Step S4: Evaluate the detection results of the model and apply the results to actual pipeline external wall corrosion detection work.

[0013] Step S1 includes:

[0014] Step S11: The pipeline external wall operation robot collects images of the outer wall of the nuclear power plant pipeline through the video inspection system, stores the collected images, and sends the images to the ground monitoring system to display the surface condition of the inspected pipeline in real time.

[0015] Step S12: Use the image processing tools in the ground monitoring system to preprocess the acquired images, highlight the damage features, and use the annotation tools in the ground monitoring system to annotate the highlighted damage feature areas.

[0016] Step S2 includes:

[0017] Step S21: The input layer of the improved RT-DETR model receives image data from the pipeline damage image dataset constructed in step S1, and feeds it into the backbone network. The backbone network extracts features and outputs multi-scale feature maps as input to the BiFPN module.

[0018] Step S22: The BiFPN module adjusts the channels through downsampling and 1x1 convolution to obtain feature map inputs at different levels. It then fuses the feature map inputs at different levels through bidirectional connectivity and weighted features to output a fused multi-scale feature map.

[0019] Step S23: The hybrid encoder converts the multi-scale feature map into a fused one-dimensional vector through intra-scale feature interaction and cross-scale feature fusion.

[0020] The calculation formula for the fused multi-scale feature map output in step S22 is as follows:

[0021]

[0022] In the formula:

[0023] This represents the output feature map of layer l;

[0024] This represents the input feature map of layer l;

[0025] and These represent input feature maps at scales higher and lower than layer l, respectively.

[0026] ω1, ω2, and ω3 represent the learnable weights of the corresponding input feature maps, respectively;

[0027] and Indicates the input and Perform size adjustment operations;

[0028] Conv represents the convolution operation, used for further processing of the fused features.

[0029] ε represents a very small number used to prevent the denominator from being zero.

[0030] In step S23, the hybrid encoder includes an AIFI module and a CCFM module. The AIFI module is used to process the feature map at each scale and capture the semantic information within each scale. The CCFM module is used to fuse the feature maps at different scales to generate a fused feature sequence.

[0031] The calculation formula for the AIFI module is as follows:

[0032] Q = K = V = Flatten (S'5)

[0033]

[0034] F5=Reshape(Attention(Q,K,V))

[0035] In the formula:

[0036] S'5 represents the multi-scale feature map obtained by fusing the feature map S5 output by the backbone network through the BiFPN module;

[0037] Q(Query) represents the query vector;

[0038] K(Key) represents the key vector;

[0039] V(Value) represents a value vector;

[0040] T represents the transpose of the key vector;

[0041] d k Indicates the dimension of the key vector;

[0042] F5 represents the globally enhanced same-scale feature map obtained after processing by the AIFI module.

[0043] The calculation formula for the CCFM module is as follows:

[0044] Fusion(x1,x2) = Concat(x1,x2)

[0045] f out =RepBlock N (f in )

[0046] Output = CCFM({S'3,S'4,F5})

[0047] In the formula:

[0048] x1 and x2 represent feature maps of different scales obtained by the BiFPN module or AIFI module. x1 and x2 are any two feature maps of different scales in S'3 and S'4.

[0049] S'3 and S'4 represent the multi-scale feature maps fused from the backbone network output feature maps S3 and S4 after processing by the BiFPN module; F5 represents the feature map processed by the AIFI module.

[0050] Concat represents the concatenation operation of feature maps at different scales;

[0051] RepBlock represents a structural reparameterization operation on the input features;

[0052] N represents the number of times RepBlock is repeated;

[0053] f in The input features represent the structural reparameterization operation;

[0054] f out This represents the output characteristics of the structure reparameterization operation.

[0055] Step S2 further includes: Step S24, IoU-aware query selection. During the decoding stage, RT-DETR dynamically selects the query most likely to contain the target based on the features output by the encoder and the IoU score of the target.

[0056] Step S3 involves training the constructed corrosion detection model using the constructed pipeline damage image dataset, specifically including:

[0057] Dataset partitioning: The constructed pipeline damage image dataset is divided into a training set, a validation set, and a test set;

[0058] Data preprocessing: Perform necessary preprocessing on the image, including resizing, normalization, and data augmentation;

[0059] Training the model: Train the model using the training set data while monitoring performance on the validation set to prevent overfitting; perform multiple rounds of iterative training until the model performance no longer improves significantly or the predetermined number of iterations is reached;

[0060] Model evaluation: Evaluate model performance on the validation set;

[0061] Hyperparameter tuning: Adjust the model's hyperparameters based on performance feedback on the validation set. Hyperparameters include learning rate, batch size, and number of network layers.

[0062] Model saving and loading: After training is complete, save the model's weights and structure.

[0063] The evaluation metrics for assessing the model's detection results in step S4 include: Precision, Recall, and Mean Precision (mAP), calculated as follows:

[0064] Accuracy calculation formula:

[0065] Recall rate calculation formula:

[0066] Formula for calculating the mean precision:

[0067] In the formula:

[0068] TP: Corrosion areas correctly identified by the model;

[0069] FP: The model incorrectly identifies non-corroded areas as corroded areas;

[0070] FN: Corrosion areas that the model failed to identify;

[0071] C: The total number of categories in the object detection task;

[0072] AP c : The average precision of the c-th category, which is the area under the precision-recall curve.

[0073] The beneficial technical effects of this invention are as follows:

[0074] 1. This invention employs an end-to-end detection technique. RT-DETR achieves the technical effect of eliminating the need for post-processing steps such as non-maximum suppression (NMS), thereby effectively reducing latency in the detection process and ensuring that the detector's inference speed is not delayed and can operate stably.

[0075] 2. This invention employs a high-efficiency hybrid encoder, IoU-aware query selection, and incorporates a BiFPN network structure. RT-DETR achieves improved detection accuracy while reducing computational costs, and can effectively extract and fuse feature information at different scales, thereby enhancing the detection capability for targets of different sizes and significantly improving the accuracy of defect detection.

[0076] 3. This invention integrates the BiFPN network into the model and, with the help of its innovative design of bidirectional feature fusion and weighted feature fusion, RT-DETR achieves the technical effects of reducing parameter computation, significantly improving the performance of the Feature Pyramid Network (FPN), and optimizing the number of parameters, making the model more efficient and concise.

[0077] 4. The RT-DETR of this invention uses optimization techniques such as sparse query mechanism, position encoding optimization and multi-scale feature fusion to predict regions where targets exist, thereby reducing unnecessary computation. It successfully solves the bottleneck problem of traditional DETR using dense query sets to predict all targets, which leads to a large amount of computational redundancy and inference speed. It achieves the technical effect of improving model computational efficiency while maintaining high accuracy, and greatly improves detection performance.

[0078] 5. This invention proposes an IoU-Aware query selection technique. RT-DETR provides higher quality initial target queries to the decoder by offering IoU constraints during training, thereby improving detection accuracy. In particular, it can more accurately identify and locate targets when facing small target detection in complex backgrounds.

[0079] 6. This invention utilizes the design of the decoder in the DETR architecture to support the technical means of flexibly adjusting the inference speed without retraining by using different decoder layers. RT-DETR achieves enhanced flexibility and adaptability, making it easier to apply to real-time detection scenarios and meet the needs of different practical applications. Attached Figure Description

[0080] Figure 1 Diagram of the improved RT-DETR model architecture;

[0081] Figure 2 Here is a diagram of the BiFPN network structure;

[0082] Figure 3This is a structural diagram of the CCFM feature fusion module. Detailed Implementation

[0083] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0084] like Figure 1 As shown, the present invention provides a vision-based method for rapid identification of pipe surface damage, which specifically includes the following steps:

[0085] Step S1: Collect images of the outer wall of pipelines in nuclear power plants, annotate the damage in the collected images, and construct a pipeline damage image dataset.

[0086] A damage detection system (including a video detection system and a ground monitoring system) is used to work on the outer wall of the pipeline. The robot is controlled wirelessly to crawl and capture close-up images of the pipeline surface. Damage in the acquired images is then labeled, and a pipeline damage image dataset is created.

[0087] Specifically, step S1 includes:

[0088] Step S11: The pipeline external wall operation robot collects images of the outer wall of the nuclear power plant pipeline through the video inspection system, stores the collected images, and sends the images to the ground monitoring system to display the surface condition of the inspected pipeline in real time.

[0089] The video inspection system for pipeline external wall operations includes cameras, local storage, and wireless image transmission equipment. The cameras have more than 2 megapixels and a maximum resolution greater than 1280*720. The wireless image transmission equipment uses a 1.2GHz or 2.4GHz communication frequency. The video inspection system captures video and images of the external walls of nuclear power plant pipelines using the cameras, storing the acquired image data in local storage. This acquired image data is then transmitted to a ground monitoring system via the wireless image transmission equipment, displaying the surface condition of the inspected pipeline in real time.

[0090] Step S12: Use the image processing tools in the ground monitoring system to preprocess the acquired images, highlight the damage features, and use the annotation tools in the ground monitoring system to annotate the highlighted damage feature areas.

[0091] The image processing tools in the ground monitoring system are used to preprocess the acquired images. Image preprocessing includes operations such as grayscale conversion, filtering, and enhancement to improve image quality and highlight damage features such as cracks and corrosion. The annotation tools in the ground monitoring system are used to select areas of damage features such as cracks and corrosion in the images using rectangular bounding boxes or pixel-level annotations, providing accurate "ground truth" data for subsequent model training. The annotation categories include: partial corrosion, protrusions, pits, cracks, and fissures.

[0092] Step S2: Using the improved RT-DETR model, the image data in the pipeline damage image dataset is used as the input of the backbone network. The output of the backbone network is first used as the input of the BiFPN module, and then the output of the BiFPN module is used as the input of the hybrid encoder (AIFI module and CCFM module) to construct the corrosion detection model.

[0093] Based on the RT-DETR model, improvements were made to its backbone network and hybrid encoder, using a dataset as input for training. The RT-DETR model consists of a backbone network, a hybrid encoder, and a Transformer decoder with an auxiliary prediction head. The improved RT-DETR model includes a backbone network, a BiFPN module, an AIFI module, and a CCFM module. Figure 1 As shown, the improved RT-DETR model first uses the output of the backbone network as the input of the BiFPN (Bi-directional Feature Pyramid Network) module, and then uses the output of the BiFPN module as the input of the hybrid encoder (AIFI module and CCFM module).

[0094] Compared to DETR, the RT-DETR basic model can:

[0095] ① Design an efficient hybrid encoder: By decoupling the intra-scale interaction and cross-scale fusion of features at different scales, the inference speed is significantly improved, and the encoder computation bottleneck problem is solved.

[0096] ② Propose a query selection method with minimal uncertainty: explicitly construct and optimize cognitive uncertainty to model the joint latent variables of encoder features, provide high-quality queries for the decoder, and reduce the difficulty of optimizing the target query.

[0097] ③ Supports flexible speed adjustment: The speed can be flexibly adjusted by changing the number of decoder layers, without the need for retraining, which improves the practicality and adaptability of the model.

[0098] The improved RT-DETR model incorporates a BiFPN network structure in the Neck region. BiFPN (Weighted Bidirectional Feature Pyramid Network) is a feature fusion layer structure. Compared to DETR, the improved RT-DETR model can:

[0099] ① Achieve simple and fast multi-scale feature fusion: BiFPN can achieve simple and fast multi-scale feature fusion. It improves accuracy through multi-level feature pyramids and bidirectional information transmission, and can better fuse multi-scale features to improve the expressive power of features.

[0100] ② Reduce computational costs: Without compromising performance metrics, computational costs are reduced, making the model more efficient.

[0101] BiFPN enables cross-level information exchange by introducing bidirectional connections within feature pyramid networks at different scales. Traditional feature pyramid networks (FPNs) only support unidirectional (top-down) information flow, which limits their ability to handle object detection in complex scenes. BiFPN, however, introduces a bidirectional connection design, preserving not only the original top-down path but also adding bottom-up connections.

[0102] Specifically, step S2 includes:

[0103] Step S21: The input layer of the improved RT-DETR model receives image data from the pipeline damage image dataset constructed in step S1 and feeds this image data into the backbone network. The backbone network extracts features from the input images and outputs multi-scale feature maps. In the improved RT-DETR, the feature maps S3, S4, and S5 (i.e., feature maps at three scales, S3, S4, and S5) output by the backbone network are used as input to the BiFPN module.

[0104] Step S22: The BiFPN module adjusts the channels through downsampling and 1x1 convolution to obtain feature map inputs at different levels. It then fuses the feature map inputs at different levels through bidirectional connectivity and weighted features to output a fused multi-scale feature map.

[0105] Bidirectional connectivity includes multi-scale feature fusion of top-down and bottom-up paths.

[0106] Top-down path: Starting from the high-level (low resolution, strong semantic information) feature map, the resolution is increased to the same level as the next level feature map through upsampling operations (such as bilinear interpolation), and then fused with the corresponding next level feature map.

[0107] Bottom-up path: Start with the feature map of the lower level (high resolution, more detailed information), reduce its resolution to the same level as the feature map of the previous level through downsampling operations (such as average pooling), and then fuse it with the corresponding feature map of the previous level.

[0108] Feature fusion: Bidirectional connectivity allows information from different levels of the feature pyramid to be fused in two directions, which helps to effectively capture multi-scale features.

[0109] Weighted Feature Fusion: BiFPN adopts a more flexible weighting strategy to combine features across layers. Each node is assigned a corresponding weight coefficient based on the different contributions of adjacent layers, thereby dynamically adjusting the importance of each part and ensuring dynamic adjustment of importance.

[0110] Repeated fusion: Treat each bidirectional (top-down and bottom-up) path as a feature network layer and repeat the same layer multiple times to enable more advanced feature fusion.

[0111] In the RT-DETR model, BiFPN is used to fuse the feature maps at three scales (S3, S4, and S5) output by the backbone network. The calculation formula for the fused multi-scale feature map is as follows:

[0112]

[0113] In the formula:

[0114] This represents the output feature map of layer l (where l represents the layer of the feature map). For the feature map of layer l, it can be one of S3, S4 or S5.

[0115] This represents the input feature map of layer l. For the feature map of layer l, it can be one of S3, S4, or S5.

[0116] and These represent input feature maps at scales higher and lower than layer l, respectively.

[0117] ω1, ω2, and ω3 represent the learnable weights of the corresponding input feature maps.

[0118] and Indicates the input and A resize operation is performed to make all input feature maps the same size so that they can be fused.

[0119] Conv stands for convolution operation, used for further processing of the fused features.

[0120] ε represents a very small number used to prevent the denominator from being zero; it is usually set to le-6 or other small values.

[0121] Specifically, such as Figure 2 As shown, the BiFPN network structure mainly consists of multiple feature layers (such as P3, P4, P5, P6, P7, etc.) and the connections and operations between them. Through this complex yet effective feature fusion network structure, BiFPN can more efficiently fuse feature information at different scales, thereby improving the performance of computer vision tasks such as object detection. Through bidirectional feature fusion and weighted operations, the network can better utilize features from different levels, enhancing its ability to detect targets of different sizes.

[0122] Figure 2 The nodes P3, P4, P5, P6, P7, etc., represent feature layers of different scales. These feature layers play different roles in the network, capturing feature information of targets of different sizes. The arrows indicate the direction of feature flow and the path for operations such as weighted fusion.

[0123] Figure 2 The paper demonstrates bidirectional connections, including top-down paths connecting features from one layer to the next (e.g., connecting P6 from P7 after resizing) and bottom-up paths connecting features from one layer to the next (e.g., connecting P6 from P3 after certain operations). This bidirectional connection method allows feature information at different scales to complement and fuse with each other.

[0124] exist Figure 2 The diagram shows several weighted fusion operations. For example, from P6 and Resize(P7) to... The fusion process, its calculation process is as follows:

[0125]

[0126] This calculation process involves processing input features. and features after the Resize operation After weighted summation, a convolution operation (Conv) is performed, where ω1 and ω2 are the weights, and ε is a very small constant to avoid the denominator being 0.

[0127] Similarly to There is also a similar weighted fusion operation, the calculation formula is:

[0128]

[0129] Step S23: The hybrid encoder converts the multi-scale feature map into a fused one-dimensional vector through intra-scale feature interaction and cross-scale feature fusion.

[0130] In RT-DETR, the hybrid encoder first processes the feature maps at each scale using an attention-based intra-scale feature interaction (AIFI) module to capture semantic information within each scale. Then, a cross-scale feature fusion (CCFM) module fuses these feature maps from different scales to generate a fused feature sequence. This feature sequence is one-dimensional and contains rich semantic and spatial information for subsequent decoder processing.

[0131] Different variants of the hybrid encoder are designed, each based on a fundamental encoder structure, with performance enhanced by adding different modules. The hybrid encoder achieves image feature extraction through an attention-based intra-scale feature interaction module (AIFI module) and a neural network-based cross-scale feature fusion module (CCFM module).

[0132] The AIFI module utilizes a self-attention mechanism to process high-level features in images and focuses on intra-scale feature interactions at high-level feature layers, reducing computational redundancy. The calculation formula for the AIFI module is as follows:

[0133] Q = K = V = Flatten (S'5)

[0134]

[0135] F5=Reshape(Attention(Q,K,V))

[0136] In the formula:

[0137] S'5 represents the multi-scale feature map obtained by fusing the feature map S5 output by the backbone network through the BiFPN module.

[0138] Q(Query) represents the query vector.

[0139] K(Key) represents the key vector.

[0140] V(Value) represents a value vector.

[0141] T represents the transpose of the key vector, used to ensure that the dimensions of the matrices match so that the attention score can be calculated correctly.

[0142] d k This represents the dimension of the key vector.

[0143] F5 represents the globally enhanced same-scale feature map obtained after processing by the AIFI module. This feature map contains enhanced features calculated through the self-attention mechanism. These features are spatially consistent but have been enhanced with global information.

[0144] In the AIFI module, the multidimensional feature map S'5 is converted into a one-dimensional vector through the Flatten operation, which serves as the input to the attention mechanism. The attention mechanism introduces d... k Used to scale the dot product operation by dividing the dot product result by... To avoid gradient vanishing or excessively large gradients during computation, a reshape operation is used to reshape the processed one-dimensional vector to the same shape as the original feature map, resulting in an enhanced, same-scale feature map F5.

[0145] The CCFM module inserts multiple fusion blocks composed of convolutional layers into the fusion path. These blocks process and fuse feature maps from two adjacent scales. The fusion process may involve various operations, such as element-wise addition and concatenation, to integrate feature information from different scales. The fused feature map is then converted into a one-dimensional vector using the Flatten operation for subsequent processing or analysis. Feature fusion can be implemented in various ways, including but not limited to element-wise addition, concatenation, or other more complex operations.

[0146] The fusion block contains two 1×1 convolutions to adjust the number of channels, and uses N repblocks composed of RepConv for feature fusion. The calculation formula for the CCFM module is as follows:

[0147] Fusion(x1,x2) = Concat(x1,x2)

[0148] f out =RepBlock N (f in )

[0149] Output = CCFM({S'3,S'4,F5})

[0150] In the formula:

[0151] x1 and x2 represent feature maps of different scales obtained through the BiFPN or AIFI modules, where x1 and x2 are any two feature maps of different scales from S'3 and S'4. S'3 and S'4 represent the fused multi-scale feature maps obtained by processing the backbone network's output feature maps S3 and S4 through the BiFPN module. F5 represents the feature map processed by the AIFI module.

[0152] Concat represents the concatenation operation of feature maps at different scales.

[0153] RepBlock represents a structural reparameterization operation on the input features.

[0154] N represents the number of times RepBlock is repeated.

[0155] f in This represents the input features of the structure reparameterization operation.

[0156] f out This represents the output characteristics of the structure reparameterization operation.

[0157] Specifically, such as Figure 3 As shown, the CCFM module is a structure for feature fusion, mainly composed of convolutional operations, a specific RepBlock module, and some feature processing operations. This CCFM feature fusion module effectively fuses two input features through a series of convolutions, concatenation, repetition modules, and feature processing operations, obtaining the final fused features for subsequent task processing.

[0158] CCFM module input: There are two input branches, and the feature dimension of each branch is labeled "c" (number of channels).

[0159] CCFM module output: After a series of processing steps, the final output is a fused feature, whose dimension is labeled "F" (the feature after the Flatten operation).

[0160] Internal processing flow:

[0161] ① Left Branch: The feature "c" input from the left branch first undergoes a "1x1 Conv" (1x1 convolution operation), and its output feature dimension remains "c". The convolved feature is then "concatenated" with the original feature "c" from the other input branch, and the concatenated feature dimension becomes "C".

[0162] ② Right Branch: The input feature "c" on the right also undergoes a "1x1 Conv" (1x1 convolution operation), and the output feature dimension is also "c". The convolved feature enters a "RepBlock" module, which is repeated "N×" times. After each pass through the RepBlock module, the feature dimension remains "c". The feature after passing through the RepBlock module "N×" times is then combined with the concatenated feature "C" from the left branch and subjected to an "Element-wise add" operation. The added feature dimension is still "c".

[0163] ③ The feature "c" after element-wise addition is then subjected to the "Flatten" operation to finally obtain the fused feature "F" and output it. The Flatten operation, also known as the flattening operation or the flattening operation, is mainly used to transform multidimensional tensors (such as multidimensional arrays, multidimensional matrices, etc.) into one-dimensional vector forms.

[0164] Step S24, IoU-aware query selection: During the decoding stage, RT-DETR dynamically selects the query most likely to contain the target based on the features output by the encoder and the IoU score of the target.

[0165] Object queries in DETR are a set of learnable embeddings optimized by the decoder and mapped to classification scores and bounding boxes by the prediction head. However, these object queries are difficult to interpret and optimize because they lack explicit physical meaning. Subsequent work improved the initialization of object queries and extended it to content queries and location queries (anchors). Specifically, query selection schemes were proposed, all characterized by using classification scores to select the top K features from the encoder to initialize object queries (or location queries only). However, due to the inconsistent distribution of classification scores and location confidence, some predicted boxes, despite having high classification scores, are not close to the ground truth (GT), leading to the selection of boxes with high classification scores but low IoU scores, while discarding boxes with low classification scores but high IoU scores. This degrades detector performance. To address this issue, we propose IoU-aware query selection, which constrains the model during training to produce high classification scores for features with high IoU scores and low classification scores for features with low IoU scores. Therefore, the model selects predicted boxes from the top K encoder features based on classification scores, resulting in boxes with both high classification scores and high IoU scores. We reformulate the detector's optimization objective as follows:

[0166]

[0167] in, y and y represent the predicted and actual values, respectively. Let y = {c, b}, where c and b represent the class and bounding box, respectively. We introduce the IOU score into the objective function of the classification branch (similar to VFL) to achieve consistency constraints on the classification and localization of positive samples.

[0168] The model uses confidence scores to select the top K features from the encoder to initialize object queries or location-only queries, minimizing query uncertainty. Query selection is optimized by incorporating uncertainty U into the loss function. Through optimization algorithms such as gradient descent, the model tends to reduce uncertainty, thus selecting features that predict more consistently in both localization and classification as queries, providing higher-quality initialization for the decoder.

[0169] Definition of uncertainty:

[0170]

[0171] Define characteristic uncertainty It is the difference between the predicted distributions of location P and classification C. Among them, The encoder features are represented by ||.|| represents a distance metric, such as Euclidean distance. This discrepancy measures the inconsistency in predictions made by the features on localization and classification tasks.

[0172] Loss function:

[0173]

[0174] in, This represents the bounding box regression loss, which measures the predicted bounding box. The difference between the actual bounding box b and the actual bounding box b. This represents the classification loss, which considers not only classification prediction. The difference between the true class c and the actual class c also takes into account feature uncertainty. This allows the model to focus more on features with higher uncertainty during training.

[0175] RT-DETR reduces unnecessary computation by introducing a sparse query mechanism that only predicts regions where the target exists.

[0176] The sparse detection mechanism is primarily implemented in the decoding stage, reducing computational redundancy through dynamic query selection and sparse attention. Optimizations in the encoding stage provide high-quality feature input for sparse detection; the two work together to achieve efficient object detection. This design allows RT-DETR to significantly improve inference speed while maintaining high accuracy.

[0177] Specifically:

[0178] 1. Sparse query mechanism in the decoding phase

[0179] The core idea of ​​sparse detection is to reduce the number of queries and computational cost in the decoder. In traditional DETR frameworks, the decoder typically uses a fixed number of queries (e.g., 100 or more) to match features from the encoder output to predict the location and class of the target. However, this dense query approach leads to significant computational redundancy, as many queries may correspond to background regions or duplicate targets.

[0180] RT-DETR achieves sparse queries during the decoding phase in the following way:

[0181] ① IoU-aware Query Selection: During the decoding stage, RT-DETR dynamically selects the query most likely to contain the target based on the features output by the encoder and the target's IoU score. This method reduces the number of unnecessary queries, allowing the decoder to focus only on the region most likely to contain the target, thereby improving computational efficiency.

[0182] ② Dynamic Attention Mechanism: RT-DETR introduces a dynamic attention mechanism in the decoder, which allows each query to focus only on the feature regions most relevant to it, rather than the entire feature map. This sparse attention mechanism further reduces computational cost.

[0183] 2. Optimization during the coding phase

[0184] Although sparse detection mechanisms are primarily implemented in the decoding stage, feature extraction and processing in the encoding stage also provide support for sparse detection:

[0185] Attention-based Intra-Scale Feature Interaction (AIFI): The encoder captures semantic information from high-level features through the AIFI module, reducing redundant computation of low-level features. This optimization makes the features output by the encoder more compact and efficient, providing high-quality input for sparse queries in the decoding stage.

[0186] Cross-scale feature fusion (CCFM): The encoder fuses features from multiple scales through the CCFM module, ensuring that the decoder can fully utilize feature information at different scales. This fusion mechanism reduces redundant computation between features and provides richer contextual information for sparse detection.

[0187] Step S3: Train the constructed corrosion detection model using the constructed pipeline damage image dataset. After training, use the constructed corrosion detection model to infer the corrosion area using the test set or actual pipeline outer wall images.

[0188] The constructed corrosion detection model was trained using the constructed pipeline damage image dataset, specifically including:

[0189] ① Dataset partitioning: The constructed pipeline damage image dataset is divided into training set, validation set and test set.

[0190] ② Data preprocessing: Perform necessary preprocessing on the images, such as resizing, normalization, and data augmentation, to improve the model's generalization ability.

[0191] ③ Train the model: Train the model using the training set data, while monitoring performance on the validation set to prevent overfitting. Perform multiple rounds of iterative training until the model performance no longer improves significantly or the predetermined number of iterations is reached.

[0192] ④ Model evaluation: Evaluate model performance on the validation set using metrics such as mAP (mean Average Precision).

[0193] ⑤ Hyperparameter tuning: Based on the performance feedback on the validation set, adjust the model's hyperparameters, such as learning rate, batch size, and number of network layers, to obtain better performance.

[0194] ⑥ Model saving and loading: After training is completed, save the model's weights and structure for subsequent inference or further training.

[0195] After training, the constructed corrosion detection model will obtain the optimal weights. The trained corrosion detection model and its weights will be loaded to perform inference on the pipeline outer wall image and detect the surface corrosion area.

[0196] Step S4: Evaluate the detection results of the model and apply the results to actual pipeline external wall corrosion detection work.

[0197] Choose appropriate evaluation metrics to measure model performance. Commonly used metrics include precision, recall, and mean average precision (mAP). Deploy the evaluated model into a real-world pipeline inspection system to detect corrosion on the pipeline's outer wall. The project technical specifications require a mean average precision (mAP) of no less than 0.7; the other metrics are commonly used for model evaluation.

[0198] The formulas for calculating accuracy, recall, and mean precision are as follows:

[0199] Accuracy calculation formula:

[0200] Recall rate calculation formula:

[0201] Formula for calculating the mean precision:

[0202] In the formula:

[0203] TP: Corrosion areas correctly identified by the model;

[0204] FP: The model incorrectly identifies non-corroded areas as corroded areas;

[0205] FN: Corrosion areas that the model failed to identify;

[0206] C: The total number of categories in the object detection task;

[0207] AP c: The average precision of the c-th category, which is the area under the precision-recall (PR) curve (i.e., the result of integrating the curve).

[0208] The present invention has been described in detail above with reference to the accompanying drawings and embodiments. However, the present invention is not limited to the above embodiments, and various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention. All contents not described in detail in the present invention can be derived from existing technologies.

Claims

1. A vision-based method for rapid identification of pipe surface damage, characterized in that, The method includes: Step S1: Collect images of the outer wall of pipelines in nuclear power plants, annotate the damage in the collected images, and construct a pipeline damage image dataset; Step S2: Using the improved RT-DETR model, the image data in the pipeline damage image dataset is used as the input of the backbone network. The output of the backbone network is first used as the input of the BiFPN module, and then the output of the BiFPN module is used as the input of the hybrid encoder to construct a corrosion detection model. Step S3: Train the constructed corrosion detection model using the constructed pipeline damage image dataset. After training, use the constructed corrosion detection model to infer the corrosion area using the test set or actual pipeline outer wall images. Step S4: Evaluate the detection results of the model and apply the results to actual pipeline external wall corrosion detection work.

2. The vision-based rapid identification method for pipe surface damage according to claim 1, characterized in that, Step S1 includes: Step S11: The pipeline external wall operation robot collects images of the outer wall of the nuclear power plant pipeline through the video inspection system, stores the collected images, and sends the images to the ground monitoring system to display the surface condition of the inspected pipeline in real time. Step S12: Use the image processing tools in the ground monitoring system to preprocess the acquired images, highlight the damage features, and use the annotation tools in the ground monitoring system to annotate the highlighted damage feature areas.

3. The vision-based rapid identification method for pipe surface damage according to claim 1, characterized in that, Step S2 includes: Step S21: The input layer of the improved RT-DETR model receives image data from the pipeline damage image dataset constructed in step S1, and feeds it into the backbone network. The backbone network extracts features and outputs multi-scale feature maps as input to the BiFPN module. Step S22: The BiFPN module adjusts the channels through downsampling and 1x1 convolution to obtain feature map inputs at different levels. It then fuses the feature map inputs at different levels through bidirectional connectivity and weighted features to output a fused multi-scale feature map. Step S23: The hybrid encoder converts the multi-scale feature map into a fused one-dimensional vector through intra-scale feature interaction and cross-scale feature fusion.

4. The vision-based rapid identification method for pipe surface damage according to claim 3, characterized in that, The calculation formula for the fused multi-scale feature map output in step S22 is as follows: In the formula: This represents the output feature map of layer l; This represents the input feature map of layer l; and These represent input feature maps at scales higher and lower than layer l, respectively. ω1, ω2, and ω3 represent the learnable weights of the corresponding input feature maps, respectively; and Indicates the input and Perform size adjustment operations; Conv represents the convolution operation, used for further processing of the fused features. ε represents a very small number used to prevent the denominator from being zero.

5. The vision-based rapid identification method for pipe surface damage according to claim 3, characterized in that, In step S23, the hybrid encoder includes an AIFI module and a CCFM module. The AIFI module is used to process the feature map at each scale and capture the semantic information within each scale. The CCFM module is used to fuse the feature maps at different scales to generate a fused feature sequence.

6. The vision-based rapid identification method for pipe surface damage according to claim 5, characterized in that, The calculation formula for the AIFI module is as follows: Q = K = V = Flatten (S'5) F5=Reshape(Attention(Q,K,V)) In the formula: S'5 represents the multi-scale feature map obtained by fusing the feature map S5 output by the backbone network through the BiFPN module; Q(Query) represents the query vector; K(Key) represents the key vector; V(Value) represents a value vector; T represents the transpose of the key vector; d k Indicates the dimension of the key vector; F5 represents the globally enhanced same-scale feature map obtained after processing by the AIFI module.

7. The vision-based rapid identification method for pipe surface damage according to claim 5, characterized in that, The calculation formula for the CCFM module is as follows: Fusion(x1,x2) = Concat(x1,x2) f out =RepBlock N (f in ) Output = CCFM({S'3,S'4,F5}) In the formula: x1 and x2 represent feature maps of different scales obtained by the BiFPN module or AIFI module. x1 and x2 are any two feature maps of different scales in S'3 and S'4. S'3 and S'4 represent the multi-scale feature maps fused from the backbone network output feature maps S3 and S4 after processing by the BiFPN module; F5 represents the feature map processed by the AIFI module. Concat represents the concatenation operation of feature maps at different scales; RepBlock represents a structural reparameterization operation on the input features; N represents the number of times RepBlock is repeated; f in The input features represent the structural reparameterization operation; f out This represents the output characteristics of the structure reparameterization operation.

8. The vision-based rapid identification method for pipe surface damage according to claim 3, characterized in that, Step S2 further includes: Step S24, IoU-aware query selection. During the decoding stage, RT-DETR dynamically selects the query most likely to contain the target based on the features output by the encoder and the IoU score of the target.

9. The vision-based rapid identification method for pipe surface damage according to claim 1, characterized in that, Step S3 involves training the constructed corrosion detection model using the constructed pipeline damage image dataset, specifically including: Dataset partitioning: The constructed pipeline damage image dataset is divided into a training set, a validation set, and a test set; Data preprocessing: Perform necessary preprocessing on the image, including resizing, normalization, and data augmentation; Training the model: Train the model using the training set data while monitoring performance on the validation set to prevent overfitting; perform multiple rounds of iterative training until the model performance no longer improves significantly or the predetermined number of iterations is reached; Model evaluation: Evaluate model performance on the validation set; Hyperparameter tuning: Adjust the model's hyperparameters based on performance feedback on the validation set. Hyperparameters include learning rate, batch size, and number of network layers. Model saving and loading: After training is complete, save the model's weights and structure.

10. A vision-based rapid identification method for pipe surface damage according to claim 1, characterized in that, The evaluation metrics for assessing the model's detection results in step S4 include: Precision, Recall, and Mean Precision (mAP), calculated as follows: Accuracy calculation formula: Recall rate calculation formula: Formula for calculating the mean precision: In the formula: TP: Corrosion areas correctly identified by the model; FP: The model incorrectly identifies non-corroded areas as corroded areas; FN: Corrosion areas that the model failed to identify; C: The total number of categories in the object detection task; AP c : The average precision of the c-th category, which is the area under the precision-recall curve.

Images