A sewer pipeline video defect time point positioning method based on a Transformer architecture

Abstract

The present application relates to a kind of based on the positioning method of sewer video defect time point of Transformer architecture, belong to the field of video understanding in computer vision, including following steps: (1) construct pipeline video dataset, by comparing the screenshot in pipeline detection report, construct pipeline detection video defect time point positioning dataset.(2) using pipeline video dataset constructs image classification auxiliary dataset.(3) training pipeline image classification model, cooperate discrete wavelet transform and pre-training ResNet151 network on ImageNet, three kinds of features are extracted from video frame by frame.(4) build video time point positioning network based on Transformer architecture.(5) the training of neural network model is carried out.(6) the optimal model weight saved is used to carry out the automatic positioning of pipeline detection video defect time point.The present application is mainly used for the automatic time positioning of pipeline defect in detection video when city sewer pipeline CCTV detection, improves detection efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to video understanding technology in computer vision, and in particular to a deep learning-based algorithm for locating time-point defects in drainage pipe videos. Background Technology

[0002] With the modernization of cities and the expansion of urban scale, the maintenance of urban infrastructure has become an important part of ensuring the long-term development of cities. Drainage pipes are an important component of urban infrastructure and play a vital role in the normal operation of cities. They are known as the "blood vessels" of cities. In order to ensure the normal operation of drainage pipes, regular inspection and maintenance are particularly important. At present, the traditional drainage pipe defect detection mainly relies on the closed-circuit television (CCTV) system [1]. This system uses a remote-controlled robot with a camera to move inside the pipe and take pictures of the internal situation. Then, the location of the pipe defect is identified and judged by manual screening. However, with the expansion of the underground pipe network, the traditional manual inspection method faces challenges such as low efficiency, weak objectivity and high labor costs.

[0003] During CCTV inspection of drainage pipelines, a large number of videos collected need to be manually reviewed to locate the time when pipeline defects occur, and then the defect type, score, level and other information are identified and a defect inspection report is written. This process is not only inefficient, but also easily affected by subjective factors, reducing the objectivity of the inspection results. Although the academic and industrial communities have conducted in-depth research on automated pipeline defect identification algorithms and made some progress [2-5], the actual application effect of these algorithms and software still falls short of the industry's strict requirements for high recall [6]. Considering the unavoidable omission problem of algorithm detection, manual inspection is still the main method at present, and it is unlikely to be completely replaced by algorithms in the short term. Therefore, the key to computer vision technology in the field of pipeline inspection at this stage is how to better assist and cooperate with workers in carrying out identification work and improve efficiency, rather than simply replacing manual labor with machines.

[0004] Studies have shown that the time cost spent locating pipeline defects accounts for the largest proportion of the entire judgment process [7]. Meanwhile, according to the pipeline inspection video dataset used in this paper, from the perspective of video duration, the total duration of labeled videos is about 100 hours. If we calculate that an average pipeline defect appears for 10 seconds, the duration containing pipeline defects accounts for about 5% of the total duration, which indicates that the task of pipeline defect temporal localization is quite difficult. In this context, it is particularly important to study the CCTV Video Temporal Defect Localization (CTDL) algorithm for drainage pipeline inspection videos. It can not only help workers quickly locate pipeline defects and improve efficiency, but also supplement the results of manual inspection and reduce omissions.

[0005] There has been extensive research on using computer vision technology to assist in the localization of defects in drainage pipes [2-5,8,9]. These studies use frame-by-frame image classification methods, combined with complex post-processing methods such as metric learning, Kalman filtering, and peak finding algorithms to achieve the final localization, without directly modeling the video sequence.

[0006] Temporal Action Localization (TAL) is a popular task in computer vision, similar to the CTDL task. High-performing networks are based on video sequences for modeling [10-12]. However, due to the significant differences between the task content and the CTDL task, and the characteristics of pipeline scenes such as simple backgrounds, insufficient lighting, lack of texture, and image blurring, directly applying methods from the TAL field to the pipeline detection video defect localization task does not yield good results [9]. Therefore, the mainstream method for the CTDL task is still to decompose the video sequence into single-frame images and then process them using image classification methods. Therefore, designing a pipeline defect localization method based on video sequence modeling, which is specific to the characteristics of drainage pipeline detection scenarios, is of great research significance for further improving the automation and intelligence of drainage pipeline defect detection.

[0007] References:

[0008] [1] Zhang Jun. On the detection and evaluation methods of drainage pipelines [J]. Water Supply and Drainage, 2011(S1):418–420.

[0009] [2]Yin

[0010] [3]Ma D, FangH, Wang N, et al.Transformer-Optimized Generation, Detection, and Tracking Network for Images with Drainage Pipeline Defects[J]. Computer-Aided Civil and Infrastructure Engineering, 2023, 38(15):2109–2127.

[0011] [4]Wang M,Kumar SS,Cheng JC P.Automated Sewer Pipe Defect Tracking in CCTV Videos Based on Defect Detection and Metric Learning[J].Automation inConstruction,2021,121:103438.

[0012] [5]Xiao B, Lin Q, Chen YA Vision-Based Method for Automatic Tracking of Construction Machines at Nighttime Based on Deep Learning IlluminationEnhancement[J]. Automation in Construction, 2021,127:103721.

[0013] [6] Ministry of Housing and Urban-Rural Development. Technical Specification for Inspection and Evaluation of Urban Drainage Pipelines: CJJ181—2012 [S]. Beijing: China Architecture & Building Press, 2012.

[0014] [7]Mohamed M A-H,Ramadan M A,El-Dash K M.Cost optimization of sewagepipelines inspection[J].Ain Shams Engineering Journal,2023,14(5):101960.

[0015] [8]Wang M,Yin X.Construction and Maintenance of Urban UndergroundInfrastructure with Digital Technologies[J].Automation in Construction,2022,141:104464.

[0016] [9]Liu Y,Zhang X,Li Y,et al.VideoPipe 2022 Challenge:Real-World VideoUnderstanding for Urban Pipe Inspection[C].In 2022 26th InternationalConference on Pattern Recognition(ICPR),Aug 2022:4967–4973.

[0017]

[10] Zhang C,Wu J,Li Y.ActionFormer:Localizing Moments of Actions withTransformers.2022.

[0018]

[11] Lee S,Lim J,Moon J,et al.An Improved Point-Level SupervisionMethod for Temporal Action Localization[J].IEEE Access,2023,11:71260–71268.

[0019]

[12] Lee P,Byun H.Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization.2021.

[0020]

[13] Apostolidis E,Balaouras G,Mezaris V,et al.Combining Global andLocal Attention with Positional Encoding for Video Summarization[C].In 2021IEEE International Symposium on Multimedia(ISM),Nov 2021:226–234. Summary of the Invention

[0021] This invention uses a neural network based on the Transformer architecture as the main model. It is trained and tested using a large amount of CCTV inspection reports and corresponding inspection video data of drainage pipes. This invention can quickly and accurately locate the time points in drainage pipe videos where pipe defects may occur. The technical solution of this invention is as follows:

[0022] A method for locating time-point defects in drainage pipe videos based on the Transformer architecture, the general steps of which are described below:

[0023] Step 1: Extract images of pipe defects from the pipe defect detection report, compare them frame by frame with the video to obtain the time points of occurrence, construct a pipeline detection video defect time point localization dataset, and divide it into training set, validation set and test set in an 8:1:1 ratio.

[0024] Step 2: Extract the corresponding video frames based on the time point of the pipe defect occurrence, as the pipe defect image. Extract video frames evenly across other time intervals as normal pipe images, constructing a pipe image classification dataset. The dataset's partitioning corresponds one-to-one with the video dataset; that is, video frames in the training set portion of the video dataset will serve as the training set for the image classification dataset, and so on.

[0025] Step 3: Train a pipeline image classification model using the AutoGluon framework as a feature extractor to extract features from the pipeline detection video. This feature is denoted as the image classification feature F. cls The video data was decomposed using Discrete Wavelet Transform (DWT). Haar wavelet basis functions were used in the time dimension, and Bior 4.4 wavelet basis functions were used in both the width and height dimensions. For the decomposed high-frequency and low-frequency sub-bands, a ResNet151 network pre-trained on ImageNet was used for frame-by-frame feature extraction. These features are denoted as DWT features F. dwt For the original video, a ResNet151 network pre-trained on ImageNet is used directly for frame-by-frame feature extraction, and the features are denoted as the basic features F. base.

[0026] Step 4: Build a video time-stamping localization network based on the Transformer architecture. The network architecture is as follows:

[0027] like Figure 1 The diagram shows the overall architecture of the network. The network consists of a video frame encoder module and a defect time-point decoder module, accepting features extracted by three different CNN feature extractors as input to obtain F. cls F dwt and F base The outputs of these feature extractors are fed into the video frame encoder module for self-attention operations, resulting in three memory tensors. These are then self-attentioned with three independent learnable query tensors. Finally, the outputs of the three decoders are fused into a single tensor, which is then used to predict the class and time point via two FFNs.

[0028] Specifically, consider a batch of video data of size B. Within this batch, the longest video has a frame count of T. max The video resolution is H×W, and each frame has C channels. First, a batch of video data undergoes sequence padding preprocessing to ensure that their lengths are all uniformly T. max Therefore, the video tensor representation is obtained as follows:

[0029] V input =[B,T max [H, W, C]

[0030] The video sequence is then fed into three feature extractors, each with a feature dimension of D. After passing through the feature extractors, three feature tensors are obtained:

[0031] F cls F dwt F base =[B,T max D]

[0032] Furthermore, these three features are input into the shared-weight video frame encoder module, and after passing through a standard self-attention mechanism, the intermediate memory tensor is obtained: M cls M dwt M base During the decoding phase, each memory tensor undergoes a cross-attention operation with an independent, learnable query tensor [B, N, D], where N is a hyperparameter defining an upper limit on the number of defect time points predicted by the network.

[0033] The feature output obtained by fusing using the maximum value strategy is:

[0034] O cls Odwt O base =[B, N, D]

[0035] After the fusion operation, the final output is formed. This output is fed into two independent FFNs to predict the class and time point, as shown below:

[0036] P cls = [B, N, 2], P time = [B, N, 1]

[0037] Among them, P cls The softmax operation yields the probability representations of defective and normal categories, while P... time The position is then converted to a relative position with respect to the video duration using the sigmoid function. During prediction, the user sets a threshold, and the time points that are classified as "defects" and whose predicted probability exceeds this threshold are the final output results.

[0038] Step 5: Train the neural network model and save the optimal model weights.

[0039] Step 6: Use the saved optimal model weights to automatically locate the time points of pipeline inspection video defects. Attached Figure Description

[0040] Figure 1 This is the network structure diagram of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and examples. Obviously, the described embodiments are only a part of the embodiments of this invention, and not an exhaustive list of all embodiments. Furthermore, the embodiments and features in the examples described herein can be combined with each other without conflict.

[0042] The processing steps of this invention include: data preparation and processing, establishing training set, validation set, and test set, training an image classification model, performing feature extraction, training a neural network for video, and using the trained network model to automatically locate defect time points in the video.

[0043] Step 1: Extract images of pipe defects from the pipe defect detection report, compare them frame by frame with the video to obtain the time points of occurrence, construct a pipeline detection video defect time point localization dataset, and divide it into training set, validation set and test set in an 8:1:1 ratio.

[0044] Step 2: Extract the corresponding video frames based on the time point of the pipe defect occurrence, as the pipe defect image. Extract video frames evenly across other time intervals as normal pipe images, constructing a pipe image classification dataset. The dataset's partitioning corresponds one-to-one with the video dataset; that is, video frames in the training set portion of the video dataset will serve as the training set for the image classification dataset, and so on.

[0045] Step 3: Train a pipeline image classification model using the AutoGluon framework as a feature extractor to extract features from the pipeline detection video. This feature is denoted as the image classification feature F. cls The video data was decomposed using Discrete Wavelet Transform (DWT). Haar wavelet basis functions were used in the time dimension, and Bior 4.4 wavelet basis functions were used in both the width and height dimensions. For the decomposed high-frequency and low-frequency sub-bands, a ResNet151 network pre-trained on ImageNet was used for frame-by-frame feature extraction. These features are denoted as DWT features F. dwt For the original video, a ResNet151 network pre-trained on ImageNet is used directly for frame-by-frame feature extraction, and the features are denoted as the basic features F. base .

[0046] Step 4: Build a video time-stamping localization network based on the Transformer architecture. The network architecture is as follows:

[0047] like Figure 1 The diagram shows the overall architecture of the network. The network consists of a video frame encoder module and a defect time-point decoder module, accepting features extracted by three different CNN feature extractors as input to obtain F. cls F dwt and F base The outputs of these feature extractors are fed into the video frame encoder module for self-attention operations, resulting in three memory tensors. These are then self-attentioned with three independent learnable query tensors. Finally, the outputs of the three decoders are fused into a single tensor, which is then used to predict the class and time point via two FFNs.

[0048] Specifically, consider a batch of video data of size B. Within this batch, the longest video has a frame count of T. max The video resolution is H×W, and each frame has C channels. First, a batch of video data undergoes sequence padding preprocessing to ensure that their lengths are all uniformly T. max Therefore, the video tensor representation is obtained as follows:

[0049] V input =[B,T max [H, W, C]

[0050] The video sequence is then fed into three feature extractors, each with a feature dimension of D. After passing through the feature extractors, three feature tensors are obtained:

[0051] F cls F dwt F base =[B,T max D]

[0052] Furthermore, these three features are input into the shared-weight video frame encoder module, and after passing through a standard self-attention mechanism, the intermediate memory tensor is obtained: M cls M dwt M base During the decoding phase, each memory tensor undergoes a cross-attention operation with an independent, learnable query tensor [B, N, D], where N is a hyperparameter defining an upper limit on the number of defect time points predicted by the network.

[0053] The feature output obtained by fusing using the maximum value strategy is:

[0054] O cls O dwt O base =[B, N, D]

[0055] After the fusion operation, the final output is formed. This output is fed into two independent FFNs to predict the class and time point, as shown below:

[0056] P cls = [B, N, 2], P time = [B, N, 1]

[0057] Among them, P cls The softmax operation yields the probability representations of defective and normal categories, while P... time The position is then converted to a relative position with respect to the video duration using the sigmoid function. During prediction, the user sets a threshold, and the time points that are classified as "defects" and whose predicted probability exceeds this threshold are the final output results.

[0058] The video frame encoder module consists of two layers, designed based on PGL-SUM. It constructs global correlations between frames within the video by combining global and local self-attention mechanisms. This allows each frame to capture not only local information but also global contextual information. This structure not only models the relationships between frames but also, by minimizing frame importance loss during training, models the importance of each frame. This helps the network identify frames that may have pipeline defects.

[0059] To enable the network to understand the order of the video frame sequence, absolute position encoding is added to the input features. In this way, the video frame encoder can learn the order information of the video frame sequence.

[0060] The video frame encoder module with shared network weights processes three different features, which can fully leverage the complementarity of multiple features. A single feature may not be able to fully capture all pipeline defect information, but when multiple features work together, they can complement each other and improve the network's recall. In particular, when a certain feature is insufficient in capturing a certain defect, other features can play a role, thereby improving the overall network performance.

[0061] The advantage of this global modeling lies not only in capturing contextual information, but also in the fact that by combining multiple features, the video frame encoder module can better capture and learn defect features that cannot be obtained by a single feature. This further ensures that the network has higher sensitivity and accuracy to potential pipeline defects.

[0062] The network's defect time-point decoder module consists of two stacked standard Transformer Decoder layers. The cross-attention mechanism plays a crucial role, enabling the network to predict all possible defect occurrence times in parallel. This design overcomes the limitations of RNN-based or frame-by-frame analysis methods, allowing the network to predict all time points in parallel, thus improving prediction efficiency. Furthermore, it eliminates the need for cumbersome post-processing steps and multiple hyperparameters; only a single hyperparameter is required to obtain the prediction result.

[0063] Absolute position encoding is also used in the decoder module. These are directly added to the Query tensor, making it easier for the network to distinguish different prediction time points. The maximum sequence length of the Query tensor determines the upper limit of the number of prediction time points.

[0064] Cross-attention allows the decoder module to retrieve relevant contextual information from the output memory of the video frame encoder module. Specifically, for each feature (e.g., F... cls F dwt and F base Each of them is configured with an independent Query tensor Q. i These Query tensors Q i Memory M, which is the corresponding feature output of the video frame encoder. i Interact with it.

[0065] The cross-attention performed by the network in the defective time-point decoder can be mathematically expressed as:

[0066]

[0067] Among them, Qi It is a Query tensor assigned to a specific feature, while M i This refers to the memory generated by the video frame encoder for this feature. Through the softmax function, the decoder's attention can be focused on the current Q-value. i Similar memory portions allow for the extraction of crucial information about defect locations from the video frame encoder's memory.

[0068] This design approach ensures that the network can not only make full use of the rich information from multiple features, but also guarantees the accuracy and efficiency of predictions.

[0069] Network loss functions include: classification loss L cls Time point loss L ts Frame importance loss L imp And the auxiliary decoder loss L aux .

[0070] During training, the network uses the Hungarian algorithm to perform optimal bipartite graph matching between the network's output and the ground truth annotations, and calculates the category and temporal losses based on this matching result. Similar to the DETR network, the network architecture employs an auxiliary loss to optimize the output of the intermediate decoder layers, preventing training instability caused by an increase in the number of decoder layers. Furthermore, inspired by video summarization tasks, the network also uses frame importance loss for the first time, obtaining the intermediate tensor M... c M dwt and M base Then, they are fused, and then passed through an FFNs to output the frame importance score, denoted as S, with shape [B, T]. max ×3, 1]. By converting the real time-point level annotations into frame-level annotations, we obtain the frame-level pseudo-labels, denoted as E, with a shape of [B, T]. max ×3,1]. When generating pseudo-labels, the midpoint of the pseudo-label interval is the true label of the dataset, and the interval length is an adjustable hyperparameter. The frame importance loss, denoted as L, is obtained by calculating the L1 loss of S and E. imp It is used to optimize the output of the video frame encoder and further improve network performance.

[0071] Specifically:

[0072] (i) Classification loss L cls To quantify the difference between the network's predicted classes and the true classes, and to address the class imbalance problem in the dataset, a weighted hyperparameter is introduced for the normal class. This hyperparameter aims to adjust the weight of the normal class in the classification loss to handle its excessive number of samples in the dataset. The specific loss calculation formula is as follows:

[0073]

[0074] Where p is the network's predicted probability for each class, q is the one-hot encoding of the true class, and w i These are the weights for each category. Specifically, the weights for the normal class are determined by the hyperparameters mentioned above, while the weight for the defective class is 1.

[0075] (ii) Time point loss L ts This describes the error between the predicted time point and the actual time point. Because the time points output by the network are relative times, which differ from the actual length of different videos, directly calculating the difference in relative time will lead to uneven difficulty in prediction across videos of different lengths. To address this issue, the network's relative time output is first converted to absolute time, in seconds. Then, to balance the loss magnitude and ensure stable convergence of the network, a scaling hyperparameter α is introduced. The specific formula for calculating the time point loss is as follows:

[0076]

[0077] Here, x represents the difference between the absolute time predicted by the network and the actual labeled time. This method ensures that the time point loss is not affected by the length of the video, thus making the network prediction more robust and fair.

[0078] (iii) Frame importance loss L imp Designed specifically to optimize the output of a video frame encoder, it aims to ensure the network effectively captures the importance of each frame, providing additional supervision signals to the encoder module. Its calculation formula is:

[0079]

[0080] Where E is the output of the video frame encoder and S is the frame-level importance score.

[0081] (iiii) Auxiliary loss L aux The auxiliary loss is similar to that in the DETR network; it originates from the prediction errors of the intermediate decoder layers. Specifically, except for the last decoder layer, the outputs of all other layers are extracted and input into the FFNs for the predicted class and the FFNs for the predicted time points, respectively. For a network containing N... dec The decoder of the layer will get N dec -1 different prediction.

[0082] For each layer of prediction, a class loss and a temporal loss are calculated. The final auxiliary loss is obtained by summing the class loss and temporal loss from all these predictions. For the decoder at layer i, its class loss and temporal loss are expressed as follows: and The formula for calculating auxiliary loss is:

[0083]

[0084] This auxiliary loss mechanism ensures that the network produces meaningful predictions across its multiple decoder layers, thereby further enhancing the network's performance.

[0085] To balance these different loss components in the final loss function, the network introduces four hyperparameters: λ cls , λ ts , λ imp and λ aux Therefore, the total loss L total The calculation formula is:

[0086] L total =λ cls L cls +λ ts L ts +λ imp L imp +λ aux L aux

[0087] During training, by appropriately adjusting these hyperparameters, the importance of each loss can be weighed as needed, thereby optimizing network performance.

[0088] Step 5: Implement the network using the PyTorch framework. Use PyTorch's functional models as the overall structure, constructing a convolutional neural network structure and internal operations such as Transformer layers, max pooling layers, fully connected layers, softmax layers, and the max function, following the syntax of the network layers in the library. Train the neural network model. The specific hyperparameter settings are as follows: use the Adam optimizer with a learning rate of 0.0001, weight decay of 0.001, and initialize using the Xavier method, λ... cls =2,λ ts =0.15, λ imp =0.55, λ aux =0.5, and save the optimal model weights.

[0089] Step 6: Use the saved optimal model weights to automatically locate the time points of pipeline inspection video defects.

[0090] The automatic time point localization method for pipeline inspection videos provided in this invention, after acquiring the pipeline inspection video to be located, eliminates the need for manual video viewing by the user. It directly utilizes a pre-trained deep learning network to determine the possible time points where pipeline defects may occur. Furthermore, because the network uses a Transformer architecture, it can process the entire video in parallel, resulting in high efficiency. Moreover, it eliminates the need for post-processing steps, directly providing the results and significantly improving the network's ability to identify pipeline defects. This allows for relatively accurate automatic time point localization of defects in pipeline inspection videos, demonstrating significant application value.

Claims

1. A method for locating time points of defects in drainage pipe videos based on the Transformer architecture, comprising the following steps: (1) Extract images of pipeline defects from pipeline defect detection reports, compare them frame by frame with the video to obtain the time points of occurrence, construct a pipeline detection video defect time point localization dataset, and divide it into training set, validation set and test set in a ratio of 8:1:1; (2) Based on the time point when the pipeline defect occurs, extract the corresponding video frame as the pipeline defect image; extract video frames evenly in other time intervals as normal pipeline images, and construct a pipeline image classification dataset. The division of the dataset corresponds one-to-one with the video dataset. That is, the video frames in the training set part of the video dataset will be used as the training set of the image classification dataset, and so on. (3) Use the AutoGluon framework to train a pipeline image classification model as a feature extractor to extract features from the pipeline detection video. This feature is denoted as the image classification feature F. cls The video data was decomposed using Discrete Wavelet Transform (DWT). Haar wavelet basis functions were used in the time dimension, and Bior 4.4 wavelet basis functions were used in both the width and height dimensions. For the decomposed high-frequency and low-frequency sub-bands, a ResNet151 network pre-trained on ImageNet was used for frame-by-frame feature extraction. These features are denoted as DWT features F. dwt For the original video, the ResNet151 network pre-trained on ImageNet is used directly for frame-by-frame feature extraction, and the features are denoted as the basic features F. base ; (4) Construct a video time-point localization network based on the Transformer architecture; the network architecture is as follows: The network consists of a video frame encoder module and a defect time point decoder module. It takes features extracted by three different CNN feature extractors as input to obtain F. cls F dwt and F base The outputs of these feature extractors are fed into the video frame encoder module for self-attention operations, resulting in three memory tensors. Then, they are self-attentioned with three independent learnable query tensors. Finally, the outputs of the three decoders are fused into a single tensor, which is then used by two FFNs to predict the class and time point. Specifically, consider a batch of video data of size B; within this batch, the longest video has a frame count of T. max The video resolution is H×W, and each frame has C channels; first, a batch of video data undergoes sequence padding preprocessing to ensure that their lengths are all uniform to T. max Therefore, the video tensor representation is obtained as follows: Subsequently, the video sequence was fed into three feature extractors with a feature dimension of D; After passing through the feature extractor, three feature tensors are obtained: Furthermore, these three features are input into the shared-weight video frame encoder module, and after passing through a standard self-attention mechanism, an intermediate memory tensor is obtained: During the decoding phase, each memory tensor undergoes a cross-attention operation with an independent, learnable query tensor [B,N,D], where N is a hyperparameter that defines the upper limit of the number of defect time points predicted by the network. The feature output obtained by fusing using the maximum value strategy is: After the fusion operation, the final output is formed; This output is fed into two independent FFNs to predict the class and time point, as follows: Among them, P cls The softmax operation yields the probability representations of defective and normal categories, while P... time The position is then converted to a relative position with respect to the video duration using the sigmoid function. During prediction, the user sets a threshold, and the time points that are classified as "defect" and whose predicted probability exceeds this threshold are the final output results. The video frame encoder module has two layers, and the structure of the layers comes from the PGL-SUM network. The absolute position encoding is added to the input features before the input decoder. The video frame encoder module with shared network weights processes three different features, which can give full play to the complementarity of multiple features. The network's defective time-point decoder module consists of two stacked standard Transformer Decoder layers, which also use absolute position encoding. They are directly added to the Query tensor, which helps the network distinguish different prediction time points. The maximum sequence length of the Query tensor determines the upper limit of the number of prediction time points. Cross-attention allows the decoder module to retrieve relevant contextual information from the output memory of the video frame encoder module; the decoder module provides image classification features F. cls DWT features F dwt and basic features F base A separate Query tensor Q was configured. i These Query tensors Q i Memory M, which is the corresponding feature output of the video frame encoder. i Interact; The cross-attention performed by the network in the defective time-point decoder can be mathematically expressed as: Among them, Q i It is a Query tensor assigned to a specific feature, while M i This refers to the memory generated by the video frame encoder for this feature; through the softmax function, the decoder's attention can be focused on the current Q-value. i Similar memory portions allow for the extraction of crucial information about defect locations from the video frame encoder's memory; Network loss functions include: classification loss L cls Time point loss L ts Frame importance loss L imp And the auxiliary decoder loss L aux ; During training, the network uses the Hungarian algorithm to perform optimal bipartite graph matching between the network output and the ground truth annotations, and calculates the category and time point losses based on this matching result. Similar to the DETR network, the network architecture employs auxiliary losses to optimize the output of intermediate decoders, preventing training instability caused by an increase in the number of decoder layers. Furthermore, inspired by video summarization tasks, the network also uses frame importance loss for the first time, after obtaining the intermediate tensor M. c, M dwt and M base Then, they are fused, and then passed through an FFNs to output the frame importance score, denoted as S, with the shape of... By converting real time-point level annotations into frame-level annotations, we obtain frame-level pseudo-labels, denoted as E, with the shape of... When generating pseudo-labels, the midpoint of the pseudo-label interval is the true label of the dataset, and the interval length is an adjustable hyperparameter; the frame importance loss, denoted as L, is obtained by calculating the L1 loss of S and E. cls These are used to optimize the output of the video frame encoder and further improve network performance; specifically, the four loss functions are as follows: (i) Classification loss L cls To quantify the difference between the network's predicted classes and the true classes, and to address the class imbalance problem in the dataset, a weighted hyperparameter is introduced for the normal class. This hyperparameter aims to adjust the weight of the normal class in the classification loss to handle its excessive number of samples in the dataset. The specific loss calculation formula is as follows: Where p is the network's predicted probability for each class, q is the one-hot encoding of the true class, and w i These are the weights for each category; for the normal class, the weight is determined by the hyperparameters mentioned above, while for the defective class, the weight is 1. (ii) Time point loss L ts This describes the error between the predicted time point and the actual time point. Because the time points output by the network are relative times, which differ from the actual length of different videos, directly calculating the difference in relative time will lead to uneven difficulty in prediction across videos of different lengths. To address this issue, the network's relative time output is first converted to absolute time, in seconds. Then, to balance the loss magnitude and ensure stable convergence of the network, a scaling hyperparameter α is introduced. The specific formula for calculating the time point loss is as follows: Where x represents the difference between the absolute time predicted by the network and the actual labeled time point; (iii) Frame importance loss L imp The calculation formula is as follows: Where E is the output of the video frame encoder, and S is the frame-level importance score; (iiii) Auxiliary loss L aux The auxiliary loss originates from the prediction error of the intermediate layer decoder; specifically, except for the last layer decoder, the outputs of all other layers are extracted and input into the FFNs for the prediction class and the FFNs for the prediction time points, respectively; for a class containing N... dec The decoder of the layer will get N dec -1 different prediction; For each layer of prediction, a class loss and a temporal loss are calculated; the final auxiliary loss is obtained by summing the class loss and temporal loss from all these predictions; for the decoder at layer i, its class loss and temporal loss are expressed as follows: and The formula for calculating auxiliary loss is: The network introduces four hyperparameters: Therefore, the total loss L total The calculation formula is: (5) Implement the network using the PyTorch framework. Use the function model in PyTorch as the overall structure. Construct the Transformer layer, max pooling layer, fully connected layer, softmax layer, max function, and other convolutional neural network structures and internal operations according to the network layer writing method in the library. Train the neural network model. The specific settings of the hyperparameters are as follows: use the Adam optimizer, the learning rate is 0.0001, the weight decay is 0.001, and the Xavier method is used for initialization. And save the optimal model weights; (6) The saved optimal model weights are used to automatically locate the time point of pipeline inspection video defects.

Images