A defect intelligent identification deduplication method for sewer pipeline detection video

Abstract

The present application relates to a kind of defects intelligent identification deduplication method for sewer detection video, comprising: for the pipe video of town sewer, obtain each frame image and frame serial number in pipe video, each frame image is labeled with the attribute information of collection time;Using pre-trained pipe defect identification model, each frame image is analyzed, and the identification result of each frame image is obtained;Based on the distance information in the attribute information of all frame images, the identification result of pipe defect, extract the effective frame image of each pipe defect and the motion trajectory of the pipe defect;The motion trajectory of the pipe defect corresponding to the effective frame image of each pipe defect, and the identification result corresponding to each effective frame image are comprehensively analyzed, and key frame image set is screened out.The above method can extract the key frame data that can represent the characteristics of each defect and the best recognition effect from a large number of detection video frames, to facilitate subsequent tracking analysis on defects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pipeline detection, in particular to a defect intelligent identification and deduplication method for drainage pipeline detection video. BACKGROUND

[0002] The detection and evaluation of existing drainage pipelines are mainly based on manual interpretation, which has the main shortcomings of high interpretation error, strong subjectivity, time and labor consumption, high cost and low efficiency.

[0003] The current YOLO-based computer vision deep learning artificial intelligence model can quickly and effectively identify pipeline defects in underground drainage pipeline detection video, which can effectively reduce the types and quantity of defects in the artificial review of drainage pipeline image data, but still requires manual recording of the types, quantity and distance position of the defects, including the following steps: 1) collecting image data of all pipeline defect types; 2) screening pipeline defect images; 3) image labeling all pipeline types; 4) forming a pipeline defect sample database; 5) building an image segmentation AI intelligent detection model framework and training; 6) building a drainage pipeline defect automatic detection, recognition, tracking and statistical model; 7) outputting interpretation video and statistical results. Even if the current AI automatic interpretation is used, it still needs to manually continue to count the defect types, quantity and longitudinal position of the drainage pipeline, and select the key frame image with the best recognition effect from the defects.

[0004] Therefore, how to further reduce manual work and realize automatic screening of defect key frame images has become a technical problem to be solved at present. SUMMARY

[0005] (I) Technical problem to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present application provides a defect intelligent identification and deduplication method for drainage pipeline detection video, which accurately selects the most representative defect key frame and improves the efficiency and accuracy of defect target identification and evaluation.

[0007] (II) Technical scheme

[0008] In order to achieve the above-mentioned purpose, the main technical scheme adopted by the present application includes:

[0009] In the first aspect, the present application provides a defect intelligent identification and deduplication method for drainage pipeline detection video, which includes:

[0010] S100, for the pipeline video of the urban drainage pipeline collected by the pipeline robot, acquiring each frame image and frame number in the pipeline video, and labeling the attribute information of the pipeline robot collection in each frame image;

[0011] S200. Analyze each frame of image using a pre-trained pipeline defect recognition model to obtain the recognition result of each frame of image. The recognition result includes: the category of pipeline defect in the current frame, the confidence level of each pipeline defect, the pipeline defect mask area, and the tracking number information used to track pipeline defects in the current frame.

[0012] S300: Based on the distance information in the attribute information of all frame images and the identification result of the pipeline defect, extract the valid frame image to which each pipeline defect belongs and the motion trajectory of the pipeline defect.

[0013] S400. Perform a comprehensive analysis on the motion trajectory of the pipeline defect corresponding to each valid frame image of the pipeline defect, as well as the category, confidence level, and mask area of ​​the pipeline defect in the recognition result corresponding to each valid frame image, and select a set of key frame images of defects for pipeline defect recognition and tracking analysis.

[0014] The intelligent defect identification and deduplication method for drainage pipeline inspection videos of this invention combines target tracking technology, identification distance information, and model confidence analysis. It sets an effective distance range and uses a sliding window mechanism to select the most representative and accurate video frame for each defect target in its motion trajectory. Compared with existing technologies, it can automatically, accurately, and efficiently extract key frame data (the best representative frame number for each defect and the distance value corresponding to that frame) from a large number of inspection video frames, which can represent the features of each defect in automatic detection and the best identification effect. This facilitates subsequent defect analysis and evaluation.

[0015] Optionally, before step S200, the method further includes:

[0016] The pipeline defect identification model was trained and validated using a pipeline defect training dataset to obtain the trained pipeline defect identification model.

[0017] Each image in the pipeline defect training dataset is a pre-labeled image with defect category and defect mask region; and for images with a single defect or multiple defect categories, image enhancement algorithms are used to process the images to obtain training data that meets the preset image quality.

[0018] Optionally, image enhancement algorithms are used to process training data to meet preset image quality, including:

[0019] DnCNN and SRCNN are used to process images with a single defect or multiple defect categories;

[0020] DnCNN consists of a first input layer, intermediate layers, and a first output layer.

[0021] The first input layer extracts features from the input image using the following formula.

[0022]

[0023] Where X is the input image with dimension 1.

[0024] Conv2d is a 2D convolution operation. k1=5 indicates that its convolution kernel is 5×5, d=2 indicates that its stride is 2, g=3 indicates that the number of groups in its grouped convolution is 3, padding=same indicates that its padding method is the same, and (3→32) indicates that its convolution operation converts 3 channels into 32 channels.

[0025] tanh is the hyperbolic tangent function, and F1(X) is the output of the first input layer.

[0026] The intermediate layers consist of 10 alternating convolutional layers.

[0027]

[0028] in, For the output of each intermediate layer, X m For the input of each intermediate layer,

[0029] This represents a 2D convolution operation with a 3×3 kernel k2 and a padding value p1 of 1, converting 32 channels into 64 channels.

[0030] This represents a 2D convolution operation with a 3×3 kernel k3 and a padding value p2 of 1, converting 64 channels into 32 channels.

[0031] BatchNorm performs batch normalization, and LeakyReLU is the activation function.

[0032] F1(X) is the output of the first input layer, F m (X) represents the output of the intermediate layer;

[0033] The first output layer is,

[0034]

[0035] Among them, F m (X) represents the output of the intermediate layer, f DnCNN (X) represents the output of the first output layer.

[0036] This represents a two-dimensional convolution operation with a kernel k4 of 5×5 and a padding value p3 of 2, converting 32 channels into 3 channels.

[0037] SRCNN takes the output of DnCNN as input, as shown in the following formula.

[0038]

[0039] Among them, f SRCNN (Y) represents the output of SRCNN, and Y represents the input of SRCNN.

[0040] This represents a 2D convolution operation with a 9×9 kernel (k5) and 4 padding (p4), converting 3 channels into 64 channels.

[0041] This represents a 2D convolution operation with a 1×1 kernel k6 and 0 padding p5, converting 64 channels into 64 channels.

[0042] This represents a 2D convolution operation with a 5×5 kernel (k7) and a padding value (p6) of 2, converting 64 channels into 3 channels.

[0043] ReLU represents the linear rectified function.

[0044] Optionally, the attribute information of each frame image includes: the time labeled in each frame image, robot identifier, distance information, and pipe identifier;

[0045] The S300 includes:

[0046] S301. Automatically identify attribute information in each frame of image;

[0047] S302. Based on the recognition results of each frame image and the distance information in the attribute information, extract the valid frame image to which each pipeline defect belongs and the motion trajectory of the pipeline defect.

[0048] Optionally, the 301 includes:

[0049] OCR recognition technology is used to obtain attribute information from each frame of the pipeline video. Based on regular expressions, the attribute information is parsed to obtain distance information.

[0050] Optionally, the 302 includes:

[0051] S302-1. For each defect, obtain the image set T of the defect based on the pipeline defect category and tracking number information in the identification results. k ,

[0052] T k ={(F i ID i D i C i ,Pi )|i=1,2,...,N}

[0053] Among them, F i ID is the frame number of the i-th frame of the image. i D represents the defect category and tracking number information corresponding to the i-th frame image. i Let D1 be the recognition distance in the distance information of the i-th frame image, and let D2 be the distance in ascending order, i.e., D1 ≤ D2 ≤ … ≤ D2. N C i C is the confidence score of the model recognition corresponding to the i-th frame image, 0 ≤ C i ≤1, P i For the i-th frame image,

[0054] S302-2, Based on the image set T of the defect occurrence k The distance information is used to obtain the movement trajectory of the defect.

[0055] D total =D N -D1

[0056] Among them, D total D represents the total distance the defect travels. N D1 is the recognition distance in the distance information of the image where the defect finally appears, and D2 is the recognition distance in the distance information of the image where the defect initially appears.

[0057] S302-3. Extract the valid frame images from the defective image set according to the following formula.

[0058]

[0059] in, For the set of valid frame images,

[0060] D min For the effective minimum distance, D min =D1 + 0.7 × D total ,

[0061] D max For the effective maximum distance, D max ×D1+0.9×D total .

[0062] Optionally, S400 includes:

[0063] S401. In the set of valid frame images, form (M-W+1) smooth windows with a fixed window length W, where M is the number of valid frame images in the set. Calculate the average confidence level of each sliding window using the following formula.

[0064]

[0065] Among them, MC j Let C be the average confidence level of the j-th sliding window. i The confidence score of the model corresponding to the i-th frame image;

[0066] S402. From all smoothed windows, select the window with the highest average confidence level as the optimal window.

[0067]

[0068]

[0069] Where, j * The j value represents the optimal window. This represents the parameter value that maximizes the following expression within the range 1 ≤ j ≤ M - W + 1. Represents the set of images within the optimal window;

[0070] S403. From the image set of the optimal window, select the image with the highest confidence level as the keyframe image of the defect.

[0071]

[0072] Among them, F best The frame number representing the keyframe image. Indicates in The parameter value within the range that makes the following expression maximize, C i The confidence score of the model corresponding to the i-th frame image;

[0073] S404. Construct a keyframe image set based on the keyframe images of each defect.

[0074] Optionally, the pipeline defect identification model is a pipeline defect identification model based on the YOLO12 algorithm, including a second input layer, a feature extraction layer, a feature fusion layer, an upsampling layer, and a second output layer;

[0075] The second input layer is used to receive the input drainage pipe image.

[0076] The feature layer includes convolutional layers, C3k2 layers, and A2C2f layers, which are used for preliminary feature extraction and attention processing.

[0077] The fusion layer is used to stitch together the feature maps output from different levels of the feature layer.

[0078] The upsampling layer is used to increase the resolution of the feature map output by the fusion layer.

[0079] The second output layer is used to output the segmentation and detection results.

[0080] Secondly, embodiments of the present invention provide a defect intelligent identification and deduplication device for drainage pipeline inspection videos, comprising:

[0081] The image acquisition module is used to acquire each frame of the video of the urban drainage pipeline collected by the pipeline robot and the frame number. Each frame of the video is marked with the attribute information of the pipeline robot at the time of acquisition.

[0082] The analysis and recognition module is used to analyze each frame of image using a pre-trained pipeline defect recognition model and obtain the recognition result of each frame of image. The recognition result includes: the category of pipeline defect in the current frame, the confidence level of each pipeline defect, the pipeline defect mask area, and the tracking number information used to track pipeline defects in the current frame.

[0083] The effective frame extraction module is used to extract the effective frame image to which each pipeline defect belongs and the motion trajectory of the pipeline defect based on the distance information in the attribute information of all frame images and the identification result of the pipeline defect.

[0084] The filtering module is used to comprehensively analyze the motion trajectory of the pipeline defect corresponding to the valid frame image of each pipeline defect, as well as the category, confidence level, and mask area of ​​the pipeline defect in the recognition result corresponding to each valid frame image, and to filter out the set of defect key frame images for pipeline defect recognition and tracking analysis.

[0085] Thirdly, embodiments of the present invention provide an electronic device, including: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program in the memory, specifically performing the steps of the intelligent defect identification and deduplication method for drainage pipe inspection videos as described in any of the first aspects above.

[0086] (III) Beneficial Effects

[0087] The beneficial effects of this invention are as follows: The intelligent defect identification and deduplication method for drainage pipeline inspection videos of this invention combines target tracking technology, identification distance information, and model confidence analysis. By setting an effective distance range and using a sliding window mechanism to select the most representative and accurate video frame in the motion trajectory of each defect target, compared with the prior art, it can automatically, accurately, and efficiently extract key frame data (the best representative frame number of each defect and the distance value corresponding to that frame) from a large number of inspection video frames, which can represent the features of each defect in automatic detection and the best identification effect, thus facilitating subsequent defect analysis and evaluation. Attached Figure Description

[0088] Figure 1This is a flowchart illustrating a method for intelligent defect identification and deduplication in drainage pipeline inspection videos according to an embodiment of the present invention.

[0089] Figure 2 This is a schematic diagram of the structure of a pipeline defect identification model;

[0090] Figure 3 This is a flowchart illustrating a method for intelligent defect identification and deduplication in drainage pipeline inspection videos, provided in another embodiment of the present invention.

[0091] Figure 4 The defective image used for training the model before it is processed using an image enhancement algorithm;

[0092] Figure 5 The image is a defective image used to train the model after being processed using an image enhancement algorithm.

[0093] Figure 6 This is a schematic diagram of one of the image frames carrying defects stored in the first stage according to another embodiment of the present invention;

[0094] Figure 7 This is a schematic diagram of one of the image frames with defect annotations stored in the first stage according to another embodiment of the present invention;

[0095] Figure 8 This is a schematic diagram of the interpretation video stored in the first stage according to another embodiment of the present invention. Detailed Implementation

[0096] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0097] This embodiment relates to a video target detection and tracking method based on computer vision and artificial intelligence technologies, and particularly to a technical method for instance segmentation of defective targets, continuous multi-target tracking, and automatic identification of numerical information within images during the inspection of urban drainage pipelines.

[0098] This embodiment employs a deep learning model based on the YOLO12 algorithm, suitable for the segmentation, detection, identification, tracking, and statistical modeling of defects in drainage pipelines. Based on image data of pipeline defects, the automatic detection and identification model is trained on the basis of the YOLO12 algorithm, improving the accuracy and stability of detection and identification.

[0099] In this embodiment, the model automatically interprets the pipeline video and automatically records the number and longitudinal location of defects in the pipeline.

[0100] In this embodiment, the pipeline videos used for screening and the pipeline defect recognition model training dataset used are all derived from pipeline robots. For example, these could be drainage pipeline image data collected by pipeline inspection robots, crawling robots equipped with cameras, or drones. These devices enter from rainwater wells, sewage wells, or open pipe ports, moving longitudinally along the pipeline interior while simultaneously capturing video or image data of the pipeline's interior. Upon reaching the other end of the pipeline, the device returns to the ground carrying the recorded image data. After acquisition, the capturing device is connected to a host computer (e.g., a computer) via a data cable to import the drainage pipeline image data (generally in MP4, AVI, or other video formats) and store it on the host computer.

[0101] Example 1

[0102] This embodiment provides an intelligent defect identification and deduplication method for drainage pipe inspection videos. It identifies pipe defects from the video and selects keyframe images with the best identification effect for each defect. This method can be implemented on any host computer device. (See [link to relevant documentation]). Figure 1 The method in this embodiment includes:

[0103] S100. For the pipeline video of the urban drainage pipeline collected by the pipeline robot, obtain each frame image and frame number in the pipeline video, and mark the attribute information of the pipeline robot at the time of collection in each frame image.

[0104] Specifically, the attribute information for each frame of the image includes: the time, robot identifier, distance information, and pipe identifier labeled in each frame of the image. This attribute information is automatically added during the pipeline robot's data acquisition process.

[0105] In step S100, the host computer automatically reads the video frames, obtains their width, height and frame rate information, and creates corresponding video storage paths and image output directories based on this information to store the video and image data generated during the detection and recognition process, as well as the corresponding text recognition and tracking information.

[0106] S200. Analyze each frame of the image using a pre-trained pipeline defect recognition model to obtain the recognition result for each frame. The recognition result includes: the category of the pipeline defect in the current frame, the confidence level of each pipeline defect, the pipeline defect mask region, and tracking number information used to track pipeline defects in the current frame. The tracking number information is unique for each defect.

[0107] Specifically, see Figure 2 The pipeline defect identification model can be a pipeline defect identification model obtained based on the YOLO12 algorithm, including a second input layer, a feature extraction layer, a feature fusion layer, an upsampling layer, and a second output layer;

[0108] The second input layer is used to receive the input drainage pipe image.

[0109] The feature extraction layer includes a convolutional layer, a C3k2 layer, and an A2C2f layer, which are used for preliminary feature extraction and attention processing.

[0110] The feature fusion layer is used to stitch together the feature maps output from different levels of the feature layer.

[0111] The upsampling layer is used to increase the resolution of the feature map output by the fusion layer.

[0112] The second output layer is used to output the segmentation and detection results.

[0113] In subsequent steps, this embodiment extracts the best effective frame and confidence level analysis of the detected pipeline defects based on the above detection and identification results.

[0114] In step S200, a pre-trained pipeline defect recognition model is loaded to analyze the input video data frame by frame, thereby achieving automatic instance segmentation and multi-target tracking of drainage pipeline defect targets. More specifically, after the host computer reads the video data frame by frame, it performs target instance segmentation detection and continuous tracking through the YOLO12 track interface.

[0115] In practical applications, the results processed by the model are stored separately as files. For example, the file names can be pre-defined, such as storage... Figure 6 The image shown is in the folder containing the original defect images. Figure 7 The images shown are stored in a folder with labeled defect images. Of course, the complete video also needs to be stored, such as... Figure 8 The video shown can be stored separately after adding defect annotation information. At this time, a separate defect tracking information record table is also set for the separately stored video, which includes: frame number, tracking number (corresponding to the tracking number in the recognition result), defect type (corresponding to the category of defect in the recognition result), confidence level, and distance (corresponding to the recognition distance in the image frame information).

[0116] This information record table can be obtained by summarizing the recognition results of each of the aforementioned frames.

[0117] S300. Based on the distance information in the attribute information of all frame images and the identification result of the pipeline defect, extract the valid frame image to which each pipeline defect belongs and the motion trajectory of the pipeline defect.

[0118] Specifically, S300 includes:

[0119] S301. Automatically identify the attribute information in each frame of the image. More specifically, use OCR recognition technology to obtain the attribute information in each frame of the pipeline video, and parse the attribute information based on regular expressions to obtain distance information.

[0120] To enable automatic recognition of specific values ​​(such as distance information) during pipeline inspection, this embodiment integrates PaddleOCR text recognition technology to perform optical character recognition (OCR) on a specified area of ​​the current video frame image (such as the bottom 10% of the image) to extract the distance value information presented by the image attribute information of each frame in the pipeline video in real time.

[0121] The text data recognized by OCR is parsed again using regular expressions to extract numerical information containing clear distance unit identifiers (such as meters, centimeters, kilometers, m, cm, km, etc.). After successful extraction, the host computer judges the validity and stability of the values: that is, by comparing the distance value recognized in the current frame with the distance value recorded in the previous frame, a corresponding change threshold is set (such as a change within 50%) to ensure data continuity and accuracy. Only when this threshold condition is met is the current distance value updated and stored, and the frame image and corresponding tracking data are stored. Frame data that fails the threshold judgment or fails recognition is automatically ignored to avoid the accumulation of error information and interference.

[0122] S302. Based on the recognition results of each frame image and the distance information in the attribute information, extract the valid frame image to which each pipeline defect belongs and the motion trajectory of the pipeline defect.

[0123] More specifically, 302 includes:

[0124] S302-1. For each defect, obtain the image set T of the defect based on the pipeline defect category and tracking number information in the identification results. k ,

[0125] T k ={(F i ID i D i C i P i )|i=1,2,...,N}

[0126] Among them, F i ID is the frame number of the i-th frame of the image. i For the defect category and tracking number information corresponding to the i-th frame image, d i Let D1 be the recognition distance in the distance information of the i-th frame image, and let D2 be the distance in ascending order, i.e., D1 ≤ D2 ≤ … ≤ D2. N C iC is the confidence score of the model recognition corresponding to the i-th frame image, 0 ≤ C i ≤1, P i This is the u-th frame image. The set of images where the defect occurs is T. k It contains N frames of images.

[0127] S302-2, Based on the image set T of the defect occurrence k The distance information is used to obtain the movement trajectory of the defect.

[0128] D total =D N -D1

[0129] Among them, D total D represents the total distance the defect travels. N D1 represents the recognition distance in the distance information of the image where the defect finally appears, while D2 represents the recognition distance in the distance information of the image where the defect initially appears.

[0130] S302-3. Extract the valid frame images from the defective image set according to the following formula.

[0131]

[0132] in, For the set of valid frame images,

[0133] D min For the effective minimum distance, D min =D1 + 0.7 × D total ,

[0134] D max For the effective maximum distance, D max =D1 + 0.9 × D total .

[0135] S400. Perform a comprehensive analysis on the motion trajectory of the pipeline defect corresponding to each effective frame image of the pipeline defect, as well as the category, confidence level, and mask area of ​​the pipeline defect in the recognition result corresponding to each effective frame image, and select the key frame image set of the defect for pipeline defect recognition and post-processing.

[0136] Specifically, S400 includes:

[0137] S401. In the set of valid frame images, form (M-W+1) smooth windows with a fixed window length W, where M is the number of valid frame images in the set. Calculate the average confidence level of each sliding window using the following formula.

[0138]

[0139] Among them, MCj Let C be the average confidence level of the j-th sliding window. i The confidence level of the model recognition corresponding to the i-th frame image.

[0140] S402. From all smoothed windows, select the window with the highest average confidence level as the optimal window.

[0141]

[0142] Where, j * The j value represents the optimal window. This represents the parameter value that maximizes the following expression within the range 1 ≤ j ≤ M - W + 1. This represents the set of images within the optimal window.

[0143] S403. From the image set of the optimal window, select the image with the highest confidence level as the keyframe image of the defect.

[0144]

[0145] Among them, F best The frame number representing the keyframe image. Indicates in The parameter value within the range that makes the following expression maximize, C i The confidence level of the model recognition corresponding to the i-th frame image.

[0146] The final selected keyframe image can be fully represented as follows:

[0147] F best ID best D best C best P best

[0148] Among them, F best The frame number, ID, represents the keyframe image. best This indicates the defect category and tracking number (one or more defects exist) corresponding to the keyframe image, D best C represents the recognition distance in the distance information of the keyframe image. best P represents the confidence level of the keyframe image (this value is the highest within the window). best This represents a keyframe image.

[0149] S404. Construct a keyframe image set based on the keyframe images of each defect.

[0150] Steps S300 to S400 are used to automatically select key video frames based on the target tracking results. Specifically, this involves a comprehensive analysis of the motion trajectory, confidence level, and distance information of target defects in urban drainage pipeline inspection videos, thereby accurately and automatically selecting the most representative image frames to improve the efficiency and accuracy of defect target identification and assessment.

[0151] It should be noted that a series of file paths can be defined first, including the original tracking information text file (track_info.txt), the folder containing the detected frame images (detected_frames), and the output folder after selecting keyframes (selected_frames), and ensure that the corresponding folder structure has been established and has the necessary access permissions.

[0152] In this specific implementation, the previously saved target tracking log file is first read. Regular expressions are used to parse each line of log information, extracting the frame number, tracking number, target class, model confidence, and real-time distance information corresponding to each frame. The parsed data is then categorized and stored according to the unique identifier of the tracked target—the tracking number and class—to facilitate unified processing of all relevant frame information for each tracked target.

[0153] Subsequently, for each tracked defective object (uniquely identified by tracking number and category), they are sorted according to frame number to determine the initial and final identification distances of the target's movement within the video segment, thereby calculating the target's total movement distance. Based on this movement distance, an effective range for keyframe selection is further defined, specifically set between 70% and 90% of the target's total movement distance. Video frames within this range typically represent stable states during the target's movement, suitable for subsequent evaluation and analysis tasks.

[0154] Within the selected effective distance range, a sliding window mechanism (e.g., a window length of 5 frames) is used to traverse all effective frames, calculate the average confidence score of the model prediction within each window, and record the window position with the highest average confidence score during the sliding process. Within this optimal window, the image frame with the highest single-frame confidence score is further selected as the representative frame to ensure that the selected image is in the optimal state in terms of target recognition accuracy.

[0155] After completing the keyframe selection process described above, the final selected key video frames are copied and saved to a predefined output folder. Simultaneously, a text log file (selected_frames.txt) containing complete tracking and recognition information is generated. Before copying the frame files, an additional file existence verification step can be added to avoid anomalies caused by missing files. Detailed debug output information is also added to each operation step to monitor the status and results of the entire processing in real time.

[0156] This embodiment combines target tracking technology, distance information recognition, and model confidence analysis to set an effective distance range and use a sliding window mechanism to select the most representative and accurate video frame in the motion trajectory of each defect target, thereby significantly improving the efficiency, accuracy, and automation level of defect identification and subsequent analysis.

[0157] The pipeline defect identification and post-processing in this embodiment can be the identification and tracking of all key frames, such as post-processing through an automatic identification and tracking model. The method in this embodiment greatly improves the intelligent detection effect of drainage pipeline defects and can track every defect. Even if N defects of a certain type are detected in a segment of image data, the type of defect, tracking number, frame number of the key frame, distance information corresponding to the key frame, and confidence level can be directly given (generally, one second contains 24-30 frames, and detection and identification may result in duplicates, which can be avoided by tracking).

[0158] Example 2

[0159] Based on Example 1, this embodiment adds a step of training and validating the pipeline defect identification model before S200.

[0160] See Figure 3 The method further includes:

[0161] The pipeline defect identification model was trained and validated using a pipeline defect training dataset to obtain the trained pipeline defect identification model.

[0162] In other words, before performing video frame analysis in the pipeline video, a pipeline defect recognition model is first trained and constructed. That is, the pipeline defect recognition model is trained and validated using a pre-given pipeline defect training dataset to obtain the trained pipeline defect recognition model.

[0163] Next, the trained pipeline defect training model is used to analyze each video frame in the pipeline video to remove duplicates, as described in the steps of Embodiment 1 above. That is, the identification results after the model analyzes each frame of the image in S200 can be such as all image frames carrying defects, all image frames with defect annotations, and all defect tracking information record tables (the record tables record the frame number, tracking sequence, defect type, confidence level, distance, etc.). After processing in S300, the key frame information of each defect is finally obtained (the key frame information may include: the original defect image to which the key frame belongs, the defect image with annotations, the defect information record table, etc.), so that it can be used in other inspection and evaluation reports, or other tracking analysis and other processing.

[0164] In the pipeline defect training dataset mentioned above, each image is pre-labeled with defect category and defect mask region; and for images with a single defect or multiple defect categories, image enhancement algorithms are used to process the images to obtain training data that meets the preset image quality.

[0165] The pipeline defect training dataset is obtained by extracting frames, filtering, and labeling pipeline videos collected by the pipeline robot for training. To better illustrate the process of obtaining the pipeline defect training dataset, steps one through three are added to the following description.

[0166] Step 1: Extract frames from the pipeline videos collected by the pipeline robot for training. Specifically, a command prompt (such as PowerShell) can be used to call a multimedia processing program (such as FFmpeg) to perform batch frame extraction on the pipeline videos used for training. Specify the source folder for storing video files and the output folder for the extracted images, set the output image format (such as png) and the frame extraction frequency (fps), recursively search all video files (.mp4 and .avi formats) in the source folder, create a subfolder named after the video file name for each video to store the corresponding extracted images, and use the FFmpeg tool to batch extract video frames at the specified frequency (fps) and save them to the corresponding subfolder.

[0167] When skipping frames, an adaptive frame skipping method can also be used. The formula for the adaptive frame skipping rate is as follows:

[0168] adaptiveFps=min(5,max(1,round(2·wres·wquality·wpipe)))

[0169] When wres = 1920×1080 / actual image resolution

[0170] wquality = Average BRISQUE / 50

[0171] wpipe = 1 / (pipe cross-section coverage + 0.1)

[0172] Range constraint: adaptiveFPS = [1, median, 5]

[0173] For the adaptive frame skipping mentioned above, an example is as follows:

[0174] If wres=0.8, wquality=1.2, wpipe=1.5: adaptiveFps=min(5,max(1,round(2×0.8×1.2×1.5)))=min(5,max(1,round(2.88)))=min(5,3)=3.

[0175] Step 2: From the pipeline image data obtained by frame extraction, images with defects are selected. These images may contain a single defect or multiple defect types. First, check whether the type and extent of the defects in the images are clearly visible. Images that meet the criteria directly proceed to the next stage of data annotation and model training.

[0176] For images with unclear defects or substandard image quality, image preprocessing techniques are used to improve image quality, ensuring that the defect type and boundary range are more clearly defined. More specifically, image enhancement algorithms (such as computer vision-based image enhancement models) are used to improve image sharpness and detail, ensuring accurate identification and analysis of defect information. For example, Deep Neural Network for Image Denoising (DnCNN) and Super Resolution Convolutional Neural Network (SRCNN) can be used to improve image quality.

[0177] Image enhancement algorithms are used to process training data that meets the preset image quality, including: using DnCNN and SRCNN to process images with a single defect or multiple defect categories.

[0178] DnCNN consists of a first input layer, intermediate layers, and a first output layer. The first input layer extracts features from the input image using the following formula.

[0179]

[0180] Where X is the input image with dimension 1.

[0181] Conv2d is a 2D convolution operation. k1=5 indicates that its convolution kernel is 5×5, d=2 indicates that its stride is 2, g=3 indicates that the number of groups in its grouped convolution is 3, padding=same indicates that its padding method is the same, and (3→32) indicates that its convolution operation converts 3 channels into 32 channels.

[0182] tanh is the hyperbolic tangent function, and F1(X) is the output of the first input layer.

[0183] The intermediate layers consist of 10 alternating convolutional layers.

[0184]

[0185] in, For the output of each intermediate layer, X m For the input of each intermediate layer,

[0186] This represents a 2D convolution operation with a 3×3 kernel k2 and a padding value p1 of 1, converting 32 channels into 64 channels.

[0187] This represents a 2D convolution operation with a 3×3 kernel k3 and a padding value p2 of 1, converting 64 channels into 32 channels.

[0188] BatchNorm performs batch normalization, and LeakyReLU is the activation function.

[0189] F1(X) is the output of the first input layer, F m (X) represents the output of the intermediate layer;

[0190] The first output layer is,

[0191]

[0192] Among them, F m (X) represents the output of the intermediate layer, F DnCNN (X) represents the output of the first output layer.

[0193] This represents a two-dimensional convolution operation with a kernel k4 of 5×5 and a padding value p3 of 2, converting 32 channels into 3 channels.

[0194] The overall definition of DnCNN is: Y = Xf DnCNN (X).

[0195] SRCNN takes the output of DnCNN as input, as shown in the following formula.

[0196]

[0197] Among them, f SRCNN (Y) represents the output of SRCNN, and Y represents the input of SRCNN.

[0198] This represents a 2D convolution operation with a 9×9 kernel (k5) and 4 padding (p4), converting 3 channels into 64 channels.

[0199] This represents a 2D convolution operation with a 1×1 kernel k6 and 0 padding p5, converting 64 channels into 64 channels.

[0200] This represents a 2D convolution operation with a 5×5 kernel (k7) and a padding value (p6) of 2, converting 64 channels into 3 channels.

[0201] ReLU represents the linear rectified function.

[0202] The overall definition of an image enhancement algorithm is:

[0203] Z = F SRCNN (XF DnCNN (X))

[0204] Where Z represents the output image obtained after the image enhancement algorithm. This formula expresses the end-to-end structure of the image enhancement algorithm: first, noise reduction (or removal of redundant information) is performed through DnCNN, and then detail enhancement is performed through the SRCNN module, thereby outputting a high-quality image Z.

[0205] See Figures 4-5 After adopting the image enhancement algorithm, the image clarity is improved and the details are more obvious, which facilitates the accurate identification and analysis of subsequent defect information.

[0206] It should be noted that other AI models can also be used to improve image quality, such as the Denoising Diffusion Probabilistic Model (DDPM).

[0207] Step 3: Annotate the images with defects.

[0208] Specifically, selected pipeline defect images can be uploaded in batches to the image data annotation module. The outlines of each defect type can be annotated according to the defect definition in the "Technical Specification for Inspection and Evaluation of Urban Drainage Pipelines", and the scope of the defect can be clearly marked. This step can be completed by professionally trained annotators.

[0209] This embodiment can automatically identify all defect types. For example, defect types can include the following:

[0210] 0: Crack; 1: Deformation; 2: Corrosion; 3: Misalignment; 4: Undulation; 5: Disconnection;

[0211] 6: Joint material falling off; 7: Concealed branch pipe connection; 8: Foreign object penetration; 9: Leakage; 10: Deposition; 11: Scale buildup; 12: Obstructions; 13: Residual wall or dam base; 14: Tree roots;

[0212] 15: Scum.

[0213] Check whether the defects outlined in the annotated image are selected according to the correct defect type and whether the defect range is completely within the selected area. Unqualified samples are returned to the defect annotation program for modification.

[0214] Export the qualified labeling data as images, labels, and annotation files (txt format) and store them in the pipeline defect sample database. The database content will not be modified further.

[0215] For example, a sample database can be stored as follows:

[0216] #Sample database path

[0217] Sample library path: path / to / dataset

[0218] Training set: path / to / dataset / train

[0219] Validation set: path / to / dataset / validation

[0220] Test set: path / to / dataset / test.

[0221] The model training is based on the YOLO12 deep learning model with multi-feature fusion, supporting various computer vision tasks, including image segmentation. It achieves fast real-time inference while maintaining high accuracy, and is compatible with various hardware environments and cross-platform deployment.

[0222] During training, the training files, i.e., the pipeline defect training dataset, are imported from the sample library into the model training module. The dataset is divided, with 80% used as the training set, 10% as the validation set, and the remaining 10% for testing, ensuring that the images and label files correspond to each other. Relevant dependencies, such as CUDA, cuDNN, PyTorch, and the Ultralytics toolkit, are installed. Next, the dataset path and defect type information are configured in the dataset.yaml file, and the number of defect types and input image sizes are set in yolo11-seg.yaml.

[0223] Load the pre-trained YOLO12 weights (e.g., yolo12-seg.pt) using the YOLO12 algorithm model, and set the training parameters, configuring hyperparameters such as mixed-precision training, optimizer, and learning rate. Enable data augmentation algorithms to improve the model's generalization ability. Training logs and model weights will be saved in a designated directory for later review and analysis.

[0224] The model was evaluated on the validation set, with key metrics including segmentation accuracy (mask mAP50), precision, and recall. Based on the evaluation results, parameter settings were adjusted, and data augmentation strategies were optimized to improve the detection model's performance.

[0225] After training, the automatic detection model will be exported in ONNX format, such as yolo12-seg-pipe defects.onnx.

[0226] This embodiment combines a convolutional neural network model of DnCNN+SRCNN to improve the quality of pipeline image data, reduce negative images such as image noise, blurring and high contrast caused by darkness and humidity inside the pipeline and shooting light source, and greatly increase the utilization rate of pipeline image data.

[0227] Example 3

[0228] This embodiment provides a defect intelligent identification and deduplication device for drainage pipeline inspection videos, including:

[0229] The image acquisition module is used to acquire each frame of the video of the urban drainage pipeline collected by the pipeline robot and the frame number. Each frame of the video is marked with the attribute information of the pipeline robot at the time of acquisition.

[0230] The analysis and recognition module is used to analyze each frame of image using a pre-trained pipeline defect recognition model and obtain the recognition result of each frame of image. The recognition result includes: the category of pipeline defect in the current frame, the confidence level of each pipeline defect, the pipeline defect mask area, and the tracking number information used to track pipeline defects in the current frame.

[0231] The effective frame extraction module is used to extract the effective frame image to which each pipeline defect belongs and the motion trajectory of the pipeline defect based on the distance information in the attribute information of all frame images and the identification result of the pipeline defect.

[0232] The filtering module is used to comprehensively analyze the motion trajectory of the pipeline defect corresponding to the valid frame image of each pipeline defect, as well as the category, confidence level, and mask area of ​​the pipeline defect in the recognition result corresponding to each valid frame image, and to filter out the set of key frame images of defects for pipeline defect recognition and tracking analysis.

[0233] The device in this embodiment can be used to filter and track key frames in pipeline video frames, enabling further analysis based on pipeline defect video frames.

[0234] This embodiment also provides an electronic device, including: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program in the memory, specifically performing the steps of the method described in Embodiment 1 or Embodiment 2 above.

[0235] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0236] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0237] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0238] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0239] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0240] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. A method for intelligent defect identification and deduplication in drainage pipe inspection videos, characterized in that, include: S100. For the pipeline video of the urban drainage pipeline collected by the pipeline robot, obtain each frame image and frame number in the pipeline video, and mark the attribute information of the pipeline robot when collecting the video in each frame image. S200. Analyze each frame of image using a pre-trained pipeline defect recognition model to obtain the recognition result of each frame of image. The recognition result includes: the category of pipeline defect in the current frame, the confidence level of each pipeline defect, the pipeline defect mask area, and the tracking number information used to track pipeline defects in the current frame. S300: Based on the distance information in the attribute information of all frame images and the identification result of the pipeline defect, extract the valid frame image to which each pipeline defect belongs and the motion trajectory of the pipeline defect. The attribute information for each frame of the image includes: the time labeled in each frame, robot identifier, distance information, and pipeline identifier; S300 includes: S301, automatically identifying attribute information in each frame of image; S302, extracting the valid frame image to which each pipeline defect belongs and the movement trajectory of the pipeline defect based on the identification result of each frame of image and the distance information in the attribute information. The 302 includes: S302-1. For each defect, obtain the set of images showing the defect based on the category and tracking number information of the pipeline defect in the identification results. , ； in, For the first The frame number of the frame image. For the first Defect category and tracking number information corresponding to the frame image. For the first The recognition distance in the distance information of the frame image satisfies the condition of being sorted in ascending order of distance, i.e. , For the first The confidence level of the model recognition corresponding to the frame image. , For the first Frame image, S302-2, Based on the set of images showing the defects The distance information is used to obtain the movement trajectory of the defect. ;in, The total distance the defect travels. The recognition distance is the distance information in the image where the defect finally appears. The recognition distance is the distance information in the image where the defect initially appears; S302-3. Extract the valid frame images from the defective image set according to the following formula. ； in, For the set of valid frame images, For the effective minimum distance, , For the effective maximum distance, ; S400. Perform a comprehensive analysis on the motion trajectory of the pipeline defect corresponding to each valid frame image of the pipeline defect, as well as the category, confidence level, and mask area of ​​the pipeline defect in the recognition result corresponding to each valid frame image, and select a set of key frame images of defects for pipeline defect recognition and tracking analysis.

2. The method according to claim 1, characterized in that, Before step S200, the method further includes: The pipeline defect identification model was trained and validated using a pipeline defect training dataset to obtain the trained pipeline defect identification model. Each image in the pipeline defect training dataset is a pre-labeled image with defect category and defect mask region; and for images with a single defect or multiple defect categories, image enhancement algorithms are used to process the images to obtain training data that meets the preset image quality.

3. The method according to claim 2, characterized in that, Image enhancement algorithms are used to process and obtain training data that meets the preset image quality, including: DnCNN and SRCNN are used to process images with a single defect or multiple defect categories; DnCNN consists of a first input layer, intermediate layers, and a first output layer. The first input layer extracts features from the input image using the following formula. ； in, The input image has dimensions of . , This is a two-dimensional convolution operation. This indicates that its convolution kernel is 5×5. This indicates that its step size is 2. This indicates that the number of groups in its grouped convolution is 3. This indicates that its filling method is "same". This indicates that its convolution operation converts 3 channels into 32 channels. It is the hyperbolic tangent function. The output of the first input layer, The intermediate layers consist of 10 alternating convolutional layers. ； in, For the output of each intermediate layer, For the input of each intermediate layer, This represents a two-dimensional convolution operation, with the convolution kernel... For a 3×3 matrix, fill values Setting it to 1 converts 32 channels to 64 channels. This represents a two-dimensional convolution operation, with the convolution kernel... For a 3×3 matrix, fill values Setting it to 1 converts 64 channels to 32 channels. This indicates that batch normalization is being performed. For activation function, The output of the first input layer, This is the output of the intermediate layer; The first output layer is, ； in, For the output of the intermediate layer, This is the output of the first output layer. This represents a two-dimensional convolution operation, with the convolution kernel... For a 5x5 grid, fill values The value is 2, converting 32 channels into 3 channels; SRCNN takes the output of DnCNN as input, as shown in the following formula. ； in, This is the output of SRCNN. As input to SRCNN, This represents a two-dimensional convolution operation, with the convolution kernel... For a 9x9 grid, fill value The value is 4, converting 3 channels into 64 channels. This represents a two-dimensional convolution operation, with the convolution kernel... For 1×1, fill value A value of 0 converts the 64-channel configuration to 64 channels. This represents a two-dimensional convolution operation, with the convolution kernel... For a 5x5 grid, fill values To convert 64 channels to 3 channels, the value is set to 2. This represents the linear rectifier function.

4. The method according to claim 1, characterized in that, The 301 includes: OCR recognition technology is used to obtain attribute information from each frame of the pipeline video. Based on regular expressions, the attribute information is parsed to obtain distance information.

5. The method according to claim 1, characterized in that, The S400 includes: S401. In the set of valid frame images, form (M-W+1) smooth windows with a fixed window length W, where M is the number of valid frame images in the set. Calculate the average confidence level of each sliding window using the following formula. ； in, For the first The average confidence level of a sliding window For the first The confidence level of the model recognition corresponding to the frame image; S402. From all smoothed windows, select the window with the highest average confidence level as the optimal window. ； ； in, For the optimal window value, Indicates in The parameter value within the range that maximizes the subsequent expression. Represents the set of images within the optimal window; S403. From the image set of the optimal window, select the image with the highest confidence level as the keyframe image of the defect. ； in, The frame number representing the keyframe image. Indicates in The parameter value within the range that maximizes the subsequent expression. For the first The confidence level of the model recognition corresponding to the frame image; S404. Construct a keyframe image set based on the keyframe images of each defect.

6. The method according to claim 1, characterized in that, The pipeline defect identification model is a pipeline defect identification model based on the YOLO12 algorithm, which includes a second input layer, a feature extraction layer, a feature fusion layer, an upsampling layer, and a second output layer. The second input layer is used to receive the input drainage pipe image. The feature extraction layer includes a convolutional layer, a C3k2 layer, and an A2C2f layer, which are used for preliminary feature extraction and attention processing. The feature fusion layer is used to stitch together the feature maps output from different levels of the feature extraction layer. The upsampling layer is used to increase the resolution of the feature map output by the feature fusion layer. The second output layer is used to output the segmentation and detection results.

7. A defect intelligent identification and deduplication device for drainage pipeline inspection videos, characterized in that, include: The image acquisition module is used to acquire each frame of the video of the urban drainage pipeline collected by the pipeline robot and the frame number. Each frame of the video is marked with the attribute information of the pipeline robot at the time of acquisition. The analysis and recognition module is used to analyze each frame of image using a pre-trained pipeline defect recognition model and obtain the recognition result of each frame of image. The recognition result includes: the category of pipeline defect in the current frame, the confidence level of each pipeline defect, the pipeline defect mask area, and the tracking number information used to track pipeline defects in the current frame. The effective frame extraction module is used to extract the effective frame image to which each pipeline defect belongs and the motion trajectory of the pipeline defect based on the distance information in the attribute information of all frame images and the identification result of the pipeline defect. The attribute information for each frame of the image includes: the time labeled in each frame, robot identifier, distance information, and pipeline identifier; The process of extracting the valid frame image to which each pipeline defect belongs and the motion trajectory of the pipeline defect includes: automatically identifying the attribute information in each frame image; and extracting the valid frame image to which each pipeline defect belongs and the motion trajectory of the pipeline defect based on the identification results of each frame image and the distance information in the attribute information. Specifically, this includes: For each defect, a set of images showing the defect is obtained based on the defect category and tracking number information in the identification results. , ； in, For the first The frame number of the frame image. For the first Defect category and tracking number information corresponding to the frame image. For the first The recognition distance in the distance information of the frame image satisfies the condition of being sorted in ascending order of distance, i.e. , For the first The confidence level of the model recognition corresponding to the frame image. , For the first Frame image, Based on the image set where the defect occurs The distance information is used to obtain the movement trajectory of the defect. ;in, The total distance the defect travels. The recognition distance is the distance information in the image where the defect finally appears. The recognition distance is the distance information in the image where the defect initially appears; Valid frame images are extracted from the defective image set using the following formula. ； in, For the set of valid frame images, For the effective minimum distance, , For the effective maximum distance, ; The filtering module is used to comprehensively analyze the motion trajectory of the pipeline defect corresponding to the valid frame image of each pipeline defect, as well as the category, confidence level, and mask area of ​​the pipeline defect in the recognition result corresponding to each valid frame image, and to filter out the set of defect key frame images for pipeline defect recognition and tracking analysis.

8. An electronic device, characterized in that, include: The system includes a memory and a processor, wherein the memory stores a computer program and the processor executes the computer program in the memory, specifically performing the steps of the intelligent defect identification and deduplication method for drainage pipe inspection videos as described in any one of claims 1 to 6.

Images