An automated pipe network defect detection system and method

Abstract

The application relates to the technical field of pipe network detection, and in particular relates to an automatic pipe network defect detection system and method, which comprises the following modules: a video preprocessing and key frame extraction module, which is used for acquiring a pipe network detection video sequence, preprocessing the pipe network detection video sequence and extracting key frames; a lightweight detection module, which is used for preliminarily screening defects in each key frame image in a key frame image set by adopting a lightweight target detection model; a large model segmentation module, which is used for inputting a candidate defect region set into a general segmentation large model, performing pixel-level segmentation on each candidate defect region and acquiring a defect pixel-level mask of each candidate defect region; and a multi-modal deep rejudgment module, which is used for inputting the defect pixel-level mask of each candidate defect region and a depth map as input tensors into a fine-tuned multi-modal vision-language model to perform joint reasoning and acquire defect information after rechecking and confirmation. The application improves the accuracy and intelligent level of pipe network detection.

Description

Technical Field

[0001] This application relates to the field of underground pipeline technology, and in particular to an automated pipeline defect detection system and method. Background Technology

[0002] With the rapid development of urban infrastructure, underground pipe networks are widely used in water supply, drainage, gas supply, and communications. Their safe operation is directly related to the stability of urban operations and public safety. However, these pipe networks are constantly exposed to closed, humid, and corrosive environments, making them highly susceptible to structural defects such as ruptures, leaks, detachments, and corrosion. If these defects are not detected and repaired in a timely manner, they can lead to water leaks, ground subsidence, environmental pollution, and even serious accidents.

[0003] Traditional pipeline inspection mainly relies on inspection robots (CCTV inspection systems) to collect video, with defects identified frame by frame by manual inspection. This method suffers from low inspection efficiency, high labor intensity, strong subjectivity in results, and high rates of missed detections and false positives, making it difficult to meet the intelligent and efficient operation and maintenance needs of modern urban pipeline systems. Therefore, the industry has gradually introduced automatic inspection technologies based on image processing and deep learning to improve identification efficiency and objectivity. For example, patent application CN121504892A (classification number G06T) provides a drainage pipeline defect detection method based on data optimization and multi-model fusion; patent application CN120182209A (classification number G06T) provides a pipeline robot-based pipeline defect detection method and system based on deep learning; and patent application CN120411043A (classification number G06T) provides a drainage pipeline video defect detection method and system based on image quality screening. However, this method generally suffers from high dependence on labeled data and weak generalization ability, resulting in a high rate of false positives and false negatives in complex real-world scenarios. Furthermore, the model training process requires a significant amount of time and manpower for data labeling, making it difficult to meet the needs of modern urban pipeline networks for efficient, accurate, and intelligent operation and maintenance.

[0004] Therefore, there is an urgent need for an automated pipeline defect detection system and method. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, this application provides an automated pipeline defect detection system and method, which solves the technical problems of low efficiency, strong subjectivity, and high false detection rate in the prior art, making it difficult to meet the needs of efficient, accurate and intelligent operation and maintenance of modern urban pipeline networks.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the main technical solutions adopted in this application include:

[0009] In a first aspect, embodiments of this application provide an automated pipeline defect detection system, comprising:

[0010] The video preprocessing and keyframe extraction module is used to acquire a pipeline network detection video sequence, preprocess the pipeline network detection video sequence and extract keyframes to obtain a keyframe image set.

[0011] A lightweight detection module is used to perform preliminary defect screening on each keyframe image in the keyframe image set using a lightweight target detection model to obtain a set of candidate defect regions; the candidate defect regions are represented in the form of rectangular boxes, and the rectangular boxes contain spatial location coordinates, width, height, category information and confidence score;

[0012] The large model segmentation module is used to input the set of candidate defect regions into a general segmentation large model, perform pixel-level segmentation on each candidate defect region, and obtain a defect pixel-level mask for each candidate defect region.

[0013] The multimodal depth re-determination module is used to obtain the depth map corresponding to the defect pixel-level mask of each candidate defect region through the depth estimation model, and to input the defect pixel-level mask and depth map of each candidate defect region as input tensors to the fine-tuned multimodal vision-language model for joint reasoning to obtain the defect information after verification.

[0014] Optionally, in some embodiments of this application, the video preprocessing and keyframe extraction module includes:

[0015] An adaptive dynamic frame rate adjustment unit is used to obtain the intensity of content change between adjacent video frames in the pipeline detection video sequence; dynamically adjust the frame rate according to the intensity of content change to obtain a first candidate frame set;

[0016] An image enhancement unit is used to obtain the quality status index of each frame in the first candidate frame set, perform dehazing and super-resolution enhancement on the candidate frames according to the quality status index, and obtain a second candidate frame set; the quality status index includes dehazing index and super-resolution index.

[0017] The redundant frame elimination unit is used to evaluate the structural consistency between consecutive frames in the second candidate frame set by using inter-frame difference and motion estimation methods, eliminate candidate frames whose structural consistency is less than a first preset threshold, and obtain a third candidate frame set.

[0018] The quality screening unit is used to obtain the information entropy, local contrast and illumination uniformity of each frame in the third candidate frame set, and to normalize and weight and fuse the information entropy, local contrast and illumination uniformity to obtain a comprehensive score for each frame. Frames with a comprehensive score less than a second preset threshold are removed to obtain a key frame image set.

[0019] Optionally, in some embodiments of this application, the adaptive dynamic frame rate adjustment unit is specifically used for,

[0020] The intensity of content change between adjacent video frames in the network detection video sequence is calculated. The intensity of content change is characterized by calculating the average of the sum of the absolute values ​​of the grayscale differences of corresponding pixels between adjacent frames.

[0021] When the intensity of the content change is less than the first change threshold, the current frame is skipped to reduce the frame rate to the first frame rate value.

[0022] When the intensity of the content change is greater than the second change threshold, the current frame is retained and the frame rate is increased to the second frame rate value, wherein the second change threshold is greater than or equal to the first change threshold.

[0023] When the intensity of the content change is greater than or equal to the first change threshold and less than or equal to the second change threshold, the baseline frame rate remains unchanged.

[0024] Optionally, in some embodiments of this application, the lightweight object detection model in the lightweight detection module is obtained through the following training process:

[0025] Obtain a training sample set, which includes multiple pipeline inspection images labeled with defect categories and defect location bounding boxes;

[0026] Perform data augmentation operations on each training image in the training sample set to obtain a data-augmented training sample set. The data augmentation operations include: brightness adjustment, contrast adjustment, blur adjustment, or simulated dirt occlusion adjustment.

[0027] The augmented training sample set is input into the lightweight object detection model. The classification loss and regression loss between the prediction results and the bounding boxes are calculated and weighted to obtain the total loss. The network parameters are updated by backpropagation based on the total loss. The above steps are repeated until the model converges, and the trained lightweight object detection model is output.

[0028] Optionally, in some embodiments of this application, the lightweight detection module is further used for:

[0029] Obtain the intersection-union ratio between any two six-dimensional rectangles in the candidate defect region set, where the intersection-union ratio is the ratio of the intersection area to the union area of ​​the two six-dimensional rectangles;

[0030] When the intersection-union ratio is greater than the third preset threshold, the six-dimensional rectangles with low confidence scores are suppressed or merged, and the processed candidate defect region set is output to avoid the same defect region being marked repeatedly.

[0031] Optionally, in some embodiments of this application, the multimodal depth re-determination module, specifically includes obtaining the depth map corresponding to the defect pixel-level mask of each candidate defect region through a depth estimation model:

[0032] Each defect pixel-level mask is mapped back to a keyframe image, and a local image patch containing the defect region corresponding to that defect pixel-level mask is extracted as input.

[0033] The local image patch is input into the depth estimation model to obtain the depth distance of each pixel in the local image patch relative to the camera device; based on the depth distance, a grayscale or pseudo-color encoded depth map is generated.

[0034] The depth estimation model is either a monocular depth estimation network or a binocular stereo matching model.

[0035] Optionally, in some embodiments of this application, the multimodal depth re-judgment module uses the defect pixel-level mask and depth map of each candidate defect region as input tensors, inputs them to a fine-tuned multimodal vision-language model for joint inference, and obtains the verified defect information, specifically including:

[0036] Construct a multimodal input tensor, wherein the input tensor includes a pixel-level mask of defects and a depth map for each candidate defect region;

[0037] The input tensor is fed into a fine-tuned multimodal vision-language model, which establishes a mapping between defect appearance representation, spatial structural relationships, and scene semantic constraints through a multimodal feature alignment and fusion mechanism.

[0038] Based on the aforementioned association mapping, a multi-task joint learning mechanism is used to perform defect authenticity verification, defect type correction, and risk level assessment, and outputs the defect type, spatial attributes, and risk description information after review and confirmation as the defect information after review and confirmation.

[0039] Optionally, in some embodiments of this application, the multimodal vision-language model is obtained through the following fine-tuning training process:

[0040] Construct a fine-tuning training set, which includes multimodal input samples and corresponding annotation information. The multimodal input samples include defect pixel-level masks, depth maps, and contextual information corresponding to defect regions. The annotation information includes defect authenticity labels, defect type labels, and risk level labels.

[0041] The multimodal input samples are input into a pre-trained visual-language model for forward inference, and the multi-task joint loss between the inference result and the labeled information is calculated. The multi-task joint loss includes defect authenticity verification loss, defect type correction loss and risk level assessment loss.

[0042] The parameters of the visual-language model are updated by backpropagation algorithm, and the training steps are repeated until the multi-task joint loss converges, outputting the fine-tuned multimodal visual-language model.

[0043] During training, hard example mining, class reweighting, and data augmentation strategies were employed to optimize model convergence.

[0044] Optionally, in some embodiments of this application, the system further includes:

[0045] The intelligent analysis and visualization module is used to integrate the verified defect information, generate a structured defect report containing defect category, spatial location, size and shape and semantic description, and display the image overlay, defect location annotation and export the detection results in the visualization interface.

[0046] Secondly, embodiments of this application provide an automated pipeline network defect detection method, including:

[0047] Obtain a pipeline network detection video sequence, preprocess the pipeline network detection video sequence and extract key frames to obtain a key frame image set;

[0048] A lightweight target detection model is used to perform preliminary defect screening on each keyframe image in the keyframe image set to obtain a set of candidate defect regions. The candidate defect regions are represented by rectangular boxes, which contain spatial location coordinates, width, height, category information, and confidence score.

[0049] The set of candidate defect regions is input into a general segmentation model, and each candidate defect region is segmented at the pixel level to obtain a defect pixel-level mask for each candidate defect region.

[0050] The depth map corresponding to the defect pixel-level mask of each candidate defect region is obtained by the depth estimation model. The defect pixel-level mask and depth map of each candidate defect region are used as input tensors and fed into the fine-tuned multimodal vision-language model for joint inference to obtain the defect information after verification.

[0051] (III) Beneficial Effects

[0052] The beneficial effects of this application are as follows: The automated pipeline defect detection system and method of this application employs a video preprocessing and keyframe extraction module to preprocess the pipeline detection video and extract keyframes, significantly reducing the amount of data processed subsequently, avoiding the waste of computational resources caused by redundant frames, and improving overall detection efficiency. A lightweight target detection model is used to perform preliminary defect screening on the keyframe images, enabling rapid location of candidate defect areas and achieving efficient and low-cost preliminary screening, reducing the computational burden of subsequent large-scale model processing. A general segmentation model is used to perform pixel-level segmentation of candidate defect areas, obtaining accurate pixel-level defect masks, allowing for fine depiction of defect boundaries and shape information, significantly improving the accuracy of defect localization and morphological description. A depth estimation model is used to obtain a depth map of the defect area, and a finely tuned multimodal vision-language model is combined to perform joint reasoning on the mask and depth map, enabling the system to integrate two-dimensional appearance information and three-dimensional spatial information, achieving in-depth verification of defect type, severity, and spatial attributes, effectively eliminating the false detection and missed detection problems of single-modal or shallow models. In summary, compared with existing technologies, it can build a fully automated, high-precision, and low-latency detection process from video input to defect output, achieving the effect of balancing real-time performance and precision, significantly reducing the cost of manual review, and improving the reliability of pipeline defect detection. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the structure of an automated pipeline defect detection system according to an embodiment of this application;

[0054] Figure 2 This is an internal structure diagram of the video preprocessing and keyframe extraction module in an embodiment of this application;

[0055] Figure 3 This is an internal flowchart of the video preprocessing and keyframe extraction module in an embodiment of this application;

[0056] Figure 4 This is a schematic diagram illustrating the final visualization result of an embodiment of this application;

[0057] Figure 5 This is a flowchart illustrating an automated pipeline defect detection method according to an embodiment of this application. Detailed Implementation

[0058] To better explain and facilitate understanding of this application, the following detailed description of the application is provided in conjunction with the accompanying drawings and specific embodiments.

[0059] In related technologies, the primary reliance is on inspection robots (CCTV inspection systems) to collect video data, with defects identified frame by frame by manual inspection. This approach suffers from low inspection efficiency, high labor intensity, strong subjectivity in results, and high rates of missed detections and false positives, making it difficult to meet the intelligent and efficient operation and maintenance needs of modern urban pipeline networks. To address this, the industry has gradually introduced automatic detection technologies based on image processing and deep learning to improve recognition efficiency and objectivity. However, these technologies generally suffer from high dependence on labeled data and weak generalization ability, resulting in high rates of false positives and missed detections even in complex real-world scenarios. Furthermore, the model training process requires significant time and manpower for data labeling, making it difficult to meet the efficient, accurate, and intelligent operation and maintenance needs of modern urban pipeline networks.

[0060] For example, patent CN112668634A proposes a cloud system for automatic identification and detection of underground pipe networks based on deep learning. This system, by constructing a sample library of pipe network defects and training a LexNet network model, can automatically identify and classify uploaded videos or images, initially achieving automation of pipe network detection. Although this approach has made some progress, it still has some shortcomings: the analysis process is relatively simple, only achieving a simple mapping from image to category, lacking a complete closed loop of detection, segmentation, re-judgment, and semantic analysis; accuracy and robustness are limited, with existing models relying heavily on two-dimensional image features, unable to adapt to detection tasks in complex environments (such as dirt occlusion, uneven lighting, etc.); it cannot perform high-precision defect boundary segmentation, limiting subsequent structural analysis; the output results are only defect category labels, lacking in-depth verification and semantic description, making it difficult to fully understand and apply the detection results.

[0061] Therefore, the automated pipeline defect detection system proposed in this application includes a video preprocessing and keyframe extraction module, a lightweight detection module, a large-scale model segmentation module, and a multimodal depth verification module. First, the pipeline inspection video is preprocessed and keyframes are extracted to obtain a keyframe image set. Then, a lightweight object detection model is used to quickly screen out candidate defect regions (rectangular boxes). Next, a general segmentation model is used to perform pixel-level segmentation on each candidate region, generating an accurate defect mask. Finally, a depth map of the defect region is obtained through a depth estimation model, and the mask and depth map are jointly input into a finely tuned multimodal vision-language model for joint inference, outputting verified defect information.

[0062] This application employs a cascaded architecture of lightweight screening—large-model fine segmentation—multimodal depth verification, enabling the detection process to balance real-time performance and high accuracy while significantly reducing computational redundancy. By introducing depth maps and a vision-language model for joint reasoning, the system can integrate 2D appearance and 3D spatial information, effectively handling complex environments such as dirt occlusion and uneven lighting, thus improving robustness. Simultaneously, the output includes pixel-level masks, depth information, and semantic verification conclusions, overcoming the limitations of existing solutions that only output category labels, enhancing the interpretability and practicality of the detection results. In summary, compared to existing technologies, this application achieves fully automated closed-loop detection from video to detailed defect information, significantly reducing reliance on manual annotation and massive training data, achieving efficient, accurate, and intelligent pipeline maintenance.

[0063] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application can be understood more clearly and thoroughly, and that the scope of this application can be fully conveyed to those skilled in the art.

[0064] Figure 1 This is a schematic diagram of an automated pipeline defect detection system according to an embodiment of this application. Figure 1 As shown, the pipeline defect detection system includes a video preprocessing and keyframe extraction module, a lightweight detection module, a large model segmentation module, and a multimodal depth re-judgment module; specifically:

[0065] The video preprocessing and keyframe extraction module is used to acquire the pipeline network detection video sequence, preprocess the pipeline network detection video sequence and extract keyframes to obtain a set of keyframe images.

[0066] The specific method for obtaining the pipeline network inspection video sequence is as follows:

[0067] A pipeline inspection robot equipped with a high-definition camera (such as a CCTV inspection system) moves inside the pipeline network to be inspected, capturing continuous video streams of the pipeline's inner wall in real time. The captured video sequence can be transmitted via wired or wireless network to a local storage device, server, or cloud platform, or the recorded video files can be read directly from the robot's own storage medium. This module obtains the raw pipeline inspection video sequence from the aforementioned data sources as input for subsequent processing.

[0068] Furthermore, such as Figure 2 As shown, the video preprocessing and keyframe extraction module includes: an adaptive dynamic frame rate adjustment unit, an image enhancement unit, a redundant frame removal unit, and a quality screening unit.

[0069] Specifically, see Figure 3 The video preprocessing and keyframe extraction module specifically includes:

[0070] An adaptive dynamic frame rate adjustment unit is used to obtain the intensity of content change between adjacent video frames in the pipeline detection video sequence; dynamically adjust the frame rate according to the intensity of content change to obtain the first candidate frame set;

[0071] The adaptive dynamic frame rate adjustment unit is specifically used for,

[0072] The intensity of content change between adjacent video frames in the pipeline detection video sequence is calculated. The intensity of content change is characterized by the average of the sum of the absolute values ​​of the grayscale differences of corresponding pixels between adjacent frames.

[0073] In the adaptive dynamic frame rate adjustment unit, the acquired pipeline detection video sequence is first converted into an image frame sequence that can be processed frame by frame. Specifically, the adaptive dynamic frame rate adjustment unit uses a video decoding library (such as FFmpeg or OpenCV's VideoCapture) to decode the original compressed video (such as H.264, MP4, etc.) frame by frame, restoring it to RGB or YUV image data. Then, each decoded frame is stored in a first-in-first-out frame buffer queue in chronological order, while recording the timestamp of each frame. Subsequently, each frame is converted into a grayscale image, resulting in a continuous grayscale frame sequence.

[0074] After the above operations, the original video sequence is converted into a continuous sequence of grayscale image frames for subsequent acquisition of the intensity of content changes.

[0075] Next, the adaptive dynamic frame rate adjustment unit reads two adjacent grayscale images in sequence and calculates the intensity of content change between them. This intensity is used to measure the degree of new structural information added between consecutive frames. The larger the value, the more drastic the pixel-level change.

[0076] In practice, the grayscale image can be downsampled (e.g., scaled to 1 / 4 or 1 / 8 resolution) before calculating the content change intensity, or methods such as random sampling points or grid region approximation can be used. For frames at the threshold boundaries, structural similarity difference methods can also be used for verification. The calculated content change intensity is then compared with the first change threshold and the second change threshold to execute dynamic frame skipping or frame advancement strategies, thereby outputting the first candidate frame set.

[0077] When the intensity of content change is less than the first change threshold, perform frame skipping on the current frame to reduce the frame rate to the first frame rate value.

[0078] When the intensity of content change is greater than the second change threshold, the current frame is retained and the frame rate is increased to the second frame rate value, where the second change threshold is greater than or equal to the first change threshold.

[0079] When the intensity of content change is greater than or equal to the first change threshold and less than or equal to the second change threshold, the baseline frame rate remains unchanged.

[0080] Optionally, the first and second frame rate values ​​are not fixed values, but are dynamically adjusted based on the degree to which the intensity of the current content change deviates from the threshold range and the frame rate status of the previous stage. Specifically, the system maintains a baseline frame rate, typically initialized to the frame rate during the original video capture, such as 30 frames per second, and sets a skipping coefficient α and a boosting coefficient β, where 0 < α < 1, β > 1. When the intensity of the content change is less than the first change threshold, the system determines that it has entered a redundant region. At this time, it performs frame skipping processing and reduces the output frame rate to the first frame rate value (original capture frame rate × α). The value of α can be further dynamically adjusted based on the degree of deviation of the content change intensity from the first change threshold: if the content change intensity is much less than the first change threshold (e.g., content change intensity < 0.5 × first change threshold), a more aggressive skipping coefficient (e.g., 0.3) is used; if the content change intensity is slightly less than the first change threshold, a milder coefficient (e.g., 0.7) is used. Similarly, when the intensity of content change exceeds the second change threshold, the system determines that it has entered a high-change region. At this time, the current frame is retained, and the output frame rate is increased to the second frame rate value (original acquisition frame rate × β). Likewise, the value of β can be dynamically adjusted according to the degree to which the intensity of content change exceeds the second change threshold: if the intensity of content change is much greater than the second change threshold (e.g., intensity of content change > 2 × second change threshold), a higher frame rate increase coefficient is used (e.g., β is 2 or 2.5); if it is slightly greater, a smaller frame rate increase coefficient is used (e.g., β is 1.2). When the intensity of content change is between the first and second change thresholds, the output frame rate remains unchanged.

[0081] Furthermore, the system incorporates a frame rate smoothing mechanism to prevent frequent frame rate fluctuations: each adjusted output frame rate undergoes low-pass filtering (e.g., weighted averaging with historical output frame rates) and is limited to a preset minimum frame rate (e.g., 5 frames / second) and maximum frame rate (e.g., the original acquisition frame rate or 60 frames / second). Through these dynamic change rules, the first and second frame rate values ​​adaptively reflect the speed of actual scene changes within the pipeline, maximizing the compression of redundant frames and improving the processing efficiency of subsequent modules while ensuring that critical defect information is not lost.

[0082] The image enhancement unit is used to obtain the quality status index of each frame in the first candidate frame set, perform dehazing and super-resolution enhancement on the candidate frames according to the quality status index, and obtain the second candidate frame set; the quality status index includes the dehazing index and the super-resolution index.

[0083] In the specific implementation process, the dehazing index is used to characterize the degree of fogging and edge degradation of an image. Its acquisition method is based on the dark channel prior theory: for each frame of the image, its dark channel image is calculated, that is, the minimum value of each pixel in the red, green, and blue color channels is taken, and then the minimum value image is subjected to minimum value filtering (the filter window size is usually set to 15×15 or adaptively adjusted according to the image resolution) to obtain the dark channel image; then the mean value of all pixels in the dark channel image is calculated, and this mean value is the dehazing index. The closer the dark channel mean value is to 0, the more fog-free the image is, the more saturated the colors are, and the clearer the edges are; conversely, if the mean value is high (e.g., greater than 0.3), it indicates that the image has obvious fogging or floating objects obstructing the view, resulting in a decrease in edge contrast, and dehazing processing is required.

[0084] Super-resolution metrics are used to determine whether an image's detail resolution is insufficient, thus deciding whether to perform super-resolution enhancement. They are obtained based on Laplacian variance: first, the current frame is converted to a grayscale image; then, a Laplacian operator (e.g., a 3×3 Laplacian kernel) is used to convolve the grayscale image to obtain a Laplacian response map; next, the variance of all pixel values ​​in this response map is calculated. Laplacian variance reflects the sharpness of image edges and the richness of texture details: a larger variance indicates that the image contains more high-frequency details and is clearer; a smaller variance (e.g., below a preset threshold, typically 100-200, the specific value depending on the image resolution and scene) indicates that the image is blurry and details are lost, requiring super-resolution enhancement to recover finer structural information.

[0085] By quantifying the above two indicators, the image enhancement unit can automatically decide whether to perform dehazing and super-resolution processing on the current frame, thereby obtaining a second set of candidate frames with improved quality.

[0086] The redundant frame elimination unit is used to evaluate the structural consistency between consecutive frames in the second candidate frame set by using inter-frame difference and motion estimation methods, eliminate candidate frames whose structural consistency is less than a first preset threshold, and obtain a third candidate frame set.

[0087] It should be noted that "structural consistency" here is defined as the degree of difference between adjacent frames. The larger the value, the more obvious the content change between the two frames; the smaller the value, the more similar or redundant the two frames are. Therefore, frames with structural consistency less than the first preset threshold are removed, that is, those frames with small differences from the preceding and following frames and redundant information are removed, while frames with significant changes are retained.

[0088] For example, suppose the second candidate frame set contains five consecutive images, denoted as frame A, frame B, frame C, frame D, and frame E, extracted from a pipeline detection video. The system sets a first preset threshold of 15 (a threshold for the degree of difference, the value range of which is determined based on the image resolution and grayscale range, for example, the range of the average grayscale difference is 0~255). The redundant frame removal unit sequentially calculates the structural consistency between adjacent frames, with the specific steps as follows:

[0089] Calculate the structural consistency between frame A and frame B: Use the inter-frame difference method to calculate the average of the absolute values ​​of the grayscale differences of corresponding pixels in the two frames, and the difference value is 3. Since 3 is much less than the threshold of 15, it means that frame B is almost identical to frame A and there is no new structural information. Therefore, frame B is judged as a redundant frame and is discarded.

[0090] Calculate the structural consistency between frame A and frame C (skip the removed frame B and directly compare with the previous frame A that was retained): The calculated difference value is 5, which is still less than 15. Frame C is also a redundant frame and is removed.

[0091] Calculate the structural consistency between frame A and frame D: At this point, the robot encounters a slight pipe interface inside the pipe, causing a slight change in the image. The calculated difference value is 12, which is still less than 15 (but close to the threshold). Since the difference value is still below the threshold, frame D is still considered redundant and is discarded.

[0092] Calculate the structural consistency between frame A and frame E: A noticeable crack appears, and the image changes significantly. The calculated difference value is 38, which is greater than the threshold of 15. Therefore, frame E is determined to be a non-redundant frame and is retained.

[0093] Furthermore, the system can also be verified using motion estimation methods. For example, the sparse optical flow method can be used to calculate the average motion vector amplitude of feature points between adjacent frames. For frames A and B, all feature points are almost stationary, and the average motion amplitude is only 0.2 pixels, far below the motion threshold (e.g., 2 pixels), further confirming that frame B is a redundant frame. However, for frames A and E, the average motion amplitude of feature points reaches 12 pixels, indicating that the inner wall structure of the pipe has undergone significant displacement or new defects have appeared, confirming that frame E should be retained.

[0094] After the above evaluation, the original frames B, C, and D were all eliminated because their structural consistency was less than the first preset threshold. Only frames A and E entered the third candidate frame set. In this way, subsequent modules do not need to process a large number of repetitive still frames, but only need to focus on the key moments when structural changes actually occur, thereby significantly improving detection efficiency.

[0095] The quality screening unit is used to obtain the information entropy, local contrast and illumination uniformity of each frame in the third candidate frame set, and to normalize and weight and fuse the information entropy, local contrast and illumination uniformity to obtain the comprehensive score of each frame. Frames with comprehensive scores less than the second preset threshold are removed to obtain the key frame image set.

[0096] Specifically, information entropy measures the amount of information contained in an image, reflecting the dispersion of grayscale distribution. For a single-channel grayscale image, the probability of each grayscale level (0-255) is first calculated by plotting the grayscale histogram to obtain the proportion of pixels corresponding to each grayscale level to the total number of pixels in the image. Then, the grayscale levels are weighted and summed according to the information entropy formula: information entropy equals the sum of the negative probability of each grayscale level multiplied by its logarithm (base 2). The larger the information entropy value, the richer the grayscale levels in the image, and the more detailed information it contains; the smaller the value, the brighter or darker the image is overall, the grayscale levels are concentrated, and the information content is poor. For color images, the image can be converted to grayscale before calculation, or the information entropy of each of the three channels can be calculated separately and then averaged.

[0097] Local contrast is used to measure the degree of difference in brightness between local areas in an image, reflecting the distinguishability between defects and the background. The method for obtaining it is as follows: First, divide the image into multiple local windows of the same size (e.g., window size 16×16 pixels, with a step size the same as the window size or overlapping division). Within each local window, calculate the difference between the maximum and minimum gray values ​​to obtain the local contrast of that window. Then, average the local contrast of all windows to obtain the local contrast of the entire image. Another common method is to calculate the standard deviation of the gray values ​​within each window. A larger standard deviation indicates more dramatic changes in local brightness and darkness, resulting in higher contrast. Then, average the standard deviations of all windows. A higher local contrast value indicates clearer defect edges and a more obvious difference from the surrounding background; conversely, a lower local contrast value indicates that the image is generally hazy and details are blurred, which is not conducive to defect identification.

[0098] Illumination uniformity is used to evaluate the flatness of illumination distribution in an image, avoiding overexposure or underexposure in certain areas due to uneven light source angles or reflections within conduits. The method is as follows: First, convert the image to grayscale, then calculate the grayscale mean of the entire image; next, divide the image into several grids (e.g., 8×8 or 16×16 grids), and calculate the grayscale mean within each grid; then calculate the standard deviation of all grid means, or calculate the average of the absolute values ​​of the deviations of each grid mean from the overall image mean. A smaller standard deviation or average deviation indicates more consistent brightness across regions, indicating good illumination uniformity; conversely, a larger standard deviation indicates the presence of distinct bright and dark areas in the image, indicating poor illumination uniformity. Additionally, the bimodality of the grayscale histogram or the distribution characteristics of the image gradient can be used to further evaluate illumination uniformity.

[0099] After obtaining information entropy, local contrast, and illumination uniformity, the quality screening unit normalizes these three indicators (e.g., using min-max normalization to map them to the 0-1 range), and then performs a weighted sum based on preset weighting coefficients to obtain a comprehensive score for each frame. A lower comprehensive score indicates poorer image quality; when the comprehensive score is less than a second preset threshold, the frame is discarded; otherwise, it is retained as the final keyframe image for use by the subsequent lightweight detection module. Optionally, sharpness can also be added, processed based on four indicators: information entropy, local contrast, illumination uniformity, and sharpness.

[0100] In the specific implementation process, optionally, when the comprehensive score is less than the second preset threshold, the comprehensive score of n consecutive frames is obtained, and the key frame with the highest comprehensive score among the n frames is also added to the key frame image set.

[0101] The aforementioned quality screening unit ensures that the keyframe image set input to the lightweight detection module has high clarity, information integrity, and lighting consistency, thereby significantly improving the robustness, accuracy, and computational efficiency of the entire pipeline defect detection system in real-world complex environments.

[0102] The lightweight detection module is used to perform preliminary defect screening on each keyframe image in the keyframe image set using a lightweight object detection model to obtain a set of candidate defect regions. The candidate defect regions are represented in the form of rectangular boxes, which contain spatial location coordinates, width, height, category information and confidence score.

[0103] The lightweight object detection model in the lightweight detection module is obtained through the following training process:

[0104] Obtain a training sample set, which includes multiple pipeline inspection images labeled with defect categories and defect location bounding boxes;

[0105] Perform data augmentation operations on each training image in the training sample set to obtain the data-augmented training sample set. Data augmentation operations include: brightness adjustment, contrast adjustment, blur adjustment, or simulated dirt occlusion adjustment.

[0106] The augmented training sample set is input into the lightweight object detection model. The classification loss and regression loss between the prediction results and the bounding boxes are calculated and weighted to obtain the total loss. The network parameters are updated by backpropagation based on the total loss. The above steps are repeated until the model converges, and the trained lightweight object detection model is output.

[0107] In the lightweight detection module, the lightweight detection model adopts a lightweight backbone network such as MobileNetV3, ShuffleNet, or a lightweight version of YOLO, EfficientDet, which significantly reduces the number of parameters and computation while ensuring detection accuracy.

[0108] Furthermore, in the lightweight detection module, candidate defect regions are represented by rectangular boxes. These rectangular boxes are the output of the lightweight object detection model, which locates and classifies the positions of potential defects in the image. Each rectangular box contains five key pieces of information, detailed below:

[0109] Spatial location coordinates: These typically represent the coordinates of the top-left vertex of the bounding box in the image, expressed as x-coordinates (x) and y-coordinates (y), usually in pixels. These coordinates determine the starting position of the defect region in the image, providing a basis for cropping or localization in subsequent large-scale model segmentation modules. In some implementations, center point coordinates plus width and height can also be used, but ultimately, they will all be converted to a coordinate form that facilitates region extraction.

[0110] Width: The horizontal length of the rectangle, measured in pixels, representing the span of the defect area across the image. Width information helps determine the lateral extent of the defect; for example, transverse cracks typically have a large width.

[0111] Height: The vertical length of the rectangle, in pixels, representing the span of the defect area along the vertical axis of the image. Combining width and height, the area and aspect ratio of the defect area can be calculated, thus helping to distinguish different types of defects (e.g., vertical deformation is often greater in height than in width, while local damage is close to a square).

[0112] Category information: Defect type labels predicted by the model, such as "crack," "deformation," "corrosion," "blockage," and "interface detachment." Category information is stored as an integer or string for use by the subsequent multimodal deep review module and the final output. This information is one of the core results of the initial screening and determines the focus of subsequent segmentation and verification stages.

[0113] Confidence score: This reflects the model's confidence in the presence of a predicted defect within the bounding box. It typically ranges from 0 to 1, with values ​​closer to 1 indicating greater confidence in the presence of the defect. The confidence score can be used for filtering in subsequent processing: for example, only bounding boxes with a confidence score above a certain threshold (e.g., 0.5) are sent to the large model segmentation module, thus filtering out low-quality false positives and saving computational resources. Furthermore, the confidence score can also serve as a prior weight for joint inference in the multimodal deep re-decision module.

[0114] All bounding boxes together form a set of candidate defect regions, which is passed to the large model segmentation module in list form. Each bounding box not only defines the spatial range that needs to be finely segmented, but also carries preliminary category information and confidence assessment, providing an efficient and compact intermediate representation for the cascaded processing of the entire system.

[0115] The lightweight detection module is also used for:

[0116] Obtain the intersection-union ratio between any two six-dimensional rectangles in the candidate defect region set. The intersection-union ratio is the ratio of the area of ​​the intersection of the two six-dimensional rectangles to the area of ​​their union.

[0117] When the crossover ratio is greater than the third preset threshold, the six-dimensional rectangles with low confidence scores are suppressed or merged, and the processed candidate defect region set is output to avoid the same defect region being marked repeatedly.

[0118] Optionally, the lightweight detection module also incorporates an Adaptive Dynamic Non-Maximum Suppression (NMS) mechanism: For each type of defect, after outputting the candidate bounding box set, the model first calculates the confidence score distribution of all boxes and counts the local density of the candidate boxes (e.g., the number of candidate boxes contained in a neighborhood centered on each box with a preset radius). For high-density regions (local density greater than a density threshold), the system adaptively lowers the NMS intersection-union threshold (e.g., from 0.5 to 0.3) to distinguish between different defects that are close to each other; for low-density regions, a higher threshold (e.g., 0.6) is maintained to quickly suppress redundant boxes. Furthermore, prior knowledge of the defect category can be incorporated: a stricter threshold is used for smaller defects (e.g., small cracks); a more lenient threshold is used for large-area defects (e.g., deformations). Through this adaptive NMS strategy, the lightweight detection module can significantly improve the recall and localization accuracy for dense, small, or adjacent defects without increasing the model's inference burden, thereby outputting a higher-quality set of candidate defect regions.

[0119] The aforementioned lightweight detection module provides a compact, reliable, and high-quality input for subsequent large-scale model segmentation and multimodal re-judgment, balancing real-time performance with screening effectiveness.

[0120] The large model segmentation module is used to input the set of candidate defect regions into the general segmentation large model, perform pixel-level segmentation on each candidate defect region, and obtain the defect pixel-level mask for each candidate defect region.

[0121] General segmentation models typically employ a base model with strong zero-shot or few-shot generalization capabilities, such as SAM (Segment Anything Model) or its variants (e.g., MobileSAM, FastSAM). These models can segment target objects in an image based on arbitrary cues (points, boxes, masks, etc.) without requiring retraining for specific defect categories, making them ideal for pipeline network detection scenarios with diverse defect morphologies and limited labeled data.

[0122] For each keyframe image and several candidate bounding boxes output by the lightweight detection module, the large-model segmentation module processes each bounding box sequentially. First, based on the bounding box's spatial coordinates, width, and height, the corresponding image region is cropped from the original keyframe image (with appropriate expansion if necessary, e.g., extending the bounding box outwards by 10-20 pixels to ensure the defect boundary is fully contained within the cropped area). The cropped sub-image serves as the input to the segmentation model. The general segmentation model requires additional cues to guide the segmentation target. This module directly converts the candidate bounding boxes into bounding box prompts, providing the normalized coordinates (top left and bottom right corners) of the bounding boxes for the model to use.

[0123] Next, the cropped sub-image (or the entire image, depending on the model input size) and bounding box cue are input into a general segmentation model. Internally, the model extracts image features through an encoder and interacts with the cue to output a binary mask (pixel values ​​of 0 represent background, and 1 represent defect areas). This mask typically has the same resolution as the input sub-image, or it can be upsampled to restore the original cropped size. For models like SAM, multiple candidate masks (such as segmentation results at different levels) can be output; this module can select the mask with the highest confidence or the most reasonable area as the final output.

[0124] After obtaining the pixel-level mask, it is mapped back to the coordinate system of the original keyframe image. If the rectangle was expanded or rotated during input, an inverse transformation (inverse cropping, inverse affine transformation) needs to be performed to restore the mask coordinates to the original image space. In addition, morphological post-processing can be performed on the mask (such as closing operations to fill small holes and opening operations to remove isolated noise) to make the defect boundaries more continuous and smooth.

[0125] After the above process, each candidate defect region yields a corresponding pixel-level defect mask. All masks are stored in association with the original bounding boxes, category information, etc., forming a defect representation set containing pixel-level fine contours, which is then passed to the multimodal depth re-determination module for subsequent depth estimation and joint inference.

[0126] Furthermore, optionally, the large model segmentation module only segments candidate boxes with a confidence level higher than a threshold (e.g., 0.5) in the output of the lightweight detection module, and skips low-confidence boxes directly.

[0127] Through the above process, the large model segmentation module realizes the transformation from coarse rectangular boxes to fine pixel-level masks, providing high-precision defect shape and boundary information for subsequent depth estimation and multimodal joint inference.

[0128] Specifically, the automated pipeline defect detection system of this embodiment also includes:

[0129] The multimodal depth re-determination module is used to obtain the depth map corresponding to the defect pixel-level mask of each candidate defect region through the depth estimation model, and to input the defect pixel-level mask and depth map of each candidate defect region as input tensors into the fine-tuned multimodal vision-language model for joint reasoning to obtain the defect information after verification.

[0130] In the multimodal depth re-determination module, the depth map corresponding to the pixel-level mask of each candidate defect region is obtained through a depth estimation model, specifically including:

[0131] Each defect pixel-level mask is mapped back to a keyframe image, and a local image patch containing the defect region corresponding to that defect pixel-level mask is extracted as input.

[0132] The local image patch is input into the depth estimation model to obtain the depth distance of each pixel in the local image patch relative to the camera device; based on the depth distance, a grayscale or pseudo-color encoded depth map is generated.

[0133] The depth estimation model can be a monocular depth estimation network or a binocular stereo matching model.

[0134] In the specific implementation process, the defect pixel-level mask output by the large model segmentation module records whether each pixel belongs to the defect region (binary mask), but its coordinates are based on the original keyframe image. The multimodal depth re-determination module first obtains this mask and determines its minimum bounding rectangle region in the original keyframe image (usually the rectangle output by the lightweight detection module can be used directly, or the bounding rectangle can be calculated based on the actual contour of the mask). Then, a local image patch corresponding to the rectangular region is cropped from the original keyframe image, ensuring that the image patch completely contains the area covered by the defect mask. If the mask shape is relatively irregular, it can be appropriately extended outward by a few pixels to avoid loss of defect edge information. The cropped local image patch is input into the pre-trained depth estimation model. The depth estimation model can be a monocular depth estimation network (such as MiDaS, DPT, ZoeDepth, etc.) or a binocular stereo matching model (if the system is equipped with a binocular camera or can obtain continuous frame disparity information). The model outputs a depth distance value for each pixel in the input image patch, which is the vertical distance from the corresponding real-world point to the camera device (or camera lens plane), typically in millimeters or meters. For monocular depth estimation models, by learning a large amount of prior depth information about the scene, they can infer relative or absolute depth from a single image; for binocular models, matching the left and right views is required to calculate disparity, which is then converted into depth.

[0135] After obtaining the depth distance for each pixel, the module maps these distance values ​​to grayscale or pseudo-color encoding to form a visualized depth map. Grayscale encoding typically maps the nearest distance to white or black, the farthest distance to its opposite color, and distributes grayscale values ​​for intermediate distances linearly or logarithmically. Pseudo-color encoding uses a heatmap, mapping closer objects to red and farther objects to blue, making depth variations more intuitive. The depth map has the same resolution as the input local image patch and spatially corresponds one-to-one with the defect pixel-level mask, thus accurately obtaining the depth information of each pixel within the defect region.

[0136] Ultimately, the depth map, along with the defect pixel-level mask and the original local image patch, constitutes the input tensor. This tensor is then fed into a finely tuned multimodal vision-language model for joint inference to verify the authenticity, type, and severity of the defect. For example, the model can determine whether a crack is structural damage rather than a surface scratch based on the difference between the depth value of the crack depression in the depth map and the surrounding normal wall surface.

[0137] In the multimodal depth re-determination module, the defect pixel-level mask and depth map of each candidate defect region are used as input tensors and fed into a fine-tuned multimodal vision-language model for joint inference to obtain the verified defect information, specifically including:

[0138] Construct a multimodal input tensor, which includes a pixel-level mask of defects and a depth map for each candidate defect region;

[0139] The input tensor is fed into a fine-tuned multimodal vision-language model, which establishes a mapping between defect appearance representation, spatial structural relationships, and scene semantic constraints through a multimodal feature alignment and fusion mechanism.

[0140] Based on association mapping, a multi-task joint learning mechanism is used to perform defect authenticity verification, defect type correction and risk level assessment, and output the defect type, spatial attributes and risk description information after review and confirmation as the defect information after review and confirmation.

[0141] The multimodal vision-language model is obtained through the following fine-tuning training process:

[0142] Construct a fine-tuning training set, which includes multimodal input samples and corresponding annotation information. The multimodal input samples include the defect pixel-level mask, depth map and context information corresponding to the defect region. The annotation information includes defect authenticity label, defect type label and risk level label.

[0143] Multimodal input samples are fed into a pre-trained visual-language model for forward inference. The multi-task joint loss between the inference result and the labeled information is calculated. The multi-task joint loss includes defect authenticity verification loss, defect type correction loss and risk level assessment loss.

[0144] The parameters of the visual-language model are updated by backpropagation algorithm, and the training steps are repeated until the multi-task joint loss converges, outputting a fine-tuned multimodal visual-language model.

[0145] During training, hard example mining, class reweighting, and data augmentation strategies were employed to optimize model convergence.

[0146] In a specific implementation, for example, suppose in a video of a drainage pipe inspection, the lightweight detection module outputs a rectangular box on a keyframe image, categorized as "crack," with a confidence level of 0.85. The large model segmentation module generates a pixel-level mask for this region (covering a curved linear area). The multimodal depth re-determination module extracts local image patches based on the mask and obtains the corresponding depth map using a monocular depth estimation model.

[0147] Construct the input tensor: the mask is a single-channel binary image (pixels at the crack are 1, and those in the background are 0), and the depth image is a single-channel grayscale image (larger pixel values ​​indicate greater distance). Stack them to obtain a 2×H×W tensor.

[0148] Input Model: This tensor is fed into a fine-tuned multimodal vision-language model (e.g., a pipeline defect-specific model fine-tuned based on BLIP-2) for question-based reasoning. The question template can be designed as follows: Given a pixel-level mask and depth map of a pipeline defect area, and the following pipeline information: material = concrete, pipe diameter = 600mm, historical inspection record = no defects two years ago, preliminary inspection category = crack. Please verify whether this area is a real defect? ​​If so, please correct the defect type and assess the risk level. The multimodal vision-language model generates an answer based on the joint representation and outputs the verified defect information.

[0149] The defect information after review and confirmation specifically includes:

[0150] Authenticity verification: Output the actual defects with a confidence level of 0.97.

[0151] Type correction: The cracks given by the lightweight model are corrected to more specific structural diagonal cracks, and a subclass of through cracks is added.

[0152] Risk Level Assessment: The average depth of the crack area in the depth map is 12 mm, and the maximum depth difference is 8 mm, while the normal pipe wall depth is 5 mm. Therefore, the crack indentation depth reaches 7 mm, exceeding the pipe wall thickness (assuming 15 mm) by 46%. The model outputs a high risk level and generates a risk description: The crack is penetrating and the indentation is obvious, which may cause leakage or pipe structural failure.

[0153] Through the above process, the multimodal depth re-determination module effectively corrects misjudgments by leveraging joint reasoning with the mask and depth map, and provides rich semantic output, greatly improving the reliability and interpretability of the detection system. Simultaneously, this question-and-answer mechanism allows the model to explicitly utilize prior environmental knowledge, avoiding misjudgments caused by relying solely on visual features. For example, when the depth map shows a depression but the context indicates that the area is a normal pipe interface structure, the model can re-examine it as non-defect and provide an explanation. By combining contextual information, the reasoning process of the multimodal depth re-determination module is more transparent, interpretable, and adaptable to the specific requirements of different pipe scenarios.

[0154] Furthermore, the automated pipeline defect detection system of this embodiment also includes:

[0155] The intelligent analysis and visualization module is used to integrate the verified defect information, generate a structured defect report containing defect category, spatial location, size and shape and semantic description, and display the image overlay, defect location annotation and export the detection results in the visualization interface.

[0156] For example, such as Figure 4As shown, the system identified three defects after inspecting a section of concrete stormwater pipe: a diagonal crack (chainage 35.2 meters, medium risk), a localized collapse (chainage 78.5 meters, high risk), and a sedimentary obstruction (chainage 102.0 meters, low risk). This module integrates the category, spatial location, size, shape, and semantic description of these defects to generate a structured report (supporting JSON, PDF, and other formats) containing a defect summary table, risk level, and maintenance recommendations. Simultaneously, on the visualization interface, a pixel-level mask is overlaid on the original image with a semi-transparent color layer, and the clock position and chainage of the defects are marked on the pipe longitudinal profile, facilitating rapid location of high-risk sections by maintenance personnel. Furthermore, the module supports exporting annotated inspection videos or GeoJSON files, enabling integration with GIS systems for asset management and decision support.

[0157] Through its intelligent analysis and visualization module, this system not only outputs machine-readable structured data but also provides engineers with intuitive and interactive tools for viewing and analyzing defects. This module greatly enhances the readability, traceability, and decision support capabilities of the detection results, extending the entire pipeline defect detection system from "automatic identification" to "intelligent analysis and decision support," truly meeting the closed-loop requirements of intelligent operation and maintenance of modern urban pipeline networks.

[0158] This embodiment of an automated pipeline defect detection system significantly reduces the amount of data required for subsequent processing by adaptively eliminating redundant frames and selecting high-quality keyframes through a video preprocessing and keyframe extraction module. A lightweight detection module performs rapid initial screening, and an adaptive dynamic non-maximum suppression mechanism is introduced to improve the recall rate of dense defects. A general segmentation model is used to perform pixel-level fine segmentation of candidate regions to obtain accurate defect masks. Then, a multimodal depth review module fuses the defect mask, depth map, and contextual information, using a finely tuned visual-language model for joint reasoning and verification, effectively correcting false detections, refining defect types, and assessing risk levels. Finally, an intelligent analysis and visualization module generates a structured report and provides intuitive visualization. The entire system forms a complete closed loop from video input to verification and confirmation, significantly reducing reliance on manual interpretation. It maintains high accuracy and robustness even in complex pipeline environments, while reducing data annotation and training costs, truly achieving efficient, intelligent, and interpretable automated pipeline defect detection and operation and maintenance decision support.

[0159] in addition, Figure 5 This is a flowchart illustrating an automated pipeline defect detection method according to an embodiment of this application. See also... Figure 5 The automated pipeline defect detection method includes:

[0160] Obtain a pipeline network detection video sequence, preprocess the pipeline network detection video sequence and extract key frames to obtain a key frame image set;

[0161] A lightweight target detection model is used to perform preliminary defect screening on each keyframe image in the keyframe image set to obtain a set of candidate defect regions. The candidate defect regions are represented by rectangular boxes, which contain spatial location coordinates, width, height, category information, and confidence score.

[0162] The set of candidate defect regions is input into a general segmentation model, and each candidate defect region is segmented at the pixel level to obtain a defect pixel-level mask for each candidate defect region.

[0163] The depth map corresponding to the defect pixel-level mask of each candidate defect region is obtained by the depth estimation model. The defect pixel-level mask and depth map of each candidate defect region are used as input tensors and fed into the fine-tuned multimodal vision-language model for joint inference to obtain the defect information after verification.

[0164] The automated pipeline defect detection method in this embodiment achieves a fully automated processing flow from raw video to accurate defect information by sequentially executing keyframe extraction, lightweight preliminary screening, large-model pixel-level segmentation, and multimodal joint inference verification. This method utilizes a lightweight model to ensure detection efficiency, while leveraging a general segmentation model and depth estimation to obtain refined geometric and depth features. Furthermore, it uses a vision-language model to fuse masks, depth maps, and contextual information for depth verification, effectively reducing false positives and false negatives, improving detection accuracy and robustness in complex pipeline environments, and simultaneously reducing reliance on manual interpretation and large amounts of labeled data, significantly enhancing the intelligence level of pipeline network operation and maintenance.

[0165] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0166] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0167] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first and second features are in direct contact, or that they are in indirect contact through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0168] In the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0169] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make modifications, alterations, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. An automated pipeline defect detection system, characterized in that, include: The video preprocessing and keyframe extraction module is used to acquire a pipeline network detection video sequence, preprocess the pipeline network detection video sequence and extract keyframes to obtain a keyframe image set. A lightweight detection module is used to perform preliminary defect screening on each keyframe image in the keyframe image set using a lightweight target detection model to obtain a set of candidate defect regions. The candidate defect region is represented by a rectangular box, which includes spatial location coordinates, width, height, category information, and confidence score. The large model segmentation module is used to input the set of candidate defect regions into a general segmentation large model, perform pixel-level segmentation on each candidate defect region, and obtain a defect pixel-level mask for each candidate defect region. The multimodal depth re-determination module is used to obtain the depth map corresponding to the defect pixel-level mask of each candidate defect region through the depth estimation model, and to input the defect pixel-level mask and depth map of each candidate defect region as input tensors to the fine-tuned multimodal vision-language model for joint reasoning to obtain the defect information after verification.

2. The automated pipeline defect detection system according to claim 1, characterized in that, The video preprocessing and keyframe extraction module includes: An adaptive dynamic frame rate adjustment unit is used to obtain the intensity of content change between adjacent video frames in the pipeline detection video sequence; dynamically adjust the frame rate according to the intensity of content change to obtain a first candidate frame set; An image enhancement unit is used to obtain the quality status index of each frame in the first candidate frame set, perform dehazing and super-resolution enhancement on the candidate frames according to the quality status index, and obtain a second candidate frame set; the quality status index includes dehazing index and super-resolution index. The redundant frame elimination unit is used to evaluate the structural consistency between consecutive frames in the second candidate frame set by using inter-frame difference and motion estimation methods, eliminate candidate frames whose structural consistency is less than a first preset threshold, and obtain a third candidate frame set. The quality screening unit is used to obtain the information entropy, local contrast and illumination uniformity of each frame in the third candidate frame set, and to normalize and weight and fuse the information entropy, local contrast and illumination uniformity to obtain a comprehensive score for each frame. Frames with a comprehensive score less than a second preset threshold are removed to obtain a key frame image set.

3. The automated pipeline defect detection system according to claim 2, characterized in that, The adaptive dynamic frame rate adjustment unit is specifically used for: The intensity of content change between adjacent video frames in the network detection video sequence is calculated. The intensity of content change is characterized by calculating the average of the sum of the absolute values ​​of the grayscale differences of corresponding pixels between adjacent frames. When the intensity of the content change is less than the first change threshold, the current frame is skipped to reduce the frame rate to the first frame rate value. When the intensity of the content change is greater than the second change threshold, the current frame is retained and the frame rate is increased to the second frame rate value, wherein the second change threshold is greater than or equal to the first change threshold. When the intensity of the content change is greater than or equal to the first change threshold and less than or equal to the second change threshold, the baseline frame rate remains unchanged.

4. The automated pipeline defect detection system according to claim 1, characterized in that, The lightweight object detection model in the lightweight detection module is obtained through the following training process: Obtain a training sample set, which includes multiple pipeline inspection images labeled with defect categories and defect location bounding boxes; Perform data augmentation operations on each training image in the training sample set to obtain a data-augmented training sample set. The data augmentation operations include: brightness adjustment, contrast adjustment, blur adjustment, or simulated dirt occlusion adjustment. The augmented training sample set is input into the lightweight object detection model. The classification loss and regression loss between the prediction results and the bounding boxes are calculated and weighted to obtain the total loss. The network parameters are updated by backpropagation based on the total loss. The above steps are repeated until the model converges, and the trained lightweight object detection model is output.

5. The automated pipeline defect detection system according to claim 1, characterized in that, The lightweight detection module is also used for: Obtain the intersection-union ratio between any two six-dimensional rectangles in the candidate defect region set, where the intersection-union ratio is the ratio of the intersection area to the union area of ​​the two six-dimensional rectangles; When the intersection-union ratio is greater than the third preset threshold, the six-dimensional rectangles with low confidence scores are suppressed or merged, and the processed candidate defect region set is output to avoid the same defect region being marked repeatedly.

6. The automated pipeline defect detection system according to claim 1, characterized in that, In the multimodal depth re-determination module, obtaining the depth map corresponding to the defect pixel-level mask of each candidate defect region through a depth estimation model specifically includes: Each defect pixel-level mask is mapped back to a keyframe image, and a local image patch containing the defect region corresponding to that defect pixel-level mask is extracted as input. The local image patch is input into the depth estimation model to obtain the depth distance of each pixel in the local image patch relative to the camera device; based on the depth distance, a grayscale or pseudo-color encoded depth map is generated. The depth estimation model is either a monocular depth estimation network or a binocular stereo matching model.

7. The automated pipeline defect detection system according to claim 1, characterized in that, In the multimodal depth re-judgment module, the defect pixel-level mask and depth map of each candidate defect region are used as input tensors and fed into a fine-tuned multimodal vision-language model for joint inference to obtain the verified defect information. Specifically, this includes: Construct a multimodal input tensor, wherein the input tensor includes a pixel-level mask of defects and a depth map for each candidate defect region; The input tensor is fed into a fine-tuned multimodal vision-language model, which establishes a mapping between defect appearance representation, spatial structural relationships, and scene semantic constraints through a multimodal feature alignment and fusion mechanism. Based on the aforementioned association mapping, a multi-task joint learning mechanism is used to perform defect authenticity verification, defect type correction, and risk level assessment, and outputs the defect type, spatial attributes, and risk description information after review and confirmation as the defect information after review and confirmation.

8. The automated pipeline defect detection system according to claim 7, characterized in that, The multimodal vision-language model was obtained through the following fine-tuning training process: Construct a fine-tuning training set, which includes multimodal input samples and corresponding annotation information. The multimodal input samples include defect pixel-level masks, depth maps, and contextual information corresponding to defect regions. The annotation information includes defect authenticity labels, defect type labels, and risk level labels. The multimodal input samples are input into a pre-trained visual-language model for forward inference, and the multi-task joint loss between the inference result and the labeled information is calculated. The multi-task joint loss includes defect authenticity verification loss, defect type correction loss and risk level assessment loss. The parameters of the visual-language model are updated by backpropagation algorithm, and the training steps are repeated until the multi-task joint loss converges, outputting the fine-tuned multimodal visual-language model. During training, hard example mining, class reweighting, and data augmentation strategies were employed to optimize model convergence.

9. The automated pipeline defect detection system according to claim 1, characterized in that, The system also includes: The intelligent analysis and visualization module is used to integrate the verified defect information, generate a structured defect report containing defect category, spatial location, size and shape and semantic description, and display the image overlay, defect location annotation and export the detection results in the visualization interface.

10. An automated pipeline network defect detection method, characterized in that, include: Obtain a pipeline network detection video sequence, preprocess the pipeline network detection video sequence and extract key frames to obtain a key frame image set; A lightweight target detection model is used to perform preliminary defect screening on each keyframe image in the keyframe image set to obtain a set of candidate defect regions. The candidate defect regions are represented by rectangular boxes, which contain spatial location coordinates, width, height, category information, and confidence score. The set of candidate defect regions is input into a general segmentation model, and each candidate defect region is segmented at the pixel level to obtain a defect pixel-level mask for each candidate defect region. The depth map corresponding to the defect pixel-level mask of each candidate defect region is obtained by the depth estimation model. The defect pixel-level mask and depth map of each candidate defect region are used as input tensors and fed into the fine-tuned multimodal vision-language model for joint inference to obtain the defect information after verification.

Images