Visual crack detection method and system for sewer pipe network detection robot

Abstract

The invention relates to the technical field of pipe network detection, and discloses a sewer pipe network detection robot visual crack detection method and system, and the method comprises the steps: collecting a first video frame sequence of a sewer pipe network through a detection robot, and obtaining the IMU acceleration data of the detection robot, calculating a frame motion saliency index, screening a second video frame sequence from the first video frame sequence, and performing polar coordinate transformation and blocking on the second video frame sequence to obtain expanded image blocks; performing edge enhancement and multi-scale feature extraction on the expanded image block to obtain a crack probability mask and a detection moment; transmission delay is predicted based on the detection time, feature point coordinate transformation is carried out on the crack probability mask, and the crack global coordinates of the sewer pipe network are obtained.According to the method, accurate transformation of the crack feature points from the camera coordinate system to the global coordinate system is achieved, and space positioning errors caused by delay are reduced; and efficient and accurate detection and accurate space calibration of the sewer pipe network crack are realized.

Description

Technical Field

[0001] This invention relates to the field of pipeline inspection technology, and in particular to a visual crack detection method and system for sewer pipeline inspection robots. Background Technology

[0002] As a crucial piece of equipment for urban infrastructure inspection, the visual crack detection capability of sewer network inspection robots directly impacts the efficiency of pipeline maintenance. Current technologies rely on continuous video streams acquired at a fixed frame rate, resulting in numerous redundant frames. These redundant frames contribute nothing to crack detection but consume significant computational resources. Existing methods lack intelligent frame filtering mechanisms tailored to the characteristics of pipeline scenarios, leading to low computational efficiency. The time-varying latency in wireless transmission caused by changes in pipeline depth severely restricts detection accuracy. During this latency, the robot continues to move; directly using the pose at the detection moment for crack spatial calibration will introduce errors. Furthermore, the cylindrical structure of the pipeline interior, coupled with low illumination and water accumulation, means that general semantic segmentation models lack structural optimization for pipeline geometric distortions and environmental characteristics, resulting in insufficient accuracy in detecting minute cracks. Summary of the Invention

[0003] This invention provides a visual crack detection method and system for sewer network detection robots. This invention achieves accurate transformation of crack feature points from the camera coordinate system to the global coordinate system, reduces spatial positioning errors caused by latency, and realizes efficient and accurate detection and precise spatial calibration of cracks in sewer networks.

[0004] In a first aspect, the present invention provides a visual crack detection method for a sewer network detection robot, the method comprising: The first video frame sequence of the sewer network is collected by the detection robot and the IMU acceleration data of the detection robot is obtained. The frame motion significance index is calculated based on the first video frame sequence and the IMU acceleration data. Based on the frame motion saliency index, a second video frame sequence is selected from the first video frame sequence, and polar coordinate transformation and block division are performed on the second video frame sequence to obtain expanded image blocks; Edge enhancement and multi-scale feature extraction are performed on the unfolded image block to obtain the crack probability mask and detection time; Based on the detection time, the transmission delay is predicted, and the feature point coordinate transformation is performed on the crack probability mask to obtain the global coordinates of the crack in the sewer network.

[0005] In conjunction with the first aspect, in a first implementation of the first aspect of the present invention, the step of collecting a first video frame sequence of the sewer network by a detection robot and obtaining IMU acceleration data of the detection robot, and calculating a frame motion saliency index based on the first video frame sequence and the IMU acceleration data, includes: The first video frame sequence of the sewer network was collected by the probe robot, and the IMU acceleration data and robot pose data of the probe robot were obtained. Based on the first video frame sequence, extract the pipe wall edge point set of adjacent frames and calculate the pipe profile deformation distance; Perform a second-order difference operation on the IMU acceleration data to obtain the robot's acceleration change rate; The contrast and entropy of the gray-level co-occurrence matrix are calculated based on the first video frame sequence, and the contrast and entropy of the gray-level co-occurrence matrix are weighted and summed to obtain the image texture complexity. The frame motion saliency index is obtained by normalizing and weighting the pipe profile deformation distance, the robot acceleration change rate, and the image texture complexity.

[0006] In conjunction with the first aspect, in a second implementation of the first aspect of the present invention, the step of extracting the pipe wall edge point set of adjacent frames based on the first video frame sequence and calculating the pipe profile deformation distance includes: Canny edge detection is performed on the i-th frame and the (i-1)-th frame in the first video frame sequence to obtain the first set of pipe wall edge points in the i-th frame and the second set of pipe wall edge points in the (i-1)-th frame. The minimum Euclidean distance to the second pipe wall edge point set is calculated by traversing each point in the first pipe wall edge point set and taking the maximum value among all the minimum values ​​to obtain the first one-way Hausdorff distance. The minimum Euclidean distance from each point in the second pipe wall edge point set to the first pipe wall edge point set is calculated, and the maximum value among all the minimum values ​​is taken to obtain the second one-way Hausdorff distance. The maximum value of the first unidirectional Hausdorff distance and the second unidirectional Hausdorff distance is taken, and the maximum value is used as the pipe profile deformation distance.

[0007] In conjunction with the first aspect, in a third implementation of the first aspect of the present invention, the step of selecting a second video frame sequence from the first video frame sequence based on the frame motion saliency index, and performing polar coordinate transformation and block division on the second video frame sequence to obtain expanded image blocks includes: Traverse each video frame in the first video frame sequence and compare the frame motion saliency index corresponding to each video frame with the first threshold and the second threshold. Video frames whose frame motion saliency index is greater than the first threshold are marked as key frames and stored in the key frame queue. Video frames whose frame motion saliency index is less than the second threshold are directly discarded; For video frames whose frame motion saliency index is between the first threshold and the second threshold, calculate the sampling probability and compare it with a random number to determine whether to store them in the key frame queue, and use the key frame queue as the second video frame sequence. The second video frame sequence is subjected to polar coordinate transformation and segmentation to obtain expanded image blocks.

[0008] In conjunction with the first aspect, in a fourth implementation of the first aspect of the present invention, the step of performing polar coordinate transformation and block division on the second video frame sequence to obtain expanded image blocks includes: Distortion removal processing is performed on the pixels of each image frame in the second video frame sequence, and normalized planar coordinates are calculated using camera intrinsic parameters. Under the constraint of the pipe cylindrical surface, the angular coordinates and axial coordinates of the pipe cylindrical coordinate system are solved. The angular coordinates and axial coordinates are mapped to rectangular planar coordinates and bilinear interpolation resampling is performed to obtain the unfolded image. Read the pipe curvature radius from the robot pose data and determine the pipe type. Calculate the adaptive window size based on the ratio of the pipe curvature radius to the camera focal length to obtain the window block parameters. The expanded image is subjected to overlap segmentation based on the window segmentation parameters to obtain expanded image blocks.

[0009] In conjunction with the first aspect, in a fifth implementation of the first aspect of the present invention, the step of performing edge enhancement and multi-scale feature extraction on the unfolded image patch to obtain the crack probability mask and detection time includes: The expanded image patch is input into the first layer of the encoder, and linear projection is performed through the Patch Embedding module to obtain feature vectors. Window self-attention calculation is then performed on the feature vectors to obtain the first layer features. The Sobel edge enhancement operation is performed on the first layer features to obtain the gradient magnitude. The gradient magnitude is then multiplied by the learnable weight matrix using the Hadamard product and activated by the Sigmoid function before being added to the first layer features to obtain the first layer edge enhancement features. The first layer edge enhancement features are then sequentially input into the second to fourth layers of the encoder. The Patch Merging module performs downsampling and Sobel edge enhancement operations to obtain the second, third, and fourth layer edge enhancement features. The first layer edge enhancement features, the second layer edge enhancement features, the third layer edge enhancement features, and the fourth layer edge enhancement features are input into the decoder to perform bilinear upsampling recovery and calculate cosine similarity. When the cosine similarity is greater than the similarity threshold, the crack probability mask and the detection time are calculated.

[0010] In conjunction with the first aspect, in the sixth implementation of the first aspect of the present invention, the step of inputting the first layer edge enhancement features, the second layer edge enhancement features, the third layer edge enhancement features, and the fourth layer edge enhancement features into the decoder to perform bilinear upsampling recovery and calculate cosine similarity, and when the cosine similarity is greater than a similarity threshold, calculating the crack probability mask and the detection time, includes: Bilinear upsampling is performed on the fourth layer edge enhancement features to restore the spatial size of the third layer edge enhancement features, resulting in the upsampled fourth layer features. The dot product of the upsampled fourth layer features and the third layer edge enhancement features is calculated to obtain the cosine similarity. Determine whether the cosine similarity is greater than the similarity threshold. If it is greater than the similarity threshold, the upsampled fourth layer features and the third layer edge enhancement features are concatenated in the channel dimension and then fused through a convolutional layer with a kernel size of 1 to obtain the first fused feature. If it is less than or equal to the similarity threshold, skip the fusion and directly use the third layer edge enhancement features as the first fused feature. The first fused feature is sequentially combined with the second-layer edge enhancement feature and the first-layer edge enhancement feature, and bilinear upsampling, cosine similarity calculation and fusion operation are repeatedly performed to generate crack probability mask and detection time.

[0011] In conjunction with the first aspect, in the seventh implementation of the first aspect of the present invention, the step of predicting the transmission delay based on the detection time and performing feature point coordinate transformation on the crack probability mask to obtain the global coordinates of the crack in the sewer network includes: Binarization and connected component analysis are performed on the crack probability mask to extract crack skeleton feature points. The historical delay sequence is read and the current transmission delay is predicted by exponential weighted moving average and the compensation time is calculated. The robot pose at the compensation time is predicted by kinematic equation based on the robot velocity and acceleration at the detection time, and the predicted pose and crack feature points are obtained. Based on the predicted pose, a homogeneous transformation matrix is ​​constructed to transform the crack feature points from the camera coordinate system to the global coordinate system, thereby obtaining the global coordinates of the cracks in the sewer network.

[0012] In conjunction with the first aspect, in the eighth implementation of the first aspect of the present invention, the step of constructing a homogeneous transformation matrix based on the predicted pose to transform the crack feature points from the camera coordinate system to the global coordinate system to obtain the global coordinates of the cracks in the sewer network includes: The roll angle, pitch angle, and yaw angle are extracted from the predicted pose. The roll rotation matrix around the x-axis, the pitch rotation matrix around the y-axis, and the yaw rotation matrix around the z-axis are calculated respectively. Matrix multiplication is performed in the order of multiplying the yaw rotation matrix by the pitch rotation matrix and then by the roll rotation matrix to obtain the three-dimensional rotation matrix. The three-dimensional translation coordinates are extracted from the predicted pose to form a translation vector. The three-dimensional rotation matrix is ​​used as the upper left third-order submatrix, the translation vector is used as the right column vector, the bottom row vector is set as the zero vector, and scalar 1 is used to assemble a homogeneous transformation matrix. The camera coordinate system three-dimensional coordinates of each point in the crack feature points are traversed and a scalar 1 is added to the end to expand them into homogeneous coordinate vectors. The homogeneous coordinate vectors are then multiplied with the homogeneous transformation matrix to extract the global coordinates of the cracks in the sewer network.

[0013] Secondly, the present invention provides a visual crack detection system for a sewer network detection robot, the visual crack detection system for a sewer network detection robot comprising: The calculation module is used to collect the first video frame sequence of the sewer network through the detection robot and obtain the IMU acceleration data of the detection robot, and calculate the frame motion significance index based on the first video frame sequence and the IMU acceleration data; The filtering module is used to filter a second video frame sequence from the first video frame sequence according to the frame motion saliency index, and perform polar coordinate transformation and block division on the second video frame sequence to obtain expanded image blocks; The feature extraction module is used to perform edge enhancement and multi-scale feature extraction on the unfolded image block to obtain the crack probability mask and detection time. The coordinate transformation module is used to predict the transmission delay based on the detection time and perform feature point coordinate transformation on the crack probability mask to obtain the global coordinates of the crack in the sewer network.

[0014] The technical solution provided by this invention calculates the significance of the overall pipeline contour deformation distance, robot acceleration change rate, and image texture complexity through a frame motion selector module. A three-level adaptive filtering strategy intelligently removes redundant frames while retaining key information frames, effectively reducing unnecessary computational load while ensuring no loss of crack features. Polar coordinate transformation unfolds the inner wall of the pipeline cylinder into a planar rectangle. Combined with an adaptive windowing strategy based on pipeline curvature, the interference of geometric distortion on detection is eliminated, enabling the CrackFormer-Pipe network to perform feature extraction on a standardized plane. A multi-layer cascaded encoder structure extracts multi-scale features and embeds a Sobel edge enhancement module in each layer to strengthen crack edge response. The decoder employs a cross-layer feature fusion mechanism based on cosine similarity judgment to avoid noise propagation and improve the detection capability of subtle cracks. An exponentially weighted moving average method is used to predict time-varying transmission delays, and kinematic equations are combined to compensate for robot pose. A homogeneous transformation matrix at the compensation time is constructed to achieve accurate transformation of crack feature points from the camera coordinate system to the global coordinate system, reducing spatial positioning errors caused by delays and achieving efficient and accurate detection and precise spatial calibration of cracks in sewer networks. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram illustrating the steps of the visual crack detection method for sewer network detection robots in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the visual crack detection system for sewer network detection robot in an embodiment of the present invention. Detailed Implementation

[0017] This invention provides a method and system for visual crack detection using a sewer network detection robot. The terms "first," "second," "third," "fourth," etc. (if applicable) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the visual crack detection method for sewer network detection robots in this invention includes: Step S1: Collect the first video frame sequence of the sewer network through the detection robot and obtain the IMU acceleration data of the detection robot. Calculate the frame motion significance index based on the first video frame sequence and IMU acceleration data. Specifically, during the stable operation of the detection robot inside the sewer pipe, it uses an onboard high-resolution industrial camera to capture a continuous sequence of first video frames at a fixed frame rate. Simultaneously, it acquires inertial data containing three-axis acceleration information via an inertial measurement unit (IMU) and robot pose data through a SLAM visual odometry system. All this data is accompanied by millisecond-level timestamps and synchronized using a hardware mechanism. During processing, adjacent frames are selected, and an edge detection algorithm (such as the Canny algorithm) is used in each frame to extract the pipe wall edge point set. By calculating the bidirectional Hausdorff distance between the current and previous frame edge point sets, the deformation degree of the pipe profile between consecutive frames is quantified, effectively reflecting whether the robot has passed through interface seams, crack areas, or pipe sections with structural abrupt changes. The second derivative operation using the central difference method is performed on the time-synchronized IMU acceleration data to calculate the rate of acceleration change per unit time, i.e., the acceleration change rate, reflecting whether the robot experiences a sudden change in motion state at that moment, such as acceleration, deceleration, or sharp turn. Simultaneously, after converting each frame image to grayscale, a grayscale co-occurrence matrix is ​​constructed with a pixel distance of 1 and directional angles of 0°, 45°, 90°, and 135°. The contrast and entropy values ​​of the image are calculated separately in each direction. The results for each direction are averaged, and the contrast and entropy values ​​are weighted with coefficients of 0.6 and 0.4, respectively, to obtain the texture complexity index corresponding to the image frame. The pipe contour deformation distance, robot acceleration change rate, and image texture complexity are each divided by their corresponding normalization constants to convert them into dimensionless forms and assigned different weight coefficients (e.g., deformation distance weighted at 0.5, acceleration change rate weighted at 0.3, and texture complexity weighted at 0.2). The three are then weighted and summed to obtain the frame motion saliency index.

[0019] Step S2: Select a second video frame sequence from the first video frame sequence based on the frame motion saliency index, and perform polar coordinate transformation and block division on the second video frame sequence to obtain expanded image blocks; Specifically, each frame in the first video frame sequence is iterated one by one, and the corresponding frame motion saliency index is read for each frame. The saliency index is compared with a preset first threshold and a second threshold. When the saliency index of a frame is greater than the first threshold, it indicates that it is representative in terms of drastic deformation, sudden motion, or significant texture, and the frame is marked as a keyframe and stored in the keyframe queue. When the saliency index is lower than the second threshold, the current frame is determined to be a structurally stable and gently changing redundant segment, and the current frame will be directly discarded to save storage and computing resources. When the saliency index is between the two thresholds, it indicates that the current frame is an atypical change area, and a sampling probability function is introduced to decide whether to retain it. The sampling probability function calculates the sampling probability value based on the deviation of the saliency index from the reference value (e.g., 2.0), and then generates a uniformly distributed random number between [0,1], which is compared with the sampling probability to determine whether to include the current frame in the keyframe queue. The constructed keyframe queue is the second video frame sequence, representing the typical moments with the most information-dense, structurally complex, and dynamic states in the entire original video. The second video frame sequence undergoes geometric normalization preprocessing. Based on the camera intrinsic parameter matrix and distortion coefficients, polar coordinate transformation is performed on the image, unfolding the cylindrical pipe inner wall in the original image into a two-dimensional rectangular plane. The polar coordinate transformation is based on projecting the image pixels back into three-dimensional space and defining angle and axial distance parameters according to the pipe's central axis, remapping it onto the planar image to eliminate distortion and deformation. The size of the block window is dynamically adjusted according to the local curvature information of the robot's operating area. The unfolded image is then divided into grids, and overlapping areas between blocks are set to prevent the loss of crack boundary information, resulting in unfolded image blocks.

[0020] Step S3: Perform edge enhancement and multi-scale feature extraction on the unfolded image patch to obtain the crack probability mask and detection time; Specifically, the image patches are unrolled and input into the first layer of the encoder. In this first layer, a linear projection operation is performed on each image patch using the Patch Embedding module. This involves first spatially segmenting the image using a fixed-size convolutional kernel (e.g., 4×4 with a stride of 4), and then projecting each small patch into a fixed-dimensional set of feature vectors to obtain the initial feature map. Based on the Swin Transformer architecture, the feature vectors are divided into multiple local windows, and a self-attention operation based on a query-key-value structure is performed within each window. Relative position encoding is introduced to preserve spatial information, resulting in the local context-sensitive features output from the first layer. After the first layer feature output, a Sobel edge enhancement module is introduced. Sobel convolutional kernels in both the horizontal and vertical directions are used to calculate the gradient of the feature map, obtaining the gradient magnitude map for each pixel. The gradient magnitude map is then multiplied element-wise by a set of learnable weight matrices using Hadamard to assign different response weights to different channels. Normalization activation is performed using the Sigmoid function. The enhanced edge response map is then added to the original feature map and fused to obtain the edge-enhanced feature map of the first layer. Edge enhancement feature maps are sequentially input into the second, third, and fourth layers of the encoder. In each layer, the previous layer's feature map is spatially downsampled and its channels expanded using the Patch Merging module to improve semantic perception. The same Sobel edge enhancement operation as in the first layer is performed in each layer, generating multi-scale edge enhancement feature maps for the second, third, and fourth layers. After obtaining the feature maps at these four levels, they are uniformly input into the decoder module. In the decoder, deep features are first upsampled using bilinear interpolation to restore their spatial resolution, and then aligned with adjacent shallow features before calculating cosine similarity. The cosine similarity value is used to evaluate the matching degree of feature fusion between different layers. When the cosine similarity is greater than a preset threshold (e.g., 0.6), a feature-level concatenation and fusion operation is performed, and the fusion channels are compressed using 1×1 convolutions to progressively construct high-resolution feature maps of the same size as the input image. A crack probability mask map with one channel is output through convolution, where each pixel value represents the probability of belonging to a crack. Simultaneously, the timestamp of the currently processed keyframe is recorded as the detection time of the crack mask.

[0021] Step S4: Based on the detection time, predict the transmission delay and perform feature point coordinate transformation on the crack probability mask to obtain the global coordinates of the cracks in the sewer network.

[0022] Specifically, the crack probability mask is binarized. A fixed threshold (e.g., 0.65) is used to convert the probability value of each pixel into a binary label for cracks and non-cracks, generating a binary mask. Connectivity analysis is then performed on the binary mask to remove isolated noise points and extract crack target regions with areas larger than a preset pixel threshold. Within each effective connected region, the Zhang-Suen thinning algorithm is used to extract skeleton lines, and feature points are sampled on the skeleton lines at fixed pixel intervals to obtain a sparse feature point set of cracks in the image space. Simultaneously, the detection times and corresponding reception processing times of several historical frames are read from the system buffer queue to calculate several delay values. An exponentially weighted moving average method is used to perform temporal smoothing prediction on the delay values. By setting a smoothing coefficient (e.g., 0.25) to give higher weight to recent delays, the predicted transmission delay corresponding to the current frame is calculated. The delay value is then added to the image acquisition time to obtain the corresponding delay compensation time. At the compensation time point, a short-time kinematic prediction model is constructed based on the pose state, instantaneous velocity, and acceleration data at the image acquisition time. The position is deduced using the assumption of constant acceleration, and the attitude angle at the compensation time is estimated by integrating the angular velocity, thereby constructing the predicted pose parameters. A homogeneous transformation matrix is ​​constructed based on the predicted pose, which includes a three-dimensional rotation matrix and translation vector constructed by Euler angles. The crack feature points are restored from the unfolded image coordinate system to the three-dimensional cylindrical surface coordinate system through inverse transformation, and then transformed into spatial coordinate points in the camera coordinate system. The predicted pose matrix is ​​used to map these points from the camera coordinate system to the global three-dimensional spatial coordinate system, realizing the projection of the image detection results onto the real physical space of the sewer network, and obtaining a global coordinate set of cracks including attitude compensation and delay correction.

[0023] In one specific embodiment, the process of performing step S1 may specifically include the following steps: The first video frame sequence of the sewer network was collected by the probe robot, and the IMU acceleration data and robot pose data of the probe robot were obtained. Extract the pipe wall edge point set of adjacent frames based on the first video frame sequence and calculate the pipe profile deformation distance; Perform second-order difference operations on the IMU acceleration data to obtain the rate of change of robot acceleration; The contrast and entropy of the gray-level co-occurrence matrix are calculated based on the first video frame sequence, and the contrast and entropy of the gray-level co-occurrence matrix are weighted and summed to obtain the image texture complexity. The frame motion significance index is obtained by normalizing and weighting the pipe profile deformation distance, robot acceleration change rate, and image texture complexity.

[0024] Specifically, inside the sewer pipes, a high-resolution industrial camera mounted on a detection robot acquires continuous images at a rate of 30 frames per second to construct the first video frame sequence. During the image acquisition process, the IMU (Inertial Measurement Unit) is simultaneously activated to acquire three-axis acceleration data at a sampling frequency of 100Hz. Combined with the robot's spatial pose data output in real time by the SLAM (Spatial Alignment and Odometry) system, a time-synchronized data structure is constructed. A hardware triggering mechanism is used to perform phase-locked control on the image acquisition pulse and the IMU sampling clock to ensure that the time deviation of the multi-source data is controlled within ±5 milliseconds. Each frame of image and the corresponding interpolated acceleration vector and pose data are encapsulated into a unified data packet. Edge extraction is performed on any two frames in the video sequence using the Canny algorithm, with a threshold range of 50 to 150 pixels. Edge point sets of the pipe wall in the current and previous frames are extracted and denoted as the current point set and the reference point set, respectively. The bidirectional Hausdorff distance between these two point sets is calculated to quantify the degree of pipe structural deformation between adjacent frames. The Hausdorff distance increases significantly when the robot passes through pipe joints, sharp bends, cracks, or areas with foreign objects, reflecting drastic changes in the environmental geometry. Regarding inertial information, the timestamp of the image frame is selected... The closest interpolated acceleration value is used, and the second derivative of the central difference formula is calculated for acceleration data at three consecutive time points. After constructing the first difference using Δa / Δt, the acceleration rate of change vector is further constructed, and its magnitude is used as a quantitative indicator of the robot's motion state changes, thereby capturing dynamic behaviors caused by turning, deceleration, or vibration. Simultaneously, the original video frames are converted to grayscale images, and grayscale co-occurrence matrices with a pixel distance of 1 are constructed in multiple directions. Image contrast and entropy indices are calculated in four directions, averaged, and then linearly weighted to form an image texture complexity evaluation value. Deformation distance, acceleration rate of change, and texture complexity are normalized by introducing normalization factors (e.g., D_norm = 15 pixels, V_norm = 2.5 m / s²). 2 (E_norm = 8.0) is converted to a dimensionless scale form and a weighted summation is performed with weight coefficients of 0.5, 0.3 and 0.2 to obtain a single frame motion saliency index.

[0025] In one specific embodiment, the process of extracting the pipe wall edge point set based on the first video frame sequence and calculating the pipe profile deformation distance can specifically include the following steps: Perform Canny edge detection on the i-th frame and the (i-1)-th frame in the first video frame sequence to obtain the first set of pipe wall edge points in the i-th frame and the second set of pipe wall edge points in the (i-1)-th frame. The minimum Euclidean distance to the second pipe wall edge point set is calculated by traversing each point in the first pipe wall edge point set and taking the maximum value among all the minimum values ​​to obtain the first one-way Hausdorff distance. Iterate through each point in the second pipe wall edge point set, calculate the minimum Euclidean distance from it to the first pipe wall edge point set, and take the maximum value among all the minimum values ​​to obtain the second one-way Hausdorff distance; The maximum value of the first unidirectional Hausdorff distance and the second unidirectional Hausdorff distance is taken, and the maximum value is used as the pipe profile deformation distance.

[0026] Specifically, consecutive frames i and (i-1) of the first video frame sequence are extracted, and edge detection is performed on these two frames respectively. The Canny edge detection algorithm is used in this process, with appropriate low and high thresholds set (e.g., T_low = 50, T_high = 150) to fully extract the boundary structure and crack contours of the pipe wall, resulting in the first set of pipe wall edge points in frame i and the second set in frame (i-1). These two sets represent the boundary information formed by high-gradient regions in the two frames, and the position of each edge point is recorded in pixel coordinates. Each edge point in the first frame is iterated through, and the Euclidean distance between each point and all edge points in the second frame is calculated. The smallest Euclidean distance is taken as the nearest matching distance from that point to the edge of the second frame. The largest of these minimum distances is taken as the first one-way Hausdorff distance, representing the maximum local difference exhibited by the point in frame i that is furthest from the edge structure of frame (i-1). Repeat the same operation, iterating through each edge point in the second frame and calculating its minimum Euclidean distance to the edge point set in the first frame. Select the largest of these minimum values ​​as the second one-way Hausdorff distance, reflecting the location of the greatest local structural change in the (i-1)th frame. Compare these two one-way distances and take the maximum value as the final two-way Hausdorff distance, thus completing the calculation of the pipe profile deformation distance between the i-th and (i-1)th frames.

[0027] In one specific embodiment, the process of performing step S2 may specifically include the following steps: Traverse each video frame in the first video frame sequence and compare the frame motion saliency index corresponding to each video frame with the first threshold and the second threshold. Video frames whose frame motion saliency index is greater than the first threshold are marked as key frames and stored in the key frame queue. Video frames with a motion significance index lower than the second threshold are discarded directly. For video frames whose frame motion significance index is between the first threshold and the second threshold, calculate the sampling probability and compare it with a random number to determine whether to store them in the key frame queue. Use the key frame queue as the second video frame sequence. The second video frame sequence is subjected to polar coordinate transformation and segmentation to obtain expanded image blocks.

[0028] Specifically, a threshold comparison operation is performed on the frame motion saliency index of each frame. When the saliency value is higher than the set first threshold (e.g., 2.8), it indicates that the current frame contains significant structural variations or motion abrupt changes, such as the appearance of cracks, the passage of seams, or sharp turns. Therefore, the frame is marked as a keyframe and its complete data structure (image, acceleration, timestamp, pose, etc.) is stored in the keyframe queue. If the saliency value is lower than the second threshold (e.g., 1.2), it indicates that the current frame is in a pipe segment region with stable structure, uniform texture, and smooth motion. Such frames are not representative and have no analytical value, so they are directly discarded to save computing resources and storage space. When the saliency value is between the first and second thresholds, the current frame is regarded as an "intermediate confidence frame." Whether to retain it is determined by a sampling probability function, which is defined as a Gaussian exponential decay function. Then, a random number between [0,1] is generated and compared with the sampling probability. If the random number is less than the sampling probability, the frame is retained and added to the keyframe queue; otherwise, it is discarded. After all frames are filtered, the set of image frames retained in the keyframe queue is used as the second video frame sequence. A polar coordinate unfolding operation is performed on each frame of the second video frame sequence. Based on the camera intrinsic matrix, distortion model, and pipe radius parameters, each pixel in the image is back-projected from (u,v) to the pipe cylindrical coordinate system (r,θ,z), and then mapped to two-dimensional plane coordinates (u′,v′) through θ and z to generate an unfolded image. The window size is set according to the local curvature (e.g., 64×64 pixels in straight pipe areas and 32×32 pixels in curved pipe areas). The unfolded image is then divided into blocks using a sliding window method with a 50% overlap rate to generate standard-sized unfolded image blocks.

[0029] In one specific embodiment, the process of performing polar coordinate transformation and block division on the second video frame sequence to obtain expanded image blocks may specifically include the following steps: Distortion removal is performed on the pixels of each image frame in the second video frame sequence, and normalized planar coordinates are calculated using camera intrinsic parameters. The angular and axial coordinates of the pipe cylindrical coordinate system are solved under the pipe cylindrical constraint. The angular and axial coordinates are mapped to rectangular planar coordinates and bilinear interpolation resampling is performed to obtain the unfolded image. Read the pipe curvature radius from the robot pose data and determine the pipe type. Calculate the adaptive window size based on the ratio of the pipe curvature radius to the camera focal length to obtain the window block parameters. The expanded image is segmented by overlapping based on the window block parameters to obtain expanded image blocks.

[0030] Specifically, distortion correction is performed pixel-by-pixel on each frame of the second video frame sequence, based on the camera's internal parameter matrix, which includes the focal lengths f_x=1200 pixels, f_y=1200 pixels, and principal point coordinates (c_x=960, c_y=540). It also includes radial and tangential distortion coefficients k_1=-0.18, k_2=0.021, p_1=0, and p_2=0, used to recover the image projection under the true geometry. After distortion correction, the position of each pixel on the camera's normalized plane is calculated, and then mapped to the angle and axial position in a cylindrical coordinate system under the pipe cylindrical assumption, where the pipe radius is fixed at 150 mm. This is used to construct the physical position of the point on the pipe cylindrical surface. The cylindrical coordinate information is uniformly mapped to a two-dimensional unfolded image plane, where the horizontal axis represents the angular distribution and the vertical axis represents the axial distribution along the pipe direction. The entire unfolded image is set to W′=1024 pixels and H′=512 pixels. The mapping process reconstructs the grayscale values ​​of the target pixels through image interpolation, transforming the original ring structure image into a flat and intuitive rectangular image. After unfolding, the window is adaptively adjusted based on the curvature information in the robot pose data. The curvature radius calculated by the SLAM module is read. When the curvature radius is less than 500 mm, the current frame is determined to be in a curved pipe section; otherwise, it is determined to be a straight pipe section. Combining the camera focal length f_x=1200 pixels, the appropriate window block size is dynamically calculated according to the different curvature radii. For example, when the curvature radius of a straight pipe section approaches infinity, the calculated window size is 8, corresponding to an image block of 64×64 pixels; while in the curved pipe section region with a curvature radius of 300 mm, the window size becomes 4, corresponding to an image block of 32×32 pixels. Based on the dynamic window parameters, the unfolded image is segmented in an overlapping manner in the horizontal and vertical directions. During the segmentation process, 50% pixel overlap between adjacent image blocks is ensured to generate a set of unfolded image blocks.

[0031] In one specific embodiment, the process of performing step S3 may specifically include the following steps: The expanded image patch is input into the first layer of the encoder. Linear projection is performed through the Patch Embedding module to obtain feature vectors. Window self-attention calculation is performed on the feature vectors to obtain the first layer features. The Sobel edge enhancement operation is performed on the first layer features to obtain the gradient magnitude. The gradient magnitude is then multiplied by the learnable weight matrix using the Hadamard product and activated by the Sigmoid function before being added to the first layer features to obtain the first layer edge enhancement features. The first layer edge enhancement features are then sequentially input into the second to fourth layers of the encoder. The Patch Merging module performs downsampling and Sobel edge enhancement operations to obtain the second, third, and fourth layer edge enhancement features. The first, second, third, and fourth layer edge enhancement features are input into the decoder to perform bilinear upsampling recovery and calculate cosine similarity. When the cosine similarity is greater than the similarity threshold, the crack probability mask and detection time are calculated.

[0032] Specifically, the image patch is input into the first layer of the encoder of the deep feature extraction network. The PatchEmbedding module divides the 2D image into non-overlapping fixed-size patches, for example, each image patch is divided into several 4×4 sub-patterns. Linear projection is performed on each sub-pattern to convert it into a set of embedded feature vectors of uniform dimension, forming an initial token sequence. Within the token sequence, self-attention operations are performed on the tokens within the local window based on the Swin Transformer architecture. After constructing the query, key, and value matrix, attention weights are calculated and the contextual information between neighboring tokens is aggregated to obtain the spatially perceptual feature map of the first layer of the encoder, which enhances the local texture expression while preserving the structural features of the original image. An edge enhancement mechanism is introduced into the first layer feature map, that is, the gradient response maps in the horizontal and vertical directions are calculated by the Sobel operator, and pixel-level gradient magnitude images are calculated based on these two directions. The gradient magnitude tensor is subjected to a Hadamard product operation with a set of learnable channel weight matrices to enhance the response of important edge channels. Then, the output is normalized by the Sigmoid activation function and added proportionally to the original first layer feature map to obtain the first layer edge-enhanced feature map. The first-layer edge enhancement feature map is sequentially input into the second to fourth layers of the encoder. At each layer, the spatial resolution is downsampled using the PatchMerging module (e.g., from 1 / 4 to 1 / 8, then to 1 / 16 and 1 / 32). Simultaneously, the Sobel edge enhancement operation is repeatedly performed on the downsampled feature map, resulting in deeper features that not only have stronger semantic expressive power but also retain sensitive responses to crack contours, forming the edge enhancement features of the second, third, and fourth layers. These four layers of features are then input into the decoder structure. First, bilinear upsampling is performed on the higher-level deep features to restore a higher spatial resolution. After upsampling at each layer, the features are paired and compared with adjacent shallow features. The semantic consistency is determined by calculating the cosine similarity between the two feature maps. When the similarity exceeds a preset threshold (e.g., 0.6), it indicates a good match between the shallow and deep layers. At this point, the two are concatenated along the channel dimension and fused into a set of multi-scale fused features. By fusing and upsampling layer by layer, a full-resolution feature map with the same size as the input image is restored. A crack probability mask map is generated under the action of a convolutional layer with the output channel set to 1. Each pixel value in the map represents the probability range of being a crack between [0,1], and the acquisition timestamp corresponding to the current image block is recorded as the detection time of the crack mask.

[0033] In one specific embodiment, the execution step involves inputting the first layer edge enhancement features, the second layer edge enhancement features, the third layer edge enhancement features, and the fourth layer edge enhancement features into the decoder to perform bilinear upsampling recovery and calculate cosine similarity. When the cosine similarity is greater than the similarity threshold, the process of calculating the crack probability mask and the detection time can specifically include the following steps: Bilinear upsampling is performed on the fourth layer edge enhancement features to restore the spatial size of the third layer edge enhancement features, resulting in the upsampled fourth layer features. The dot product of the upsampled fourth layer features and the third layer edge enhancement features is calculated to obtain the cosine similarity. Determine if the cosine similarity is greater than the similarity threshold. If it is greater than the similarity threshold, the upsampled fourth layer features and the third layer edge enhancement features are concatenated in the channel dimension and then fused through a convolutional layer with a kernel size of 1 to obtain the first fused feature. If it is less than or equal to the similarity threshold, skip the fusion and directly use the third layer edge enhancement features as the first fused feature. The first fused feature is sequentially combined with the second-layer edge enhancement feature and the first-layer edge enhancement feature, and the bilinear upsampling, cosine similarity calculation and fusion operation are repeatedly performed to generate the crack probability mask and the detection time.

[0034] Specifically, for the fourth-layer edge enhancement feature, which has the smallest spatial resolution and the highest semantic level, a bilinear upsampling operation is used to restore the fourth-layer edge enhancement feature to the same spatial size as the third-layer edge enhancement feature, achieving feature alignment and obtaining an upsampled fourth-layer feature with consistent spatial dimensions. Using the channel vector corresponding to each spatial location as a unit, a dot product operation is performed on the upsampled fourth-layer feature and the third-layer edge enhancement feature. Combined with L2 norm normalization of each channel vector, cosine similarity is calculated. Cosine similarity reflects the consistency of expression between the two layers of features in a local region. When the similarity is higher than a preset threshold (e.g., 0.6), it indicates a good match between deep semantic features and shallow structural features in that region. The two feature maps are then concatenated along the channel dimension and input into a convolutional layer with a kernel size of 1. The convolutional layer acts as a channel fusion processor, compressing, transforming, and recombining the channel dimensions through learnable parameters to generate the first fused feature. However, when the cosine similarity is not higher than the threshold, it is determined that fusion in that region leads to information pollution or noise accumulation. Therefore, this fusion is skipped, and the third-layer edge enhancement feature is directly used as the first fused feature. The first fusion feature is then subjected to bilinear upsampling to restore it to the same spatial resolution as the second-layer edge enhancement feature. The cosine similarity judgment and conditional fusion strategy are repeated. When the similarity meets the requirements, channels are concatenated and fusion is performed using a 1×1 convolution; otherwise, the second-layer feature is used directly. This process is repeated up to the first-layer edge enhancement feature, achieving a step-by-step recursive integration of the four-layer fusion paths, resulting in a full-resolution fusion feature map with the same spatial size as the original image. A 1×1 convolutional probability mapping layer with one output channel is added to the fusion feature map, mapping the activation value of each pixel to the [0,1] interval, representing the probability intensity of its belonging to the crack region, forming a crack probability mask map. Simultaneously, the acquisition timestamp corresponding to the current image patch is used as the detection time of the mask map.

[0035] In one specific embodiment, the process of performing step S4 may specifically include the following steps: Binarization and connected component analysis are performed on the crack probability mask to extract crack skeleton feature points. The historical delay sequence is read and the current transmission delay is predicted by exponential weighted moving average and the compensation time is calculated. The robot pose at the compensation time is predicted by kinematic equation based on the robot velocity and acceleration at the detection time, and the predicted pose and crack feature points are obtained. Based on the predicted pose, a homogeneous transformation matrix is ​​constructed to transform the crack feature points from the camera coordinate system to the global coordinate system, thereby obtaining the global coordinates of the cracks in the sewer network.

[0036] Specifically, pixel-level binarization is performed on the crack probability mask image. A fixed threshold (e.g., 0.65) is set, and all pixels with values ​​greater than or equal to the threshold are assigned a value of 1, indicating a possible crack region. The remaining pixels are set to 0, representing the background region, thus forming a binary mask image. Connectivity analysis is performed on the binary image. All interconnected regions are marked using Flood Fill or an equivalent algorithm, and noisy connected regions with too small an area are removed, retaining only the target regions with actual crack structural significance. For each effective region, the Zhang-Suen skeleton extraction algorithm is applied to refine crack regions with a width greater than 1 pixel into a centerline skeleton curve with a width of 1 pixel. Then, the skeleton curve is sampled along its main path direction at fixed intervals (e.g., every 10 pixels) to obtain sparse crack skeleton feature points. Simultaneously, a historical transmission delay sequence is maintained, where each term is the time difference between the system processing time for image detection and the actual image acquisition time. The time difference is recorded by a software clock and updated in real time. The delay sequence is recursively filtered using an exponentially weighted moving average method, with more recent terms given higher weights to enhance adaptability to dynamic network environments, thus obtaining the predicted transmission delay value at the current moment. The predicted delay is summed with the acquisition timestamp of the crack image frame to calculate the compensation time, which is the observation time point that the system believes the crack feature point should correspond to in the real physical space. Based on the compensation time, combined with the robot's velocity and acceleration vectors corresponding to the detection time, the predicted pose of the robot at that moment is calculated using basic kinematic formulas under the assumption that the acceleration is constant in the short time. The attitude angle is calculated by Euler integral based on the angular velocity vector and added to the initial attitude, resulting in a complete six-degree-of-freedom predicted pose including spatial position and attitude angle. A homogeneous transformation matrix is ​​constructed based on the predicted pose, which includes a rotation matrix constructed with Euler angles and a translation vector of the predicted position, combined into a 4×4 homogeneous coordinate transformation matrix. The coordinates of the crack skeleton feature points are then back-projected from the planar coordinates in the image unfolded diagram back to the three-dimensional cylindrical model. The angle θ and axis z are restored through polar coordinates, and the radius r is set to a fixed value of the inner wall of the pipe. This is then converted to three-dimensional point coordinates in the camera coordinate system. Finally, matrix multiplication is used to project them onto the global three-dimensional coordinate system to complete the coordinate system transformation, obtaining the global spatial distribution result of the crack under the time-varying delay compensation and spatial attitude prediction mechanism.

[0037] In one specific embodiment, the process of constructing a homogeneous transformation matrix based on the predicted pose to transform the crack feature points from the camera coordinate system to the global coordinate system, thereby obtaining the global coordinates of the cracks in the sewer network, can specifically include the following steps: The roll angle, pitch angle, and yaw angle are extracted from the predicted pose. The roll rotation matrix around the x-axis, the pitch rotation matrix around the y-axis, and the yaw rotation matrix around the z-axis are calculated respectively. Matrix multiplication is performed in the order of multiplying the yaw rotation matrix by the pitch rotation matrix and then by the roll rotation matrix to obtain the three-dimensional rotation matrix. Extract the three-dimensional translation coordinates from the predicted pose to form a translation vector. Assemble the three-dimensional rotation matrix as the top left third-order submatrix, the translation vector as the right column vector, the bottom row vector as the zero vector, and scalar 1 to form a homogeneous transformation matrix. Iterate through the camera coordinate system three-dimensional coordinates of each point in the crack feature points and add a scalar 1 to the end to expand it into a homogeneous coordinate vector. Perform matrix multiplication between the homogeneous coordinate vector and the homogeneous transformation matrix to extract the global coordinates of the cracks in the sewer network.

[0038] Specifically, the robot's attitude information obtained through motion prediction after acquiring crack images is decomposed into three independent Euler angles: roll, pitch, and yaw. These three angles are then used to construct basic rotation matrices about the x, y, and z axes in three-dimensional space. The roll rotation matrix about the x-axis is defined as R_x = [[1, 0, 0], [0, cos(roll), -sin(roll)], [0, sin(roll), cos(roll)]], the pitch rotation matrix about the y-axis is defined as R_y = [[cos(pitch), 0, sin(pitch)], [0, 1, 0], [-sin(pitch),0, cos(pitch)]], and the yaw rotation matrix about the z-axis is R_z = [[cos(yaw), -sin(yaw), 0], [sin(yaw), cos(yaw), 0], [0, 0, Following a fixed matrix multiplication order—first multiplying R_z by R_y, then multiplying the product by R_x—a 3D rotation matrix is ​​generated to transform the local camera coordinates to the predicted global coordinate system. Simultaneously, translational position coordinates [x, y, z] are extracted from the same predicted pose and constructed as a 3D column vector T_trans = [x; y; z]. This vector is concatenated with the aforementioned rotation matrix to form a 3x4 spatial transformation matrix. A homogeneous coordinate vector [0, 0, 0, 1] is added below this matrix to construct a 4×4 homogeneous transformation matrix. This matrix can transform the coordinates of points in any camera coordinate system to the global world coordinate system at the prediction time. The 3D coordinates [x, y, z] of each feature point obtained during the crack skeleton extraction process are then used. T Process each point individually, expanding it to four-dimensional homogeneous coordinates [x, y, z, 1]. TThen, perform matrix multiplication on the vector and the homogeneous transformation matrix to obtain the precise position coordinates of the point in the global three-dimensional space. Repeat the above steps until all crack feature points are processed, and construct a global coordinate set of crack three-dimensional space containing spatial pose compensation and time delay correction mechanisms.

[0039] The above describes the visual crack detection method of the sewer network detection robot in the embodiments of the present invention. The following describes the visual crack detection system of the sewer network detection robot in the embodiments of the present invention. Please refer to [link / reference]. Figure 2 One embodiment of the visual crack detection system for sewer network detection robots in this invention includes: The calculation module 201 is used to collect the first video frame sequence of the sewer network through the detection robot and obtain the IMU acceleration data of the detection robot, and calculate the frame motion significance index based on the first video frame sequence and the IMU acceleration data; The filtering module 202 is used to filter the second video frame sequence from the first video frame sequence according to the frame motion saliency index, and perform polar coordinate transformation and block division on the second video frame sequence to obtain expanded image blocks; Feature extraction module 203 is used to perform edge enhancement and multi-scale feature extraction on the unfolded image patch to obtain crack probability mask and detection time; The coordinate transformation module 204 is used to predict the transmission delay based on the detection time and perform feature point coordinate transformation on the crack probability mask to obtain the global coordinates of the crack in the sewer network.

[0040] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0041] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0042] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for visual crack detection using a sewer network detection robot, characterized in that, include: The first video frame sequence of the sewer network is collected by the detection robot and the IMU acceleration data of the detection robot is obtained. The frame motion significance index is calculated based on the first video frame sequence and the IMU acceleration data. Based on the frame motion saliency index, a second video frame sequence is selected from the first video frame sequence, and polar coordinate transformation and block division are performed on the second video frame sequence to obtain expanded image blocks; Edge enhancement and multi-scale feature extraction are performed on the unfolded image block to obtain the crack probability mask and detection time; Based on the detection time, the transmission delay is predicted, and the feature point coordinate transformation is performed on the crack probability mask to obtain the global coordinates of the crack in the sewer network.

2. The method for visual crack detection using a sewer network detection robot according to claim 1, characterized in that, The process of acquiring a first video frame sequence of the sewer network using a detection robot and obtaining the robot's IMU acceleration data, and calculating a frame motion saliency index based on the first video frame sequence and the IMU acceleration data, includes: The first video frame sequence of the sewer network was collected by the probe robot, and the IMU acceleration data and robot pose data of the probe robot were obtained. Based on the first video frame sequence, extract the pipe wall edge point set of adjacent frames and calculate the pipe profile deformation distance; Perform a second-order difference operation on the IMU acceleration data to obtain the robot's acceleration change rate; The contrast and entropy of the gray-level co-occurrence matrix are calculated based on the first video frame sequence, and the contrast and entropy of the gray-level co-occurrence matrix are weighted and summed to obtain the image texture complexity. The frame motion saliency index is obtained by normalizing and weighting the pipe profile deformation distance, the robot acceleration change rate, and the image texture complexity.

3. The method for visual crack detection using a sewer network detection robot according to claim 2, characterized in that, The step of extracting the pipe wall edge point set of adjacent frames based on the first video frame sequence and calculating the pipe profile deformation distance includes: Canny edge detection is performed on the i-th frame and the (i-1)-th frame in the first video frame sequence to obtain the first set of pipe wall edge points in the i-th frame and the second set of pipe wall edge points in the (i-1)-th frame. The minimum Euclidean distance to the second pipe wall edge point set is calculated by traversing each point in the first pipe wall edge point set and taking the maximum value among all the minimum values ​​to obtain the first one-way Hausdorff distance. The minimum Euclidean distance from each point in the second pipe wall edge point set to the first pipe wall edge point set is calculated, and the maximum value among all the minimum values ​​is taken to obtain the second one-way Hausdorff distance. The maximum value of the first unidirectional Hausdorff distance and the second unidirectional Hausdorff distance is taken, and the maximum value is used as the pipe profile deformation distance.

4. The method for visual crack detection using a sewer network detection robot according to claim 1, characterized in that, The step of selecting a second video frame sequence from the first video frame sequence based on the frame motion saliency index, and performing polar coordinate transformation and block division on the second video frame sequence to obtain expanded image blocks includes: Traverse each video frame in the first video frame sequence and compare the frame motion saliency index corresponding to each video frame with the first threshold and the second threshold. Video frames whose frame motion saliency index is greater than the first threshold are marked as key frames and stored in the key frame queue. Video frames whose frame motion saliency index is less than the second threshold are directly discarded; For video frames whose frame motion saliency index is between the first threshold and the second threshold, calculate the sampling probability and compare it with a random number to determine whether to store them in the key frame queue, and use the key frame queue as the second video frame sequence. The second video frame sequence is subjected to polar coordinate transformation and segmentation to obtain expanded image blocks.

5. The method for visual crack detection using a sewer network detection robot according to claim 4, characterized in that, The step of performing polar coordinate transformation and segmentation on the second video frame sequence to obtain expanded image blocks includes: Distortion removal processing is performed on the pixels of each image frame in the second video frame sequence, and normalized planar coordinates are calculated using camera intrinsic parameters. Under the constraint of the pipe cylindrical surface, the angular coordinates and axial coordinates of the pipe cylindrical coordinate system are solved. The angular coordinates and axial coordinates are mapped to rectangular planar coordinates and bilinear interpolation resampling is performed to obtain the unfolded image. Read the pipe curvature radius from the robot pose data and determine the pipe type. Calculate the adaptive window size based on the ratio of the pipe curvature radius to the camera focal length to obtain the window block parameters. The expanded image is subjected to overlap segmentation based on the window segmentation parameters to obtain expanded image blocks.

6. The method for visual crack detection using a sewer network detection robot according to claim 1, characterized in that, The step of performing edge enhancement and multi-scale feature extraction on the unfolded image patch to obtain the crack probability mask and detection time includes: The expanded image patch is input into the first layer of the encoder, and linear projection is performed through the Patch Embedding module to obtain feature vectors. Window self-attention calculation is then performed on the feature vectors to obtain the first layer features. The Sobel edge enhancement operation is performed on the first layer features to obtain the gradient magnitude. The gradient magnitude is then multiplied by the learnable weight matrix using the Hadamard product and activated by the Sigmoid function before being added to the first layer features to obtain the first layer edge enhancement features. The first layer edge enhancement features are then sequentially input into the second to fourth layers of the encoder. The PatchMerging module performs downsampling and Sobel edge enhancement operations to obtain the second, third, and fourth layer edge enhancement features. The first layer edge enhancement features, the second layer edge enhancement features, the third layer edge enhancement features, and the fourth layer edge enhancement features are input into the decoder to perform bilinear upsampling recovery and calculate cosine similarity. When the cosine similarity is greater than the similarity threshold, the crack probability mask and the detection time are calculated.

7. The method for visual crack detection using a sewer network detection robot according to claim 6, characterized in that, The process involves inputting the first layer edge enhancement features, the second layer edge enhancement features, the third layer edge enhancement features, and the fourth layer edge enhancement features into the decoder, performing bilinear upsampling recovery, and calculating cosine similarity. When the cosine similarity is greater than a similarity threshold, the crack probability mask and detection time are calculated, including: Bilinear upsampling is performed on the fourth layer edge enhancement features to restore the spatial size of the third layer edge enhancement features, resulting in the upsampled fourth layer features. The dot product of the upsampled fourth layer features and the third layer edge enhancement features is calculated to obtain the cosine similarity. Determine whether the cosine similarity is greater than the similarity threshold. If it is greater than the similarity threshold, the upsampled fourth layer features and the third layer edge enhancement features are concatenated in the channel dimension and then fused through a convolutional layer with a kernel size of 1 to obtain the first fused feature. If it is less than or equal to the similarity threshold, skip the fusion and directly use the third layer edge enhancement features as the first fused feature. The first fused feature is sequentially combined with the second-layer edge enhancement feature and the first-layer edge enhancement feature, and bilinear upsampling, cosine similarity calculation and fusion operation are repeatedly performed to generate crack probability mask and detection time.

8. The method for visual crack detection using a sewer network detection robot according to claim 1, characterized in that, The step of predicting the transmission delay based on the detection time and performing feature point coordinate transformation on the crack probability mask to obtain the global coordinates of the cracks in the sewer network includes: Binarization and connected component analysis are performed on the crack probability mask to extract crack skeleton feature points. The historical delay sequence is read and the current transmission delay is predicted by exponential weighted moving average and the compensation time is calculated. The robot pose at the compensation time is predicted by kinematic equation based on the robot velocity and acceleration at the detection time, and the predicted pose and crack feature points are obtained. Based on the predicted pose, a homogeneous transformation matrix is ​​constructed to transform the crack feature points from the camera coordinate system to the global coordinate system, thereby obtaining the global coordinates of the cracks in the sewer network.

9. The method for visual crack detection using a sewer network detection robot according to claim 8, characterized in that, The step of constructing a homogeneous transformation matrix based on the predicted pose to transform the crack feature points from the camera coordinate system to the global coordinate system, thereby obtaining the global coordinates of the cracks in the sewer network, includes: The roll angle, pitch angle, and yaw angle are extracted from the predicted pose. The roll rotation matrix around the x-axis, the pitch rotation matrix around the y-axis, and the yaw rotation matrix around the z-axis are calculated respectively. Matrix multiplication is performed in the order of multiplying the yaw rotation matrix by the pitch rotation matrix and then by the roll rotation matrix to obtain the three-dimensional rotation matrix. The three-dimensional translation coordinates are extracted from the predicted pose to form a translation vector. The three-dimensional rotation matrix is ​​used as the upper left third-order submatrix, the translation vector is used as the right column vector, the bottom row vector is set as the zero vector, and scalar 1 is used to assemble a homogeneous transformation matrix. The camera coordinate system three-dimensional coordinates of each point in the crack feature points are traversed and a scalar 1 is added to the end to expand them into homogeneous coordinate vectors. The homogeneous coordinate vectors are then multiplied with the homogeneous transformation matrix to extract the global coordinates of the cracks in the sewer network.

10. A visual crack detection system for sewer pipe networks, characterized in that, A method for performing visual crack detection using a sewer network detection robot as described in any one of claims 1-9, comprising: The calculation module is used to collect the first video frame sequence of the sewer network through the detection robot and obtain the IMU acceleration data of the detection robot, and calculate the frame motion significance index based on the first video frame sequence and the IMU acceleration data; The filtering module is used to filter a second video frame sequence from the first video frame sequence according to the frame motion saliency index, and perform polar coordinate transformation and block division on the second video frame sequence to obtain expanded image blocks; The feature extraction module is used to perform edge enhancement and multi-scale feature extraction on the unfolded image block to obtain the crack probability mask and detection time. The coordinate transformation module is used to predict the transmission delay based on the detection time and perform feature point coordinate transformation on the crack probability mask to obtain the global coordinates of the crack in the sewer network.

Images