A method for identifying water leakage in tunnel structures using robot dog detection

Abstract

This invention discloses a method for identifying water leakage in tunnel structures using a robot dog. First, an environmental perception algorithm based on the fusion of LiDAR point clouds and camera-based LiDAR point clouds is constructed to ensure the safety of photos taken by the inspection robot dog. Then, the mobile inspection robot dog is used to take photos of water leakage in the tunnel structure, and the images are preprocessed. Finally, features for identifying water leakage within the tunnel are constructed, and an improved Yolov8+HorNet+DASI+CBAM model is proposed to identify missing water leakage. This invention embeds the improved Yolov8+HorNet+DASI+CBAM model into a camera-equipped inspection robot to address the challenge of accessing underground structures by inspection workers. It accurately identifies water leakage in complex underground environments, eliminating safety hazards caused by water leakage and ensuring the operational safety of underground structures.

Description

Technical Field

[0001] This invention belongs to the field of underground tunnel structure operation monitoring, and relates to a method for identifying internal water leakage during the operation of underground structures such as tunnels, pipelines, and pipe corridors. Specifically, it relates to a method for identifying water leakage in tunnel structures using a machine dog-embedded deep convolution and semantic segmentation combined recognition model. Background Technology

[0002] The construction cycle of underground tunnel infrastructure is generally long. Early-built facilities not only show signs of aging, but the long-term effects of water seepage on the tunnel structure lead to corrosion and deterioration of the lining concrete and other structural materials. This causes concrete expansion, resulting in cracks and a decrease in the structural strength of the underground tunnel, potentially causing structural failure in severe cases. Simultaneously, water seepage into the tunnel structure can accumulate, damaging electrical, lighting, and ventilation equipment, causing equipment malfunctions or fires. Furthermore, the long-term presence of seepage in the tunnel structure softens the surrounding soil, increasing the risk of collapses and landslides, seriously threatening the safety of the surrounding environment. Summary of the Invention

[0003] To address the challenge of accurately identifying water leakage in underground structures such as tunnels, pipelines, and utility tunnels in complex environments, this invention provides a method for identifying water leakage in tunnel structures using a robot-based inspection system. This method leverages a camera-equipped inspection robot to overcome the limitations of human inspectors accessing underground structures. It proposes a semantic segmentation and recognition method for water leakage based on an improved Yolov8 model (Yolov8+HorNet+BoTNet+MSFN), and embeds this method into the camera-equipped inspection robot. This allows for accurate identification of water leakage in complex underground environments, eliminating safety hazards caused by water leakage and ensuring the operational safety of underground structures.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A method for identifying water leakage in tunnel structures for robot dog detection includes the following steps:

[0006] Step 1: Construct an environmental perception algorithm for inspection robot dogs based on the fusion of LiDAR point clouds and camera-based LiDAR point clouds, and realize the safe operation of inspection robot dogs in complex environments by fusing point cloud data with grid maps;

[0007] Step 2: Median filtering is used to enhance the moving images captured by the inspection robot dog. The images are then denoised using grayscale thresholding and particle swarm optimization image denoising algorithms. After denoising, the images are expanded to construct data features for identifying water leakage inside the tunnel using image scaling methods, thereby expanding the image dataset.

[0008] Step 3: Based on the Yolov8 model, add HorNet and BoTNet modules, embed the MSFN module, and build a tunnel structure leakage identification model based on the Yolov8+HorNet+BoTNet+MSFN model;

[0009] Step 4: Annotate the images of water leakage in the pipes inside the expanded underground tunnel to generate grayscale images containing green masked areas of water leakage on the walls and expansion joints; perform area recognition on the water leakage images identified by the tunnel structure water leakage recognition model based on the Yolov8+HorNet+BoTNet+MSFN model; and propose a water leakage area compensation function based on the correlation between the actual leakage area, the identified area, and the perimeter of the identified water leakage area to correct the actual water leakage area of ​​the tunnel structure.

[0010] Compared with the prior art, the present invention has the following advantages:

[0011] This invention provides a precise method for identifying water leakage in tunnel structures using robot dog detection. Based on computer vision recognition technology and combined with dynamic image data collected by robot dog detection, it offers two approaches: one for robot dog-based safety inspections in complex environments, and the other for identifying water leakage inside tunnel structures using a combination of deep convolution and attention mechanisms. Compared with existing methods, this approach has higher detection accuracy and is more suitable for identifying water leakage in tunnel structures in harsh and complex environments. Attached Figure Description

[0012] Figure 1 Flowchart of a method for identifying tunnel water leakage by equipping a robot dog with an improved Yolov8.

[0013] Figure 2 This is a diagram of the Yolov8+HorNet+BoTNet+MSFN model architecture.

[0014] Figure 3 Photo of the inspection robot dog.

[0015] Figure 4 Images of a camera and a mobile robotic arm for 8K shooting.

[0016] Figure 5 Moving images showing water leakage in pipes and underground structures.

[0017] Figure 6 Before and after photos processed by median filtering algorithm and grayscale threshold transformation (left: before processing, right: after processing).

[0018] Figure 7 Images processed by the Particle Swarm Optimization (PSO) algorithm for image denoising (left: before processing, right: after processing).

[0019] Figure 8 This image shows the result of manually marking water leakage using the Labelimg annotation software.

[0020] Figure 9 This is a grayscale image containing the masked area of ​​water leakage.

[0021] Figure 10 Images used to validate the validation set using the Yolov8+HorNet+BoTNet+MSFN model.

[0022] Figure 11 The figure shows the area correction fitting curve for the water leakage area inside the underground structure in the embodiment.

[0023] Figure 12 The image shows the leakage identification effect of the Yolov8+HorNet+DASI+CBAM model.

[0024] Figure 13 The image shows a comparison of the recognition performance of the Yolov8 model, Yolov8+HorNet model, Yolov8+CAFM model, Yolov8+BoTNet model, and Yolov8+BiFRN model in the embodiments.

[0025] Figure 14 This is a comparison chart of the recognition performance of the Yolov8+AKConv model, Yolov8+HorNet+BoTNet model, Yolov8+HorNet+BoTNet+CBAM model, Yolov8+HorNet+BoTNet+CAFM model, and Yolov8+HorNet+BoTNet+DASI model in the embodiments.

[0026] Figure 15 The image shows a comparison of the recognition performance of the Yolov8+HorNet+BoTNet+EMA model, Yolov8+HorNet+BoTNet+MDCR model, Yolov8+HorNet+BoTNet+MSFN model, Yolov8+HorNet+BoTNet+PPA model, and Yolov8+HorNet+BoTNet+CARAFE model in the embodiments.

[0027] Figure 16 Comparison chart showing the recognition performance of the Yolov8+ C2f_LSKA model, Yolov8+ DualConv model, Yolov8+ DWConv model, Yolov8+ EfficientNetV2 model, and Yolov8+ GhostConv model.

[0028] Figure 17The images show the recognition results of the Yolov8+ MobileOne model, Yolov8+ODConv model, Yolov8+RepConv model, Yolov8+RepLKNet model, and Yolov8+RepViTblock model in the embodiments. Detailed Implementation

[0029] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.

[0030] This invention provides a method for identifying water leakage in tunnel structures using a robotic dog-based detection system. First, an environmental perception algorithm based on the fusion of LiDAR point clouds and camera-based LiDAR point clouds is constructed to ensure the safety of photos taken by the inspection robotic dog. Then, the mobile inspection robotic dog is used to capture images of water leakage in the tunnel structure, and the images are preprocessed. Finally, features for identifying water leakage within the tunnel are constructed, and an improved Yolov8 + HorNet + DASI + CBAM model is proposed to identify missing water leakage. Figure 1 As shown, the specific steps include the following:

[0031] Step 1: Utilizing a motion distortion removal method for inspection robots based on leg-based odometry, an environmental perception algorithm for inspection robots is constructed based on the fusion of LiDAR point clouds and camera-based LiDAR point clouds. This algorithm, based on point cloud data fusion with a grid map, enables safe operation of the inspection robot in complex environments, addressing the problem of incomplete obstacle detection information in complex tunnel structures. Specific steps are as follows:

[0032] Step 11: Equip the quadrupedal inspection robot with a lidar to achieve a 360°×90° hemispherical ultra-wide-angle perception capability and acquire three-dimensional information of the surrounding environment in real time.

[0033] Steps 1 and 2: Construct a motion distortion removal method for the inspection robot dog based on leg-based odometry. The specific steps are as follows:

[0034] Step 121: Take the laser point cloud after the inspection robot dog scans 360° of the underground structure as one frame output, and record it. , Here, represents the start and end times of a single frame of LiDAR point cloud data, and represents the time interval between two frames. Let be the odometer's th... One and Data points, start time and end time The corresponding inspection robot dog changes position. , They are respectively:

[0035]

[0036] Step 122: To address the issue of poor signal transmission in underground tunnels, which causes discrepancies between two frames of lidar point cloud data and lacks corresponding consistent moments, linear data interpolation is performed on the lidar point cloud data using odometry data. Assume... Given odometer data at two adjacent times, and Interpolation is performed according to the following formula to obtain... The robot dog's pose corresponding to the LiDAR point cloud data at any given time:

[0037]

[0038] Steps one through three: Encapsulate the interpolation results into new LiDAR point cloud data in odometry coordinates. In one frame of LiDAR point cloud data... The pose corresponding to each laser point All of these are obtained through the linear data interpolation described above, p x p y These are the poses corresponding to the x-th and y-th laser points, respectively. and These are the coordinates before and after the transformation, respectively. The coordinate transformation calculation method is as follows:

[0039]

[0040] Steps 1-4: Repackage and output the motion-distorted LiDAR point cloud data to complete the LiDAR point cloud distortion removal process, thereby accurately reflecting the surrounding environment information of the inspection robot dog.

[0041] Step 13: Due to differences in measurement angles and data frequencies between LiDAR point cloud data and camera data, a distributed feature layer fusion is performed using point cloud information from both LiDAR and camera data. The specific steps are as follows:

[0042] Step 131: Convert the point cloud data from the two lidar systems into PCL format in the world coordinate system. Point cloud data is obtained, and the PCL point cloud library is used to merge the two into a point cloud data consisting of three-dimensional coordinate points. and use it as a three-dimensional space vector matrix Represented in the form of .

[0043] Step 1, 3, 2: Traverse the vector matrix, perform planar projection on all vectors, and calculate the horizontal distance from the projection point to the base coordinate system of the inspection robot dog. and the angle with the x-axis. .

[0044] Step 133: According to Calculate the current projection point in the predefined fused laser data .Index number in the array If this If a distance value already exists, select the distance value for this index according to the principle of minimum distance value and assign it to the specified value. middle.

[0045] Steps one, three, and four: After traversing all column vectors, assign values ​​to the... The array is returned as is. .

[0046] Step Two: Median filtering is used to enhance the moving images captured by the inspection robot. Noise reduction is then performed based on grayscale thresholding and particle swarm optimization image denoising algorithms. After noise reduction, image expansion methods such as photo illumination intensity, translation transformation, mirror transformation, and rotation transformation are used to construct tunnel internal water leakage identification data features, thereby expanding the image dataset. The specific steps are as follows:

[0047] Step 21: Median filtering is used to enhance the moving images captured by the inspection robot dog. Median filtering can not only reduce the image blurring effect and highlight the outline edge of the leaking water, but also enhance the color of the leaking water. It is suitable for image processing in complex environments such as underground tunnels. The principle is as follows: establish a square pixel grid centered on the target pixel, sort the n (n is an odd number) pixels according to the size of the gray value, and take the gray value corresponding to the median as the new gray value of the center pixel.

[0048] Step 22: Convert the acquired grayscale image into a binary image using grayscale thresholding. Thresholding separates the leaking water target from the background, which helps extract the shape and edge features of the leaking water target during subsequent image processing of the leaking water mask area, improving the model's leaking water recognition performance. The grayscale thresholding calculation formula is:

[0049]

[0050] Where T is the manually set grayscale threshold.

[0051] Steps 2 and 3: Denoising is applied to images captured during the movement of the inspection robot dog using a particle swarm optimization (PSO) image denoising algorithm. The fitness of the image particles represents the restoration effect under the current blur kernel parameters. During restoration, a convolution operation is typically used to apply the blur kernel to image patches, and the fitness between the restored image and the original image is calculated. The principle of the PSO image denoising algorithm is as follows:

[0052] Each particle updates its velocity in each iteration using the following formula:

[0053]

[0054] In the formula, Let i be the velocity of i in the t-th iteration; The inertial weight controls how well the particle stays in the current direction; and The learning factor controls the particle's trajectory toward its optimal position. and global optimal position The degree of trend; and Use random numbers to ensure exploratory nature; The optimal position found for particle i; The optimal position found in the particle swarm.

[0055] The position update formula in each iteration is as follows:

[0056]

[0057] In the formula, Let i be the current position of particle i in the t-th iteration, which is the current parameter of the fuzzy kernel; This is the updated speed.

[0058] During each update, the particle adjusts its velocity and position based on its own optimal position and the global optimal position, gradually approaching the optimal solution. This is achieved through inertia weighting. Control the exploration and development of particles to ensure the search for the optimal solution on a global scale.

[0059] Gradient is a primary indicator of grayscale changes in an image, reflecting image details, edges, etc. A larger gradient indicates richer details and relative clarity in an image region, while a smaller gradient indicates a more blurred region. The gradient is calculated for each image patch, and the standard deviation of that patch is also calculated. Regions with a large standard deviation, i.e., rich image details, are given less restoration power, while regions with a small standard deviation, i.e., blurred regions, require more restoration power.

[0060]

[0061] In the formula, Local weighted values; The gradient of the image patch; denoted as the standard deviation of the gradient.

[0062] In global weighting, a global structural similarity index is typically relied upon. Based on this index, the overall structural similarity between the restored image and the original image is calculated. A high global structural similarity index indicates a good restoration result, while a low index suggests the need for further restoration optimization.

[0063] The global structural similarity index can be used to measure the structural similarity between the restored image and the original image. The formula for the global structural similarity index is as follows:

[0064]

[0065] In the formula, and For image and The mean; and Let be the variance of the image and . For image and covariance; and These are two constants used to stabilize the denominator and prevent the denominator from being zero.

[0066] Based on the global structural similarity index, the overall structural similarity between the restored image and the original image can be calculated. A higher global structural similarity index indicates a better restoration result, while a lower index indicates that more restoration optimization is needed.

[0067] Based on local and global weighting, the final weighted fusion formula is:

[0068]

[0069] In the formula, The initial image after restoration; The weighted value is calculated based on local features; This is a weighted value calculated based on global structural similarity; This is the final image after restoration.

[0070] The processed image patches are stitched together to synthesize the restored image back into a complete image. This process is crucial because it determines the final result of the image restoration.

[0071] When processing overlapping areas, a weighted average method will be used to fuse the restored results, ensuring that there are no obvious transition issues at the seams. The weighting formula is as follows:

[0072]

[0073] In the formula, and The restored image of the overlapping left and right parts; and These are the weighting coefficients.

[0074] The processed image patches are stitched together according to their original positions in the image to reconstruct the complete image. This is usually achieved using simple array indexing, where image patches are sequentially filled into their corresponding positions. After stitching together all the image patches, a complete image is obtained.

[0075] Step 24: Based on the preprocessed images of water leakage, manually annotate the image data using Labelimg annotation software and output a label file in .txt format.

[0076] Step 25: Modify the illumination intensity of the collected image data and the generated .txt tag file. Based on the illumination intensity of the collected image data itself, modify the illumination intensity of the image data from 0.25 to 1.5 times.

[0077] Step 26: Based on the position of the pixels in the image data itself, change the pixel position of the image data within the range of 5 pixels to 25 pixels, changing the pixel position of the image data through horizontal mirroring, vertical mirroring, and horizontal and vertical mirroring.

[0078] Step 27: Based on the position of the pixels in the image data itself, rotate the pixel position of the image data within a range of 5 to 25 degrees, using the center pixel of the image as the center.

[0079] Step 3: Based on the Yolov8 model, add the HorNet general vision module to improve the spatial interaction of the object detection task; add the BoTNet module to reduce computational overhead and improve the attention focus of object recognition; embed the MSFN module to enhance the model's ability to enrich contextual information and the model's attention to the object, thereby constructing a tunnel structure seepage water identification method based on the Yolov8+HorNet+BoTNet+MSFN model. The specific steps are as follows:

[0080] Step 31: Add the HorNet general visual convolutional module to the Yolov8 backbone network to improve spatial interaction in the object detection task, where:

[0081] HorNet's general visual convolutional module is an efficient operation for achieving long-term and high-order spatial interactions. n Conv gated convolutions perform spatial interactions through simple operations such as convolution and fully connected layers.

[0082] In order to make g n Conv is able to capture long-term interactions and uses depthwise convolutions. The two implementations both use a 7×7 convolution kernel. The global filter (GF) multiplies the frequency domain features with a learnable global filter, processing half of the channels with the global filter and the other half with a 3×3 depthwise convolution.

[0083] g n Conv can replace the spatial blending layer in visual Transformers or modern CNNs. It follows the same meta-architecture as HorNet, where the basic blocks contain spatial blending layers and feedforward networks (FFNs). For object detection, the 3×3 convolutions following the top-down path are replaced with HorNet modules to improve spatial interaction for object detection tasks.

[0084] Step 32: Addressing the issues of significantly larger image sizes (640×640) in the general vision module of the YOLOv8 model for object detection and semantic segmentation, and the substantial training and inference overhead caused by the quadratic scale of self-attention memory and computation in the spatial dimension, a BoTNet module is embedded in the YOLOv8 model. This reduces parameters in semantic segmentation and object detection during tunnel structure image recognition, minimizing latency overhead. The specific steps are as follows:

[0085] Generate the vector Query, matching vector Key, and actual representation Value:

[0086] Input feature map Flattened into a sequence Generate Query, Key, and Value through linear transformation:

[0087]

[0088] In the formula, ; Dimensions for each attention head; The number of heads.

[0089] Calculate attention weights by scaling the dot product:

[0090]

[0091] In the formula, K is the scaling factor. T It is the transpose of K(Key), used to alleviate the problem of gradient vanishing due to excessively large dot product values.

[0092] The Q, K, and V values ​​are split into h heads, and the outputs of these heads are concatenated and linearly projected to obtain the final result.

[0093]

[0094] In the formula, Concat means concatenation, head is the header, and W0 is the output projection matrix. .

[0095] The calculation for each head is as follows:

[0096]

[0097] BoTNet uses relative position encoding to enhance spatial location awareness, and its formula is as follows:

[0098]

[0099] In the formula, The relative positional offset between positions i and j is achieved through learnable parameters.

[0100] BoTNet's Bottleneck structure can be formalized as follows:

[0101]

[0102] Expanded into three aspects: Conv 1×1 For convolution, MHSA is for processing and dimensionality-upgrading convolution.

[0103] Dimensionality reduction convolution is:

[0104]

[0105] In the formula, This is for the dimensionality reduction ratio.

[0106] MHSA processing is as follows:

[0107]

[0108] Upward convolution is:

[0109]

[0110] BoTNet reduces complexity through two strategies: 1) Reducing resolution. Used in deeper layers, such as when H=W=14, N=196. 2) Multi-head decomposition, distributing computation across h heads.

[0111] BoTNet retains the residual structure of ResNet, and its output is:

[0112]

[0113] In the formula, This is a combination of convolution and self-attention operations in BottleNeck.

[0114] Step 33: To enhance the nonlinear feature transformation in image processing, a multi-scale feedforward network (MSFN) is introduced. After each mixing block, the output of C2f is input into the MSFN to aggregate multi-scale features and enhance the nonlinear information transformation. The MSFN module uses two 1×1 convolutions to expand the feature channels with a scaling factor of 2. The input features are processed on two parallel paths, and a gating mechanism is introduced to enhance the nonlinear transformation through the element-wise product of the features from the two paths. In the lower path, depthwise convolutions are used for feature extraction. In the upper path, multi-scale expanded convolutions are used for multi-scale feature extraction. Two 3×3 expanded convolutions are used with scaling factors of 2 and 3, respectively.

[0115] Steps 3 and 4: To more intuitively and effectively demonstrate the recognition results, we select Precision, Recall, and F1-Score to evaluate the model's detection performance for a single class. mAP@0.5 represents the mean precision of all classes in the entire dataset.

[0116] The formula for calculating accuracy is as follows:

[0117]

[0118] Where TP represents the number of samples correctly identified as positive by the model, and FP represents the number of negative samples incorrectly identified as positive by the model.

[0119] The recall rate is calculated using the following formula:

[0120]

[0121] The formula for calculating the F1 score is as follows:

[0122]

[0123] The formula for calculating mAP@0.5 is shown below:

[0124]

[0125] N is the number of all classes in the entire dataset, AP i The recognition accuracy for each category.

[0126] This allows for the verification of the recognition accuracy of the proposed model.

[0127] Step 4: Annotate the images of water leakage in the expanded underground tunnel, generating grayscale images containing green masked areas of wall and expansion joint leakage; perform area recognition on the leakage images identified by the tunnel structure leakage recognition model based on Yolov8+HorNet+BoTNet+MSFN, and propose a leakage area compensation function based on the correlation between the actual leakage area, the identified area, and the perimeter of the identified leakage area to correct the actual leakage area of ​​the tunnel structure and improve the model's leakage recognition accuracy. The specific steps are as follows:

[0128] Step 41: Combine the on-site images of water leakage in the underground tunnel pipes after being annotated by Labelimg in Step 2 with the JSON tag file generated after annotation to generate a grayscale image containing green masked areas of water leakage on the wall and expansion joints.

[0129] Step 42: If the green area mask generated by combining the grayscale image with the JSON label file after annotation is pure green, then when defining the BGR range of the green area, the low threshold and high threshold are set to [B,G,R]=[0,255,0] for green processing.

[0130] Step 43: Load the grayscale image containing the green mask area of ​​wall leakage and expansion joint leakage, create a binary mask, with the green area as white (255) and other areas as black (0), and calculate the area of ​​the green area in pixels, that is, calculate the area of ​​the white (255) area in the binary mask, create a blank list to store the results, add the results to the list and convert them into a DataFrame, and save them as an Excel file.

[0131] Step 44: Load the leakage image after identification based on the Yolov8+HorNet+BoTNet+MSFN model, and create binary masks for the red and green regions by defining the BGR color range of the red and green regions. The function is used to create a binary mask, that is, for each pixel, if its color is within a given range (...). , ) lower limit and ( , If the mask position is within the upper limit, the corresponding mask position is set to white (255), otherwise it is set to black (0), which helps to isolate areas of a specific color.

[0132] Steps four and five: Locate the outlines of the red and green areas. The function is used to find the outlines of red and green regions in a binary image. The function retrieves the outermost outlines of the red and green regions and compresses redundant points in the horizontal, vertical and diagonal directions to save memory.

[0133] Step 46: Calculate the area and perimeter of the red and green areas. Calculate the area of ​​the region enclosed by the outlines of the red and green regions. Calculate the perimeter of the outlines of the red and green regions. Save the calculated areas and perimeters of the red and green regions as an Excel file. The area of ​​the green segmented region is denoted as... ; Calculate the areas of the red and green segmented regions. and perimeter .

[0134] Step 47: Calculate the area of ​​the identified red and green segmented regions. and the area of ​​the green segmented region before verification Find the difference to get the value. and compared with the perimeter obtained after identification. Establish connections and form a consensus on the difference. With perimeter The fitted curve.

[0135] Step 48: Evaluate the fit of the data using the sum of squared residuals, using the following formula:

[0136]

[0137] In the formula, This is the actual value; This is a predicted value; This represents the number of samples.

[0138] The smaller the sum of squared residuals, the closer the predicted value is to the actual value, the better the fitting effect, and the more accurately the fitted curve can express the relationship between the difference / perimeter / perimeter and the perimeter.

[0139] The relation is obtained as follows:

[0140]

[0141] In the formula, C, B1, B2, B3, B4, B5, B6, B7, and B8 are the parameters of the scatter plot and the fitted curve obtained by comparing the difference with the perimeter, respectively, and L is the perimeter.

[0142] The actual area was obtained by sorting. The relation is:

[0143]

[0144] Example:

[0145] Unitree Robotics' GO2 quadruped robot dog was used to identify water leakage inside underground structures. Equipped with a 4D LiDAR L1 sensor, the GO2 quadruped robot dog achieves a 360°×90° hemispherical ultra-wide-angle perception capability with an ultra-low blind zone and a minimum detection distance as low as 0.05m. This allows the GO2 robot dog to achieve blind-spot-free coverage, acquire real-time 3D information about the surrounding environment, and intelligently avoid obstacles during movement using an environmental perception algorithm that fuses LiDAR point clouds with camera-based LiDAR point clouds. This prevents collisions with obstacles and ensures the safety of both the robot dog and its surroundings.

[0146] It comes with the new Unitree GO App, offering omnidirectional ultra-wide-angle video transmission, real-time viewing of the captured footage, and built-in 4G and eSIM. The robot dog is equipped with an 8K camera, which mainly consists of a lens, image sensor, image processing chip, and storage module, etc. Figure 3 As shown. It also features a Phantom Technology robotic arm, which employs a three-in-one control system. This control system includes an STM32 microcontroller, an Arduino, a 51 microcontroller, and a Bluetooth module. It consists of an LDX-335MG anti-blocking servo motor, an LDF-06 anti-blocking servo motor, an LDX-218 high-precision digital servo motor, an alloy mechanical gripper, and an all-metal rotating chassis. Figure 4 As shown.

[0147] A mobile robotic camera was used to photograph water leaks at the connection points between pipes and the underground utility tunnel at Harbin Airport. A total of 307 images were collected, including images of water leaks on the tunnel walls and at expansion joints. Figure 5 As shown.

[0148] Based on step two, image data preprocessing is performed on the captured moving images. The effects before and after image processing are as follows: Figure 7 , Figure 8 As shown.

[0149] Based on the image data expansion processing in steps two and three, four transformation methods were selected: 0.25 to 0.5 times illumination intensity transformation, 15-degree horizontal translation transformation, horizontal mirror transformation, and rotation transformation within the range of [-10, 10]. These four transformation methods were combined to expand the dataset for image recognition of underground integrated pipe gallery pipes and water leakage. The expanded dataset contains 3458 images of wall leakage and expansion joint leakage. These 3458 images were divided into 3112 training images and 346 verification images at a 9:1 ratio.

[0150] The Yolov8+HorNet+BoTNet+MSFN model constructed in step three was trained and validated. To verify the performance gains brought to the model by the three optimization strategies of adding the HorNet convolutional module, the BOTNet module, and the MSFN module, an ablation experiment was designed on the dataset, as shown in Table 2.

[0151] Table 2 Ablation Experiment Results

[0152]

[0153] As shown in Table 1, adding the HorNet module to the YOLOv8 model improves the accuracy of wall leakage by 1.1% compared to the YOLOv8 model, decreases the accuracy of expansion joint leakage by 3%, and improves mAP@0.5 by 2%. Adding the HorNet and BoTNet modules to the YOLOv8 model respectively improves the accuracy of wall leakage by 1.8%, increases the accuracy of expansion joint leakage by 5%, and improves mAP@0.5 by 3.2%. This indicates that the addition of the BoTNet and HorNet modules helps improve network performance. Finally, adding the MSFN module to the YOLOv8 + HorNet + BoTNet model improves the accuracy of wall leakage by 1.4%, increases the accuracy of expansion joint leakage by 5.6%, and improves mAP@0.5 by 3.5% compared to YOLOv8.

[0154] To verify the identification effect of the proposed method in the leakage area, it was compared with the identification effect of 25 traditional models. First, the convergence effect of the models was compared and analyzed using precision, recall, mAP@0.5, and mAP@[0.5:0.95]. The comparison results are as follows. Figures 13-17 As shown in Table 3.

[0155] Table 3 Comparison of training set AP and mAP values

[0156]

[0157] By observing the loss curves of 25 models on the training set of the dataset, we can identify the Yolov8 model, Yolov8+HorNet model, Yolov8+CAFM model, Yolov8+BoTNet model, Yolov8+BiFRN model, Yolov8+AKConv model, Yolov8+HorNet+BoTNet model, Yolov8+HorNet+BoTNet+CBAM model, Yolov8+HorNet+BoTNet+CAFM model, Yolov8+HorNet+BoTNet+DASI model, Yolov8+HorNet+BoTNet+EMA model, Yolov8+HorNet+BoTNet+MDCR model, Yolov8+HorNet+BoTNet+MSFN model, Yolov8+HorNet+BoTNet+PPA model, Yolov8+HorNet+BoTNet+CARAFE model, and Yolov8+ The loss values ​​of the C2f_LSKA model, YOLOv8+DualConv model, YOLOv8+DWConv model, YOLOv8+EfficientNetV2 model, YOLOv8+GhostConv model, YOLOv8+MobileOne model, YOLOv8+ODConv model, YOLOv8+RepConv model, YOLOv8+RepLKNet model, and YOLOv8+RepViTblock model all remained low. Among them, the classification loss and confidence loss values ​​of the YOLOv8+HorNet+BoTNet+MSFN model were slightly lower than those of YOLOv8. Comparisons of the Precision curve, Recall curve, mAP@0.5 curve, and mAP@[0.5:0.95] curve show that the YOLOv8+HorNet+BoTNet+MSFN model performs better in both precision and recall, with a smoother convergence curve and better convergence performance. Furthermore, the mAP@0.5 and mAP@[0.5:0.95] values ​​are higher for the Yolov8+HorNet+BoTNet+MSFN model.

[0158] Experimental results show that, compared with the Yolov8+HorNet+BoTNet+MSFN model, other models are not ideal in detecting wall leakage and expansion joint leakage. Compared with the Yolov8 model, the Yolov8+HorNet+BoTNet+MSFN model improves accuracy by 1.4% for wall leakage, 5.6% for expansion joint leakage, and 3.5% for mAP@0.5. In conclusion, the Yolov8+HorNet+BoTNet+MSFN model performs best overall.

[0159] Table 4 Comparison of F1 values

[0160]

[0161] According to the accuracy formula, although the Yolov8+HorNet+BoTNet+MSFN model does not have the highest precision and R-value compared to other models, its calculated F1 score exceeds that of other models, with an F1 score 2.826% higher than that of the Yolov8 model, indicating that this model is the best.

[0162] Table 5 Comparison of validation set AP and mAP values

[0163]

[0164] Experimental results show that, compared to the Yolov8 model, the Yolov8+HorNet+BoTNet+MSFN model improves performance by 1.6% in wall leakage, 5.6% in expansion joint leakage, and 3.6% in mAP@0.5. In conclusion, the Yolov8+HorNet+BoTNet+MSFN model performs best overall.

[0165] Based on steps four one through four three, after labeling the images with Labelimg, the labeled images of the on-site verification set of water leakage in the internal structure of the underground utility tunnel are combined with the generated JSON tag file to generate a grayscale image containing green masked areas of wall leakage and expansion joint leakage, such as... Figure 9 As shown.

[0166] Using steps 4.4 to 4.6, the algorithm for identifying water leakage in underground pipe corridors based on the Yolov8+HorNet+BoTNet+MSFN model is validated on the validation set, and the original background image is changed to black. The image after validation by the Yolov8+HorNet+BoTNet+MSFN recognition model is shown below. Figure 10 As shown.

[0167] After image 10 is verified by the recognition model, the area of ​​the water leakage area inside the pipe gallery in the image is corrected according to steps 46 to 48, forming a relationship between the difference S2-S1 and the perimeter. The fitted curve, such as Figure 11 As shown in Table 6, the sum of squared residuals of the fitted curve obtained above is 0.00675.

[0168] Table 6. Difference / Perimeter / Perimeter vs. Parameters of the Fitted Curve

[0169]

Claims

1. A tunnel structure water leakage identification method for a robot dog detection, characterized by The method includes the following steps: Step 1: Construct an environmental perception algorithm for inspection robot dogs based on the fusion of LiDAR point clouds and camera-based LiDAR point clouds, and realize the safe operation of inspection robot dogs in complex environments by fusing point cloud data with grid maps; Step 2: Median filtering is used to enhance the moving images captured by the inspection robot dog, and noise reduction is performed on the images based on grayscale thresholding and particle swarm image denoising algorithm. After noise reduction, the images are used to construct data features for identifying water leakage inside the tunnel using image expansion methods, thereby expanding the image dataset. Step 3: Based on the Yolov8 model, add HorNet and BoTNet modules, embed the MSFN module, and build a tunnel structure leakage identification model based on the Yolov8+HorNet+BoTNet+MSFN model; Step 4: Annotate the images of water leakage in the pipes inside the expanded underground tunnel to generate grayscale images containing green masked areas of water leakage on the walls and expansion joints; perform area recognition on the water leakage images identified by the tunnel structure water leakage recognition model based on the Yolov8+HorNet+BoTNet+MSFN model; and propose a water leakage area compensation function based on the correlation between the actual leakage area, the identified area, and the perimeter of the identified water leakage area to correct the actual water leakage area of ​​the tunnel structure. 2.The tunnel structure water leakage identification method for a robot dog detection, according to claim 1, wherein The specific steps of step one are as follows: Step 11: Equip the quadrupedal inspection robot dog with a lidar to achieve a 360°×90° hemispherical ultra-wide-angle perception capability and acquire three-dimensional information of the surrounding environment in real time. Steps 1 and 2: Construct a motion distortion removal method for inspection robot dogs based on leg-based odometers; Step 13: Perform distributed feature layer fusion using point cloud information from LiDAR point cloud data and camera data.

3. The method for identifying water leakage in tunnel structures based on robot dog detection according to claim 2, characterized in that... The specific steps of steps one and two are as follows: Step 121: Take the laser point cloud after the inspection robot dog scans 360° of the underground structure as one frame output, and record it. , Here, represents the start and end times of a single frame of LiDAR point cloud data, and represents the time interval between two frames. Let be the odometer's th... Individual and Data points, start time and end time The corresponding inspection robot dog changes position. , They are respectively: Step 122: Assumption Given odometer data at two adjacent times, and Interpolation is performed according to the following formula to obtain... The robot dog's pose corresponding to the LiDAR point cloud data at any given time: Steps one through three: Encapsulate the interpolation results into new LiDAR point cloud data in odometry coordinates. In one frame of LiDAR point cloud data... The pose corresponding to each laser point All values ​​are obtained through interpolation in steps one through two. , The first The, the The pose corresponding to each laser point and These are the coordinates before and after the transformation, respectively. The coordinate transformation calculation method is as follows: Steps 1-4: Repackage and output the motion-distorted LiDAR point cloud data to complete the LiDAR point cloud distortion removal process, thereby accurately reflecting the surrounding environment information of the inspection robot dog.

4. The method for identifying water leakage in tunnel structures based on robot dog detection according to claim 2, characterized in that... The specific steps of steps one and three are as follows: Step 131: Convert the point cloud data from the two lidar systems into PCL format in the world coordinate system. Point cloud data is obtained, and the PCL point cloud library is used to merge the two into a point cloud data consisting of three-dimensional coordinate points. and use it as a three-dimensional spatial vector matrix The formal representation; Step 1, 3, 2: Traverse the vector matrix, perform planar projection on all vectors, and calculate the horizontal distance from the projection point to the base coordinate system of the inspection robot dog. and with Angle between axes ; Step 133: According to Calculate the current projection point in the predefined fused laser data Index number in the array If this If a distance value already exists, select the distance value for this index according to the principle of minimum distance value and assign it to the specified value. middle; Steps one, three, and four: After traversing all column vectors, assign values ​​to the... The array is returned as is. .

5. The method for identifying water leakage in tunnel structures based on robot dog detection according to claim 1, characterized in that... The specific steps of step two are as follows: Step 21: Use median filtering to enhance the moving images captured by the inspection robot dog; Step 22: Convert the acquired grayscale image into a binary image using grayscale thresholding. Steps 2 and 3: Denoise the images captured during the movement of the inspection robot dog based on the particle swarm image denoising algorithm; Step 24: Based on the preprocessed images of water leakage, manually annotate the image data using Labelimg annotation software and output a label file in .txt format; Step 25: Modify the light intensity of the collected image data and the generated .txt tag file. Based on the light intensity of the collected image data itself, modify the light intensity of the image data from 0.25 to 1.5 times. Step 26: Based on the position of the pixels in the image data itself, change the pixel position of the image data within the range of 5 pixels to 25 pixels, and change the pixel position of the image data by horizontal mirroring, vertical mirroring and horizontal and vertical mirroring. Step 27: Based on the position of the pixels in the image data itself, rotate the pixel position of the image data within a range of 5 to 25 degrees, using the center pixel of the image as the center.

6. The method for identifying water leakage in tunnel structures based on robot dog detection according to claim 1, characterized in that... The specific steps of step three are as follows: Step 31: Add the HorNet general visual convolutional module to the Yolov8 backbone network to improve spatial interaction in the object detection task; Step 32: Embed the BoTNet module into the Yolov8 model; Step 33: Introduce a multi-scale feedforward network (MSFN). After each mixing block, the output of C2f is input into the MSFN. The MSFN module uses two 1×1 convolutions to expand the feature channels with an expansion ratio of 2. The input features are processed on two parallel paths, and a gating mechanism is introduced to enhance the nonlinear transformation by the element-wise product of the features from the two paths. In the lower path, depthwise convolutions are used for feature extraction, and in the upper path, multi-scale expanded convolutions are used for multi-scale feature extraction. Two 3×3 expanded convolutions are used with expansion ratios of 2 and 3, respectively. Steps 3 and 4: To more intuitively and effectively demonstrate the recognition results, we select Precision, Recall, and F1-Score to evaluate the model's detection performance for a single class. We select mAP@0.5 to represent the mean precision of all classes in the entire dataset.

7. The method for identifying water leakage in tunnel structures based on robot dog detection according to claim 6, characterized in that... The specific steps of step 3.2 are as follows: Generate the vector Query, matching vector Key, and actual representation Value: Input feature map Flattened into a sequence Generate Query, Key, and Value through linear transformation: In the formula, ; Dimensions for each attention head; Number of heads; Calculate attention weights by scaling the dot product: In the formula, Scaling factor for transpose; Will , , Split into The outputs of one or more heads are concatenated and then linearly projected to obtain the final result: In the formula, The meaning is splicing. For the head, To output the projection matrix, ; Each head The calculation is as follows: BoTNet uses relative position encoding to enhance spatial location awareness, and its formula is as follows: In the formula, For position and The relative positional offset between them; BoTNet's Bottleneck structure can be formalized as follows: This can be broken down into three aspects: For convolution, MHSA is for processing and increasing the dimensionality of convolution; Dimensionality reduction convolution is: In the formula, To reduce the dimensionality ratio; MHSA processing is as follows: Upward convolution is: BoTNet reduces complexity through two strategies: 1) reducing resolution; 2) multi-head decomposition, distributing computation across... Size; BoTNet retains the residual structure of ResNet, and its output is: In the formula, This is a combination of convolution and self-attention operations in BottleNeck.

8. The method for identifying water leakage in tunnel structures based on robot dog detection according to claim 1, characterized in that... The specific steps of step four are as follows: Step 41: Combine the on-site images of water leakage in the underground tunnel pipes after being annotated by Labelimg in Step 2 with the JSON tag file generated after annotation to generate a grayscale image containing green mask areas of water leakage on the wall and water leakage at the expansion joint; Step 42: If the green area mask generated by combining the grayscale image with the JSON label file after annotation is pure green, then when defining the BGR range of the green area, the low threshold and high threshold are set to [B,G,R]=[0,255,0] for green processing; Step 43: Load the grayscale image containing the green mask area of ​​wall leakage and expansion joint leakage, create a binary mask, with green areas as white and other areas as black, and calculate the area of ​​the green area in pixels, that is, calculate the area of ​​the white area in the binary mask, create a blank list to store the results, add the results to the list and convert them into a DataFrame, and save them as an Excel file; Step 44: Load the water leakage image after identification based on the Yolov8+HorNet+BoTNet+MSFN model, and create binary masks for the red and green regions by defining the BGR color range of the red and green regions; Steps four and five: Locate the outlines of the red and green areas; Step 46: Calculate the area and perimeter of the red and green regions. Save the calculated areas and perimeters of the red and green regions as an Excel file. The area of ​​the green segmented region is denoted as... ; Calculate the areas of the red and green segmented regions. and perimeter ; Step 47: Calculate the area of ​​the identified red and green segmented regions. and the area of ​​the green segmented region before verification Find the difference to get the difference value. and compared with the perimeter obtained after identification. Establish connections and form a consensus on the difference. With perimeter The fitted curve; Step 48: Evaluate the fit of the data using the sum of squared residuals, using the following formula: In the formula, This is the actual value; This is a predicted value; The number of samples; The relation is obtained as follows: In the formula, , , , , , , , , The parameters are shown in the scatter plot of the difference versus the perimeter and the parameters of the fitted curve, respectively. Perimeter; The actual area was obtained by sorting. The relation is: 。

Images