Pipeline robot and multi-sensor fusion method for detecting inner diameter defects and silt in pipelines

Abstract

The present invention discloses a pipeline robot and a pipeline inner diameter, defect and sludge detection method based on multi-sensor fusion, including a mobile chassis, a lifting mechanism, a pan-tilt, a sensor module, a pipeline inner diameter and defect detection method, and a sludge detection method. The sensor module includes a laser radar, an ultrasonic sensor and an RGB-D camera. The laser radar detects the portion without water on the upper part of the inner wall of the pipeline, and the ultrasonic sensor detects the portion with water on the lower part of the pipeline. The RGB-D camera is installed on the pan-tilt, and can achieve vertical lifting, pitch angle and yaw angle adjustment with the movement of the lifting mechanism and the pan-tilt. The pipeline inner diameter detection method is based on the laser radar, the ultrasonic sensor module and the RGB-D camera, the sludge detection method is based on the ultrasonic sensor, and the pipeline inner wall defect detection is based on the RGB-D camera. The multi-sensor fusion of the present invention improves the pipeline inner diameter detection accuracy, as well as the intelligence level and operation capability of the pipeline robot.

Description

Technical Field

[0001] The present invention relates to the technical field of pipeline inspection robots, and specifically to a pipeline robot and a multi-sensor fusion method for detecting inner diameter defects and silt in pipelines. Background Art

[0002] Drainage pipelines are of great significance to human society in terms of hygiene, environmental protection, urban planning, infrastructure, and water resource management, and play a key role in maintaining health, protecting the environment, promoting development, and improving the quality of life of residents. The accumulation of silt and inner wall defects in drainage pipelines will have a significant impact on aspects such as pipeline flow rate, transmission efficiency, and structural stability. The accumulation of silt reduces the effective cross-sectional area of the pipeline, causing a decrease in flow rate and blockage, while inner wall defects such as corrosion and cracks weaken the structural strength and stability of the pipeline, which may lead to leakage and pipeline damage. Therefore, regular maintenance and inspection are crucial for the normal operation and reliability of the pipeline system. Measuring the inner diameter of the pipeline, the deposition height of silt, envelope detection, and inner wall defect detection are the basic tasks of pipeline robot inspection. Currently, the main solutions for pipeline robots to measure the inner diameter of the pipeline and the deposition height of silt are CCTV imaging devices or lidar. The CCTV imaging device needs to be calibrated before each measurement in the pipeline inspection task, resulting in low efficiency; while the laser pulses emitted by the lidar will be interfered with when reaching the water, seriously affecting the accuracy of pipeline inner diameter measurement, and thus unable to accurately calculate the deposition height of silt. Therefore, it is of great practical value to design a multi-sensor fusion method for detecting the inner wall diameter, defects, and silt of a pipeline robot that can adapt to different water levels.

[0003] Patent CN201710874866.9 designed a silt detection system and a silt detection method, which cannot be used in the underground drainage pipeline environment due to engineering deployment problems. Therefore, the present invention designs a multi-sensor fusion method for detecting the inner wall diameter, defects, and silt of a pipeline robot, which is suitable for detecting the inner wall diameter, defects, and silt under different pipeline water levels, and is of great significance for the pipeline robot inspection task. Summary of the Invention

[0004] The present invention overcomes the problem that the existing pipeline robots cannot adapt to the detection of inner wall defects and silt under different pipeline water level environments, and designs a pipeline robot and a multi-sensor fusion method for detecting inner diameter defects and silt in pipelines, which can enable the pipeline robot to efficiently and quickly measure the inner diameter of the pipeline, measure the deposition height of silt, perform envelope detection, and detect the defects of the pipeline inner wall, meeting the requirements of the inspection task.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] Pipe robot, characterized in that: it includes a mobile chassis, an operating mechanism module and a sensor module, the operating mechanism module includes a servo, a pan-tilt and a lifting mechanism, the lifting mechanism is installed above the mobile chassis, the pan-tilt is installed on the lifting mechanism, the servo is installed on the front side of the mobile chassis, the sensor module includes an ultrasonic sensor, a lidar and an RGB-D camera, the lidar is installed on the upper side of the mobile chassis, the ultrasonic sensor is installed on the servo disk, and the RGB-D camera is installed on the pan-tilt.

[0007] Multi-sensor fusion method for detecting pipe inner diameter, defects and silt, characterized in that: it includes the following steps:

[0008] Step S1: Obtain lidar point cloud data;

[0009] Step S2: Perform circular fitting;

[0010] Step S3: Judge whether the circular fitting of the point cloud is successful. If successful, go to step S4; if failed, adjust the point cloud fitting angle range and perform circular fitting again, where the point cloud fitting angle range is from α degrees to 360 - α degrees;

[0011] Step S4: The servo rotates the ultrasonic sensor to complete the fitting angle;

[0012] Step S5: Construct ultrasonic data;

[0013] Step S6: Perform circular fitting, where the preliminary fitting angle range of the selected ultrasonic data is from 0 degrees to α degrees and from 360 - α degrees to 360 degrees;

[0014] Step S7: Judge whether the circular fitting of the ultrasonic data is successful. If successful, go to step S7; if failed, adjust the ultrasonic data fitting angle range and perform circular fitting again, where the final fitting angle range of the ultrasonic data is from β degrees to α degrees and from 360 - α degrees to 360 - β degrees;

[0015] Step S8: The RGB-D camera measures the pipe inner diameter;

[0016] Step S9: Fuse the RGB-D camera data, ultrasonic data and point cloud data to calculate the pipe inner diameter;

[0017] Step S10: Calculate the water surface height and the silt deposition height;

[0018] Step S11: Silt envelope detection;

[0019] Step S12: RGB-D camera defect detection.

[0020] As a preferred technical solution of the present invention: in steps S1 - S3, the inner diameter R of the pipeline inner wall is obtained by fitting the point cloud data constructed by the lidar 1 and the center position (x 1 , y 1 ) of the pipeline cross-section, specifically:

[0021] The lidar vertically integrated on the pipeline robot scans the cross-section of the drainage pipeline 360 degrees to construct the point cloud data of the pipeline cross-section. Circular fitting is performed on the point cloud data. First, an objective function is constructed, including initial estimates including the center coordinates (x 1 , y 1 ) and the inner diameter R 1 of the circle. The lidar point cloud data is input, and the point cloud data includes the coordinates (x i , y i ) of each point. The distance from each point in a round of point cloud data to the center of the circle is calculated

[0022]

[0023] Finally, the sum of the squares output of each distance deviation is returned and used as the objective function:

[0024]

[0025] Using the conjugate gradient method as the optimization algorithm, with the objective function, initial estimates, and point cloud data as inputs, the optimization process continuously iteratively adjusts the parameters to make the objective function reach the minimum value. Finally, the optimization result, that is, the parameters x 1 , y 1 and R 1 of the fitted circle, are obtained. The fitting angle range of the point cloud is dynamically adjusted. Assume that the first selection of 360-degree point cloud data for fitting fails, then the fitting angle range is reduced by 2α degrees, that is, the point cloud within the range of α degrees to 360 - α degrees is selected for fitting until the fitting is successful.

[0026] As a preferred technical solution of the present invention: in steps S4 - S7, the inner diameter R of the pipeline inner wall is obtained by fitting the distance measured by the ultrasonic sensor 2 and the center position (x 2 , y 2 ) of the pipeline cross-section, specifically:

[0027] The servo rotates, and the distance information returned by the ultrasonic sensor is read every time the servo rotates by θ degrees. Ultrasonic data is constructed based on the angle of rotation of the servo and the returned distance information

[0028] x i = d i .cos(angle_min + θ.i)

[0029] y i = d i .sin(angle_min + θ.i)

[0030] where i is the pulse count returned by the ultrasonic sensor during one full rotation of the servo, di is the distance information returned by this pulse, angle_min is the starting angle of the servo scan, θ is the angle of each rotation of the servo, and from the angle range finally fitted by the above lidar, the angle range of the preliminary circular fitting of the ultrasonic data is obtained. Assuming that the final fitting angle range of the lidar is from α degrees to 360 - α degrees, the preliminary angle range of the circular fitting of the ultrasonic data is from 0 degrees to α degrees and from 360 - α degrees to 360 degrees. A target function is constructed, including the initial estimated values including the center coordinates x 2 、y 2 and the inner diameter R of the circle 2 . The ultrasonic data is input, and the ultrasonic data includes the coordinates x i 、y i of each point. Calculate the distance from each point in a round of point cloud data to the center of the circle

[0031]

[0032] Finally, return the sum of the squares of each distance deviation output, and use it as the target function:

[0033]

[0034] Use the conjugate gradient method as the optimization algorithm, take the target function, the initial estimated values, and the ultrasonic data as inputs. The optimization process will continuously iterate and adjust the parameters to make the target function reach the minimum value. Finally, the optimization result is obtained, that is, the parameters x 2 、y 2 and R 2 of the fitted circle. Dynamically adjust the angle range of the ultrasonic data fitting. Assume that the first selection of the ultrasonic data fitting in the range of 0 degrees to α degrees and from 360 - α degrees to 360 degrees fails, then reduce the fitting angle range by 2β degrees, that is, select the ultrasonic data in the range of β degrees to α degrees and from 360 - α degrees to 360 - β degrees for fitting until the fitting is successful.

[0035] As a preferred technical solution of the present invention: In steps S8 - S9, use the RGB - D camera to measure the distance and fit to obtain the inner diameter R 3 of the inner wall of the pipeline and the center position (x 3 , y 3 ) of the pipeline cross - section. x is the horizontal position and y is the vertical position. Assume that the starting position of the RGB - D camera is (x c0 , y c0 ). Specifically:

[0036] The RGB-D camera adjusts the yaw angle to face the left side of the pipeline, and then adjusts the lifting mechanism to continuously rise from the lowest point with a step size of Δh. At the same time, record the discrete distance data {L 1 , L 2 , …, L N} measured by the RGB-D camera from the left pipe wall until the lifting mechanism reaches the highest point. Similarly, the RGB-D camera adjusts the yaw angle to face the right side of the pipeline, and then adjusts the lifting mechanism to continuously rise from the lowest point with a step size of Δh. At the same time, record the discrete distance data {R 1 , R 2 , …, R N} measured by the RGB-D camera from the left pipe wall until the lifting mechanism reaches the highest point. Assuming that the center position of the RGB-D camera remains unchanged in the horizontal position of the pipeline cross-section during the yaw angle adjustment process, the following system of equations can be obtained:

[0037]

[0038] According to the above 2N equations, select 5 of them in turn to calculate and obtain x c0 , y c0 , R 3 , x 3 and y 3 . There are groups of results, and then take the group of means to obtain the final x c0 , y c0 , R 3 , x 3 and y 3 results.

[0039] Finally, fuse the inner diameters of the pipeline inner wall obtained by the three sensors to obtain the pipeline inner diameter R.

[0040] R = w 1 *R 1 + w 2 *R 2 + w 3 *R 3

[0041] where w 1 , w 2 and w 3 are weight coefficients, and w 1 + w 2 + w 3 = 1.

[0042] As a preferred technical solution of the present invention: in step S9, the errors of the three sensors are all millimeter-level. Therefore, the weight of the ultrasonic data is 1 / 3, the weight of the point cloud data is 1 / 3, and the weight of the RGB-D camera data is 1 / 3.

[0043] As a preferred technical solution of the present invention: in steps S10 - S11, by the starting angle of the fitting angle range of the lidar point cloud and the parameters of the fitted circle, the height of the water surface in the pipeline is calculated.

[0044]

[0045] Where α’ is the starting angle of the fitting angle range of the point cloud data in the center coordinate system of the circle. By the starting angle and the ending angle of the fitting angle range of the ultrasonic data, the angle range where the silt is located can be obtained. From the ultrasonic data and the parameters of the fitted circle, the average height of the silt is calculated.

[0046]

[0047] Where d i is the distance from the coordinate returned by a beam of ultrasonic data in the center coordinate system of the circle to the center of the circle, angle_min is the starting angle of the servo scanning in the center coordinate system of the circle, θ’ is the angle of each rotation of the servo in the center coordinate system of the circle, and n is the number of ultrasonic pulses within the silt range. Based on the water surface height and the silt deposition height, silt envelope detection is performed to judge the silt deposition situation.

[0048]

[0049] By quantifying the silt deposition situation with b, a b threshold can be set to judge whether silt cleaning is required, meeting the task requirements of the pipeline robot inspection.

[0050] As a preferred technical solution of the present invention: in step S10, the theoretical height of the water surface is 0.5, and the silt deposition height is 0.13.

[0051] As a preferred technical solution of the present invention: in step S11, the silt deposition situation b is 0.26.

[0052] As a preferred technical solution of the present invention: in step S12, the RGB - D camera moves with the pan - tilt head and adjusts the angle. The data collected by the RGB - D camera includes depth information and RGB information. The RGB - D camera scans the inner wall of the pipeline and returns the RGB information and depth information of the inner wall of the pipeline, and performs real - time processing on the collected data. The real - time processing method used is the YOLOv5 algorithm. The dataset used in the YOLOv5 algorithm is the dataset collected in the pipeline. Before running the program in the actual pipeline, the collected data is processed so that the dataset can meet the requirements of the specific usage scenario. The processed dataset is used to train the model. When actually using the RGB - D camera for real - time detection, the YOLOv5 algorithm will display the defect information detected in each frame of the image and make marks on the picture.

[0053] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0054] Through the method of multi-sensor fusion, the present invention realizes the measurement of the inner diameter of the pipeline, the measurement of the sediment height of the silt, the envelope detection and the defect detection on the inner wall under different water level environments, thereby improving the operation ability and efficiency of the robot in the underground pipeline environment. Description of the Drawings

[0055] Figure 1 It is a functional framework diagram of the pipeline robot and the multi-sensor fusion pipeline inner diameter defect and silt detection method in the embodiment of the present invention;

[0056] Figure 2 It is a three-dimensional view of the pipeline robot in the embodiment of the present invention;

[0057] Figure 3 It is a schematic installation diagram of the lidar of the pipeline robot in the embodiment of the present invention;

[0058] Figure 4 It is a schematic installation diagram of the servo and ultrasonic sensor of the pipeline robot in the embodiment of the present invention;

[0059] Figure 5 It is a schematic installation diagram of the RGB-D camera, lifting mechanism and pan-tilt of the pipeline robot in the embodiment of the present invention;

[0060] Figure 6 It is a schematic diagram of the pipeline inner diameter detection method based on the RGB-D camera in the implementation of the present invention;

[0061] Figure 7 It is a schematic diagram of the pipeline inner diameter detection method based on the lidar in the embodiment of the present invention;

[0062] Figure 8 It is a schematic diagram of the pipeline inner diameter detection method based on the ultrasonic sensor module in the embodiment of the present invention;

[0063] Figure 9 It is a schematic diagram of the water surface height measurement of the multi-sensor fusion silt detection method of the pipeline robot in the embodiment of the present invention;

[0064] Figure 10 It is a schematic diagram of the silt deposition height measurement of the multi-sensor fusion silt detection method of the pipeline robot in the embodiment of the present invention;

[0065] Figure 11 It is a flow chart of the pipeline robot and the multi-sensor fusion pipeline inner diameter defect and silt detection method in the embodiment of the present invention.

[0066] List of Reference Numerals:

[0067] 1. Sensor module; 1-1. Ultrasonic sensor; 1-2. LiDAR; 1-3. RGB-D camera; 2. Operating mechanism module; 2-1. Servo; 2-2. Pan-tilt head; 2-3. Lifting mechanism; 3. Pipeline inner diameter detection module; 4. Silt detection module; 5. Inner wall defect detection module. Detailed implementation manners

[0068] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners:

[0069] To overcome the problem that in the prior art, pipeline robots cannot adapt to the detection of inner wall defects and silt under different pipeline water level environments, the present invention designs a pipeline robot and a multi-sensor fusion method for pipeline inner diameter defects and silt detection, which can enable the pipeline robot to efficiently and quickly measure the pipeline inner diameter, measure the deposition height of silt and perform envelope detection, as well as detect the defects of the pipeline inner wall, meeting the requirements of the inspection task.

[0070] Refer to Figures 1-5 As shown, the pipeline robot includes a mobile chassis, an operating mechanism module 2 and a sensor module 1. The operating mechanism module 2 includes a servo 2-1, a pan-tilt head 2-2 and a lifting mechanism 2-3. The lifting mechanism 2-3 is installed above the mobile chassis. The pan-tilt head 2-2 is installed on the lifting mechanism 2-3. The servo 2-1 is installed on the front side of the mobile chassis. The sensor module 1 includes an ultrasonic sensor, a LiDAR 1-2 and an RGB-D camera 1-3. The LiDAR 1-2 is installed on the upper side of the mobile chassis. The ultrasonic sensor is installed on the servo disk of the servo 2-1. The RGB-D camera 1-3 is installed on the pan-tilt head 2-2.

[0071] Among them, the mobile chassis is in a four-wheel drive form, and a lifting mechanism 2-3 is installed on its top. The pan-tilt head 2-2 has two degrees of freedom and is placed on the top of the lifting mechanism 2-3.

[0072] The sensor unit includes a LiDAR 1-2, an ultrasonic sensor 1-1 and an RGB-D camera 1-3. The LiDAR 1-2 is a single-line LiDAR. The LiDAR 1-2 is installed on the upper side of the mobile chassis and can perform 360-degree scanning on the pipeline inner wall to obtain point cloud data, realizing the detection of the part of the upper part of the pipeline inner wall without water.

[0073] The ultrasonic sensor 1-1 includes an ultrasonic sensor and a servo 2-1. The servo 2-1 is installed on the front side of the mobile chassis. The ultrasonic sensor is installed on the servo disk of the servo 2-1 and can realize the scanning and ranging of the pipeline inner wall in the vertical plane, realizing the detection of the part of the lower part of the pipeline with water.

[0074] The RGB-D cameras 1-3 are installed on the pan-tilt 2-2 and can achieve vertical lifting, as well as adjustment of the pitch angle and yaw angle, along with the movement of the lifting mechanism 2-3 and the pan-tilt 2-2.

[0075] There are also a pipeline inner diameter detection module 3, a silt detection module 4, and an inner wall defect detection module 5. The function of the pipeline inner diameter detection module 3 is to detect the inner diameter of the pipeline, and the inner diameter detection is achieved through the lidar 1-2, ultrasonic sensor 1-1, and RGB-D cameras 1-3. The function of the silt detection module 4 is to detect the deposition height and envelope information of the silt in the pipeline, and the silt detection module 4 is achieved through the servo 2-1, ultrasonic sensor 1-1, lidar 1-2, RGB-D cameras 1-3, pan-tilt 2-2, and lifting mechanism 2-3. The function of the inner wall defect detection module 5 is to detect the defects on the inner wall of the pipeline, display the defect information detected in each frame of the image on the terminal interface, and make marks on the pictures, which is achieved through the RGB-D cameras 1-3.

[0076] Refer to Figure 1 、 Figure 2 、 Figure 3 and Figure 4 As shown in, the silt detection module 4 is achieved through the lidar 1-2, servo 2-1, ultrasonic sensor 1-1, RGB-D cameras 1-3, pan-tilt 2-2, and lifting mechanism 2-3. The ultrasonic sensor 1-1 is installed on the servo 2-1 and rotates through the servo 2-1. The point cloud data obtained by the lidar 1-2 and the ultrasonic data constructed by the ultrasonic sensor 1-1 are respectively subjected to dynamic circular fitting, and the RGB-D camera 1-3 data, point cloud data, and ultrasonic data are fused to fit out the center and inner diameter of the pipeline. As shown in Figure 7 ,the water surface height of the pipeline is calculated from the starting angle and ending angle of the fitting angle range of the fitted pipeline information and point cloud data. As shown in Figure 8 ,the silt deposition height of the pipeline is calculated from the starting angle and ending angle of the fitting angle range of the fitted pipeline information and ultrasonic data.

[0077] Refer to Figure 1 、 Figure 2 and Figure 5 ,the inner wall defect detection module 5 is achieved through the RGB-D cameras 1-3 and the pan-tilt 2-2. The RGB-D cameras 1-3 are installed on the pan-tilt 2-2 and rotate through the pan-tilt 2-2. The RGB information and depth information obtained by the RGB-D cameras 1-3 are processed to frame out the defects inside the pipeline in the real-time image.

[0078] Refer to Figures 6-11 As shown in, the pipeline robot and the multi-sensor fusion pipeline inner diameter defect and silt detection method include the following steps:

[0079] Step S1: Obtain the point cloud data of lidar 1-2;

[0080] Step S2: Perform circular fitting;

[0081] Step S3: Determine whether the circular fitting of the point cloud is successful. If successful, proceed to Step S4; if failed, adjust the point cloud fitting angle range and perform circular fitting again, where the point cloud fitting angle range is from α degrees to 360 - α degrees;

[0082] Step S4: The servo 2-1 rotates the ultrasonic sensor 1-1 to complete the fitting angle;

[0083] Step S5: Construct ultrasonic data;

[0084] Step S6: Perform circular fitting, where the preliminary fitting angle range of the selected ultrasonic data is from 0 degrees to α degrees and from 360 - α degrees to 360 degrees;

[0085] Step S7: Determine whether the circular fitting of the ultrasonic data is successful. If successful, proceed to Step S7; if failed, adjust the ultrasonic data fitting angle range and perform circular fitting again, where the final fitting angle range of the ultrasonic data is from β degrees to α degrees and from 360 - α degrees to 360 - β degrees;

[0086] Step S8: The RGB-D camera 1-3 measures the inner diameter of the pipeline;

[0087] Step S9: Integrate the data of the RGB-D camera 1-3, the ultrasonic data, and the point cloud data to calculate the inner diameter of the pipeline;

[0088] Step S10: Calculate the water surface height and the silt deposition height;

[0089] Step S11: Silt envelope detection;

[0090] Step S12: Defect detection of the RGB-D camera 1-3.

[0091] The detection of the inner diameter of the pipeline is based on the lidar 1-2, the ultrasonic sensor 1-1, and the RGB-D camera 1-3

[0092] The detailed detection of the inner diameter of the pipeline based on the lidar 1-2 is as follows:

[0093] Refer to Figure 7 , in Steps S1 - S3, the inner diameter R of the pipeline inner wall and the center position (x 1 , y 1 , y 1 ) of the pipeline cross-section are obtained by fitting the point cloud data constructed by the lidar 1-2, specifically:

[0094] The lidar vertically integrated on the pipeline robot scans the cross-section of the drainage pipeline 1-2360 degrees, constructs the point cloud data of the pipeline cross-section, and performs circular fitting on the point cloud data. First, a target function is constructed, including the initial estimate values including the center coordinates (x 1 、y 1 ) and the inner diameter R of the circle 1 . The lidar 1-2 point cloud data is input, and the point cloud data includes the coordinates (x i 、y i ) of each point. Calculate the distance from each point in a round of point cloud data to the center of the circle

[0095]

[0096] Finally, return the sum of the squares of each distance deviation output, and use it as the target function:

[0097]

[0098] Use the conjugate gradient method as the optimization algorithm, take the target function, the initial estimate value and the point cloud data as the input, and continuously iterate and adjust the parameters in the optimization process to make the target function reach the minimum value. Finally, obtain the optimization result, that is, the parameters x 1 、y 1 and R 1 of the fitted circle. Because the laser pulses emitted by the lidar 1-2 will be interfered by water, circular fitting using 360-degree point cloud data in the drainage pipeline often fails. Therefore, the present invention dynamically adjusts the point cloud fitting angle range. Assuming that the first selection of 360-degree point cloud data for fitting fails, the fitting angle range is reduced by 2α degrees, that is, the point cloud within the range of α degrees to 360-α degrees is selected for fitting until the fitting is successful.

[0099] Refer to Figure 8 , the detailed detection of the pipeline inner diameter based on the ultrasonic sensor 1-1 is as follows:

[0100] In steps S4-S7, the inner diameter R 2 of the pipeline inner wall and the center position (x 2 , y 2 ) of the pipeline cross-section are obtained by ultrasonic sensor 1-1 ranging and fitting, specifically:

[0101] Since the ultrasonic sensor 1-1 is installed on the servo 2-1 and rotates through the servo 2-1, the distance information returned by the ultrasonic sensor 1-1 is read every time the servo 2-1 rotates by θ degrees, and ultrasonic data is constructed according to the rotation angle of the servo 2-1 and the returned distance information,

[0102] x i =d i.cos(angle_min + θ.i)

[0103] y i = d i .sin(angle_min + θ.i)

[0104] where i is the pulse count returned by the ultrasonic sensor 1-1 during one full rotation of the servo 2-1, di is the distance information returned by this pulse, angle_min is the starting angle of the servo 2-1 scan, θ is the angle of each rotation of the servo 2-1, and from the angle range finally fitted by the above lidar 1-2, the angle range for the preliminary circular fitting of the ultrasonic data is obtained. Assuming that the final fitting angle range of the single-line lidar 1-2 is from α degrees to 360 - α degrees, the preliminary angle range for the circular fitting of the ultrasonic data is from 0 degrees to α degrees and from 360 - α degrees to 360 degrees. A target function is constructed, including the initial estimates including the center coordinates x 2 , y 2 and the inner diameter R of the circle 2 . Input the ultrasonic data, where the ultrasonic data includes the coordinates x i , y i of each point. Calculate the distance from each point in a round of point cloud data to the center of the circle

[0105]

[0106] Finally, return the sum of the squares of each distance deviation output, and use it as the target function:

[0107]

[0108] Use the conjugate gradient method as the optimization algorithm. Take the target function, the initial estimates, and the ultrasonic data as inputs. The optimization process will continuously iterate and adjust the parameters to make the target function reach the minimum value. Finally, the optimization result is obtained, that is, the parameters x 2 , y 2 and R of the fitted circle 2 . Due to the accumulation of silt in the drainage pipe, which interferes with the fitting result of the ultrasonic data, it is also necessary to dynamically adjust the angle range for the ultrasonic data fitting. Assume that the first selection of the ultrasonic data fitting in the range from 0 degrees to α degrees and from 360 - α degrees to 360 degrees fails. Then reduce the fitting angle range by 2β degrees, that is, select the ultrasonic data in the range from β degrees to α degrees and from 360 - α degrees to 360 - β degrees for fitting until the fitting is successful.

[0109] Refer to Figure 6 and Figure 11 , use the RGB-D camera 1-3 to measure the inner diameter of the pipe and perform data fusion, so as to calculate the inner diameter of the pipe, as detailed below:

[0110] In steps S8 - S9, the inner diameter R of the pipeline wall is obtained by fitting the distance measurement of RGB - D cameras 1 - 3. 3 and the center position (x 3 , y 3 ) of the pipeline cross - section, where x is the horizontal position and y is the vertical position. Assuming the starting positions of RGB - D cameras 1 - 3 are (x c0 , y c0 ), specifically:

[0111] RGB - D cameras 1 - 3 adjust the yaw angle to face the left side of the pipeline, then adjust the lifting mechanisms 2 - 3 to continuously rise from the lowest point with a step size of Δh. At the same time, record the discrete distance data {L 1 , L 2 , …, L N} measured by RGB - D cameras 1 - 3 from the left pipe wall until the lifting mechanisms 2 - 3 reach the highest point. Similarly, RGB - D cameras 1 - 3 adjust the yaw angle to face the right side of the pipeline, then adjust the lifting mechanisms 2 - 3 to continuously rise from the lowest point with a step size of Δh, and record the discrete distance data {R 1 , R 2 , …, R N} measured by RGB - D cameras 1 - 3 from the left pipe wall until the lifting mechanisms 2 - 3 reach the highest point. Assuming that the center position of RGB - D cameras 1 - 3 remains unchanged in the horizontal position of the pipeline cross - section during the yaw angle adjustment, the following equations can be obtained:

[0112]

[0113] According to the above 2N equations, select 5 of them in turn, and x c0 , y c0 , R 3 , x 3 and y 3 can be solved. There are groups of results. Then take the mean of groups to obtain the final x c0 , y c0 , R 3 , x 3 and y 3 results.

[0114] Finally, the inner diameter R of the pipeline is obtained by fusing the inner diameters of the pipeline wall obtained by the three sensors.

[0115] R = w 1 * R 1 + w 2 * R 2 + w 3 * R 3

[0116] Among them, w 1 , w 2 and w 3 are weight coefficients, w 1 + w 2 + w 3 = 1, and its specific value is determined according to the detection accuracy of the sensor and experimental tests.

[0117] In step S9, the errors of all three sensors are in the millimeter level. Therefore, the weight of the ultrasonic data is 1 / 3, the weight of the point cloud data is 1 / 3, the weight of the RGB-D camera 1-3 data is 1 / 3, and the theoretical inner diameter of the pipeline is 1.

[0118] The method for silt detection is based on the ultrasonic sensor 1-1 and the lidar:

[0119] In steps S10 - S11, the height of the water surface inside the pipeline is calculated by the starting angle of the point cloud fitting angle range of the lidar 1-2 and the parameters of the fitted circle.

[0120]

[0121] Among them, α’ is the starting angle of the point cloud fitting angle range in the center coordinate system. By the starting angle and the ending angle of the ultrasonic data fitting angle range, the angle range where the silt is located can be obtained, and the average height of the silt is calculated from the ultrasonic data and the parameters of the fitted circle.

[0122]

[0123] Among them, d i is the distance from the coordinate returned by a beam of ultrasonic data to the center in the center coordinate system, angle_min is the starting angle of the servo 2-1 scanning in the center coordinate system, θ’ is the angle of each rotation of the servo 2-1 in the center coordinate system, and n is the number of ultrasonic pulses within the silt range. The silt envelope detection is performed based on the water surface height and the silt deposition height to judge the silt deposition situation.

[0124]

[0125] The silt deposition situation is quantified by b, and the b threshold can be set to judge whether silt cleaning is required to meet the task requirements of the pipeline robot inspection.

[0126] In step S10, the theoretical water surface height is 0.5, and the silt deposition height is 0.13.

[0127] In step S11, the silt deposition situation b is 0.26.

[0128] The method for detecting defects on the inner wall of the pipeline is detailed as follows:

[0129] Referring to Figure 1 , in step S12, the RGB-D camera 1-3 moves with the pan-tilt 2-2 and adjusts its angle. The data collected by the RGB-D camera 1-3 includes depth information and RGB information. The RGB-D camera 1-3 scans the inner wall of the pipeline and returns the RGB information and depth information of the inner wall of the pipeline, and performs real-time processing on the collected data. The real-time processing method used is the YOLOv5 algorithm. The dataset used in the YOLOv5 algorithm is the dataset collected in the pipeline. Before running the program in the actual pipeline, the collected data is processed so that the dataset can meet the requirements of the specific usage scenario. The processed dataset is used to train the model. When the RGB-D camera 1-3 is actually used for real-time detection, the YOLOv5 algorithm will display the defect information detected in each frame of the image and make marks on the picture.

[0130] The present invention realizes the measurement of the inner diameter of the pipeline, the measurement of the deposition height of silt, the envelope detection, and the defect detection on the inner wall under different water level environments through the method of multi-sensor fusion, thereby improving the operation ability and efficiency of the robot in the underground pipeline environment.

[0131] The above are only the preferred embodiments of the present invention, and are not any other form of limitation to the present invention. Any modification or equivalent change made according to the technical essence of the present invention still belongs to the scope protected by the present invention.

Claims

1. A multi-sensor fusion method for detecting inner diameter defects and silt in pipelines based on a pipeline robot. The pipeline robot includes a mobile chassis, an operating mechanism module (2), and a sensor module (1). The operating mechanism module (2) includes a servo motor (2-1), a pan-tilt (2-2), and a lifting mechanism (2-3). The lifting mechanism (2-3) is installed above the mobile chassis. The pan-tilt (2-2) is installed on the lifting mechanism (2-3). The servo motor (2-1) is installed on the front side of the mobile chassis. The sensor module (1) includes an ultrasonic sensor, a lidar (1-2), and an RGB-D camera (1-3). The lidar (1-2) is installed on the upper side of the mobile chassis. The ultrasonic sensor is installed on the servo disk of the servo motor (2-1). The RGB-D camera (1-3) is installed on the pan-tilt (2-2). It is characterized in that: It includes the following steps: Step S1: Obtain the lidar (1-2) point cloud data; Step S2: Perform circular fitting; Step S3: Judge whether the point cloud circular fitting is successful. If successful, proceed to Step S4; if failed, adjust the point cloud fitting angle range and perform circular fitting again, where the point cloud fitting angle range is from α degrees to 360 - α degrees; Step S4: The servo motor (2-1) rotates the ultrasonic sensor (1-1) to complete the fitting angle; Step S5: Construct ultrasonic data; Step S6: Perform circular fitting, where the preliminary fitting angle range of the selected ultrasonic data is from 0 degrees to α degrees and from 360 - α degrees to 360 degrees; Step S7: Judge whether the ultrasonic data circular fitting is successful. If successful, proceed to Step S7; if failed, adjust the ultrasonic data fitting angle range and perform circular fitting again, where the final fitting angle range of the ultrasonic data is from β degrees to α degrees and from 360 - α degrees to 360 - β degrees; Step S8: The RGB-D camera (1-3) measures the inner diameter of the pipeline; Step S9: Fuse the RGB-D camera (1-3) data, ultrasonic data, and point cloud data to calculate the inner diameter of the pipeline; Step S10: Calculate the water surface height and the silt deposition height; Step S11: Silt envelope detection; Step S12: RGB-D camera (1-3) defect detection.

2. The multi-sensor fusion method for detecting inner diameter defects and silt in pipelines according to claim 1, It is characterized in that: In steps S1 - S3, the inner diameter R of the pipeline inner wall is obtained by fitting the point cloud data constructed by the lidar (1 - 2). 1 and the position (x 1 , y 1 ) of the center of the pipeline cross-section, specifically as follows: The lidar (1-2) vertically integrated on the pipeline robot scans the cross-section of the drainage pipeline 360 degrees, constructs the point cloud data of the pipeline cross-section, and performs circular fitting on the point cloud data. First, a target function is constructed, including the initial estimate values including the center coordinates (x 1 , y 1 ) and the inner diameter R of the circle 1 . The lidar (1-2) point cloud data is input, and the point cloud data includes the coordinates (x i , y i ) of each point. Calculate the distance from each point in a round of point cloud data to the center of the circle Finally, return the sum of the squares of each distance deviation output, and use it as the objective function: Using the conjugate gradient method as the optimization algorithm, taking the objective function, the initial estimate, and the point cloud data as inputs, the optimization process iteratively adjusts the parameters to minimize the objective function, and finally obtains the optimization result, that is, the parameters x of the fitted circle 1 , y 1 and R 1 . By dynamically adjusting the point cloud fitting angle range, assuming that the first attempt to fit the 360-degree point cloud data fails, the fitting angle range is reduced by 2α degrees, that is, the point cloud within the range of α degrees to 360-α degrees is selected for fitting until the fitting is successful.

3. The multi-sensor fusion method for detecting inner diameter defects and silt in pipelines according to claim 1, It is characterized in that: In steps S4 - S7, the inner diameter R of the pipe wall is obtained by fitting the distance measured by the ultrasonic sensor (1 - 1). 2 and the position (x 2 , y 2 ) of the center of the pipe cross-section, specifically as follows: Rotate through the servo motor (2-1). The servo motor (2-1) reads the distance information returned by the ultrasonic sensor (1-1) every time it rotates by θ degrees. Construct ultrasonic data according to the rotation angle of the servo motor (2-1) and the returned distance information, x i = d i .cos(angle_min + θ﹒i) y i = d i .sin(angle_min + θ﹒i) where i is the pulse count returned by the ultrasonic sensor (1-1) during one full rotation of the servo (2-1), di is the distance information returned by the pulse, angle_min is the starting angle of the servo (2-1) scan, θ is the angle of each rotation of the servo (2-1), and the angle range of the initial circular fitting of the ultrasonic data is obtained from the angle range finally fitted by the lidar (1-2). Assuming that the final fitted angle range of the lidar (1-2) is from α degrees to 360-α degrees, the preliminary angle range of the circular fitting of the ultrasonic data is from 0 degrees to α degrees and from 360-α degrees to 360 degrees. A target function is constructed, including the initial estimated values including the center coordinates x 2 , y 2 and the inner diameter R of the circle 2 . The ultrasonic data is input, and the ultrasonic data includes the coordinates x i , y i of each point. Calculate the distance from each point in a round of point cloud data to the center of the circle Finally, return the sum of the squares of each distance deviation output, and use it as the objective function: Using the conjugate gradient method as the optimization algorithm, with the objective function, initial estimate, and ultrasonic data as inputs, the optimization process iteratively adjusts the parameters to minimize the objective function, and finally obtains the optimization result, that is, the parameters x of the fitted circle 2 , y 2 and R 2 , dynamically adjust the ultrasonic data fitting angle range. Assume that the first attempt to fit the ultrasonic data in the range of 0 degrees to α degrees and 360 - α degrees to 360 degrees fails. Then, reduce the fitting angle range by 2β degrees, that is, select the ultrasonic data in the range of β degrees to α degrees and 360 - α degrees to 360 - β degrees for fitting until the fitting is successful.

4. The multi-sensor fusion method for detecting inner diameter defects and silt in pipelines according to claim 1, It is characterized in that: In steps S8 - S9, the inner diameter R of the pipeline wall is obtained by ranging and fitting using the RGB - D camera (1 - 3). 3 and the position (x 3 , y 3 ) of the center of the pipeline cross - section, where x is the horizontal position and y is the vertical position. Assuming the starting position of the RGB - D camera (1 - 3) is (x c0 , y c0 ), specifically: The RGB-D camera (1-3) adjusts the yaw angle to face the left side of the pipeline, and then adjusts the lifting mechanism (2-3) to continuously rise from the lowest point with a step size of Δh. At the same time, record the discrete distance data {L 1 , L 2 , …, L N} measured by the RGB-D camera (1-3) from the left pipe wall until the lifting mechanism (2-3) reaches the highest point. Similarly, the RGB-D camera (1-3) adjusts the yaw angle to face the right side of the pipeline, and then adjusts the lifting mechanism (2-3) to continuously rise from the lowest point with a step size of Δh. At the same time, record the discrete distance data {R 1 , R 2 , …, R N} measured by the RGB-D camera (1-3) from the left pipe wall until the lifting mechanism (2-3) reaches the highest point. Assuming that the center position of the RGB-D camera (1-3) remains unchanged in the horizontal position of the pipeline cross-section during the yaw angle adjustment process, the following system of equations can be obtained: Based on the above 2N equations, select 5 of them in sequence, and x can be solved c0 , y c0 , R 3 , x 3 and y 3 , there are groups of results, and then take the mean of the groups to obtain the final x c0 , y c0 , R 3 , x 3 and y 3 results Finally, fuse the inner diameters of the pipeline inner wall obtained by the three sensors to obtain the pipeline inner diameter R, R = w 1 *R 1 +w 2 *R 2 +w 3 *R 3 where w 1 , w 2 and w 3 are weight coefficients, and w 1 + w 2 + w 3 = 1.

5. The multi-sensor fusion method for detecting inner diameter defects and silt in pipelines according to claim 4, characterized in that: In step S9, the errors of all three sensors are in the millimeter range. Therefore, the weight of the ultrasonic data is 1 / 3, the weight of the point cloud data is 1 / 3, and the weight of the RGB-D camera (1-3) data is 1 / 3.

6. The multi-sensor fusion method for detecting inner diameter defects and silt in pipelines according to claim 1, characterized in that: In steps S10 - S11, the height of the water surface inside the pipeline is calculated through the starting angle of the point cloud fitting angle range of the lidar (1-2) and the parameters of the fitted circle. Where α’ is the starting angle of the point cloud data fitting angle range in the center coordinate system of the circle. Through the starting angle and the ending angle of the ultrasonic data fitting angle range, the angle range where the silt is located can be obtained. From the ultrasonic data and the parameters of the fitted circle, the average height of the silt is calculated. where d i is the distance from the coordinate returned by a beam of ultrasonic data to the center of the circle in the center coordinate system, angle_min is the starting angle of the servo (2-1) scanning in the center coordinate system, θ’ is the angle of each rotation of the servo (2-1) in the center coordinate system, n is the number of ultrasonic pulses within the silt range. The silt envelope detection is performed based on the water surface height and the silt deposition height to determine the silt deposition situation The silt deposition situation is quantified by b. The b threshold can be set to determine whether silt cleaning is required, meeting the task requirements of pipeline robot inspection.

7. The multi-sensor fusion method for detecting inner diameter defects and silt in pipelines according to claim 6, characterized in that: In step S10, the theoretical height of the water surface is 0.5, and the silt deposition height is 0.

13.

8. The multi-sensor fusion method for detecting inner diameter defects and silt in pipelines according to claim 6, characterized in that: In step S11, the silt deposition situation b is 0.

26.

9. The multi-sensor fusion method for detecting inner diameter defects and silt in pipelines according to claim 1, characterized in that: In step S12, the RGB-D camera (1-3) moves with the pan-tilt (2-2) and adjusts the angle. The data collected by the RGB-D camera (1-3) includes depth information and RGB information. The RGB-D camera (1-3) scans the inner wall of the pipeline, returns the RGB information and depth information of the inner wall of the pipeline, and performs real-time processing on the collected data. The real-time processing method used is the YOLOv5 algorithm. The dataset used in the YOLOv5 algorithm is the dataset collected in the pipeline. Before running the program in the actual pipeline, the collected data is processed so that the dataset can meet the requirements of the specific usage scenario. The processed dataset is used to train the model. When the RGB-D camera (1-3) is actually used for real-time detection, the YOLOv5 algorithm will display the defect information detected in each frame of the image and make marks on the picture.

Images