Real-time autonomous identification method for typical defects in municipal pipelines

Abstract

The present invention relates to the field of municipal pipeline detection technology, and in particular to a method for real-time autonomous identification of typical defects in municipal pipelines. The method comprises the following steps: S1, collecting and acquiring original sonar data of the municipal pipeline environment through a mechanical scanning sonar mounted on a crawling robot; S2, real-time analysis of the original sonar data obtained in step S1 to obtain a data matrix containing beam angles and sound intensity values ​​of sampling points; S3, real-time generation of a mechanical scanning sonar image from the sonar data matrix obtained in step S2; S4, denoising the mechanical scanning sonar image; S5, outputting the denoised image to a target monitoring model for defect identification. The method can not only realize concise and effective denoising and pipeline contour detection, but also realize accurate identification of various defects in municipal pipelines.

Description

Technical Field

[0001] The invention relates to the technical field of municipal pipeline detection, and in particular to a real-time autonomous recognition method for typical defects of municipal pipelines. Background Art

[0002] With the expansion of urban scale and the continuous increase of urban population, pipelines have been in peak high-load operation for a long time, causing a series of problems such as pipeline rupture and damage. Therefore, as an important infrastructure of the city, it is necessary to regularly inspect the safety of the pipeline structure itself during long-term operation. The existing maintenance method is to send experienced municipal workers to enter the pipeline for on-site inspection. However, in order to improve the safety of operations, work efficiency and reduce costs. More and more pipeline inspection robots are being used. These robots can automatically navigate, collect data and transmit it back to the analysis center in real time, which can improve the efficiency and safety of inspection.

[0003] Since there are many types of typical defects in municipal pipelines and the received data is large, the recognition accuracy and real-time requirements are high. Traditional machine learning methods or other methods are difficult to meet the requirements. Deep learning convolutional neural networks are the most appropriate. Although CNN has developed to this day and there are many mature networks, it is a major technical difficulty to achieve a high recognition accuracy when the input data is acoustic rather than optical images, and there are many image categories to be recognized and real-time requirements. Not only that, the representation of different types of defects in municipal pipelines on sonar images is very different. Although foreign body insertion and protrusion, depression and hole are completely different types of defects, their representations on acoustic images are very similar. It is a major technical challenge to effectively identify these two different levels of defects.

[0004] Most pipeline defect detection methods use electromagnetic, acoustic wave and other detection methods. For example, corrosion and leakage of submarine oil and gas pipelines, as well as pipeline layout and direction, are usually detected remotely from outside the pipeline using sensors, but it is impossible to achieve precise extraction of internal features of the pipeline.

[0005] For the detection of defects in pipelines, CCTV detection methods are currently mostly adopted, that is, using optical cameras to obtain the inner wall conditions of the pipeline. However, in municipal pipelines filled with water, CCTV detection methods have great limitations, because light scatters and attenuates strongly in water, and the reflection ability of targets in the water at a slightly longer distance is sharply weakened, so CCTV detection methods are not good at extracting features in water. If it is in muddy water conditions, CCTV detection methods may even be completely incapable.

[0006] The propagation ability of sound waves in water is much stronger than that of light, and it is also less affected by water turbidity. The existing target detection methods based on sonar images have the following defects when detecting defects in pipelines: First, the complex and changeable underwater environment causes serious interference from reverberation and clutter noise, and underwater sound waves will attenuate. Sonar images are usually difficult to accurately determine the boundaries of the target area or the boundaries are incomplete, the target grayscale is low, and the information loss is serious; secondly, for the detection environment of silt accumulation and high water level in pipeline tunnels, sonar image data is difficult to obtain and the cost is high, so there are fewer data sets. In addition, the texture features presented by pipeline defects are also relatively few, making it difficult to implement defect detection and the accuracy is not high. There are many impurities similar to the target in the pipeline, which will lead to a high misidentification rate and leakage rate.

[0007] In summary, there is currently no universal mechanical scanning sonar image target detection method for municipal pipelines. Summary of the invention

[0008] The purpose of the present invention is to overcome the above-mentioned defects of the prior art and propose a real-time autonomous identification method for typical defects of municipal pipelines, which can not only realize concise and effective denoising and pipeline contour detection, but also realize accurate identification of various defects of municipal pipelines.

[0009] The technical solution of the present invention is: a real-time autonomous identification method for typical defects of municipal pipelines, which includes the following steps:

[0010] S1. Collect and obtain the original sonar data of the municipal pipeline environment through the mechanical scanning sonar mounted on the crawler robot;

[0011] S2, performing real-time analysis on the original sonar data obtained in step S1 to obtain a data matrix including beam angles and sound intensity values ​​of sampling points;

[0012] S3, generating a mechanical scanning sonar image in real time using the sonar data matrix obtained in step S2;

[0013] S4, performing denoising processing on the mechanical scanning sonar image;

[0014] S5. Output the denoised image to the target monitoring model for defect identification.

[0015] In the present invention, the circular highlight part in the sonar image is an invalid feature, and the pixel values ​​whose distance from the image center is less than a specified threshold are all set to zero. The specified threshold is related to the range of the mechanical scanning sonar.

[0016] Step S2 includes the following specific steps:

[0017] S2.1, inspecting and removing the original sonar data packets obtained in step S1;

[0018] S2.2. Analyze and process the data packets that have been inspected and eliminated.

[0019] In the above step S2.1, the received original sonar data packets are processed one by one, and the first byte '0xfe' of the ping header is first found;

[0020] If '0xfe' cannot be found, the next byte is searched. If the byte cannot be found within the specified time, the data packet is considered to be incorrect and the entire data packet is discarded.

[0021] If the '0xfe' byte is found in the ping header, get the full byte length of the data portion from the ping header, and then compare the actual data length with the full length:

[0022] If the actual length is less than the complete length, it indicates that the ping is the last ping in the data packet and is incomplete, so several zero values ​​are padded at the end to make it reach the complete data length; if the actual length is equal to the complete length and the ping is not the last ping, jump to the next ping for processing; if the actual length is equal to the complete length and the ping is the last ping, the entire data packet has been processed.

[0023] In the above step S2.2, the data packets are processed one by one by ping:

[0024] Read the beam angle and data length information in the ping header. The beam angle information is stored in the 5th and 6th bytes, which are the lower 7 bits and higher 7 bits of the beam angle respectively. The calculation formula of the beam angle is:

[0025] angle=((AngleHi&0x7f)<<7)|((AngleLo&0x7f));

[0026] Among them, AngleHi represents the high byte of the beam angle; AngleLo represents the low byte of the beam angle;

[0027] The data length information is stored in the 3rd and 4th bytes, which are the lower 7 bits and higher 7 bits of the data length respectively. The calculation formula for the data length is:

[0028] data_length=((DataHi&0x7f)<<7)|(DataLo&0x7f);

[0029] Among them, DataHi represents the high byte of data length; DataLo represents the low byte of data length;

[0030] Each sampling point data occupies 4 bits, and one byte contains two sampling point data. The calculation formulas for the two sampling point data are:

[0031] first_point=echo_data&0xf0, second_point=((echo_data&0xf0)>>4);

[0032] Among them, echo_data represents a byte containing two sampling point data;

[0033] Create a list for each ping data. The first element of the list is the beam angle angle read by the above method. The following elements are the sampling point data read from near to far from the sonar transducer by the above method. Each ping data in the data packet is spliced ​​into a two-dimensional numpy array by row.

[0034] When concatenating each ping data row by row into a two-dimensional numpy array, compare the current ping length with the previous ping length;

[0035] When the data length of the current ping is smaller than that of the previous ping, zeros are added to the right end of the current ping to make it the same length as the previous ping and then the data is spliced ​​row by row. When the data length of the current ping is larger than that of the previous ping, the extra part of the right end of the current ping compared to the previous ping is removed. If the current ping is the same length as the previous ping, they are directly spliced ​​row by row.

[0036] In the above step S3,

[0037] The first data in each row of the data matrix is ​​the beam angle angle, which takes an odd number from 1 to 799, with a total of 400 values. The actual angle value is:

[0038] actual_angle=(angle / 799)*360,

[0039] Each row of the data matrix starting from the second data is the echo sampling point data of the ping. The calculation formula for the horizontal and vertical coordinates of the pixel point corresponding to a certain sampling point on the image is:

[0040] x=┌index (s) *cos(actual_angle)┐,

[0041] y=┌index (s) *sin(actual_angle)┐,

[0042] Where index represents the column coordinate of the sth data in each row of the data matrix, s = 1, 2, 3, ..., t; t represents that each row of the data matrix contains t data;

[0043] Interpolation is performed between two adjacent pings, and the number of interpolated pings is 7; each ping data is filled in the order of beam angle from small to large on a square canvas, and 400 ping data are required to generate a complete picture;

[0044] When the number of pings received by the mechanical scanning sonar within a UDP protocol receiving time is less than 400, all data will be retained and spliced ​​to the front of the next data packet; when the number of pings received is an integer multiple of 400, the quotient of the number of pings and 400 is the number of images that can be continuously imaged; when the number of pings received is greater than 400 but not an integer multiple of 400, the number of pings modulo 400 is taken, the last remainder of the pings in the data packet is retained and spliced ​​to the front of the next data packet, and the quotient of the number of pings and 400 is the number of images that can be continuously imaged;

[0045] When the previous data packet has extra data that needs to be spliced ​​to the front of the current data packet, compare the ping length of the data packet to be spliced ​​with the ping length of the current data packet:

[0046] When the length of the ping of the data packet to be spliced ​​is greater than the length of the ping of the current data packet, fill zeros at the right end of all pings of the current data packet to make it equal to the length of the ping of the data packet to be spliced, and then splice it by row; when the length of the ping of the data packet to be spliced ​​is smaller than the length of the ping of the current data packet, cut off the part of the right end of all pings of the current data packet that is longer than the data packet to be spliced, and then splice it by row; when the length of the ping of the data packet to be spliced ​​is equal to the length of the ping of the current data packet, directly splice it by row.

[0047] Finally, all the above sonar data matrices are input into the display and control interface implemented by Pyqt5, and real-time sonar images are displayed.

[0048] The specific implementation steps of step S4 are as follows:

[0049] S4.1. The circular highlight part in the center of the sonar image is an invalid feature, and the values ​​of the pixels whose distance from the center of the sonar image is within the specified threshold are set to zero;

[0050] S4.2. Determine the circular outline of the pipeline using Hough transform, and set the pixel values ​​of the area outside the circular outline and with a distance from the circular outline greater than 83 pixels to zero;

[0051] Hough transform circle detection method using opencv library function

[0052] |cv2.HoughCircles(image,cv2.HOUGH_GRADIENT,dp,minDist,param1,param2,minRadius,maxRadius),

[0053] Among them, minRadius represents the minimum radius of the detected circular contour, and its parameter value is 203;

[0054] maxRadius represents the minimum radius of the detected circular contour, and its parameter value is 240; param1 represents the minimum radius of the detected circular contour, and its parameter value is 100; param2 represents the minimum radius of the detected circular contour, and its parameter value is 30; minDist represents the minimum radius of the detected circular contour, and its parameter value is 200;

[0055] S4.3, using the regional connectivity detection based on image morphology to determine the interconnected regions in the sonar image, ignoring the connected regions of the pipeline itself, and setting the pixel values ​​of the connected regions outside the circular contour of the pipeline determined in step S4.2 and more than 7 pixels away from the circular contour to zero;

[0056] S4.4. Use bilateral filter and median filter in sequence to filter out noise points.

[0057] The target detection model used in step S5 is the yolov5s target detection model, and the specific construction steps of the model are as follows:

[0058] S5.1. Use mechanical scanning sonar to collect sonar images of several municipal pipeline defects in the experimental pipeline. Municipal pipeline defects include deposits, depressions, holes, and foreign body insertions;

[0059] S5.2, performing data augmentation on the sonar image collected in step S5.1, including upside-down flipping, left-right flipping, translation, random angle rotation, and HSV image enhancement;

[0060] S5.3, use labelimg to label the dataset augmented in step S5.2. The labeling file is in XML format. The labeling content includes the position, size and category information of the anchor box.

[0061] S5.4, the data set annotated in step S5.3 is randomly divided into a training set and a validation set in a ratio of 9:1;

[0062] S5.5, use the training set and validation set generated in step S5.4 to train the yolov5s target detection model and build a target detection model;

[0063] The yolov5s target detection model consists of four modules: input, backbone, Neck network and Head.

[0064] The model outputs the category of municipal pipeline defects, the location and size of the anchor box, and the confidence of the category.

[0065] Determine the confidence level of the output category. If the confidence level is greater than 0.5, draw the anchor box on the input sonar image based on its position and size information, and then send the image to the display interface for display;

[0066] If the confidence level is less than 0.5, it is considered as a misidentification and the input sonar image is sent directly to the display interface without any processing.

[0067] The beneficial effects of the present invention are:

[0068] (1) This application is based on mechanical scanning sonar and can solve the problem of pipeline defect detection in turbid water environments with many interferences;

[0069] (2) The present application performs preprocessing on the collected sonar images and denoises the sonar images. For the sonar images, a variety of image denoising algorithms are used to filter the background noise in the specific environment of the pipeline, while better retaining the required defect features;

[0070] (3) The present application solves the problem of filling sparsity at the periphery of the image by performing an interpolation operation on the obtained sonar image to enrich the image information;

[0071] (4) This application uses mechanical scanning of sonar images combined with deep learning to identify multiple types of typical pipeline defects, greatly broadening the application scope of sonar in pipeline inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a flow chart of the method of the present invention;

[0073] Figure 2 is the sonar image before interpolation;

[0074] Figure 3 It is the sonar image after interpolation processing;

[0075] Figure 4 It is a sonar image of the circular pipe contour obtained by the Hough transform circular detection method;

[0076] Figure 5 It is a sonar image of different connected area components obtained through the regional connectivity detection algorithm;

[0077] Figure 6It is the sonar image after denoising;

[0078] Figure 7 It is the sonar image of the identification result of the hole defect;

[0079] Figure 8 It is the sonar image of the identification result of the concave defect;

[0080] Fig. 9 It is the sonar image of the identification result of the deposition defect;

[0081] Fig.10 It is the sonar image of the identification result of foreign body insertion defect. DETAILED DESCRIPTION

[0082] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0083] In the following description, specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in a variety of other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0084] The present invention discloses a method for real-time autonomous identification of typical defects in municipal pipelines. The flowchart of the method is as follows: Figure 1 As shown, the specific steps are as follows.

[0085] The first step is to collect raw sonar data of the pipeline environment.

[0086] When the crawling robot crawls along the inner and outer walls of the pipeline, the mechanical scanning sonar mounted on the crawling robot collects and acquires the sonar raw data packets. The mechanical scanning sonar used in this application is a single-beam sonar. In the working state, the transducer of the mechanical scanning sonar rotates at a fixed time interval, 0.9 degrees at a time, and emits a 2.25MHZ fan beam each time and collects echo signals at a sampling period of 2.67us. The above time interval is related to the set range: the larger the range, the longer the time interval.

[0087] At the same time, the UDP protocol continuously sends sonar echo data in ping units to the host computer. 1 ping represents an echo data sequence at a certain beam angle. The larger the value of 1 ping, the stronger the echo of the corresponding sampling point.

[0088] In this embodiment, every 2 seconds, the NVIDIA Jetson AGX Orin GPU is used as the client to receive the original sound intensity data packet from the mechanical scanning sonar through the UDP protocol and send it to the protocol parsing process at the same time.

[0089] The second step is to perform real-time analysis on the mechanical scanning sonar data packets obtained in the first step.

[0090] Since the UDP protocol is not oriented to reliable connections, in order to ensure that there are no errors in the data packets received by the host computer program to affect the subsequent processing steps, it is necessary to first check and eliminate the sonar data packets received in step one.

[0091] The entire received data packet is processed one by one, and the first byte of the ping header, '0xfe', is first found. If '0xfe' cannot be found, the next byte is searched. If the byte cannot be found within the specified time, the data packet is judged to be incorrect and the entire data packet is discarded.

[0092] If the '0xfe' byte is found in the ping header, the complete byte length of the data part is obtained from the ping header, and then the actual data length is compared with the complete length. If the actual length is less than the complete length, it means that the ping is the last ping in the data packet and is incomplete, and several zero values ​​are filled at the end to make it reach the complete data length; if the actual length is equal to the complete length and the ping is not the last ping, jump to the next ping processing; if the actual length is equal to the complete length and the ping is the last ping, the entire data packet is processed. Determined by the sonar working mechanism, the actual length will not be greater than the complete length.

[0093] Next, the data packets that have been inspected and eliminated are parsed. The data frame sent by the mechanical scanning sonar used in this application consists of two parts: an 8-byte ping header and a data part. The ping header contains the header byte '0xfe' and the tail byte '0xfd' for identification, a sonar head number with a value of 1 to 255, a two-byte data length information, and a two-byte beam angle information.

[0094] Process the data packets one by one. First, read the beam angle and data length information in the ping header. The beam angle information is stored in the 5th and 6th bytes, which are the lower 7 bits and higher 7 bits of the beam angle respectively. Therefore, the beam angle is calculated as follows:

[0095] angle=((AngleHi&0x7f)<<7)|((AngleLo&0x7f)).

[0096] Among them, AngleHi represents the high byte of the beam angle; AngleLo represents the low byte of the beam angle.

[0097] The data length information is stored in the 3rd and 4th bytes, which are the lower 7 bits and higher 7 bits of the data length respectively. Therefore, the data length is calculated as follows:

[0098] data_length=((DataHi&0x7f)<<7)|(DataLo&0x7f).

[0099] Among them, DataHi represents the high byte of the data length; DataLo represents the low byte of the data length.

[0100] Each sampling point data occupies 4 bits, so one byte contains two sampling point data. The calculation methods of the two sampling point data are:

[0101] first_point=echo_data&0xf0, second_point=((echo_data&0xf0)>>4).

[0102] Among them, echo_data represents a byte containing two sampling point data.

[0103] Create a list for each ping data. The first element of the list is the beam angle angle read by the above method. The following elements are the sampling point data read from near to far from the sonar transducer by the above method. Convert the list storing all the sampling point data of the ping into a numpy array.

[0104] Each ping data in the data packet is spliced ​​into a two-dimensional numpy array by row. In actual work, there is often a demand to adjust the sonar range, and the number of sampling points of each ping is different for different sonar ranges. The larger the range, the more sampling points. The range of the mechanical scanning sonar used in this embodiment is set to nine gears, namely 0.2m, 0.4m, 0.6m, 0.8m, 1m, 2m, 3m, 4m, and 5m. When the range is set differently, the data length of each 1ping is different. The larger the range, the larger the data length. Therefore, the sonar image size of the sound field with different ranges is also different. Numpy arrays of different lengths cannot be spliced ​​by row, which will also affect subsequent imaging. Therefore, in the parsing process, it is necessary to compare the current ping length with the size of the previous ping length.

[0105] When the range decreases, the data length of the current ping is shorter than that of the previous ping, so the right end of the current ping is padded with zeros to make it the same length as the previous ping and then spliced ​​in rows; when the range increases, the data length of the current ping is longer than that of the previous ping, so the extra part of the right end of the current ping compared to the previous ping is removed; if the current ping is the same length as the previous ping, they are directly spliced ​​in rows. Finally, a data matrix containing beam angles and sampling point sound intensity values ​​is obtained, and each row of the data matrix corresponds to 1 ping of sonar data.

[0106] The third step is to generate a sonar image in real time using the sonar data matrix obtained in the second step.

[0107] Since the data matrix is ​​square and the image of the mechanical scanning sonar is a circular image with the transducer as the center, in the process of restoring the sonar image, every point in the data matrix must be mapped to the correct position.

[0108] Since each row of the data matrix corresponds to 1 ping of the sonar, the circular sonar image is formed by scanning in a fixed direction with the transducer position as the center of the circle. The first data in each row of the data matrix is ​​the beam angle angle, which represents the angle value of the ping from the sonar reference direction in a clockwise direction. The value is an odd number from 1 to 799, with a total of 400 values. Therefore, the actual angle value is:

[0109] actual_angle=(angle / 799)*360,

[0110] Each row of the data matrix starting from the second data is the echo sampling point data of the ping. The larger the index value, the farther the echo sampling point is from the transducer. Therefore, the index value can be used as the radial position of the pixel corresponding to the sampling point relative to the center of the circle. Based on this principle, the calculation formula for the horizontal and vertical coordinates of the pixel corresponding to a certain sampling point on the image is:

[0111] x=┌index (s) *cos(actual_angle)┐,

[0112] y=┌index (s) *sin(actual_angle)┐,

[0113] Among them, index represents the column coordinate of the sth data in each row of the data matrix, s = 1, 2, 3, ..., t; t means that each row of the data matrix contains t data.

[0114] Since the coordinates of pixels in an image can only be integers, the above x and y are rounded up.

[0115] The farther away from the sonar transducer, the greater the distance between two adjacent pings. Therefore, in order to solve the problem of filling sparsity, interpolation is required between two adjacent pings. The interpolation method used in this application is the neighboring interpolation method, that is, inserting a number of pings between two adjacent pings, and the value of the inserted ping is the value closest to its original ping. If the number of inserted pings is too large, it will cause unnecessary data redundancy and affect the real-time performance of the imaging algorithm; if the number of inserted pings is too small, it will be difficult to solve the problem of filling sparsity. In this embodiment, the optimal value of the number of inserted pings between two adjacent pings is determined to be 7. The sonar image before interpolation processing is as follows Figure 2 As shown, the sonar image obtained after the interpolation processing of this embodiment is as follows Figure 3 shown.

[0116] Imaging is to fill each ping of data on a square canvas in order from small to large beam angles. Since the mechanical scanning sonar rotates 400 times in one circle and 7 pings are inserted between each adjacent ping, a total of 2800 pings are required to form a complete image. In order to improve the dynamic effect of imaging and form a visual effect of several circles of scanning, instead of transferring it to the display and control interface for imaging after filling all 2800 pings of data, this embodiment is set to transfer it to the display and control interface for imaging every time 200 pings of data are filled, so a picture can be imaged 14 times, greatly improving the visual effect. Each time the canvas used for filling is not a blank canvas, but continues to be updated on the basis of the canvas formed after the previous data matrix is ​​filled. In this way, there will be no black unfilled areas on the image that have not been scanned.

[0117] Each time a complete image is generated, 400 pings of data are required. In theory, the number of rows in the data matrix cannot be less than 400. However, the length of each ping is related to the range. The larger the range, the more data each ping contains, and the longer it takes for the UDP protocol to receive the same 400 pings. If the receiving data duration of the UDP client protocol in the first step is set to be shorter, when the sonar works in a large range state, the data packets received at one time are less than 400 pings, and a complete sonar image cannot be formed. If the receiving data duration of the UDP client protocol in the first step is set to be longer, then at a shorter range, the time interval between receiving and processing data will be too long, seriously affecting the real-time performance of imaging.

[0118] To this end, this application performs the following processing. When the number of pings received by the mechanical scanning sonar within a UDP protocol receiving time is less than 400, the data is incomplete and cannot be imaged. All data is retained and spliced ​​to the front end of the next data packet. When the number of pings received is an integer multiple of 400, imaging can be achieved, and the quotient of the number of pings and 400 is the number of images that can be imaged continuously. When the number of pings received is greater than 400 but not an integer multiple of 400, the number of pings is modulo 400, and the last remainder pings of the data packet are retained and spliced ​​to the front end of the next data packet. The quotient of the number of pings and 400 is the number of images that can be imaged continuously. In this way, when the duration of the UDP protocol client receiving data is set to a smaller fixed value, the data integrity and real-time requirements under different ranges can be taken into account.

[0119] Although the range conversion problem is considered and handled in the second step of parsing the protocol, it is not applicable to the special case where the node changes the range between receiving two adjacent UDP protocol packets in the first step. Therefore, when there is extra data in the previous packet that needs to be spliced ​​in front of the current packet, it is necessary to compare the ping length of the packet to be spliced ​​with the ping length of the current packet.

[0120] When the length of the ping of the data packet to be spliced ​​is longer than the length of the ping of the current data packet, add zeros to the right end of all pings of the current data packet to make it equal to the length of the ping of the data packet to be spliced, and then splice it by row. When the length of the ping of the data packet to be spliced ​​is shorter than the length of the ping of the current data packet, cut off the part of the right end of all pings of the current data packet that is longer than the data packet to be spliced, and then splice it by row. When the length of the ping of the data packet to be spliced ​​is equal to the length of the ping of the current data packet, splice it directly by row.

[0121] Finally, all the above sonar data matrices are input into the display and control interface implemented by Pyqt5, and real-time sonar images are displayed.

[0122] The fourth step is to denoise the sonar image generated in the third step.

[0123] In actual pipeline detection application scenarios, there is noise both inside and outside the pipeline. Since the pipeline contains the target parameters, it is necessary to preserve the features inside the pipeline as much as possible. Therefore, pipeline image denoising mainly focuses on removing the noise outside the pipeline in the image.

[0124] The denoising process specifically includes the following four steps.

[0125] First, remove the invalid features located at the center of the image due to the vibration of the transducer itself.

[0126] The sampling point closest to the sonar is generated by the vibration of the transducer of the mechanical scanning sonar itself, rather than the backscattering of the sound waves after reaching the target. Therefore, the circular highlight part in the center of the sonar image is an invalid feature.

[0127] Due to the different sonar ranges, the length of each ping is different, and the length of the invalid features generated by the transducer's own vibration is also different. Therefore, different thresholds should be set according to different sonar ranges, and all pixel values ​​​​that are less than the threshold from the center point of the image are set to zero.

[0128] Second, use Hough transform to determine the wall position of the circular pipe and filter out the noise beyond a certain distance from the wall position.

[0129] The basic principle of Hough transform circle detection method: Assume that the parameters of the circle to be detected are (x c ,y c , r), where (x c ,y c ) is the coordinate of the center of the circle, and r is the radius of the circle. Starting from each edge point in the image, calculate the distance from the edge point to the center of the circle and represent it in the parameter space as (x c ,y c ) as the center. The radius of this circle in parameter space is the radius r of the circle. Therefore, in parameter space, each edge point corresponds to a circle with the center (x c ,y c ) are accumulated at each location. A location with a high cumulative value means that there are more edge points corresponding to the boundary of the circle at that location, indicating that a circle may exist.

[0130] Use the Hough transform circular detection method to detect the pipe wall and filter out the noise outside the pipe wall. Since the pipe wall has a certain thickness on the sonar image, many concentric circles can be detected; sometimes due to noise interference and motion distortion, "false circles" will appear outside the pipe. Therefore, accurately identifying the pipe wall is the most important task in this step. How to determine the parameters of the Hough transform circular detection method is the key.

[0131] In this application, the Hough transform circle detection method uses the opencv library function

[0132] |cv2.HoughCircles(image,cv2.HOUGH_GRADIENT,dp,minDist,param1,param2,minRadius,maxRadius)

[0133] The two parameters minRadius (the minimum radius of the detected circular outline) and maxRadius (the maximum radius of the detected circular outline) are determined according to the approximate radius of the pipeline on the image. In this embodiment, the above two parameter values ​​are 203 and 240 respectively. The two parameters param1 (the high threshold of the Canny operator for edge detection) and param2 (accumulator threshold) jointly determine the difficulty of detecting the circle. If the circle detection difficulty is too high, it is difficult to detect the circle. If the detection difficulty is too low, it is easy to detect a false circle. In this embodiment, the above two parameters are 100 and 30 respectively. The minDist (minimum distance between circle centers) parameter is used to prevent the simultaneous detection of multiple very close circles. When the distance between the centers of two circles is less than minDist, the circle with lower confidence will be filtered out. In this embodiment, the parameter value is selected as 200.

[0134] The Hough transform circular detection method will output many results, and the best detection result needs to be screened out. In actual pipeline detection operations, the center of the pipeline is usually near the center point of the sonar image, especially a little above the center point. In this application, the center of the circle is assumed to be within 200 above the midpoint of the image. The detection results outside the interval of 200 above the midpoint to the midpoint are determined as false circles and filtered out. The sonar image of the circular pipeline contour obtained by the Hough transform circular detection method is as follows Figure 4 shown.

[0135] Then, the one with the largest radius is selected from the multiple concentric circles detected as the final detection result. Since the concave defect appears on the outside of the pipe and is connected to the pipe wall, all features located on the outside of the pipe wall cannot be simply regarded as noise. In this application, all features with a distance greater than 83 from the outer wall of the pipe are regarded as noise and set to zero for filtering.

[0136] Third, a regional connectivity detection algorithm based on image morphology is applied to retain the features located outside the tube wall and connected to the tube wall, and the features located outside the tube wall and not connected to the tube wall are regarded as noise and filtered out.

[0137] The image morphology-based regional connectivity detection algorithm is used to filter out the noise located outside the pipe wall and very close to the pipe wall. This step is to distinguish defects such as dents from noise that is less than 83 away from the outer wall of the pipe. The main difference between these two types of features is that dents are connected to the pipe, while most of the noise is isolated outside the pipe, and connectivity detection can distinguish these two types of situations.

[0138] The regional connectivity detection algorithm will output multiple results. All feature domains of the connected area with a distance greater than 7 from the outer wall of the pipe, except for the connected area of ​​the pipe wall itself, are regarded as noise and set to zero for filtering. The sonar images of different connected area components obtained by the regional connectivity detection algorithm are as follows: Figure 5shown.

[0139] Fourth, bilateral filter and median filter are used successively to remove isolated noise points. The sonar image after denoising is as follows: Figure 6 shown.

[0140] The fifth step is to output the denoised image to the target detection model for defect recognition.

[0141] The target detection model used in this application adopts the yolov5s target detection model, through which the category of municipal pipeline defects, the location and size of the anchor frame, and the confidence level can be output.

[0142] The yolov5s target detection model is constructed through the following specific steps.

[0143] Step 1: Use mechanical scanning sonar to collect sonar images of several common municipal pipeline defects in the experimental pipeline, including deposition, depression, hole, and foreign body insertion.

[0144] Step 2: In order to prevent overfitting and improve the generalization of the model, the sonar images collected in step 1 are augmented, including up and down flipping, left and right flipping, translation, random angle rotation, and HSV image enhancement.

[0145] Step 3: Use labelimg to label the dataset augmented in step 2. The labeling file is in XML format. The labeling content includes the position, size and category information of the anchor box.

[0146] Step 4: The dataset annotated in step 3 is randomly divided into a training set and a validation set in a ratio of 9:1.

[0147] Step 5, using the training set and verification set generated in step 4 to train the yolov5s target detection model adopted by the present invention, to build a target detection model.

[0148] The yolov5s target detection model in this application is a single-stage target detection model, which includes four modules, namely, input end, backbone (backbone network), Neck network, and Head (output head).

[0149] The input end is used to scale the size of the input sonar image to the size of the network input and perform normalization operations. When the sonar range is changed using the display platform, the size of the generated image is also different. The scaling of the image size effectively solves the scale inconsistency problem in target detection and improves detection accuracy and robustness. In addition, the Mosaic data enhancement operation is used in the training stage to expand the training sample set.

[0150] Backbone includes Focus structure and CSP structure. Among them, the Focus structure performs a slice operation on the input image. When the range is 0.8m, the sonar image size is (880,880) and the number of channels is 3. The slicing operation is performed on it, and it becomes a feature map with a size of (440,440) and a number of channels of 12. The slicing operation is used to split the high-resolution image / feature map into multiple low-resolution images / feature maps. The information on the hw plane is converted to the channel dimension, and then different features are extracted through 3×3 convolution. This method can reduce the information loss caused by downsampling and improve the accuracy of the model.

[0151] The CSP structure divides the input feature map into two parts, one part is processed by a small convolutional network (called a subnetwork), and the other part is directly processed by the next layer. The two parts of the feature map are then concatenated as the input of the next layer. Specifically, the following steps are included:

[0152] (I) The input feature map is divided into two parts, one part enters the sub-network for further processing, and the other part enters the next layer of network processing together with the output of the sub-network. In the sub-network, a convolution layer is used to compress the input feature map, and then a series of convolution operations are performed, and finally a convolution layer is used to expand it. In this way, relatively few high-level features can be extracted.

[0153] (ii) In the next layer, the feature map processed by the sub-network is concatenated with the directly processed feature map, and then a series of convolution operations are performed. This can combine low-level detail features with high-level abstract features, improving the efficiency of feature extraction.

[0154] The Neck network is an intermediate feature extraction network added to the backbone to enhance the model's feature expression ability and receptive field, and further improve the model's detection performance. The main structure used by the Neck network is the SPP structure, which is a pyramid pooling structure that can pool feature maps of different sizes, thereby enhancing the model's perception of targets of different scales. It mainly fuses information of different scales by performing pooling operations of different sizes on the input feature maps.

[0155] The output head is responsible for multi-scale object detection on the feature maps extracted by the backbone network. This module includes the following parts:

[0156] (1) Anchors: used to define target boxes of different sizes and aspect ratios. They are usually obtained by clustering the target boxes of the training set using K-means clustering. They can be calculated before model training and stored in the model to generate detection boxes during prediction.

[0157] (2) Convolutional Layers: The detection layer of YOLOv5s contains a series of convolutional layers for processing feature maps and extracting features. These convolutional layers can adapt to different detection tasks by adjusting the number of channels and kernel size;

[0158] (3) Regression: used to regress each detection box to obtain its position and size. The feature is usually regressed in the form of a fully connected layer.

[0159] (4) Prediction Layers: Each prediction layer is responsible for predicting a set of anchor boxes and categories. Each prediction layer usually consists of a convolutional layer and an output layer. The number of channels and shape of the output layer determines the number of predicted anchor boxes and categories.

[0160] In the output anchor boxes, the non-maximum suppression algorithm is used to suppress overlapping anchor boxes and only retain the most representative anchor boxes. Assume that B = {b1, b2, …, b n} is the set of anchor boxes to be processed, each anchor box b i Including coordinate information and confidence scores. Then the anchor box set B' after NMS processing can be calculated as follows:

[0161] (I) Sort the confidence scores to obtain the sorted anchor frame index sequence S = {S1, S2, ..., S n}, where b s1 score ≥ b s2 score≥…≥b sn score,b sn score represents the confidence score of the nth anchor box.

[0162] (ii) Initialize an empty output anchor box list B'={}.

[0163] (III) For each anchor box index s in the sorted anchor box index sequence S i , perform the following steps:

[0164] (1) The current anchor box b si Add to the output anchor box list B'.

[0165] (2) for b si Traverse the remaining anchor frames except b si The loU is greater than the threshold loU threshold , then remove it from the candidate list.

[0166] (iv) Return the output anchor box list.

[0167] In this application, in the above target detection model, the loss function of the target detection task generally includes classification loss (classification loss function), Box_loss (regression loss) and obj_cls (confidence loss). Given an input image X∈R H×EW×3 , with a height of H and a width of W. When input into the target detection model, three feature layers will be obtained, with sizes of (80,80) (40,40) (20,20) respectively. Feature layers of different sizes will have different receptive fields. The receptive field of the feature layer size (80,80) is small, one feature point corresponds to 8 pixels on the original input image, which is suitable for detecting small targets, and its corresponding anchor size is [10,13], [16,30], [33,23]; the receptive field of the feature layer size (40,40) is slightly larger, one feature point corresponds to 16 pixels on the original input image, which is suitable for detecting medium-sized targets, and its corresponding anchor size is [30,61], [62,45], [59,119]; the receptive field of the feature layer size (20,20) is larger, one feature point corresponds to 32 pixels on the original input image, which is suitable for detecting large targets, and its corresponding anchor size is [116,90], [156,198], [373,326]. The number of dataset categories E = 4. Assume that the ground truth box of a target in the sonar image is (x1 (i) , y1 (i) , x2 (i) , y2 (y) ), then the corresponding center point coordinates are

[0168] However, the cell where the center point is located also has a preset anchor box. The resulting anchor box position error is measured using the GIOU (Generalized Intersection over Union) loss function, and its mathematical expression is as follows:

[0169]

[0170] Where: IOU represents the standard Intersection over Union between the predicted box and the true box; A represents the area intersection of the minimum enclosing rectangles of the predicted box and the true box; C represents the area union of the minimum enclosing rectangles of the predicted box and the true box.

[0171] The design of the GIOU loss function takes into account not only the matching degree of the position and size of the box, but also the matching degree of the overall position and size of the box when calculating the loss. This loss function can more comprehensively measure the difference between the predicted box and the true box, which helps the model learn the prediction of the target box more accurately.

[0172] The classification loss function is defined as:

[0173]

[0174] Among them, p represents the true label (usually 0 or 1), and q represents the probability value output by the model (usually between 0 and 1). i represents the value of the i-th sample in the true label, q i Represents the predicted probability value of the i-th sample in the model output. In the target detection task, BCE Loss can be used to determine whether the target exists in a target box. Similar to the cross entropy loss, BCELoss can also be combined with the position information of the target box to achieve joint training.

[0175] The obj_cls confidence loss function is as follows:

[0176]

[0177] Among them, λ obj and λ cls represents the weight parameter, which controls the relative importance of target existence loss and target category classification loss. BCE represents the binary cross entropy loss function, p obj and They represent the probability of the real target existence and the target existence predicted by the model, p cls and They represent the probabilities of the true target category and the target category predicted by the model, respectively.

[0178] In summary, the loss function of the entire target detection model can be expressed as:

[0179] LOSS=L GIOU +LOSS cla (p,q)+LOSS obj_cls ,

[0180] The sonar image processed by the denoising algorithm in step 4 is input into the trained recognition network. The output content includes the following parts: the position, size, category, and confidence of the anchor box.

[0181] In order to reduce the misrecognition rate, the output results of the target detection model are post-processed. The confidence of the output category is determined. If the confidence is greater than 0.5, the anchor frame is drawn on the input sonar image according to its position and size information, and the image is sent to the display interface for display. If the confidence is less than 0.5, it is judged as a misrecognition, and the input sonar image is sent to the display interface for display without any processing.

[0182] The schematic diagrams of various defects of municipal pipelines output by the method described in this application are shown below. When a hole defect occurs in a municipal pipeline, the image characteristics obtained by the method described in this invention are that bulges and gaps appear on the pipe wall, such as Figure 7 When a municipal pipeline has a concave defect, the image feature obtained by the method of the present invention is that a bulge is formed on one side of the pipe wall, such as Figure 8 When a municipal pipeline has a deposition defect, the image feature obtained by the method of the present invention is that bulges are formed on both sides of the pipe wall, such as Fig. 9 When a foreign body is inserted into a municipal pipeline, the image characteristic obtained by the method of the present invention is that a bulge is formed on the pipe wall, but the bulge is elongated and has a weaker pullback strength, as shown in FIG. Fig.10 shown.

[0183] As shown in Table 1, compared with the existing Faster R-CNN algorithm, the method described in this application has an obvious speed advantage; compared with the lightweight YOLOv3tiny and YOLOv4tiny algorithms, the PmA of the method described in this application is significantly improved; compared with the classic YOLOv3, YOLOv4, and SSD algorithms, the comprehensive performance of the method described in this application is stronger.

[0184] Table 1 Performance comparison between the method described in this application and the existing method

[0185]

[0186] The above is a detailed introduction to the real-time autonomous identification method for typical defects of municipal pipelines provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined in this article can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown in this article, but will conform to the widest range consistent with the principles and novel features disclosed in this article.

Claims

1. A real-time autonomous identification method for typical defects in municipal pipelines. It is characterized in that The following steps are involved: S1. Collect and obtain the original sonar data of the municipal pipeline environment through the mechanical scanning sonar mounted on the crawler robot; S2, performing real-time analysis on the original sonar data obtained in step S1 to obtain a data matrix including beam angles and sound intensity values ​​of sampling points; S3, generating a mechanical scanning sonar image in real time using the sonar data matrix obtained in step S2; S4, performing denoising processing on the mechanical scanning sonar image; S5, outputting the denoised image to the target monitoring model for defect identification; Step S2 includes the following specific steps: S2.1, inspecting and removing the original sonar data packets obtained in step S1; S2.2, parsing and processing the data packets that have been inspected and eliminated; In the above step S2.2, the data packets are processed one by one by ping: Read the beam angle and data length information in the ping header. The beam angle information is stored in the 5th and 6th bytes, which are the lower 7 bits and higher 7 bits of the beam angle respectively. The calculation formula of the beam angle is: angle=((AngleHi&0x7f)<<7)|((AngleLo&0x7f)); Among them, AngleHi represents the high byte of the beam angle; AngleLo represents the low byte of the beam angle; The data length information is stored in the 3rd and 4th bytes, which are the lower 7 bits and higher 7 bits of the data length respectively. The calculation formula for the data length is: data_length=((DataHi&0x7f)<<7)|(DataLo&0x7f); Among them, DataHi represents the high byte of data length; DataLo represents the low byte of data length; Each sampling point data occupies 4 bits, and one byte contains two sampling point data. The calculation formulas for the two sampling point data are: first_point=echo_data&0xf0, second_point=((echo_data&0xf0)>>4); Among them, echo_data represents a byte containing two sampling point data; Create a list for each ping data. The first element of the list is the beam angle angle read by the above method. The following elements are the sampling point data read from near to far from the sonar transducer by the above method. Each ping data in the data packet is spliced ​​into a two-dimensional numpy array by row. In the above step S3, The first data in each row of the data matrix is ​​the beam angle angle, which takes an odd number from 1 to 799, with a total of 400 values. The actual angle value is: actual_angle=(angle / 799)*360, Each row of the data matrix starting from the second data is the echo sampling point data of the ping. The calculation formula for the horizontal and vertical coordinates of the pixel point corresponding to a certain sampling point on the image is: x=┌index (s) *cos(actual_angle)┐, y=┌index (s) *sin(actual_angle)┐, Where index represents the column coordinate of the sth data in each row of the data matrix, s = 1, 2, 3, ..., t; t represents that each row of the data matrix contains t data; Interpolation is performed between two adjacent pings, and the number of interpolated pings is 7; each ping data is filled in the order of beam angle from small to large on a square canvas, and 400 ping data are required to generate a complete picture; When the number of pings received by the mechanical scanning sonar within a UDP protocol receiving time is less than 400, all data will be retained and spliced ​​to the front of the next data packet; when the number of pings received is an integer multiple of 400, the quotient of the number of pings and 400 is the number of images that can be continuously imaged; when the number of pings received is greater than 400 but not an integer multiple of 400, the number of pings modulo 400 is taken, the last remainder of the pings in the data packet is retained and spliced ​​to the front of the next data packet, and the quotient of the number of pings and 400 is the number of images that can be continuously imaged; When the previous data packet has extra data that needs to be spliced ​​to the front of the current data packet, compare the ping length of the data packet to be spliced ​​with the ping length of the current data packet: When the length of the ping of the data packet to be spliced ​​is longer than the length of the ping of the current data packet, zeros are added to the right end of all pings of the current data packet to make it equal to the length of the ping of the data packet to be spliced, and then the data is spliced ​​in rows; when the length of the ping of the data packet to be spliced ​​is shorter than the length of the ping of the current data packet, the part of the right end of all pings of the current data packet that is longer than the data packet to be spliced ​​is cut off, and then the data is spliced ​​in rows; when the length of the ping of the data packet to be spliced ​​is equal to the length of the ping of the current data packet, the data is spliced ​​directly in rows; Finally, all the above sonar data matrices are input into the display control interface implemented by Pyqt5, and the real-time sonar image is displayed; The specific implementation steps of step S4 are as follows: S4.

1. The circular highlight part in the center of the sonar image is an invalid feature, and the values ​​of the pixels whose distance from the center of the sonar image is within the specified threshold are set to zero; S4.

2. Determine the circular outline of the pipeline using Hough transform, and set the pixel values ​​of the area outside the circular outline and with a distance from the circular outline greater than 83 pixels to zero; Hough transform circle detection method using opencv library function |cv2.HoughCircles(image,cv2.HOUGH_GRADIENT,dp,minDist,paraml,param2,minRadius,maxRadius), Among them, minRadius represents the minimum radius of the detected circular contour, and its parameter value is 203; maxRadius represents the maximum radius of the detected circular contour, and its parameter value is 240; paraml represents the high threshold of the Canny operator used for edge detection, and its parameter value is 100; param2 represents the accumulator threshold, and its parameter value is 30; minDist represents the minimum distance between circle centers, and its parameter value is 200; S4.3, using the regional connectivity detection based on image morphology to determine the interconnected regions in the sonar image, ignoring the connected regions of the pipeline itself, and setting the pixel values ​​of the connected regions outside the circular contour of the pipeline determined in step S4.2 and more than 7 pixels away from the circular contour to zero; S4.4, use bilateral filter and median filter to filter out noise points in sequence; The target detection model used in step S5 is the yolov5s target detection model, and the specific construction steps of the model are as follows: S5.

1. Use mechanical scanning sonar to collect sonar images of several municipal pipeline defects in the experimental pipeline. Municipal pipeline defects include deposits, depressions, holes, and foreign body insertions; S5.2, performing data augmentation on the sonar image collected in step S5.1, including upside-down flipping, left-right flipping, translation, random angle rotation, and HSV image enhancement; S5.3, use labelimg to label the dataset augmented in step S5.

2. The labeling file is in XML format. The labeling content includes the position, size and category information of the anchor box. S5.4, the data set annotated in step S5.3 is randomly divided into a training set and a validation set in a ratio of 9:1; S5.5, use the training set and validation set generated in step S5.4 to train the yolov5s target detection model and build a target detection model; The yolov5s target detection model consists of four modules: input, backbone, Neck network and Head; The model outputs the category of municipal pipeline defects, the location and size of the anchor box, and the confidence of the category.

2. According to claim 1, the real-time autonomous identification method for typical defects of municipal pipelines, It is characterized in that The circular highlighted part in the sonar image is an invalid feature. All pixel values ​​whose distance from the image center is less than a specified threshold are set to zero. The specified threshold is related to the range of the mechanical scanning sonar.

3. According to claim 1, the real-time autonomous identification method for typical defects of municipal pipelines, It is characterized in that In the above step S2.1, the received original sonar data packets are processed ping by ping, and the first byte '0xfe' of the ping header is first searched; If '0xfe' cannot be found, the next byte is searched. If the byte cannot be found within the specified time, the data packet is considered to be incorrect and the entire data packet is discarded. If the '0xfe' byte is found in the ping header, get the full byte length of the data portion from the ping header, and then compare the actual data length with the full length: If the actual length is less than the complete length, it indicates that the ping is the last ping in the data packet and is incomplete, so several zero values ​​are padded at the end to make it reach the complete data length; if the actual length is equal to the complete length and the ping is not the last ping, jump to the next ping for processing; if the actual length is equal to the complete length and the ping is the last ping, the entire data packet has been processed.

4. According to claim 1, the real-time autonomous identification method for typical defects of municipal pipelines, It is characterized in that When concatenating each ping data row by row into a two-dimensional numpy array, compare the current ping length with the previous ping length; When the data length of the current ping is smaller than that of the previous ping, zeros are added to the right end of the current ping to make it the same length as the previous ping and then the data is spliced ​​row by row. When the data length of the current ping is larger than that of the previous ping, the extra part of the right end of the current ping compared to the previous ping is removed. If the current ping is the same length as the previous ping, they are directly spliced ​​row by row.

5. According to claim 1, the real-time autonomous identification method for typical defects of municipal pipelines, It is characterized in that Determine the confidence level of the output category. If the confidence level is greater than 0.5, draw the anchor box on the input sonar image based on its position and size information, and then send the image to the display interface for display; If the confidence level is less than 0.5, it is considered as a misidentification and the input sonar image is sent directly to the display interface without any processing.

Images