Pipeline defect detection method, electronic equipment and storage medium

Abstract

The invention provides a pipeline defect detection method, electronic equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: performing coordinate conversion on an initial fisheye video stream to obtain a multi-frame plane image; for any frame of plane image, determining a binarization mask of the frame of plane image based on a semantic segmentation network, and extracting feature points of a pipe wall region in the frame of plane image by taking the binarization mask as a constraint to form a feature point set; determining an optimal suture line between any two adjacent frames of feature point sets by adopting a graph cut algorithm based on energy minimization, and carrying out splicing to obtain a pipeline panorama; and identifying the pipeline panorama by using a deep neural network model to perform defect identification to obtain a target defect. According to the method, the binarization mask is used as a constraint, and non-pipe wall area interference is eliminated; an optimal suture line is determined and splicing is completed based on an energy minimization graph cut algorithm, so that inter-frame splicing gaps and artifacts are effectively eliminated, and rapid and accurate detection of target defects can be realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a pipeline defect detection method, electronic device, and storage medium. Background Technology

[0002] Underground drainage networks are the "lifeline" ensuring the normal operation of cities. With the acceleration of urbanization, problems such as aging, corrosion, cracking, and siltation of these networks are becoming increasingly serious, leading to frequent road collapses, sewage leaks, and urban flooding. Regularly conducting health checks on the network is a core task of municipal operation and maintenance. Typically, CCTV robots equipped with fisheye lenses with a field of view exceeding 180° are used to enter the pipelines, acquire video footage, and then use computer vision algorithms to identify defects.

[0003] In existing technologies, the bottom of drainage pipes typically contains continuously flowing sewage. The ripples on the water surface, reflections, and waves caused by robot movement manifest as high-frequency noise and dynamic textures in computer vision algorithms. Existing image stitching algorithms (such as those based on SIFT / SURF feature matching) often struggle to distinguish between the static texture of the pipe wall and the dynamic texture of the water surface, leading to mismatched feature points. This results in severe "ghosting" or misalignment at the bottom of the generated panoramic image, obscuring sediment or crack defects and significantly impacting the accuracy of defect detection. Summary of the Invention

[0004] This invention provides a pipeline defect detection method, electronic device, and storage medium to address the problem of low accuracy in existing defect identification algorithms.

[0005] In a first aspect, embodiments of the present invention provide a pipeline defect detection method, comprising: Obtain the initial fisheye video stream, and perform coordinate transformation on the initial fisheye video stream to obtain multiple frames of planar images; For any given frame of planar image, a binary mask for that frame of planar image is determined based on a semantic segmentation network, and feature points of the pipe wall region in that frame of planar image are extracted using the binary mask as a constraint, forming the feature point set corresponding to that frame of planar image; An energy-minimum-based graph cut algorithm is used to determine the optimal stitching line between any two adjacent feature point sets, and the images are stitched together based on the corresponding optimal stitching line to obtain a panoramic view of the pipeline. Defects are identified by performing a panoramic view of the pipeline to determine the target defects.

[0006] In a second aspect, embodiments of the present invention provide an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the pipeline defect detection method as described in the first aspect or any possible implementation of the first aspect.

[0007] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the pipeline defect detection method as described in the first aspect or any possible implementation thereof.

[0008] This invention provides a pipeline defect detection method, electronic device, and storage medium. The pipeline defect detection method includes: acquiring an initial fisheye video stream; performing coordinate transformation on the initial fisheye video stream to obtain multiple frames of planar images; for any given frame of planar image, determining a binarization mask for that frame of planar image based on a semantic segmentation network, and extracting feature points of the pipe wall region in that frame of planar image using the binarization mask as a constraint to form a feature point set corresponding to that frame of planar image; using an energy-minimizing graph cut algorithm to determine the optimal stitching line between any two adjacent feature point sets, and stitching them together based on the corresponding optimal stitching line to obtain a panoramic view of the pipeline; and performing defect identification on the panoramic view of the pipeline to obtain the target defect. This invention uses a binary mask generated by a semantic segmentation network as a constraint to extract feature points from the pipe wall, achieving accurate localization and effective filtering of feature points and eliminating interference from non-pipe wall areas. Simultaneously, based on an energy-minimizing graph cut algorithm, the optimal stitching line is determined and stitching is completed, which can effectively eliminate inter-frame stitching gaps and artifacts. The generated panoramic image of the pipeline is complete, continuous, and has strong visual consistency, fully presenting the overall shape of the pipeline. Finally, a deep neural network model is combined to identify defects in the panoramic image, enabling rapid and accurate detection of target defects. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating the implementation of a pipeline defect detection method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a coordinate transformation provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of feature point extraction provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the determination of an optimal suture line according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a spatiotemporal consistency calibration provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of a weighting factor provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of an evaluation report provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of the pipeline defect detection device provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0010] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0011] In this application, a pipeline robot is equipped with a fisheye camera to enter the pipeline and capture raw fisheye video streams. It also features an IMU and odometry to acquire high-frequency odometer data and camera attitude angles, providing geometric constraints for image correction. For example, to address robot posture changes, a virtual gimbal correction algorithm based on IMU data is introduced to electronically stabilize the optical axis before image unfolding, ensuring the texture continuity of the unfolded image.

[0012] See Figure 1 The diagram illustrates the implementation flowchart of the pipeline defect detection method provided in this embodiment of the invention, which is described in detail below: refer to Figure 1 The above-mentioned pipeline defect detection methods include: S101: Obtain the initial fisheye video stream and perform coordinate transformation on the initial fisheye video stream to obtain multiple frames of planar images; Fisheye cameras have ultra-wide-angle lenses, which produce barrel distortion (the edges of the image are stretched outwards and the center is compressed), making it impossible to directly obtain a planar image that conforms to the visual habits of the human eye. Therefore, coordinate transformation is required.

[0013] In one possible implementation, S101 may include: S1011: The mapping relationship between the polar coordinate system of the fisheye image and the cylindrical coordinate system of the pipe is established using an equidistant projection model; S1012: Based on the mapping relationship, convert each frame of fisheye image in the initial fisheye video stream into a planar image; The mapping relationship can be:

[0014]

[0015] in, The optical center of the fisheye image. The coordinates of a pixel in the fisheye image. This is the pixel distance from this pixel in the fisheye image to the optical center; Focal length Angle of incidence; For the pipe radius, This is the distance from the current observation point along the optical axis to the lens.

[0016] Firstly, based on the equidistant projection model, referencing... Figure 2 Establish an inverse mapping between the polar coordinate system of the fisheye image and the cylindrical coordinate system of the pipe.

[0017] For example, refer to Figure 2 Specifically, it includes the following steps: 1. Parameter calibration: Using the checkerboard calibration method, calculate the camera's intrinsic parameter matrix and distortion coefficients. ; 2. Inverse mapping: Construct a planar image (unfolded image), with a width of... Gao Wei For any pixel in the unfolded image : Normalized coordinates: ; For circumferential angle, It is the axial angle.

[0018] Spherical coordinate transformation: transforming normalized coordinates Convert to three-dimensional coordinates on a unit sphere .

[0019] Fisheye projection: based on Calculate the polar radius of this point on the original fisheye image. and polar angle The three-dimensional coordinates on the unit sphere Mapping back to the polar coordinates of the original fisheye image ( , ).

[0020] Distortion correction: adjust the angle of incidence. Substitute into the distorted polynomial Corrections are made to correct the barrel distortion of the fisheye lens. Among these, It is the extreme radius The corresponding angle, after correction, is obtained Then use it to calculate the corrected polar radius. .

[0021] Pixel index: based on ( , Calculate the coordinates of the original image. And use bilinear interpolation to obtain pixel values.

[0022] The reverse mapping method provided in this application avoids the voids that may be generated by forward mapping and ensures the compactness of the unfolded graph.

[0023] S102: For any frame of planar image, determine the binarization mask of the frame of planar image based on the semantic segmentation network, and extract the feature points of the pipe wall region in the frame of planar image with the binarization mask as a constraint to form the feature point set corresponding to the frame of planar image; refer to Figure 3In drainage pipes, the bottom 1 / 5 to 1 / 3 of the area is usually covered by sewage. To avoid splicing artifacts caused by water surface reflection and flow, this application uses a binarized mask as a constraint to extract feature points of the pipe wall region in the frame planar image.

[0024] In one possible implementation, refer to Figure 3 The semantic segmentation network is the Mask R-CNN semantic segmentation network; S102 may include: S1021: Input the planar image of this frame into the Mask R-CNN semantic segmentation network to obtain a binary mask; wherein, the mask value of the pipe wall region is 1, and the mask value of the water surface region is 0; Train a lightweight Mask R-CNN semantic segmentation network. The training set contains 3000 images of pipes labeled "water" and "wall". The network output layer generates a binary mask of the same size as the input image, with a mask value of 1 for the wall region and a mask value of 0 for the water region.

[0025] S1022: The ORB algorithm is used to extract feature points in the pipe wall region to form the feature point set corresponding to the planar image of this frame.

[0026] The ORB algorithm is used to extract feature points, and a judgment condition is added: if a candidate feature point falls in an area with a mask value of 0, it is discarded directly to ensure that all feature points come from the pipe wall area, thus eliminating the interference of fluid dynamic texture on subsequent panoramic image stitching from the root.

[0027] S103: The optimal stitching line between any two adjacent feature point sets is determined by using a graph cut algorithm based on energy minimization, and the corresponding optimal stitching line is used to stitch together to obtain a panoramic view of the pipeline. In one possible implementation, S103 may include: S1031: For any two adjacent feature point sets, perform feature point matching on the two adjacent feature point sets to determine the overlapping region; construct a Markov random field in the overlapping region and establish an energy function based on the Markov random field; optimize the solution with the minimum energy function as the objective function to obtain the optimal stitching line between the two adjacent feature point sets.

[0028] In one possible implementation, the energy function can be:

[0029] in, It is an energy function; For data items, representing pixels Differences in luminosity between different source images; For pixels Corresponding source image labels; For smoothing, this indicates a penalty for cutting stitch lines in areas with similar textures or at structural edges. For pixels Corresponding source image labels; For all adjacent pixel pairs A set of.

[0030] refer to Figure 4 This application constructs a Markov random field in the overlapping area of ​​adjacent frames. By minimizing the energy function containing color difference terms and geometric structure terms, the stitching line is allowed to pass through smooth areas (such as textureless pipe walls) where the differences between the two images are small, while avoiding high-frequency areas such as cracks or text. A non-linear stitching path that avoids significant defect textures is found. Cylindrical unfolding and multi-band fusion stitching are performed to fuse the multi-frame unfolded planar images into a continuous long strip-shaped pipe panoramic image that removes water surface interference.

[0031] For example, the maximum flow algorithm can be used to solve this energy minimization problem, obtain the optimal stitching path, and achieve seamless fusion.

[0032] S104: Perform defect identification on the pipeline panorama to obtain the target defect.

[0033] To improve the accuracy of defect identification, reduce the rate of missed and false detections, and take into account identification efficiency, a method of "preliminary model identification + hierarchical verification and screening" can be adopted.

[0034] In one possible implementation, S104 may include: S1041: Input the panoramic view of the pipeline into a deep neural network model for identification to obtain multiple suspected defects and their corresponding confidence levels; By leveraging the feature extraction and pattern matching capabilities of deep neural network models, a comprehensive scan of the pipeline panorama can be performed quickly, initially identifying potential defect areas and assigning confidence scores. This provides a basis for subsequent screening and verification, effectively reducing the scope and workload of manual inspections.

[0035] S1042: For any suspected defect, if the confidence level of the suspected defect is not less than the preset confidence level, then the suspected defect is taken as the target defect; if the confidence level of the suspected defect is less than the preset confidence level, then the spatiotemporal consistency check is performed on the suspected defect; if the check passes, then the suspected defect is determined to be the target defect; otherwise, the suspected defect is not the target defect.

[0036] For any suspected defect, a stratified screening process is performed by combining a confidence threshold and spatiotemporal consistency verification to accurately identify the target defect. If the confidence level of the suspected defect is not less than the preset confidence level, it indicates that the model's identification result for the defect is highly reliable, and the suspected defect can be directly identified as the target defect; if the confidence level of the suspected defect is less than the preset confidence level, it indicates that there is a certain degree of uncertainty in the model's identification result.

[0037] Since real structural defects (such as cracks) are stationary relative to the pipe wall, while interfering objects (such as flying insects or water droplets) exhibit high-frequency motion characteristics, the system can effectively distinguish between real and false defects by analyzing their continuity in the time dimension. Based on this, for suspected defects with low confidence, further verification is performed through spatiotemporal consistency checks—combining information such as the spatial location of the suspected defect in the panoramic view of the pipeline and the morphological continuity in adjacent frames for cross-validation. If the verification passes, it indicates that the suspected defect is not a misjudged interference item by the model and can be identified as the target defect; otherwise, the suspected defect is determined to be an interference item and excluded from the target defect list.

[0038] This application employs a two-tiered mechanism of "preliminary screening with confidence threshold + secondary verification of low-confidence defects." This ensures efficient identification of high-confidence defects while overcoming the limitations of a single confidence level through spatiotemporal consistency verification. It effectively filters out misjudgments caused by factors such as image noise and lighting variations, while avoiding the omission of low-confidence but real defects. This significantly improves the accuracy and reliability of target defect identification, providing precise data support for subsequent pipeline defect assessment and maintenance decisions, and ensuring the safety and stability of pipeline operation.

[0039] In one possible implementation, S1042 includes: 1. Determine the multiple frames of images corresponding to the suspected defect in the initial fisheye video stream, and crop them to obtain a spatiotemporal cube containing the suspected defect; refer to Figure 5 For the suspected defect in the panoramic image, based on the mapping table of the stitching record, it was found that the point originated from frames tk to t+k in the initial fisheye video. A small segment of a three-dimensional spatiotemporal cube containing the defect was then cropped from the initial fisheye video.

[0040] 2. Input the spatiotemporal cube into the ConvLSTM network to determine the defect type of the suspected defect; the defect types include: static defects and dynamic defects; The spatiotemporal cube is directly input into the trained ConvLSTM network. The network extracts and matches spatiotemporal features through forward propagation, and finally outputs the classification result of the suspected defect—static defect or dynamic defect.

[0041] 3. If the suspected defect is a static defect, the verification passes; 4. If the suspected defect is a dynamic defect, the verification will fail.

[0042] This application traces back to the temporal frame segment corresponding to the suspected defect in the initial fisheye video stream, and uses a spatiotemporal sequence analysis network to determine the temporal stability of the candidate defect in order to eliminate transient noise interference.

[0043] refer to Figure 5 This application constructs a spatiotemporal cube by cropping multiple frames of images corresponding to suspected defects, which can completely preserve the inter-frame correlation of defects in the time dimension and the morphological features in the spatial dimension. It makes full use of the context information of the video stream and can effectively avoid the verification deviation caused by the partial information of a single frame image.

[0044] The ConvLSTM network combines the spatial feature extraction capability of convolutional neural networks with the temporal dependency capture capability of long short-term memory networks. It can accurately uncover the dynamic change patterns of defects in a spatiotemporal cube and efficiently distinguish between static defects and dynamic interference items. In pipeline scenarios, static defects (such as cracks, corrosion, and damage) are mostly inherent defects of the pipeline itself and are the key targets to focus on. Dynamic interference items (such as flying insects and water droplets) are not defects of the pipeline itself. This network can effectively filter out such interference and reduce the false detection rate from the root.

[0045] Because pipeline defect data exhibits a typical long-tail distribution, samples of severe defects are scarce. Furthermore, the top and bottom edges of the fisheye unfolded image correspond to the center and outermost edge of the original image's field of view; due to projection transformation, the shapes of objects in these regions are stretched. For example, a circular hole may become an ellipse at the edge. Ordinary CNN networks struggle to adapt to this non-rigid deformation. Conventional loss functions treat all regions of the image equally, causing the model to tend to learn features from the central region, resulting in extremely low recognition rates and high false positive rates for defects located at the top or bottom of the pipeline. Therefore, this application employs the YOLOv8 detection network and designs a centrality decay loss function.

[0046] In one possible implementation, the loss function of the deep neural network model can be:

[0047]

[0048] in, For the total loss function, The total number of samples, For the first The attenuation factor for each sample, For hyperparameters, For the first Normalized vertical distance of each sample center from the horizontal center line of the pipeline panorama; Let be the classification loss, representing the predicted class probability. and real category labels The differences between them; The bounding box regression loss represents the predicted regression objective. Regression to the true target The differences between them.

[0049] This application introduces a weighting factor that decays with increasing distance, referencing... Figure 6 , The larger, The smaller; conversely, The smaller, The larger the value, the higher the weight of samples at the image center and the lower the weight of samples at the edges. This allows the model to focus on learning clear, distortion-free defect features at the image center, reducing the contribution of distorted areas at the fisheye edges to the model weights and decreasing gradient noise from blurred edge samples, thereby improving the model's robustness under complex conditions. This strategy reduced the model's false alarm rate by approximately 18%, particularly significantly suppressing false crack alarms at the edges, effectively improving the model's accuracy.

[0050] Furthermore, it can map target defects to standard codes, extract defect geometric parameters and quantify their classification, automatically calculate the repair index and maintenance index of pipe segments, and achieve accurate classification of defects.

[0051] Parameter extraction includes: 1. Calculate the length and width of the crack using the pixel resolution of the panoramic image.

[0052] 2. Using sparse point clouds generated by 3D laser point clouds (if carried by the robot) or visual SLAM, fit the cross-sectional circle of the pipe and calculate the deformation rate.

[0053] Quantitative grading includes: 1. If the defect is a crack, and the width is greater than 1.5mm but not greater than 15mm, and there is no soil loss, the grade is determined to be level 2, with a score of 2, by referring to the table.

[0054] 2. If the defect is deformation and the deformation rate is greater than 15%, the grade is determined to be level 3, with a score of 5, by referring to the table.

[0055] Repair Index Calculation: Repair Index The calculation formula can be:

[0056] in, For the first The score for each defect, For the first The location weights of each defect. For example, the location weight at the vault is 1.1, and the location weight at the sidewall is 1.

[0057] Finally, based on the grade and score, a complete assessment report is generated, including a disease distribution map, a panoramic view, and statistical data. Figure 7 .

[0058] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0059] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0060] Figure 8 A schematic diagram of the pipeline defect detection device provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: like Figure 8 As shown, the pipeline defect detection device includes: The coordinate transformation module 21 is used to acquire the initial fisheye video stream and perform coordinate transformation on the initial fisheye video stream to obtain multiple frames of planar images. The feature point extraction module 22 is used to determine the binarization mask of any frame planar image based on the semantic segmentation network, and extract the feature points of the pipe wall region in the frame planar image with the binarization mask as a constraint, so as to form the feature point set corresponding to the frame planar image. The panoramic image stitching module 23 is used to determine the optimal stitching line between any two adjacent feature point sets using a graph cut algorithm based on energy minimization, and to stitch the images based on the corresponding optimal stitching line to obtain a panoramic image of the pipeline. The defect identification module 24 is used to identify defects in the pipeline panorama to obtain the target defects.

[0061] In one possible implementation, the panoramic image stitching module 23 may include: The seam line determination unit is used to perform feature point matching on any two adjacent feature point sets to determine the overlapping region; construct a Markov random field in the overlapping region and establish an energy function based on the Markov random field; optimize the solution with the minimum energy function as the objective function to obtain the optimal seam line between the two adjacent feature point sets.

[0062] In one possible implementation, the energy function is:

[0063] in, It is an energy function; For data items, representing pixels Differences in luminosity between different source images; For pixels Corresponding source image labels; For smoothing, this indicates a penalty for cutting stitch lines in areas with similar textures or at structural edges. For pixels Corresponding source image labels; For all adjacent pixel pairs A set of.

[0064] In one possible implementation, the defect identification module 24 may include: The identification unit is used to input the panoramic image of the pipeline into a deep neural network model for identification, and obtain multiple suspected defects and their corresponding confidence levels. The verification unit is used to determine the following for any suspected defect: if the confidence level of the suspected defect is not less than the preset confidence level, then the suspected defect is designated as the target defect; if the confidence level of the suspected defect is less than the preset confidence level, then a spatiotemporal consistency verification is performed on the suspected defect; if the verification passes, then the suspected defect is determined to be the target defect; otherwise, the suspected defect is not the target defect.

[0065] In one possible implementation, the verification unit can be specifically used for: 1. Determine the multiple frames of images corresponding to the suspected defect in the initial fisheye video stream, and crop them to obtain a spatiotemporal cube containing the suspected defect; 2. Input the spatiotemporal cube into the ConvLSTM network to determine the defect type of the suspected defect; the defect types include: static defects and dynamic defects; 3. If the suspected defect is a static defect, the verification passes; 4. If the suspected defect is a dynamic defect, the verification will fail.

[0066] In one possible implementation, the loss function of the deep neural network model can be:

[0067]

[0068] in, For the total loss function, The total number of samples, For the first The attenuation factor for each sample. For the first Normalized vertical distance of each sample center from the horizontal center line of the pipeline panorama; Let be the classification loss, representing the predicted class probability. and real category labels The differences between them; The bounding box regression loss represents the predicted regression objective. Regression to the true target The differences between them.

[0069] In one possible implementation, the semantic segmentation network is a Mask R-CNN semantic segmentation network; the feature point extraction module 22 may include: The mask output unit is used to input the planar image of the frame into the Mask R-CNN semantic segmentation network to obtain a binary mask; wherein, the mask value of the pipe wall region is 1, and the mask value of the water surface region is 0. The feature point output unit is used to extract feature points of the pipe wall region using the ORB algorithm to form the feature point set corresponding to the planar image of that frame.

[0070] In one possible implementation, the coordinate transformation module 21 may include: The mapping relationship establishment unit is used to establish the mapping relationship between the polar coordinate system of the fisheye image and the cylindrical coordinate system of the pipe using an equidistant projection model; The coordinate transformation unit is used to convert each frame of fisheye image in the initial fisheye video stream into a planar image based on the mapping relationship. The mapping relationship can be:

[0071]

[0072] in, The optical center of the fisheye image. The coordinates of a pixel in the fisheye image. This is the pixel distance from this pixel in the fisheye image to the optical center; Focal length Angle of incidence; For the pipe radius, This is the distance from the current observation point along the optical axis to the lens.

[0073] Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. For example... Figure 9 As shown, the electronic device 3 of this embodiment includes a processor 30 and a memory 31. The memory 31 stores a computer program 32. When the processor 30 executes the computer program 32, it implements the steps in the various method embodiments described above. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the various device embodiments described above.

[0074] For example, computer program 32 may be divided into one or more modules / units, which are stored in memory 31 and executed by processor 30 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 32 in electronic device 3.

[0075] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 3 may also include input / output devices, network access devices, buses, etc.

[0076] The processor 30 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0077] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store the computer program 32 and other programs and data required by the electronic device 3. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0078] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0079] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0080] This invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0081] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0082] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0083] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting pipeline defects, characterized in that, include: Obtain an initial fisheye video stream, and perform coordinate transformation on the initial fisheye video stream to obtain multiple frames of planar images; For any given frame of planar image, a binary mask for that frame of planar image is determined based on a semantic segmentation network, and feature points of the pipe wall region in that frame of planar image are extracted using the binary mask as a constraint, forming a feature point set corresponding to that frame of planar image; An energy-minimum-based graph cut algorithm is used to determine the optimal stitching line between any two adjacent feature point sets, and the images are stitched together based on the corresponding optimal stitching line to obtain a panoramic view of the pipeline. Defect identification is performed on the panoramic view of the pipeline to obtain the target defect.

2. The pipeline defect detection method according to claim 1, characterized in that, The step of using an energy-minimization-based graph cut algorithm to determine the optimal stitching line between any two adjacent feature point sets includes: For any two adjacent feature point sets, feature point matching is performed on the two adjacent feature point sets to determine the overlapping region; a Markov random field is constructed in the overlapping region, and an energy function is established based on the Markov random field; optimization is performed with the minimum of the energy function as the objective function to obtain the optimal stitching line between the two adjacent feature point sets.

3. The pipeline defect detection method according to claim 2, characterized in that, The energy function is: in, The energy function is... For data items, representing pixels Photometric differences between different source images; For pixels Corresponding source image labels; For smoothing, this indicates a penalty for cutting stitch lines in areas with similar textures or at structural edges. For pixels Corresponding source image labels; For all adjacent pixel pairs A set of.

4. The pipeline defect detection method according to any one of claims 1 to 3, characterized in that, The defect identification process for the pipeline panorama, to obtain the target defect, includes: The panoramic view of the pipeline is input into a deep neural network model for identification, resulting in multiple suspected defects and their corresponding confidence levels. For any suspected defect, if the confidence level of the suspected defect is not less than the preset confidence level, then the suspected defect is designated as the target defect; if the confidence level of the suspected defect is less than the preset confidence level, then a spatiotemporal consistency check is performed on the suspected defect; if the check passes, then the suspected defect is determined to be the target defect; otherwise, the suspected defect is not the target defect.

5. The pipeline defect detection method according to claim 4, characterized in that, The spatiotemporal consistency verification of the suspected defect includes: The suspected defect is identified in the initial fisheye video stream by identifying multiple frames and cropping them to obtain a spatiotemporal cube containing the suspected defect. The spatiotemporal cube is input into a ConvLSTM network to determine the defect type of the suspected defect; wherein, the defect type includes: static defects and dynamic defects; If the suspected defect is a static defect, the verification passes. If the suspected defect is a dynamic defect, the verification will fail.

6. The pipeline defect detection method according to claim 4, characterized in that, The loss function of the deep neural network model is: in, For the total loss function, The total number of samples, For the first The attenuation factor for each sample. For the first Normalized vertical distance of each sample center from the horizontal center line of the panoramic view of the pipeline; Let be the classification loss, representing the predicted class probability. and real category labels The differences between them; The bounding box regression loss represents the predicted regression objective. Regression to the true target The differences between them.

7. The pipeline defect detection method according to any one of claims 1 to 3, characterized in that, The semantic segmentation network is Mask R-CNN semantic segmentation network; the step of determining the binarization mask of the frame planar image based on the semantic segmentation network, and extracting feature points of the pipe wall region in the frame planar image using the binarization mask as a constraint, forming the feature point set corresponding to the frame planar image, includes: The planar image of this frame is input into the Mask R-CNN semantic segmentation network to obtain the binarized mask; wherein, the mask value of the pipe wall region is 1, and the mask value of the water surface region is 0; The ORB algorithm is used to extract feature points in the pipe wall region to form the feature point set corresponding to the frame planar image.

8. The pipeline defect detection method according to any one of claims 1 to 3, characterized in that, The process of performing coordinate transformation on the initial fisheye video stream to obtain multiple frames of planar images includes: An equidistant projection model is used to establish the mapping relationship between the polar coordinate system of the fisheye image and the cylindrical coordinate system of the pipe; Based on the mapping relationship, each frame of fisheye image in the initial fisheye video stream is converted into the planar image; The mapping relationship is as follows: in, The optical center of the fisheye image is... Let be the coordinates of a pixel in the fisheye image. The pixel distance from this pixel in the fisheye image to the optical center; Focal length Angle of incidence; For the pipe radius, This is the distance from the current observation point along the optical axis to the lens.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the pipeline defect detection method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the pipeline defect detection method as described in any one of claims 1 to 8.

Images