Pipeline defect detection method and system and electronic equipment

Abstract

The invention provides a pipeline defect detection method and system and electronic equipment, and the method comprises the following steps: collecting and preprocessing a pipeline video, and obtaining a preprocessed pipeline image; detecting and positioning pipeline defects existing in the preprocessed pipeline image by adopting a pipeline real-time detection algorithm; studying and judging the type of the pipeline defect to obtain a pipeline defect detection result; and generating a pipeline defect detection report according to the pipeline defect detection result. According to the scheme provided by the invention, the pipeline defect is detected by using the pipeline real-time detection algorithm, so that the detection accuracy is improved and the manual operation is reduced; a ChatGLM3-6B technology and an RAG technology are applied to generate a more accurate defect detection report, and a pipeline CCTV detection technology is combined to realize one-button generation of the pipeline defect detection report, so that resource configuration can be optimized, pipelines can be managed and maintained in a targeted manner, and the management capability of an urban drainage system is effectively improved while the cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of pipeline detection, and in particular to a pipeline defect detection method, system and electronic equipment. Background Art

[0002] With the acceleration of my country's urbanization process, the steady growth of urban population, and the continuous expansion of urban scale, the existing drainage network can no longer meet the needs of development. At present, the amount of water supply and drainage in cities is in a continuous growth, but the transformation of municipal pipelines has not kept pace with it. Due to the relatively poor quality of early pipeline materials and the long burial period, defects in drainage pipelines have become inevitable. Therefore, a technology that can effectively detect and identify these defects is urgently needed.

[0003] Municipal drainage pipes are generally located underground, and their concealment makes it difficult to detect defects in the early stages. Once defects are revealed, they often cause serious harm and have adverse effects on society. At present, pipeline television inspection (CCTV inspection) is the most widely used inspection technology, which mainly obtains data through mobile video and images, and these data need to be processed and identified later. Traditionally, manual visual inspection methods are relied upon to identify defects and assess their severity. This process is not only cumbersome and time-consuming, but also has a low degree of intelligence and requires a lot of manpower investment. In addition, the defect identification process is highly dependent on the personal experience and working status of technicians, and is highly subjective. For workers who are inexperienced or fatigued due to long-term work, false detection or misjudgment is prone to occur, followed by missed detection and missed judgment, which further aggravates the hidden dangers of pipeline safety. Summary of the invention

[0004] The embodiments of the present application provide a pipeline defect detection method, system and electronic device, which perform defect detection on pipeline images by constructing a neural network model, thereby improving detection efficiency and detection accuracy and improving productivity.

[0005] The embodiment of the present application provides a pipeline defect detection method, comprising the following steps: collecting pipeline videos and performing preprocessing to obtain preprocessed pipeline images; using pipeline real-time detection algorithms to detect and locate pipeline defects in the preprocessed pipeline images; determining the categories of the pipeline defects to obtain pipeline defect detection results; and generating a pipeline defect detection report based on the pipeline defect detection results; wherein the network structure of the pipeline real-time detection algorithm comprises a backbone network, a neck network and a detection head network, the neck network comprises a CBLL module, a lightweight dynamic upsampling DySample module, and a C2f-OPERA module; the CBLL module is used to reduce the network model size of the pipeline real-time detection algorithm, the DySample module is used to focus the focus area of ​​the pipeline real-time detection algorithm on the inner wall of the pipeline, and the C2f-OPERA module is used to compress the training module of the pipeline real-time detection algorithm into a single convolution.

[0006] In one embodiment, the capturing of pipeline video and preprocessing to obtain a preprocessed pipeline image includes: using a pipeline inspection robot to capture pipeline video; performing frame extraction processing on the pipeline video to obtain an initial pipeline image; and using a dark light image enhancement technology based on deep learning to optimize illumination to enhance the initial pipeline image to obtain a preprocessed pipeline image.

[0007] In one embodiment, the use of a pipeline real-time detection algorithm to detect and locate pipeline defects in the preprocessed pipeline image includes: scaling and normalizing the preprocessed pipeline image to obtain a pipeline image to be detected; using a pipeline real-time detection algorithm to determine whether the pipeline defect exists in the pipeline image to be detected; if the pipeline defect exists in the pipeline image to be detected, determining the position of the pipeline defect in the pipeline image to be detected; and using an image plotting algorithm to frame and mark the defect position to obtain a framed and marked pipeline image.

[0008] In one embodiment, the workflow of the CBLL module includes: using half of the convolution to extract feature information to obtain a first feature map; using a linear transformation to process the remaining feature information to obtain a second half feature map; combining the first feature map and the second feature map to obtain a combined feature map.

[0009] In one embodiment, the determining the category of the pipeline defect to obtain the pipeline defect detection result includes: cropping the framed and annotated pipeline image according to the framed area to obtain the pipeline defect area; inputting the pipeline defect area into a defect determination model to obtain the category of the pipeline defect; and using an image plotting algorithm to mark the category of the pipeline defect on the framed and annotated pipeline image to obtain the pipeline defect detection result.

[0010] In one embodiment, the process of obtaining the defect assessment model includes: establishing a pipeline defect category data set, wherein the pipeline defect category data set includes pipeline defect areas and corresponding pipeline defect categories; dividing the pipeline defect category data set into a pipeline defect category training set and a pipeline defect category verification set; inputting the pipeline defect category training set into the initial defect assessment model for training to obtain a trained defect assessment model; inputting the pipeline defect category verification set into the trained defect assessment model for verification until the model converges to obtain a defect assessment model.

[0011] In one embodiment, after inputting the pipeline defect area into the defect assessment model to obtain the category of the pipeline defect, the method further includes: when detecting and locating the pipeline defect existing in the preprocessed pipeline image, each frame of the processed pipeline image is detected and located according to the playback order of the pipeline video, if the category of the pipeline defect in the current frame is consistent with the category of the pipeline defect in the previous frame, the current frame is discarded.

[0012] In one embodiment, generating a pipeline defect detection report according to the pipeline defect detection result includes: collecting historical pipeline defect data, using the bilingual dialogue language model ChatGLM3-6B technology and the retrieval generation enhanced RAG technology to build a pipeline defect detection report generation model; forming a pipeline defect report outline according to the pipeline defect detection result; calling the pipeline defect detection report generation model to generate the pipeline defect detection report according to the pipeline defect report outline.

[0013] The embodiment of the present application also provides a pipeline defect detection system, including: a data acquisition unit, used to collect pipeline videos and upload them to a preprocessing agent; a preprocessing agent, used to preprocess the pipeline videos to obtain a preprocessed pipeline image; a defect detection agent, used to detect and locate pipeline defects in the preprocessed pipeline image; a defect assessment agent, used to assess the category of the pipeline defects and obtain pipeline defect detection results.

[0014] An embodiment of the present application also provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the above-mentioned pipeline defect detection method.

[0015] The solution provided in the above-mentioned embodiment of the present application uses a real-time pipeline detection algorithm to detect pipeline defects, improves detection accuracy and reduces manual operations, and uses an image plotting algorithm to mark defects, which facilitates the analysis and classification of different defects; ChatGLM3-6B technology and RAG technology are used to generate more accurate defect detection reports, and combined with pipeline CCTV detection technology, one-click generation of pipeline defect detection reports is achieved, which can optimize resource allocation, manage and maintain pipelines in a targeted manner, reduce costs, and effectively improve the management capabilities of urban drainage systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.

[0017] Figure 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application; Figure 2 It is a flow chart of a pipeline defect detection method provided in an embodiment of the present application; Figure 3 is a schematic diagram of an initial pipeline image provided by an embodiment of the present application; Figure 4 is a schematic diagram of a preprocessed pipeline image provided in an embodiment of the present application; Figure 5 is a schematic diagram of the network structure of the pipeline real-time detection algorithm provided in an embodiment of the present application; Figure 6 is a schematic diagram of the network structure of the C2f-OPERA module provided in an embodiment of the present application; Figure 7 is a schematic diagram of the network structure of the Bottleneck-OPERA module provided in an embodiment of the present application; Figure 8 is a schematic diagram of the workflow of the DySample module provided in an embodiment of the present application; Fig. 9 is a schematic diagram of the working process of the CBLL module provided in an embodiment of the present application; Fig.10 is a schematic diagram of the network structure of the DyHead-DCNv3 module provided in an embodiment of the present application; Fig.11 is a schematic diagram of pipeline defect detection results provided by an embodiment of the present application; Fig.12 Schematic diagram of a pipeline defect detection system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0019] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0020] Figure 1 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 100 can be used to execute the pipeline defect detection method provided in an embodiment of the present application. Figure 1 As shown, the electronic device 100 includes: one or more processors 102, and one or more memories 104 storing processor executable instructions. The processor 102 is configured to execute the pipeline defect detection method provided in the following embodiment of the present application.

[0021] The processor 102 can be a gateway, a smart terminal, or a device including a central processing unit (CPU), a graphics processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process data of other components in the electronic device 100 and control other components in the electronic device 100 to perform desired functions.

[0022] The memory 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the pipeline defect detection method described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0023] In one embodiment, Figure 1 The electronic device 100 shown may also include an input device 106, an output device 108, and a data acquisition device 110, which are interconnected via a bus system 112 and / or other forms of connection mechanisms (not shown). Figure 1The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device 100 may also have other components and structures as required.

[0024] The input device 106 may be a device for a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, etc. The output device 108 may output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, a speaker, etc. The data acquisition device 110 may acquire pipeline videos, and store the acquired data in the memory 104 for use by other components. Exemplarily, the data acquisition device 110 may be a pipeline robot.

[0025] In one embodiment, the various components in the example electronic device 100 for implementing the pipeline defect detection method of the embodiment of the present application can be integrated or dispersed, such as integrating the processor 102, memory 104, input device 106 and output device 108 into one, while separating the data acquisition device 110.

[0026] In one embodiment, the example electronic device 100 for implementing the pipeline defect detection method of the embodiment of the present application can be implemented as an intelligent terminal such as a smart phone, a tablet computer, a desktop computer, a server, a vehicle-mounted device, etc.

[0027] Figure 2 Schematic diagram of a pipeline defect detection method according to an embodiment of the present application. Figure 2 As shown, the method includes the following steps 210-250.

[0028] Step 210: Capture pipeline video and perform preprocessing to obtain a preprocessed pipeline image.

[0029] The above step 210 specifically includes: Step 2101: Collect pipeline video.

[0030] Pipeline CCTV inspection technology can be used to collect pipeline videos using pipeline inspection robots.

[0031] Step 2101: preprocess the pipeline video to obtain a preprocessed pipeline image.

[0032] The pipeline video is subjected to frame extraction processing to obtain an initial pipeline image. Therefore, the initial pipeline image is a plurality of frames of the pipeline video, and each frame of the pipeline video corresponds to an initial pipeline image.

[0033] In one embodiment, when performing frame extraction processing, the frame extraction processing can be performed in the playback order of the pipeline video. The obtained initial pipeline image is a plurality of frames in the playback order of the pipeline video, and each frame has a sequential relationship in time sequence. The preprocessed pipeline image, the pipeline image to be detected, the pipeline image after selection and annotation, the pipeline defect area, and the pipeline defect detection result obtained according to the initial pipeline image all have the above-mentioned characteristics of the initial pipeline image. Each pipeline image corresponds to a frame, and each frame has a sequential order in time sequence. The image being processed is called the current frame.

[0034] After obtaining the initial pipeline image, the dark light image enhancement technology based on deep learning optimized illumination is used to enhance the initial pipeline image to obtain a preprocessed pipeline image.

[0035] Figure 3 is a frame obtained by extracting the pipeline video, that is, an example of an initial pipeline image. Figure 3 By enhancing the initial pipeline image, we can get Figure 4 The preprocessed pipeline image in .

[0036] Video can effectively reflect the movement trajectory of an object, but this application is a method for detecting pipeline defects. The defects of the pipeline are fixed, and the defect judgment is based on the characteristics of the defects. Therefore, it is necessary to extract frames from the pipeline video to facilitate feature analysis of the defects in each frame and improve the accuracy of defect screening and classification.

[0037] In actual applications, although the CCTV pipeline inspection robot is used to collect the internal appearance status data of the drainage pipeline, the clarity and frame rate of the collected data can meet the requirements, but the light inside the drainage pipeline is dim, and it is necessary to use an image enhancement algorithm to enhance the pipeline image under dim conditions.

[0038] Step 220: using a pipeline real-time detection algorithm to detect and locate pipeline defects in the preprocessed pipeline image.

[0039] The pipeline real-time detection algorithm is an algorithm that can detect pipeline defects in real time based on pipeline images. The input of the algorithm model is the pipeline image, and the output is the pipeline defects existing in the pipeline image, that is, the specific location of the pipeline defects in the pipeline image.

[0040] In one embodiment, the network structure of the pipeline real-time detection algorithm includes a backbone network, a neck network and a detection head network, and the neck network includes a CBLL module, a lightweight dynamic upsampling DySample module, and a C2f-OPERA module; the CBLL module is used to reduce the network model size of the pipeline real-time detection algorithm, the DySample module is used to focus the focus area of ​​the pipeline real-time detection algorithm around the inner wall of the pipeline, and the C2f-OPERA module is used to compress the training module of the pipeline real-time detection algorithm into a single convolution.

[0041] Specifically, the CBLL module first uses half of the original convolution to extract feature information, uses linear transformation to process the remaining feature information, obtains the other half of the feature map, and then combines the two to generate a new feature map. Using the CBLL module can greatly reduce the size of the network model and reduce computing resources. The DySample module concentrates the sampling points in the pipeline area and ignores the background part, enhancing the algorithm's anti-interference ability. The C2f-OREPA module compresses complex training modules into a single convolution, reducing training costs and ensuring feature expression capabilities.

[0042] Figure 5 is a schematic diagram of the network structure of the pipeline real-time detection algorithm provided in the embodiment of the present application. Figure 5 In the example, the input can be a pipeline image and the output can be a pipeline defect. The network structure of the pipeline real-time detection algorithm includes a backbone network, a neck network and a detection head network.

[0043] The backbone network consists of 10 components: 5 CBL modules, 4 C2f modules and one SPPF module. The CBL module is used for convolution calculations, the C2f module improves the model detection capability through cross-layer feature fusion, and the SPPF module is used to improve the model's multi-scale feature extraction capability and speed up calculation efficiency. The workflow of the backbone network is as follows: the output of the first CBL module is input to the second CBL module, the output of the second CBL module is input to the first C2f module, the output of the first C2f module is input to the third CBL module, the output of the third CBL module is input to the second C2f module, the output of the second C2f module is input to the fourth CBL module, the output of the fourth CBL module is input to the third C2f module, the output of the third C2f module is input to the fifth CBL module, the output of the fifth CBL module is input to the fourth C2f module, and the output of the fourth C2f module is input to the final SPPF module.

[0044] The neck network consists of 8 components: 4 C2f-OPERA modules, 2 Dysample modules and 2 CBLL modules. The C2f-OPERA module converts the complex structure parameters into a single convolution layer to reduce the model calculation consumption, the Dysample module is used to enhance the anti-interference ability of the model, and the CBLL module can effectively reduce the consumption of convolution calculation. The input of the neck network is the output of the SPPF module in the backbone network and the third and second C2f modules. The output of the SPPF module is input to the first Dysample module, the output of the first Dysample module is merged with the output of the third C2f module and input to the first C2f-OPERA module, the output of the first C2f-OPERA is input to the second Dysample module, the output of the second Dysample module is merged with the output of the second C2f module and input to the second C2f-OPERA module, the input of the second C2f-OPERA module is output to the first CBLL module, the output of the first CBLL module is merged and input to the third C2f-OPERA module, the output of the third C2f-OPERA module is input to the second CBLL module, the output of the second CBLL module is merged with the output of the SPPF module and input to the fourth C2f-OPERA module. Finally, the outputs of the 2nd, 3rd, and 4th C2f-OPERA modules are input to the DyHead-DCNv3 detection head (i.e., DyHead-DCNv3 module) in the detection head network. The DyHead-DCNv3 detection head used in this application is used to identify multi-scale pipeline defect targets. It has excellent long-distance modeling capabilities and adaptive spatial aggregation capabilities, and has significant advantages in pipeline defect detection tasks.

[0045] Figure 6 is a schematic diagram of the network structure of the C2f-OPERA module provided in the embodiment of the present application, Figure 6 In the paper, the C2f-OPERA module contains 6 components: 2 CBL modules, 1 Split module and 3 Bottleneck-OPERA modules. The CBL module is used for convolution calculation, the Split module decomposes the output into different scales, and the Bottleneck-OPERA module improves the calculation efficiency by reducing the dimension and compressing the features.

[0046] Figure 7 is a schematic diagram of the network structure of the Bottleneck-OPERA module provided in the embodiment of the present application, Figure 7In the figure, each Bottleneck-OPERA module contains 2 convolutional layers, 2 linear scaling layers, 1 BN layer and a ReLU layer. The BN layer is used for layer normalization to improve model stability, and the ReLU layer uses the ReLU activation function to perform nonlinear transformation on the input.

[0047] The workflow of the C2f-OPERA module is as follows: the output of the CBL module is input into the Split module, the output of the Split module is input into the first Bottleneck-OPERA module, the output of the first Bottleneck-OPERA module is input into the second Bottleneck-OPERA module, the output of the second Bottleneck-OPERA module is input into the third Bottleneck-OPERA module, and the output of the third Bottleneck-OPERA module is merged and input into the second CBL module.

[0048] The workflow of the Bottleneck-OPERA module is as follows: the input feature map is output to the first convolution layer, the output of the first convolution layer is input to the first linear scaling layer, the output of the first linear scaling layer is input to the second convolution layer, the output of the second convolution layer is input to the second linear scaling layer, and the output and input features of the second linear scaling layer are combined. Figure 1 The output of the BN layer is input to the ReLU layer to generate the output of the Bottleneck-OPERA module.

[0049] The Bottleneck-OPERA module partly involves simplifying the sequential structure, and the C2f-OPERA module partly involves simplifying the parallel structure.

[0050] The simplified sequential structure process is as follows: The convolution size is recorded as ,enter , output , the convolution process is , Considering a series of convolutional layers, the sequential convolution process is , Simplify by compressing the equation: , in, W e Represents the end-to-end mapping matrix.

[0051] The process of simplifying the parallel structure by equation compression is as follows:

[0052] Through the above simplification, the C2f-OREPA module can compress complex training modules into a single convolution, reducing training costs and ensuring feature expression capabilities.

[0053] Figure 8 is a schematic diagram of the workflow of the DySample module provided in the embodiment of the present application. Figure 8 The workflow of the Dysample module in the neck network is as follows: Let the input feature map be χ , the size is , point sampling set δ Size . Use the grid_sample function to map the features χ Resample to size of χ′ The generation process of the point sampling set is as follows: The sampling scale factor is s , first use the input and output channels as and 2 respectively gs 2 The linear layer generates a size of The offset , and then reshaped to size The high-resolution raw sampling grid G , and finally the offset O and the original sampling grid G Sum to get the sampling point set .

[0054] Through the above workflow, the DySample module concentrates the sampling points on the pipeline area and ignores the background part, enhancing the algorithm's anti-interference ability.

[0055] Fig. 9 is a schematic diagram of the working process of the CBLL module provided in the embodiment of the present application. Fig. 9 In the figure, the CBLL module in the neck network contains three components: 1 GhostConv module, 1 BN module and 1 ReLU module. The GhostConv module is used to compress the network model size and reduce the number of parameters and calculations. The BN layer is used for layer normalization to improve model stability. The ReLU layer uses the ReLU activation function to perform nonlinear transformation on the input. Among them, the GhostConv module contains 1 Conv layer, 1 Cheapoperation layer and 1 Concat layer. The Conv layer is used for convolution calculation, the Cheap operation layer performs linear transformation on the convolution, and the Concat layer performs convolution merge calculation.

[0056] The workflow of the CBLL module includes: using half of the convolution to extract feature information to obtain the first feature map; using linear transformation to process the remaining feature information to obtain the second half feature map; combining the first feature map and the second feature map to obtain a combined feature map. More specifically, Fig. 9 The workflow of the CBLL module shown is as follows: the GhostConv module first uses a convolution operation of half size to extract feature information and generate half of the feature map; then the other half of the convolution operation generates the other half of the feature map through the Cheapoperation layer, and finally the two halves of the feature map are merged to get the output of the GhostConv layer, the output of the GhostConv module is input to the BN layer, and then the output of the BN layer is input to the ReLU layer.

[0057] Through the above workflow, using the CBLL module can greatly reduce the size of the network model and reduce computing resources.

[0058] Fig.10 is a schematic diagram of the network structure of the DyHead-DCNv3 module provided in the embodiment of the present application. Fig.10 The DyHead-DCNv3 module in the detection head network includes three parts: scale-aware attention calculation, space-aware attention calculation, and task-aware attention calculation. The tensor in the input detection head network is F Size , let the scale attention function on dimension L be π L (·), the spatial perception attention function on dimension S is π S (·), and its task-aware attention function on dimension C is π C (·), and calculate the scale-aware attention, space-aware attention, and task-aware attention in turn. The attention function of the dynamic detection head of DyHead is obtained as follows: .

[0059] The attention function is used to increase the weight of key features, thereby improving the accuracy and efficiency of the model. The calculation process of scale-aware attention is as follows: first, the tensor F is input to the average pooling Avg pool layer, the output of the Avg pool layer is input to the DCNv3 layer, the output of the average pooling layer is input to the ReLU layer, and the output of the ReLU layer is input to the Hardsigmoid layer. The average pooling Avgpool layer is used to compress the feature dimension, the ReLU layer uses the ReLU activation function to perform nonlinear transformation on the input, the DCNv3 layer improves the flexibility of the receptive field by introducing an offset, and the Hard sigmoid layer uses the Hard sigmoid activation function to perform nonlinear transformation on the input.

[0060] The calculation process of spatial perception attention is as follows: the dot product of scale perception attention and tensor F is calculated and input into the Index layer, the output of the Index layer is input into the DCNv3 layer, and the output of the DCNv3 layer is input into the Sigmoid layer and the Offset layer to obtain spatial perception attention and offset respectively. The Sigmoid layer uses the Sigmoid activation function to perform nonlinear transformation on the output, and the Offset layer is used to calculate the offset.

[0061] Among them, the calculation process of task-aware attention is as follows: add the spatial-aware attention and the offset, and calculate the dot product with the tensor F, input it into the average pooling Avg pool layer, input the output of the Avg pool layer into the first fully connected fc layer, input the output of the fc layer into the Relu layer, input the output of the ReLU layer into the second fc layer, and input the output of the fc layer into the normalization Normalize layer to obtain the final attention score. The average pooling Avgpool layer is used to compress the feature dimension, the fully connected fc layer is responsible for converting the feature map into a one-dimensional vector, the ReLU layer uses the ReLU activation function to perform a nonlinear transformation on the input, and the normalization Normalize layer normalizes the input. Finally, an offset of (1,0,0,0) is added to the result, which can strengthen the weight of the channel. The above step 220 specifically includes: Step 2201: scaling and normalizing the preprocessed pipeline image to obtain a pipeline image to be detected.

[0062] In practical applications, the preprocessed pipeline image needs to be detected for defects using a target detection algorithm. For example, the preprocessed image needs to be scaled and normalized to meet the requirements of the target detection algorithm.

[0063] Step 2202: using a pipeline real-time detection algorithm to determine whether the pipeline defect exists in the pipeline image to be detected; if the pipeline defect exists in the pipeline image to be detected, determining the position of the pipeline defect in the pipeline image to be detected.

[0064] Step 2203: Use an image mapping algorithm to select and mark the defect location to obtain a pipeline image after the selection and marking.

[0065] The image labeling algorithm can be OpenCV. OpenCV is an open source computer vision library that contains many algorithms and functions for image processing and computer vision tasks. The image labeling included in OpenCV can perform box selection and labeling on pipeline images to improve the efficiency of defect detection.

[0066] Step 2204: If the pipeline defect does not exist in the pipeline image to be inspected, then inspect the next frame of the pipeline image to be inspected.

[0067] If no pipeline defect is identified in the pipeline image to be inspected, it is considered that the current frame image has no defect, and the current frame image is no longer analyzed and processed, but the next frame pipeline image is inspected.

[0068] Step 230: Determine the type of the pipeline defect and obtain a pipeline defect detection result.

[0069] For the pipeline defects selected and marked in step 220, the categories to which these pipeline defects belong are determined in step 203, and then the pipeline defect detection result is obtained. The pipeline defect detection result can be a pipeline image obtained by marking the categories to which the determined pipeline defects belong on the pipeline image after the selection and marking.

[0070] The above step 230 specifically includes: Step 2301: According to the framed area, the framed and labeled pipeline image is cropped to obtain the pipeline defect area.

[0071] The framed and annotated pipeline image is cropped with a fixed image size. During the cropping process, it should be noted that the cropped image needs to include the area framed in step 220 to ensure that the pipeline defect is included in the cropped image, and the cropped image is the pipeline defect area.

[0072] Step 2302: Input the pipeline defect area into the defect assessment model to obtain the category of the pipeline defect.

[0073] Each type of pipeline defect can be given an independent defect identifier (ID), and a pipeline defect type has a defect ID that uniquely corresponds to it. If the defect ID is introduced when training the defect assessment model, then a defect ID can be obtained by inputting the pipeline defect area into the defect assessment model.

[0074] In one embodiment, it is necessary to detect whether the defect ID of the current frame is consistent with the defect ID of the previous frame. If the defect IDs are consistent, the current frame is discarded to solve the repeated recording of defects in the same video image and avoid affecting the final defect detection.

[0075] Step 2303: using an image marking algorithm to mark the category of the pipeline defect on the framed and marked pipeline image to obtain the pipeline defect detection result.

[0076] Fig.11 It is a pipeline image obtained after marking the category of the determined pipeline defects on the selected and labeled pipeline image, that is, the pipeline defect detection result.

[0077] The process of obtaining the defect analysis model in step 2302 includes the following steps: Step 1: Establish a pipeline defect category dataset, which includes pipeline defect areas and corresponding pipeline defect categories.

[0078] The pipeline defect area can be obtained according to the above steps 200 and 2301, or the pipeline defect area can be cut out by manually observing the pipeline image. The category of the pipeline defect in the pipeline defect area can be determined according to manual observation. The above operation can be performed on many pipeline images collected in history to obtain the pipeline defect area and the corresponding category of the pipeline defect to establish a pipeline defect category data set. At the same time, in order to facilitate the representation of the category of pipeline defects, each type of pipeline defect can also be given an independent defect ID. In this case, the pipeline defect category data set includes the pipeline defect area and the corresponding defect ID.

[0079] Step 2: Divide the pipeline defect category dataset into a pipeline defect category training set and a pipeline defect category verification set.

[0080] 80% of the pipeline defect category data set can be used as the pipeline defect category training set, and 20% can be used as the pipeline defect category verification set for subsequent model training.

[0081] Step 3: Input the pipeline defect category training set into the initial defect analysis model for training to obtain a trained defect analysis model.

[0082] The initial defect analysis model can be a neural network model.

[0083] Step 4: Input the pipeline defect category verification set into the trained defect analysis model for verification until the model converges to obtain a defect analysis model.

[0084] Input the pipeline defect area in the pipeline defect category verification set into the trained defect analysis model to obtain the defect ID output by the model. Compare the defect ID output by the model with the defect ID in the pipeline defect category verification set. If the comparison result meets the preset conditions, the defect analysis model of the current training round will be used as the final defect analysis model. If the comparison result does not meet the preset conditions, the various parameters of the model will be adjusted, and then the model will continue to be trained until the comparison result meets the preset conditions and the model converges. The above-mentioned comparison of the defect ID output by the model with the defect ID in the pipeline defect category verification set can be a comparison of the similarity of the defect ID. Correspondingly, the above-mentioned preset condition is a similarity threshold set according to the empirical value. If the similarity comparison result is greater than or equal to the threshold, it indicates that the model has converged; if the similarity is less than the threshold, it indicates that the model has not converged and further model training is required.

[0085] The solution provided by the above embodiment of the present application uses a target detection algorithm to detect pipeline defects, improves detection accuracy and reduces manual operations, uses an image plotting algorithm to mark defects, and facilitates the analysis and classification of different defects; uses a defect analysis model to classify the categories of pipeline defects to obtain pipeline defect detection results. In summary, by constructing multiple neural network models to detect defects in pipeline images, the detection efficiency and accuracy are improved, and productivity is improved.

[0086] After step 230 is completed, although the pipeline defect detection result can be obtained, the result is not systematic and the visibility is not good. Therefore, the pipeline defect detection method of the embodiment of the present application may also include step 240.

[0087] Step 240: Generate a pipeline defect detection report based on the pipeline defect detection result.

[0088] The above step 240 may specifically include: Step 2401: Collect historical pipeline defect data, use the bilingual conversation language model ChatGLM3-6B technology and retrieval generation enhanced RAG technology to build a pipeline defect detection report generation model.

[0089] Historical pipeline defect data includes records, reports, defect descriptions and other documentation materials of pipeline defects in the past, and is the original data for building a report generation model.

[0090] ChatGLM3-6B is the latest generation of open source models in the ChatGLM series. It retains many excellent features of the previous two generations of models, such as smooth conversation and low deployment threshold, while introducing a more powerful basic model with stronger language processing and generalization capabilities.

[0091] RAG (Retrieval-augmented Generation) is a technology that combines information retrieval and text generation. It can enhance the text generation capability by retrieving relevant external knowledge. Through the RAG plug-in knowledge base in the field of pipeline defect detection, the ChatGLM3-6B model can quickly generate pipeline defect detection reports.

[0092] Step 2402: Based on the pipeline defect detection results, a pipeline defect report outline is formed.

[0093] The pipeline defect detection result can be a pipeline image obtained by marking the category of the determined pipeline defects on the framed and annotated pipeline image. Therefore, the pipeline defect report outline can be formed according to a predetermined format through these pipeline images, the framed areas on the images, and various annotations. The predetermined format needs to meet the generation requirements of the pipeline defect detection report generation model.

[0094] Step 2403: calling the pipeline defect detection report generation model to generate the pipeline defect detection report according to the pipeline defect report outline.

[0095] The solution provided in the above-mentioned embodiment of the present application uses ChatGLM3-6B technology and RAG technology to generate more accurate defect detection reports, and combines pipeline CCTV detection technology to achieve one-click generation of pipeline defect detection reports, which can optimize resource allocation, manage and maintain pipelines in a targeted manner, reduce costs, and effectively improve the management capabilities of urban drainage systems.

[0096] Step 205: The pipeline defect detection report is backed up and stored and then exported.

[0097] Specifically, the pipeline defect detection report is backed up and stored locally and in the cloud, and then exported in an editable format, such as Word format, PDF format, etc., to facilitate modification and annotation of the defect detection report according to actual conditions, and to more conveniently apply it to the management of urban drainage systems.

[0098] The solution provided in the above-mentioned embodiments of the present application can not only store pipeline defect detection reports for later analysis of pipeline data, but also modify and annotate the exported reports, so as to facilitate their application in urban drainage system management.

[0099] Fig.12 Schematic diagram of a pipeline defect detection system according to an embodiment of the present application. Fig.12 As shown, the system includes a data collection unit 310, a preprocessing agent 320, a defect detection agent 330, a defect analysis agent 340, a report generation agent 350 and a report export unit 360.

[0100] The data collection unit 310 is used to collect pipeline videos and upload them to the pre-processing agent.

[0101] The main body of the data acquisition unit 310 is a pipeline robot, which collects pipeline videos and uploads the pipeline videos to the preprocessing agent.

[0102] The preprocessing agent 320 is used to preprocess the pipeline video to obtain a preprocessed pipeline image.

[0103] Specifically, a pipeline inspection robot is used to collect pipeline videos; the pipeline video is subjected to frame extraction processing to obtain an initial pipeline image; and a dark light image enhancement technology based on deep learning to optimize illumination is used to enhance the initial pipeline image to obtain a preprocessed pipeline image.

[0104] The defect detection agent 330 is used to detect and locate pipeline defects in the preprocessed pipeline image.

[0105] Specifically, the preprocessed pipeline image is scaled and normalized to obtain a pipeline image to be detected; a target detection algorithm is used to determine whether the pipeline defect exists in the pipeline image to be detected; if the pipeline defect exists in the pipeline image to be detected, an image plotting algorithm is used to select and mark the defect position to obtain a selected and marked pipeline image.

[0106] The defect assessment agent 340 is used to assess the category of the pipeline defect and obtain pipeline defect detection results.

[0107] Specifically, according to the framed area, the framed and annotated pipeline image is cropped to obtain the pipeline defect area; the pipeline defect area is input into the defect analysis model to obtain the category of the pipeline defect; and the image marking algorithm is used to mark the category of the pipeline defect on the framed and annotated pipeline image to obtain the pipeline defect detection result.

[0108] The report generation agent 350 is used to generate a pipeline defect detection report according to the pipeline defect detection results.

[0109] Specifically, historical pipeline defect data is collected, and a pipeline defect detection report generation model is constructed using the bilingual conversational language model ChatGLM3-6B technology and the retrieval generation enhanced RAG technology; a pipeline defect report outline is formed based on the pipeline defect detection results; the pipeline defect detection report generation model is called to generate the pipeline defect detection report based on the pipeline defect report outline.

[0110] The report export unit 360 is used to export the pipeline defect detection report after backup storage.

[0111] Specifically, the pipeline defect detection report is backed up and stored locally and in the cloud, and then exported in an editable format, such as Word format, PDF format, etc., to facilitate modification and annotation of the defect detection report according to actual conditions, and to more conveniently apply it to the management of urban drainage systems.

[0112] The system provided in the above-mentioned embodiments of the present application can manage and maintain pipelines in a targeted manner, thereby reducing costs and effectively improving the management capabilities of the urban drainage system.

[0113] In several embodiments provided in the present application, the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and a part of a module, a program segment or a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0114] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0115] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

Claims

1. A pipeline defect detection method, characterized in that: The following steps are involved: Collect pipeline videos and perform preprocessing to obtain preprocessed pipeline images; Using a pipeline real-time detection algorithm to detect and locate pipeline defects in the preprocessed pipeline image; Determine the type of the pipeline defect and obtain the pipeline defect detection result; Generate a pipeline defect detection report according to the pipeline defect detection result; The network structure of the pipeline real-time detection algorithm includes a backbone network, a neck network and a detection head network. The neck network includes a CBLL module, a lightweight dynamic upsampling DySample module and a C2f-OPERA module. The CBLL module is used to reduce the network model size of the pipeline real-time detection algorithm, the DySample module is used to focus the focus area of ​​the pipeline real-time detection algorithm around the inner wall of the pipeline, and the C2f-OPERA module is used to compress the training module of the pipeline real-time detection algorithm into a single convolution.

2. The pipeline defect detection method according to claim 1, characterized in that: The collecting pipeline video and preprocessing it to obtain the preprocessed pipeline image includes: Use pipeline inspection robots to collect pipeline videos; Performing frame extraction processing on the pipeline video to obtain an initial pipeline image; The initial pipeline image is enhanced using a dark light image enhancement technology based on deep learning to optimize illumination, thereby obtaining a preprocessed pipeline image.

3. The pipeline defect detection method according to claim 1, characterized in that: The real-time pipeline detection algorithm is used to detect and locate pipeline defects in the preprocessed pipeline image, including: Scaling and normalizing the preprocessed pipeline image to obtain a pipeline image to be detected; Using a pipeline real-time detection algorithm, judging whether the pipeline defect exists in the pipeline image to be detected, and if the pipeline defect exists in the pipeline image to be detected, determining the position of the pipeline defect in the pipeline image to be detected; The defect location is framed and annotated using an image mapping algorithm to obtain a framed and annotated pipeline image.

4. The pipeline defect detection method according to claim 1, characterized in that: The workflow of the CBLL module includes: Use half of the convolution to extract feature information and obtain the first feature map; The remaining feature information is processed by linear transformation to obtain the second half feature map; The first feature map and the second feature map are combined to obtain a combined feature map.

5. The pipeline defect detection method according to claim 3, characterized in that: The determining the type of the pipeline defect and obtaining the pipeline defect detection result includes: According to the framed area, the framed and annotated pipeline image is cropped to obtain a pipeline defect area; Inputting the pipeline defect area into the defect analysis model to obtain the category of the pipeline defect; An image plotting algorithm is used to mark the category of the pipeline defect on the framed and marked pipeline image to obtain the pipeline defect detection result.

6. The pipeline defect detection method according to claim 5, characterized in that: The process of obtaining the defect analysis model includes: Establishing a pipeline defect category data set, wherein the pipeline defect category data set includes pipeline defect areas and corresponding pipeline defect categories; The pipeline defect category data set is divided into a pipeline defect category training set and a pipeline defect category verification set; Inputting the pipeline defect category training set into the initial defect analysis model for training to obtain a trained defect analysis model; The pipeline defect category verification set is input into the trained defect analysis model for verification until the model converges to obtain a defect analysis model.

7. The pipeline defect detection method according to claim 5, characterized in that: After inputting the pipeline defect area into the defect analysis model to obtain the category of the pipeline defect, the method further includes: When detecting and locating the pipeline defect existing in the preprocessed pipeline image, each frame of the processed pipeline image is detected and located according to the playback order of the pipeline video. If the category of the pipeline defect in the current frame is consistent with the category of the pipeline defect in the previous frame, the current frame is discarded.

8. The pipeline defect detection method according to claim 2, characterized in that: Generating a pipeline defect detection report according to the pipeline defect detection result includes: Collect historical pipeline defect data, use the bilingual conversational language model ChatGLM3-6B technology and retrieval generation enhanced RAG technology to build a pipeline defect detection report generation model; Forming a pipeline defect report outline based on the pipeline defect detection results; The pipeline defect detection report generation model is called to generate the pipeline defect detection report according to the pipeline defect report outline.

9. A pipeline defect detection system, characterized in that: include: The data acquisition unit is used to collect pipeline videos and upload them to the preprocessing agent; A preprocessing agent is used to preprocess the pipeline video to obtain a preprocessed pipeline image; A defect detection agent, used to detect and locate pipeline defects in the preprocessed pipeline image; The defect assessment agent is used to assess the category of the pipeline defect and obtain pipeline defect detection results.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the pipeline defect detection method described in any one of claims 1-9.

Images