A pipeline defect detection method, system and electronic device

Abstract

The present application provides a pipeline defect detection method, system and electronic device. The method includes the following steps: collecting pipeline videos and performing preprocessing to obtain preprocessed pipeline images; using a pipeline real-time detection algorithm to detect and locate pipeline defects existing in the preprocessed pipeline images; judging the categories of the pipeline defects to obtain pipeline defect detection results; and generating a pipeline defect detection report according to the pipeline defect detection results. The solution provided by the present application uses a pipeline real-time detection algorithm for pipeline defect detection, improving the detection accuracy and reducing manual operations; applying ChatGLM3-6B technology and RAG technology to generate more accurate defect detection reports, and combining with pipeline CCTV detection technology to realize one-key generation of pipeline defect detection reports, which can optimize resource allocation, manage and maintain pipelines in a targeted manner, reduce costs and effectively improve the management ability of the urban drainage system at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of pipeline detection, and particularly to a pipeline defect detection method, system and electronic device. Background Art

[0002] With the acceleration of the urbanization process in China, the urban population has been steadily increasing, and the urban scale has been continuously expanding. The existing drainage pipe networks can no longer meet the development needs. At present, the water supply and drainage volume in cities is continuously increasing, but the renovation of municipal pipelines has not kept pace. Due to the relatively poor pipeline materials and long burial years in the early stage, defects in drainage pipelines are inevitable. Therefore, there is an urgent need for a technology that can effectively detect and identify these defects.

[0003] Municipal drainage pipelines are generally located underground, and their concealment makes it difficult to detect problems in the initial stage of defects. Once the defects are revealed, they often cause major hazards and have an adverse impact on society. Currently, pipeline television inspection (CloseCircuit Television Inspection, CCTV inspection) is the most widely used inspection technology, mainly obtaining data through mobile - collected videos and images, and these data need to be processed and identified later. Traditionally, manual visual inspection is relied on to identify defects and evaluate their severity. This process is not only cumbersome and time - consuming, but also has a low degree of intelligence and requires a large amount of manpower. In addition, the defect identification process highly depends on the personal experience and working state of technicians, which is highly subjective. For technicians with little experience or those fatigued due to long - term work, it is easy to have misdetection or misjudgment, followed by missed detection and missed judgment phenomena, further exacerbating the hidden danger of pipeline safety. Summary of the Invention

[0004] The embodiments of the present application provide a pipeline defect detection method, system and electronic device. By constructing a neural network model to detect pipeline images, the detection efficiency and detection accuracy are improved, and the productivity is enhanced.

[0005] The embodiments of the present application provide a method for pipeline defect detection, including the following steps: collecting a pipeline video and performing preprocessing to obtain a preprocessed pipeline image; using a pipeline real-time detection algorithm to detect and locate pipeline defects existing in the preprocessed pipeline image; judging the categories of the pipeline defects to obtain a pipeline defect detection result; generating a pipeline defect detection report according to the pipeline defect detection result; wherein, the network structure of the pipeline real-time detection algorithm includes a backbone network, a neck network, and a detection head network, and the neck network includes a CBLL module, a lightweight dynamic upsampling DySample module, and a C2f-OPERA module; the CBLL module is used to reduce the size of the network model of the pipeline real-time detection algorithm, the DySample module is used to make the attention area of the pipeline real-time detection algorithm focus on the surrounding of the pipeline inner wall, and the C2f-OPERA module is used to compress the training module of the pipeline real-time detection algorithm into a single convolution.

[0006] In one embodiment, the step of collecting a pipeline video and performing preprocessing to obtain a preprocessed pipeline image includes: using a pipeline detection robot to collect a pipeline video; performing frame extraction on the pipeline video to obtain an initial pipeline image; using a low-light image enhancement technology based on deep learning to optimize illumination to enhance the initial pipeline image to obtain a preprocessed pipeline image.

[0007] In one embodiment, the step of using a pipeline real-time detection algorithm to detect and locate pipeline defects existing in the preprocessed pipeline image includes: performing scaling and normalization processing on the preprocessed pipeline image to obtain a pipeline image to be detected; using a pipeline real-time detection algorithm to judge whether there are pipeline defects in the pipeline image to be detected; if there are pipeline defects in the pipeline image to be detected, determining the position of the pipeline defects in the pipeline image to be detected; using an image plotting algorithm to box and label the defect position to obtain a boxed and labeled pipeline image.

[0008] In one embodiment, the working process of the CBLL module includes: extracting feature information using half of the convolution to obtain a first feature map; processing the remaining feature information using a linear transformation to obtain a second half feature map; combining the first feature map and the second feature map to obtain a combined feature map.

[0009] In one embodiment, the step of judging the categories of the pipeline defects to obtain a pipeline defect detection result includes: cropping the boxed and labeled pipeline image according to the boxed area to obtain a pipeline defect area; inputting the pipeline defect area into a defect judgment model to obtain the categories of the pipeline defects; using an image plotting algorithm to label the categories of the pipeline defects on the boxed and labeled pipeline image to obtain the pipeline defect detection result.

[0010] In one embodiment, the process of obtaining the defect judgment model includes: establishing a pipeline defect category data set, which includes pipeline defect regions and corresponding pipeline defect categories; dividing the pipeline defect category data set into a pipeline defect category training set and a pipeline defect category validation set; inputting the pipeline defect category training set into an initial defect judgment model for training to obtain a trained defect judgment model; inputting the pipeline defect category validation set into the trained defect judgment model for validation until the model converges to obtain the defect judgment model.

[0011] In one embodiment, after inputting the pipeline defect region into the defect judgment model to obtain the pipeline defect category, the following steps are further included: when detecting and locating the pipeline defects existing in the preprocessed pipeline image by detecting and locating each frame of the processed pipeline image in the order of the pipeline video playback, if the category of the pipeline defect in the current frame is the same as that in the previous frame, the current frame is discarded.

[0012] In one embodiment, generating a pipeline defect detection report according to the pipeline defect detection result includes: collecting historical pipeline defect data, using the bilingual dialogue language model ChatGLM3-6B technology and the retrieval generation enhancement RAG technology to construct a pipeline defect detection report generation model; forming a pipeline defect report outline according to the pipeline defect detection result; calling the pipeline defect detection report generation model to generate the pipeline defect detection report according to the pipeline defect report outline.

[0013] The embodiment of the present application also provides a pipeline defect detection system, including: a data acquisition unit, configured to acquire a pipeline video and upload it to a preprocessing agent; the preprocessing agent, configured to preprocess the pipeline video to obtain a preprocessed pipeline image; a defect detection agent, configured to detect and locate the pipeline defects existing in the preprocessed pipeline image; a defect judgment agent, configured to judge the category of the pipeline defect to obtain a pipeline defect detection result.

[0014] The embodiment of the present application also provides an electronic device, which includes: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to execute the above-mentioned pipeline defect detection method.

[0015] The solution provided in the above embodiments of the present application uses a pipeline real-time detection algorithm to detect pipeline defects, improving the detection accuracy and reducing manual operations. The image plotting algorithm is used to label the defects, facilitating the judgment and classification of different defects. By applying the ChatGLM3-6B technology and the RAG technology, a more accurate defect detection report can be generated. Combining with the pipeline CCTV detection technology, a one-key generation of the pipeline defect detection report can be realized, which can optimize resource allocation, manage and maintain the pipeline in a targeted manner, reduce costs, and effectively improve the management ability of the urban drainage system at the same time. Description of the Drawings

[0016] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for use in the embodiments of the present application will be briefly introduced below.

[0017] Figure 1 is a schematic structural diagram of the electronic device provided by the embodiment of the present application;

[0018] Figure 2 is a schematic flow diagram of a pipeline defect detection method provided by the embodiment of the present application;

[0019] Figure 3 is a schematic diagram of the initial pipeline image provided by the embodiment of the present application;

[0020] Figure 4 is a schematic diagram of the preprocessed pipeline image provided by the embodiment of the present application;

[0021] Figure 5 is a schematic diagram of the network structure of the pipeline real-time detection algorithm provided by the embodiment of the present application;

[0022] Figure 6 is a schematic diagram of the network structure of the C2f-OPERA module provided by the embodiment of the present application;

[0023] Figure 7 is a schematic diagram of the network structure of the Bottleneck-OPERA module provided by the embodiment of the present application;

[0024] Figure 8 is a schematic diagram of the working process of the DySample module provided by the embodiment of the present application;

[0025] Figure 9 is a schematic diagram of the working process of the CBLL module provided by the embodiment of the present application;

[0026] Figure 10 is a schematic diagram of the network structure of the DyHead-DCNv3 module provided by the embodiment of the present application;

[0027] Figure 11It is a schematic diagram of the pipeline defect detection result provided by the embodiment of the present application;

[0028] Figure 12 It is a schematic diagram of a pipeline defect detection system provided by the embodiment of the present application. Specific embodiments

[0029] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application.

[0030] Similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present application, terms such as "first" and "second" are only used for differential description and cannot be construed as indicating or implying relative importance.

[0031] Figure 1 It is a schematic diagram of the structure of an electronic device provided by the embodiment of the present application. The electronic device 100 can be used to execute the pipeline defect detection method provided by the embodiment of the present application. As Figure 1 shown, the electronic device 100 includes: one or more processors 102, and one or more memories 104 for storing instructions executable by the processor. Among them, the processor 102 is configured to execute the pipeline defect detection method provided in the following embodiments of the present application.

[0032] The processor 102 can be a gateway, or a smart terminal, or a device including a central processing unit (CPU), an image processing unit (GPU), or other forms of processing units with data processing capabilities and / or instruction execution capabilities. It can process the data of other components in the electronic device 100 and can also control other components in the electronic device 100 to perform desired functions.

[0033] The memory 104 may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the pipeline defect detection method described below. Various application programs and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the application programs, etc.

[0034] In one embodiment, Figure 1The illustrated electronic device 100 may further include an input device 106, an output device 108, and a data acquisition device 110, and these components are interconnected through a bus system 112 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the illustrated electronic device 100 are merely exemplary and not restrictive. According to requirements, the electronic device 100 may also have other components and structures.

[0035] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, etc. The output device 108 may output various information (such as images or sounds) to the outside (for example, to the user), and may include one or more of a display, a speaker, etc. The data acquisition device 110 may acquire pipeline videos and store the acquired data in the memory 104 for use by other components. Exemplarily, the data acquisition device 110 may be a pipeline robot.

[0036] In one embodiment, the various devices in the exemplary electronic device 100 for implementing the pipeline defect detection method of the embodiments of the present application may be integrally arranged or dispersedly arranged. For example, the processor 102, the memory 104, the input device 106, and the output device 108 may be integrally arranged, while the data acquisition device 110 is separately arranged.

[0037] In one embodiment, the exemplary electronic device 100 for implementing the pipeline defect detection method of the embodiments of the present application may be implemented as a smart terminal such as a smart phone, a tablet computer, a desktop computer, a server, a vehicle-mounted device, etc.

[0038] Figure 2 is a schematic flowchart of a pipeline defect detection method according to an embodiment of the present application. As Figure 2 shown, the method includes the following steps 210 - step 250.

[0039] Step 210: Acquire a pipeline video and perform preprocessing to obtain a preprocessed pipeline image.

[0040] The above step 210 specifically includes:

[0041] Step 2101: Acquire a pipeline video.

[0042] The pipeline CCTV detection technology may be used, and a pipeline detection robot may be used to acquire the pipeline video.

[0043] Step 2101: Perform preprocessing on the pipeline video to obtain a preprocessed pipeline image.

[0044] Perform frame extraction on the pipeline video to obtain the initial pipeline images. Therefore, the initial pipeline images are multiple frames of the pipeline video, and each frame of the pipeline video corresponds to an initial pipeline image.

[0045] In one embodiment, when performing frame extraction, frame extraction can be carried out in the playing order of the pipeline video. The obtained initial pipeline images are multiple frames in the playing order of the pipeline video, and each frame has a chronological relationship. The preprocessed pipeline images, the pipeline images to be detected, the framed and labeled pipeline images, the pipeline defect regions, and the pipeline defect detection results obtained from the initial pipeline images all have the above characteristics inherited from the initial pipeline images. Each pipeline image corresponds to a frame, and each frame has a chronological order. The image being processed is called the current frame.

[0046] After obtaining the initial pipeline images, use the dark image enhancement technology based on deep learning to optimize illumination to enhance the initial pipeline images and obtain the preprocessed pipeline images.

[0047] Figure 3 is an example of a certain frame obtained by performing frame extraction on the pipeline video, that is, an initial pipeline image. Figure 3 Enhance the initial pipeline image of Figure 4 to obtain the preprocessed pipeline image in

[0048] Videos can effectively reflect the movement trajectories of objects. However, this application is a method for detecting pipeline defects. The defects in the pipeline are fixed, and defect judgment is based on the characteristics of the defects. Therefore, it is necessary to perform frame extraction on the pipeline video to facilitate feature analysis of the defects in each frame and improve the accuracy of defect screening and classification.

[0049] In practical applications, although a CCTV pipeline inspection robot is used to collect the internal appearance state data of the drainage pipeline, the clarity and frame rate of the collected data can meet the requirements. However, the light inside the drainage pipeline is dim, and an image enhancement algorithm is still needed to enhance the pipeline images under dim conditions.

[0050] Step 220: Use a pipeline real-time detection algorithm to detect and locate the pipeline defects existing in the preprocessed pipeline images.

[0051] Regarding the pipeline real-time detection algorithm, it is an algorithm that can detect pipeline defects in real time according to pipeline images. The input of the algorithm model is the pipeline image, and the output is the pipeline defects existing in the pipeline image, that is, the specific positions of the pipeline defects in the pipeline image.

[0052] In one embodiment, the network structure of the pipeline real-time detection algorithm includes a backbone network, a neck network, and a detection head network. The neck network includes a CBLL module, a lightweight dynamic upsampling DySample module, and a C2f-OPERA module. The CBLL module is used to reduce the size of the network model of the pipeline real-time detection algorithm. The DySample module is used to focus the region of interest of the pipeline real-time detection algorithm around the inner wall of the pipeline. The C2f-OPERA module is used to compress the training module of the pipeline real-time detection algorithm into a single convolution.

[0053] Specifically, the CBLL module first uses half of the original convolution to extract feature information, processes the remaining feature information using a linear transformation to obtain the other half of the feature map, and then combines the two to generate a new feature map. Using the CBLL module can greatly reduce the size of the network model and reduce computational resources. The DySample module makes the sampling points concentrate on the pipeline area and ignores the background part, enhancing the anti-interference ability of the algorithm. The C2f-OREPA module compresses the complex training module into a single convolution, reducing the training cost and ensuring the feature expression ability.

[0054] Figure 5 is a schematic diagram of the network structure of the pipeline real-time detection algorithm provided by the embodiment of the present application. In Figure 5 Input can be a pipeline image, and Output can be a pipeline defect. The network structure of the pipeline real-time detection algorithm includes a backbone network, a neck network, and a detection head network.

[0055] The backbone network contains 10 components: 5 CBL modules, 4 C2f modules, and one SPPF module. The CBL module is used for convolutional calculation. The C2f module improves the model detection ability through cross-layer feature fusion. The SPPF module is used to improve the multi-scale feature extraction ability of the model and speed up the calculation efficiency. The working process of the backbone network is as follows: the output of the first CBL module is input to the second CBL module, the output of the second CBL module is input to the first C2f module, the output of the first C2f module is input to the third CBL module, the output of the third CBL module is input to the second C2f module, the output of the second C2f module is input to the fourth CBL module, the output of the fourth CBL module is input to the third C2f module, the output of the third C2f module is input to the fifth CBL module, the output of the fifth CBL module is input to the fourth C2f module, and the output of the fourth C2f module is input to the final SPPF module.

[0056] The neck network consists of 8 components: 4 C2f-OPERA modules, 2 Dysample modules, and 2 CBLL modules. The C2f-OPERA module reparameterizes complex structures into a single convolutional layer to reduce the model's computational consumption. The Dysample module is used to enhance the model's anti-interference ability, and the CBLL module can effectively reduce the consumption of convolutional calculations. The inputs to the neck network are the outputs of the SPPF module, the 3rd and 2nd C2f modules in the backbone network respectively. The output of the SPPF module is input into the 1st Dysample module. The output of the 1st Dysample module is merged with the output of the 3rd C2f module and input into the 1st C2f-OPERA module. The output of the 1st C2f-OPERA is input into the 2nd Dysample module. The output of the 2nd Dysample module is merged with the output of the 2nd C2f module and input into the 2nd C2f-OPERA module. The output of the 2nd C2f-OPERA module is input into the 1st CBLL module. The output of the 1st CBLL module is merged and input into the 3rd C2f-OPERA module. The output of the 3rd C2f-OPERA module is input into the 2nd CBLL module. The output of the 2nd CBLL module is merged with the output of the SPPF module and input into the 4th C2f-OPERA module. Finally, the outputs of the 2nd, 3rd, and 4th C2f-OPERA modules are input into the DyHead-DCNv3 detector (i.e., the DyHead-DCNv3 module) in the detector head network. The DyHead-DCNv3 detector adopted in this application is used to identify multi-scale pipeline defect targets, has excellent long-distance modeling ability and adaptive spatial aggregation ability, and has significant advantages in the pipeline defect detection task.

[0057] Figure 6 It is a schematic diagram of the network structure of the C2f-OPERA module provided by the embodiment of this application. In Figure 6 it, the C2f-OPERA module contains 6 components: 2 CBL modules, 1 Split module, and 3 Bottleneck-OPERA modules. The CBL module is used for convolutional calculations. The Split module decomposes the output into different scales. The Bottleneck-OPERA module improves the computational efficiency by dimension reduction and feature compression.

[0058] Figure 7 It is a schematic diagram of the network structure of the Bottleneck-OPERA module provided by the embodiment of this application. In Figure 7Among them, each Bottleneck-OPERA module contains 2 convolutional layers, 2 linear scaling layers, 1 BN layer, and 1 ReLU layer. The BN layer is used for layer normalization to improve the stability of the model, and the ReLU layer uses the ReLU activation function to perform a non-linear transformation on the input.

[0059] The working process of the C2f-OPERA module is as follows: The output of the CBL module is input into the Split module, the output of the Split module is input into the first Bottleneck-OPERA module, the output of the first Bottleneck-OPERA module is input into the second Bottleneck-OPERA module, the output of the second Bottleneck-OPERA module is input into the third Bottleneck-OPERA module, and after the output of the third Bottleneck-OPERA module is merged, it is input into the second CBL module.

[0060] Among them, the working process of the Bottleneck-OPERA module is as follows: The input feature map is output to the first convolutional layer, the output of the first convolutional layer is input into the first linear scaling layer, the output of the first linear scaling layer is input into the second convolutional layer, the output of the second convolutional layer is input into the second linear scaling layer, and the output of the second linear scaling layer and the input Figure 1 feature are input into the BN layer together. The output of the BN layer is input into the ReLU layer to generate the output of the Bottleneck-OPERA module.

[0061] The Bottleneck-OPERA module part involves simplifying the sequential structure, and the C2f-OPERA module part involves simplifying the parallel structure.

[0062] The process of simplifying the sequential structure is as follows: Denote the convolution size as The process of simplifying the sequential structure is as follows: Denote the convolution size as , input , output , and its convolution process is

[0063] ,

[0064] Considering a series of convolutional layers, the sequential convolution process is

[0065] ,

[0066] Simplify through equation compression:

[0067] ,

[0068] Among them, W eRepresents the end-to-end mapping matrix.

[0069] The process of simplifying the parallel structure through equation compression is as follows:

[0070]

[0071] Through the above simplification, the C2f-OREPA module can compress the complex training module into a single convolution, reducing the training cost and ensuring the feature expression ability.

[0072] Figure 8 is a schematic diagram of the working process of the DySample module provided by the embodiment of the present application. In Figure 8 The working process of the Dysample module in the neck network is as follows: Denote the input feature map as χ , with a size of , and the point sampling set δ with a size of . Use the grid_sample function to resample the feature map χ to a size of of χ′ . Among them, the generation process of the point sampling set is as follows: Denote the upsampling scale factor as s , first use a linear layer with input and output channels of and 2 respectively gs 2 to generate an offset with a size of , then reshape it into a high-resolution original sampling grid with a size of G through pixel recombination, and finally sum the offset O and the original sampling grid G to obtain the sampling point set .

[0073] Through the above working process, the DySample module makes the sampling points concentrated in the pipeline area and ignores the background part, enhancing the anti-interference ability of the algorithm.

[0074] Figure 9 is a schematic diagram of the working process of the CBLL module provided by the embodiment of the present application. In Figure 9In it, the CBLL module in the neck network contains 3 components: 1 GhostConv module, 1 BN module, and 1 ReLU module. The GhostConv module is used to compress the size of the network model and reduce the number of parameters and computational volume. The BN layer is used for layer normalization to improve the model stability. The ReLU layer uses the ReLU activation function to perform a non-linear transformation on the input. Among them, the GhostConv module contains 1 Conv layer, 1 Cheapoperation layer, and 1 Concat layer. The Conv layer is used to perform convolution calculations. The Cheap operation layer performs a linear transformation on the convolution. The Concat layer performs a combined calculation on the convolution.

[0075] The working process of the CBLL module includes: extracting feature information using half of the convolution to obtain the first feature map; processing the remaining feature information using a linear transformation to obtain the second half feature map; combining the first feature map and the second feature map to obtain the combined feature map. More specifically, as Figure 9 shown, the working process of the CBLL module is as follows: The GhostConv module first uses a convolution operation of half the size to extract feature information and generate half of the feature map; then the other half of the convolution operation generates the other half of the feature map through the Cheapoperation layer. Finally, the two half feature maps are combined to obtain the output of the GhostConv layer. The output of the GhostConv module is input into the BN layer, and then the output of the BN layer is input into the ReLU layer.

[0076] Through the above working process, using the CBLL module can greatly reduce the size of the network model and reduce computational resources.

[0077] Figure 10 It is a schematic diagram of the network structure of the DyHead-DCNv3 module provided by the embodiment of the present application. In Figure 10 it, the DyHead-DCNv3 module in the detection head network contains three parts: scale-aware attention calculation, space-aware attention calculation, and task-aware attention calculation. Denote the tensor F input into the detection head network as in size, and denote the scale attention function on its dimension L as π L (·), the space-aware attention function on dimension S as π S (·), and the task-aware attention function on its dimension C as π C (·). Calculate the scale-aware attention, space-aware attention, and task-aware attention in sequence, and the attention function of the dynamic detection head of the obtained DyHead is

[0078] .

[0079] The attention function is used to increase the weight of key features, thereby improving the accuracy and efficiency of the model. Among them, the calculation process of scale-aware attention is as follows: First, input the tensor F into the average pooling Avg pool layer, input the output of the Avg pool layer into the DCNv3 layer, input the output of the average pooling layer into the ReLU layer, and input the output of the ReLU layer into the Hardsigmoid layer. The average pooling Avgpool layer is used to compress the feature dimension, the ReLU layer uses the ReLU activation function to perform a non-linear transformation on the input, the DCNv3 layer improves the flexibility of the receptive field by introducing offsets, and the Hard sigmoid layer uses the Hard sigmoid activation function to perform a non-linear transformation on the input.

[0080] Among them, the calculation process of spatial-aware attention is as follows: Take the dot product of the scale-aware attention and the tensor F and input it into the Index layer, input the output of the Index layer into the DCNv3 layer, input the output of the DCNv3 layer into the Sigmoid layer and the Offset layer to obtain the spatial-aware attention and the offset respectively. The Sigmoid layer uses the Sigmoid activation function to perform a non-linear transformation on the output, and the Offset layer is used to calculate the offset.

[0081] Among them, the calculation process of task-aware attention is as follows: Add the spatial-aware attention and the offset, take the dot product with the tensor F, input it into the average pooling Avg pool layer, input the output of the Avg pool layer into the first fully connected fc layer, input the output of the fc layer into the Relu layer, input the output of the ReLU layer into the second fc layer, and input the output of the fc layer into the normalization Normalize layer to obtain the final attention score. Among them, the average pooling Avgpool layer is used to compress the feature dimension, the fully connected fc layer is responsible for converting the feature map into a one-dimensional vector, the ReLU layer uses the ReLU activation function to perform a non-linear transformation on the input, the normalization Normalize layer normalizes the input, and finally add an offset of (1, 0, 0, 0) to the result, which can strengthen the weight of the channel. The above step 220 specifically includes:

[0082] Step 2201: Scale and normalize the preprocessed pipeline image to obtain the pipeline image to be detected.

[0083] In practical applications, when defect detection is to be performed on the preprocessed pipeline image, a target detection algorithm needs to be used for detection. For example, a target detection algorithm can be used for detection, and the preprocessed image needs to be scaled and normalized to meet the usage requirements of the target detection algorithm.

[0084] Step 2202: Use a pipeline real-time detection algorithm to determine whether there is such a pipeline defect in the pipeline image to be detected; if there is such a pipeline defect in the pipeline image to be detected, determine the position of the pipeline defect in the pipeline image to be detected.

[0085] Step 2203: Use an image plotting algorithm to box and label the defect position to obtain the pipeline image after boxing and labeling.

[0086] The image plotting algorithm can be OpenCV. OpenCV is an open-source computer vision library that contains many algorithms and functions for image processing and computer vision tasks. The image plotting it contains can box and label on the pipeline image, improving the efficiency of defect detection.

[0087] Step 2204: If there is no such pipeline defect in the pipeline image to be detected, perform the detection on the next frame of the pipeline image to be detected.

[0088] If it is not recognized that there is a pipeline defect in the pipeline image to be detected, it is considered that there is no defect in the current frame image, and the current frame image is no longer analyzed and processed. Instead, the detection of the next frame of the pipeline image is performed.

[0089] Step 230: Analyze and judge the category of the pipeline defect to obtain the pipeline defect detection result.

[0090] For the pipeline defects boxed and labeled in Step 220, in Step 203, the categories to which these pipeline defects belong will be analyzed and judged, and then the pipeline defect detection result will be obtained. The pipeline defect detection result can be the pipeline image obtained after labeling the category to which the judged pipeline defect belongs on the pipeline image after boxing and labeling.

[0091] The above Step 230 specifically includes:

[0092] Step 2301: According to the boxed area, crop the pipeline image after boxing and labeling to obtain the pipeline defect area.

[0093] Crop the pipeline image after boxing and labeling with a fixed picture size. During the cropping process, it should be noted that the cropped image needs to include the area boxed in Step 220 to ensure that the pipeline defect is included in the cropped image. The cropped image is the pipeline defect area.

[0094] Step 2302: Input the pipeline defect area into the defect analysis and judgment model to obtain the category of the pipeline defect.

[0095] An independent defect identifier (ID) can be assigned to each type of pipeline defect, and there is a unique defect ID corresponding to each type of pipeline defect. If the defect ID is introduced during the training of the defect judgment model, then when the pipeline defect area is input into the defect judgment model, a defect ID can be obtained.

[0096] In one embodiment, it is necessary to detect whether the defect ID of the current frame is the same as the defect ID of the previous frame. If the defect IDs are the same, the current frame is discarded to solve the problem of duplicate recording of defects in the same video image and avoid affecting the final defect detection.

[0097] Step 2303: Use an image plotting algorithm to label the category of the pipeline defect on the boxed and labeled pipeline image to obtain the pipeline defect detection result.

[0098] Figure 11 It is the pipeline image obtained by labeling the category to which the judged pipeline defect belongs on the boxed and labeled pipeline image, that is, the pipeline defect detection result.

[0099] The process of obtaining the defect judgment model in the above step 2302 includes the following steps:

[0100] Step 1: Establish a pipeline defect category dataset, which includes pipeline defect areas and the corresponding categories of pipeline defects.

[0101] The pipeline defect area can be obtained according to the above steps 200 and 2301, or the pipeline defect area can be cropped by manually observing the pipeline image. The category of the pipeline defect in the pipeline defect area can be determined by manual observation. The above operations can be performed on a large number of historical pipeline images to obtain pipeline defect areas and the corresponding categories of pipeline defects to establish a pipeline defect category dataset. At the same time, in order to facilitate the representation of the categories of pipeline defects, an independent defect ID can also be assigned to each type of pipeline defect. At this time, the pipeline defect category dataset includes pipeline defect areas and the corresponding defect IDs.

[0102] Step 2: Divide the pipeline defect category dataset into a pipeline defect category training set and a pipeline defect category validation set.

[0103] 80% of the pipeline defect category dataset can be used as the pipeline defect category training set, and 20% can be used as the pipeline defect category validation set for subsequent model training.

[0104] Step 3: Input the pipeline defect category training set into the initial defect judgment model for training to obtain the trained defect judgment model.

[0105] The initial defect judgment model can be a neural network model.

[0106] Step 4: Input the pipeline defect category verification set into the trained defect judgment model for verification until the model converges, and obtain the defect judgment model.

[0107] Input the pipeline defect areas in the pipeline defect category verification set into the trained defect judgment model to obtain the defect IDs output by the model. Compare the defect IDs output by the model with the defect IDs in the pipeline defect category verification set. If the comparison result meets the preset conditions, the defect judgment model of the current training round will be used as the final defect judgment model. If the comparison result does not meet the preset conditions, various parameters of the model will be adjusted, and then the model will continue to be trained until the comparison result meets the preset conditions and the model converges. The above comparison of the defect IDs output by the model with the defect IDs in the pipeline defect category verification set can be a comparison of the similarity of the defect IDs. Correspondingly, the above preset condition is a similarity threshold set according to empirical values. If the similarity comparison result is greater than or equal to the threshold, it indicates that the model has converged; if the similarity is less than the threshold, it indicates that the model has not converged and further model training is required.

[0108] The solution provided in the above embodiments of the present application uses an object detection algorithm to detect pipeline defects, improves the detection accuracy and reduces manual operations, uses an image plotting algorithm to label defects, which is convenient for the judgment and classification of different defects; uses a defect judgment model to classify the categories of pipeline defects to obtain the pipeline defect detection results. Generally speaking, by constructing multiple neural network models to detect defects in pipeline images, the detection efficiency and detection accuracy are improved, and the productivity is increased.

[0109] After step 230 is executed, although the pipeline defect detection result can be obtained, the result is not systematic and the visibility is not good. Therefore, the pipeline defect detection method of the embodiments of the present application may further include step 240.

[0110] Step 240: Generate a pipeline defect detection report according to the pipeline defect detection result.

[0111] The above step 240 may specifically include:

[0112] Step 2401: Collect historical pipeline defect data, and use the bilingual dialogue language model ChatGLM3-6B technology and the retrieval generation enhancement RAG technology to construct a pipeline defect detection report generation model.

[0113] Historical pipeline defect data includes document materials such as records, reports, and defect descriptions of pipeline defects in the past, which are the original data for constructing the report generation model.

[0114] ChatGLM3-6B is the latest generation of open source models in the ChatGLM series. It retains many excellent features of the previous two generations of models, such as smooth conversation and low deployment threshold, while introducing a more powerful basic model with stronger language processing and generalization capabilities.

[0115] RAG (Retrieval-augmented Generation) is a technology that combines information retrieval and text generation. It can enhance the text generation capability by retrieving relevant external knowledge. Through the RAG plug-in knowledge base in the field of pipeline defect detection, the ChatGLM3-6B model can quickly generate pipeline defect detection reports.

[0116] Step 2402: Based on the pipeline defect detection results, a pipeline defect report outline is formed.

[0117] The pipeline defect detection result can be a pipeline image obtained by marking the category of the pipeline defect determined on the framed and annotated pipeline image. Therefore, the pipeline defect report outline can be formed according to a predetermined format through these pipeline images, the framed areas on the images, and various annotations. The predetermined format needs to meet the generation requirements of the pipeline defect detection report generation model.

[0118] Step 2403: calling the pipeline defect detection report generation model to generate the pipeline defect detection report according to the pipeline defect report outline.

[0119] The solution provided in the above-mentioned embodiment of the present application uses ChatGLM3-6B technology and RAG technology to generate more accurate defect detection reports, and combines pipeline CCTV detection technology to achieve one-click generation of pipeline defect detection reports, which can optimize resource allocation, manage and maintain pipelines in a targeted manner, reduce costs, and effectively improve the management capabilities of urban drainage systems.

[0120] Step 205: The pipeline defect detection report is backed up and stored and then exported.

[0121] Specifically, the pipeline defect detection report is backed up and stored locally and in the cloud, and then exported in an editable format, such as Word format, PDF format, etc., to facilitate modification and annotation of the defect detection report according to actual conditions, and to more conveniently apply it to the management of urban drainage systems.

[0122] The solution provided in the above-mentioned embodiments of the present application can not only store pipeline defect detection reports for later analysis of pipeline data, but also modify and annotate the exported reports, so as to facilitate their application in urban drainage system management.

[0123] Figure 12 It is a schematic diagram of a pipeline defect detection system according to an embodiment of the present application. As Figure 12 shown, the system includes a data acquisition unit 310, a preprocessing agent 320, a defect detection agent 330, a defect judgment agent 340, a report generation agent 350, and a report export unit 360.

[0124] The data acquisition unit 310 is used to collect pipeline videos and upload them to the preprocessing agent.

[0125] The main body of the data acquisition unit 310 is a pipeline robot, which collects pipeline videos through the pipeline robot and uploads the pipeline videos to the preprocessing agent.

[0126] The preprocessing agent 320 is used to preprocess the pipeline videos to obtain preprocessed pipeline images.

[0127] Specifically, a pipeline inspection robot is used to collect pipeline videos; the pipeline videos are frame-extracted to obtain initial pipeline images; the initial pipeline images are enhanced using a deep learning-optimized low-light image enhancement technology for illumination to obtain preprocessed pipeline images.

[0128] The defect detection agent 330 is used to detect and locate pipeline defects existing in the preprocessed pipeline images.

[0129] Specifically, the preprocessed pipeline images are scaled and normalized to obtain pipeline images to be detected; a target detection algorithm is used to determine whether there are the pipeline defects in the pipeline images to be detected; if there are the pipeline defects in the pipeline images to be detected, an image plotting algorithm is used to frame and label the defect positions to obtain framed and labeled pipeline images.

[0130] The defect judgment agent 340 is used to judge the categories of the pipeline defects to obtain pipeline defect detection results.

[0131] Specifically, according to the framed area, the framed and labeled pipeline images are cropped to obtain pipeline defect areas; the pipeline defect areas are input into a defect judgment model to obtain the categories of the pipeline defects; an image plotting algorithm is used to label the categories of the pipeline defects on the framed and labeled pipeline images to obtain the pipeline defect detection results.

[0132] The report generation agent 350 is used to generate a pipeline defect detection report according to the pipeline defect detection results.

[0133] Specifically, historical pipeline defect data is collected, and a pipeline defect detection report generation model is constructed using the bilingual dialogue language model ChatGLM3-6B technology and the retrieval-augmented generation (RAG) technology; according to the pipeline defect detection results, an outline of the pipeline defect report is formed; the pipeline defect detection report generation model is called to generate the pipeline defect detection report according to the outline of the pipeline defect report.

[0134] A report export unit 360 is configured to back up and store the pipeline defect detection report and then export it.

[0135] Specifically, the pipeline defect detection report is backed up and stored locally and in the cloud, and then the pipeline defect detection report is exported in an editable format, such as Word format, PDF format, etc., which is convenient for modifying and annotating the defect detection report according to the actual situation and can be more conveniently applied to the management work of the urban drainage system.

[0136] The system provided in the above embodiments of the present application can specifically manage and maintain the pipeline, reduce costs and effectively improve the management ability of the urban drainage system at the same time.

[0137] In several embodiments provided by the present application, the disclosed devices and methods can also be implemented in other ways. The device and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and a module, a program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0138] In addition, in each embodiment of the present application, the various functional modules may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

[0139] If a function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

Claims

1. A pipeline defect detection method, characterized in that: The following steps are involved: Collect pipeline videos and perform preprocessing to obtain preprocessed pipeline images; Using a pipeline real-time detection algorithm to detect and locate pipeline defects in the preprocessed pipeline image; Determine the type of the pipeline defect and obtain the pipeline defect detection result; Generate a pipeline defect detection report according to the pipeline defect detection result; The network structure of the pipeline real-time detection algorithm includes a backbone network, a neck network and a detection head network, and the neck network includes a CBLL module, a lightweight dynamic upsampling DySample module, and a C2f-OPERA module; the CBLL module is used to reduce the network model size of the pipeline real-time detection algorithm, the DySample module is used to focus the focus area of ​​the pipeline real-time detection algorithm around the inner wall of the pipeline, and the C2f-OPERA module is used to compress the training module of the pipeline real-time detection algorithm into a single convolution; The C2f-OPERA module includes 6 components: 2 CBL modules, 1 Split module and 3 Bottleneck-OPERA modules; the CBL module is used to perform convolution calculations, the Split module decomposes the output into different scales, and the Bottleneck-OPERA module improves computing efficiency by dimensionality reduction and feature compression.

2. The pipeline defect detection method according to claim 1, characterized in that: The collecting pipeline video and preprocessing it to obtain the preprocessed pipeline image includes: Use pipeline inspection robots to collect pipeline videos; Performing frame extraction processing on the pipeline video to obtain an initial pipeline image; The initial pipeline image is enhanced using a dark light image enhancement technology based on deep learning to optimize illumination, thereby obtaining a preprocessed pipeline image.

3. The pipeline defect detection method according to claim 1, characterized in that: The real-time pipeline detection algorithm is used to detect and locate pipeline defects in the preprocessed pipeline image, including: Scaling and normalizing the preprocessed pipeline image to obtain a pipeline image to be detected; Using a pipeline real-time detection algorithm, judging whether the pipeline defect exists in the pipeline image to be detected, and if the pipeline defect exists in the pipeline image to be detected, determining the position of the pipeline defect in the pipeline image to be detected; The defect location is framed and annotated using an image plotting algorithm to obtain a framed and annotated pipeline image.

4. The pipeline defect detection method according to claim 1, characterized in that: The workflow of the CBLL module includes: Use half of the convolution to extract feature information and obtain the first feature map; The remaining feature information is processed using linear transformation to obtain the second half feature map; The first feature map and the second feature map are combined to obtain a combined feature map.

5. The pipeline defect detection method according to claim 3, characterized in that: The determining the type of the pipeline defect and obtaining the pipeline defect detection result includes: According to the framed area, the framed and annotated pipeline image is cropped to obtain a pipeline defect area; Inputting the pipeline defect area into the defect analysis model to obtain the category of the pipeline defect; An image plotting algorithm is used to mark the category of the pipeline defect on the framed and marked pipeline image to obtain the pipeline defect detection result.

6. The pipeline defect detection method according to claim 5, characterized in that: The process of obtaining the defect analysis model includes: Establishing a pipeline defect category data set, wherein the pipeline defect category data set includes pipeline defect areas and corresponding pipeline defect categories; The pipeline defect category data set is divided into a pipeline defect category training set and a pipeline defect category verification set; Inputting the pipeline defect category training set into the initial defect analysis model for training to obtain a trained defect analysis model; The pipeline defect category verification set is input into the trained defect analysis model for verification until the model converges to obtain a defect analysis model.

7. The pipeline defect detection method according to claim 5, characterized in that: After inputting the pipeline defect area into the defect analysis model to obtain the category of the pipeline defect, the method further includes: When detecting and locating the pipeline defect existing in the preprocessed pipeline image, each frame of the processed pipeline image is detected and located according to the playback order of the pipeline video. If the category of the pipeline defect in the current frame is consistent with the category of the pipeline defect in the previous frame, the current frame is discarded.

8. The pipeline defect detection method according to claim 2, characterized in that: Generating a pipeline defect detection report according to the pipeline defect detection result includes: Collect historical pipeline defect data, use the bilingual conversational language model ChatGLM3-6B technology and retrieval generation enhanced RAG technology to build a pipeline defect detection report generation model; Forming a pipeline defect report outline based on the pipeline defect detection results; The pipeline defect detection report generation model is called to generate the pipeline defect detection report according to the pipeline defect report outline.

9. A pipeline defect detection system, characterized in that: include: The data acquisition unit is used to collect pipeline videos and upload them to the preprocessing agent; A preprocessing agent is used to preprocess the pipeline video to obtain a preprocessed pipeline image; A defect detection agent, which is used to detect and locate pipeline defects in the preprocessed pipeline image using a real-time pipeline detection algorithm; A defect assessment agent is used to assess the type of pipeline defects and obtain pipeline defect detection results; A report generation agent, used to generate a pipeline defect detection report according to the pipeline defect detection result; The network structure of the pipeline real-time detection algorithm includes a backbone network, a neck network and a detection head network, and the neck network includes a CBLL module, a lightweight dynamic upsampling DySample module, and a C2f-OPERA module; the CBLL module is used to reduce the network model size of the pipeline real-time detection algorithm, the DySample module is used to focus the focus area of ​​the pipeline real-time detection algorithm around the inner wall of the pipeline, and the C2f-OPERA module is used to compress the training module of the pipeline real-time detection algorithm into a single convolution; The C2f-OPERA module includes 6 components: 2 CBL modules, 1 Split module and 3 Bottleneck-OPERA modules; the CBL module is used to perform convolution calculations, the Split module decomposes the output into different scales, and the Bottleneck-OPERA module improves computing efficiency by dimensionality reduction and feature compression.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to execute the pipeline defect detection method described in any one of claims 1-8.

Images