CCTV pipeline disease intelligent identification method based on deep learning and related device

Abstract

The invention discloses a CCTV pipeline disease intelligent identification method based on deep learning and a related device. The method comprises the following steps: acquiring an image frame sequence in a pipeline; extracting a key frame from the image frame sequence, wherein the key frame reflects a stable state in the pipeline; carrying out disease area identification on the key frame based on a pre-trained target detection model to obtain an abnormal frame marked with a disease area and a disease type and a coordinate of the abnormal frame, and collecting the abnormal frame and the coordinate into a queue; and comparing and fusing the queues obtained by multiple times of recognition to obtain a defect detection result form. According to the invention, key frame automatic extraction and recognition can be carried out on the image frame sequence in the pipeline, intelligent recognition of pipeline diseases is realized, and the efficiency and accuracy of underground drainage pipeline defect detection are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of underground drainage pipe defect detection, and in particular to a deep learning-based CCTV pipeline disease intelligent identification method and related devices. Background Art

[0002] Underground drainage pipeline defect detection is crucial for ensuring the safe operation of urban infrastructure. Traditionally, this process relies primarily on manual interpretation, which is subject to low efficiency and accuracy influenced by the individual interpreter. With the acceleration of urbanization, the underground drainage pipeline network is growing larger, creating an increasingly urgent need for efficient and accurate pipeline defect detection technology. Summary of the Invention

[0003] The purpose of this application is to provide a deep learning-based CCTV pipeline defect intelligent identification method and related devices to solve the technical problems in the existing technology that underground drainage pipeline defect detection mainly relies on manual interpretation, which has low efficiency and accuracy affected by the personal factors of the interpreter.

[0004] To achieve the above objectives, the first aspect of the present application provides a method for intelligently identifying CCTV pipeline defects based on deep learning, comprising:

[0005] Get the image frame sequence inside the pipeline;

[0006] Extracting key frames from the image frame sequence, wherein the key frames reflect a stable state inside the pipeline;

[0007] Based on the pre-trained target detection model, the key frame is subjected to diseased area identification, abnormal frames marked with diseased areas and disease types and the coordinates of the abnormal frames are obtained, and the abnormal frames are collected into a queue;

[0008] The queues obtained by multiple identifications are compared and integrated to obtain a defect detection result form.

[0009] In one or more embodiments, the step of extracting key frames from the image frame sequence specifically includes:

[0010] Calculate the optical flow vector changes of adjacent frame images based on the optical flow method and collect them into an array;

[0011] The array is traversed to find the local minimum value, and the corresponding frame is extracted as the key frame.

[0012] In one or more embodiments, the step of calculating the optical flow vector changes of adjacent frame images based on the optical flow method and collecting them into an array further includes:

[0013] Extract features from each frame and calculate the feature difference between adjacent frames;

[0014] Based on the feature difference values, a difference score value of each frame is calculated;

[0015] The frame whose difference score value is greater than the first threshold is regarded as a mutation frame;

[0016] The sudden change frame is used as a shot end frame and a next shot start frame, so as to divide the image frame sequence into a plurality of sub-image frame sequences.

[0017] In one or more embodiments, in the step of identifying the diseased area of ​​the key frame based on a pre-trained target detection model, the target detection model includes at least one or more combinations of YOLOv8, Faster R-CNN, U-Net and DeepLabv3+.

[0018] In one or more embodiments, the object detection model training method includes:

[0019] Obtaining a sample target set and dividing it into a sample training set and a sample verification set, wherein the sample target set includes images marked with disease labels;

[0020] Using the images in the sample training set as input and the corresponding disease labels as targets, training the target detection model until the loss function converges;

[0021] The effectiveness of the target detection model is tested using the sample validation set to obtain the optimal model parameters.

[0022] In one or more embodiments, in the step of comparing and fusing the queues obtained by multiple identifications, the multiple identifications are obtained by different target detection models;

[0023] The step of comparing and fusing the queues obtained by multiple identifications includes:

[0024] Traversing the queue obtained by multiple identifications, and calculating the number of times each abnormal frame is identified;

[0025] Calculating a confidence score for each of the abnormal frames based on the number of times the abnormal frames are identified;

[0026] The abnormal frames with the confidence scores greater than the second threshold are taken as recognition results, and a defect detection result form is output.

[0027] In one or more embodiments, in the step of comparing and fusing the queues obtained by multiple identifications, the multiple identifications are obtained by the same target detection model with different parameters;

[0028] The step of comparing and fusing the queues obtained by multiple identifications includes:

[0029] Based on the weights of target detection models with different parameters, weighted averaging or weighted voting is performed on the queues obtained by multiple recognitions to obtain a confidence score for each abnormal frame;

[0030] The abnormal frames with the confidence scores greater than the third threshold are taken as recognition results, and a defect detection result form is output.

[0031] To achieve the above objectives, the second aspect of the present application provides a CCTV pipeline disease intelligent identification device based on deep learning, comprising:

[0032] An acquisition module is used to acquire the image frame sequence inside the pipeline;

[0033] a frame extraction module, configured to extract key frames from the image frame sequence, wherein the key frames reflect the stable state inside the pipeline;

[0034] The target recognition module is used to identify the diseased area of ​​the key frame based on the pre-trained target detection model, obtain abnormal frames marked with the diseased area and disease type, and the coordinates of the abnormal frames, and collect them into a queue;

[0035] The output module is used to compare and fuse the queues obtained by multiple identifications to obtain a defect detection result form.

[0036] To achieve the above-mentioned objectives, the third aspect of the present application provides an electronic device, comprising: at least one processor; and a memory, wherein the memory stores instructions, and when the instructions are executed by the at least one processor, the at least one processor executes the CCTV pipeline disease intelligent identification method as described in any of the above-mentioned embodiments.

[0037] In order to achieve the above-mentioned objectives, the fourth aspect of the present application provides a machine-readable storage medium storing executable instructions, which, when executed, enable the machine to execute the CCTV pipeline disease intelligent identification method as described in any of the above-mentioned embodiments.

[0038] Different from the prior art, the present invention has the following advantages:

[0039] This application can automatically extract and identify key frames from the image frame sequence inside the pipeline, realize intelligent identification of pipeline diseases, and effectively improve the efficiency and accuracy of underground drainage pipeline defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of an implementation method of an intelligent CCTV pipeline disease identification method based on deep learning in this application;

[0041] Figure 2 yes Figure 1 A schematic flow chart of an embodiment corresponding to S200;

[0042] Figure 3 yes Figure 1 A schematic flow chart of an implementation method corresponding to S400;

[0043] Figure 4 yes Figure 1 A schematic flow chart of another embodiment corresponding to S400;

[0044] Figure 5 This is a structural diagram of an embodiment of the CCTV pipeline disease intelligent identification device based on deep learning in this application;

[0045] Figure 6 It is a structural diagram of an embodiment of the electronic device of the present application. DETAILED DESCRIPTION

[0046] The following embodiments of the technical solution of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore only examples and are not intended to limit the scope of protection of the present invention.

[0047] It should be noted that, unless otherwise specified, the technical or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0048] In order to solve the problems that the current detection of underground drainage pipeline defects mainly relies on manual interpretation, which has low efficiency and accuracy affected by the personal factors of the interpreter, the applicant has developed a CCTV pipeline disease intelligent identification method based on deep learning. This method can automatically identify defects based on the video inside the pipeline, greatly improving the detection efficiency of underground drainage pipeline defects. At the same time, the detection standards are unified and are not affected by the personal factors of the interpreter, which significantly improves the detection accuracy.

[0049] For details, please refer to Figure 1 , Figure 1 This is a flow chart of an implementation method of an intelligent CCTV pipeline disease identification method based on deep learning in this application.

[0050] like Figure 1 As shown, the method includes:

[0051] S100: Acquire an image frame sequence inside a pipeline.

[0052] Specifically, the image frame sequence may be a motion video of the interior of the pipe. The image frame sequence may be obtained by shooting a single camera independently or by cooperating with multiple cameras, both of which can achieve the effect of this embodiment.

[0053] S200 , extracting key frames from an image frame sequence.

[0054] Among them, the key frame reflects the stable state inside the pipeline.

[0055] Since the image frame sequence is a motion video, there may be blurred frames, frames outside the pipe, etc. It is necessary to extract the key frames that can best reflect the stable state inside the pipe for subsequent recognition operations to avoid defective frames affecting the recognition effect.

[0056] For details, please refer to Figure 2 , Figure 2 yes Figure 1 A flow chart of an implementation method corresponding to S200.

[0057] like Figure 2 As shown, the method of extracting key frames includes:

[0058] S201. Calculate the optical flow vector changes of adjacent frame images based on the optical flow method and collect them into an array.

[0059] In this embodiment, the frame with the local minimum motion is identified as the key frame. Specifically, in this embodiment, the optical flow method is used to detect the movement of pixels in the image to estimate the motion vector. The optical flow field can be calculated for each pair of adjacent frames. Each pixel in the optical flow field has a corresponding motion vector, which represents the movement direction and size of the pixel from the previous frame to the current frame.

[0060] Then, the motion vectors of all pixels in the optical flow field are counted to obtain the changes in the optical flow vectors of adjacent frames. The changes in the optical flow vectors represent the amount of motion.

[0061] S202: traverse the array, find the local minimum, and extract the corresponding frame as the key frame.

[0062] By traversing and finding the local minimum of the optical flow vector change, we can obtain the frame with the local minimum motion, that is, the key frame. The key frame has the local minimum motion, which can reflect the stable state of the scene and help improve the accuracy of subsequent recognition.

[0063] In one embodiment, when the target pipeline is long, the length of the image frame sequence is too long. In order to improve the efficiency and accuracy of key frame extraction, the following steps may be further included before S201:

[0064] S203: Extract features from each frame and calculate feature difference values ​​between adjacent frames.

[0065] Among them, feature extraction can be specifically to calculate the grayscale or HSV histogram of each frame and count the color distribution; or, it can also be to extract the edge structure information of each frame, count the edge direction distribution, or it can be a combination of multiple features, all of which can achieve the effect of this embodiment, and will not be repeated here.

[0066] S204: Calculate a difference score value for each frame based on the feature difference value.

[0067] S205 : Frames with difference scores greater than a first threshold are regarded as mutation frames.

[0068] S206 : Using the sudden change frame as the shot end frame and the next shot start frame, so as to divide the image frame sequence into a plurality of sub-image frame sequences.

[0069] It can be understood that a score can be assigned to each frame based on the feature difference value, and then the frame with a difference score value greater than the first threshold can be used as a mutation frame. The difference between the mutation frame and the adjacent frames is too large, and it can be identified as the end frame of the shot and the start frame of the next shot, thereby realizing the segmentation of the image frame sequence and obtaining multiple sub-image frame sequences.

[0070] Afterwards, key frames may be extracted from each sub-image frame sequence based on the methods of S201 and S02 to improve extraction efficiency and accuracy.

[0071] S300 , based on a pre-trained target detection model, the key frame is subjected to diseased area identification, abnormal frames marked with diseased areas and disease types and the coordinates of the abnormal frames are obtained, and the abnormal frames are collected into a queue.

[0072] In one embodiment, the target detection model can be trained based on the underground drainage pipe disease dataset built by the applicant.

[0073] Specifically, the training method of the target detection model may include:

[0074] Obtain a sample target set and divide it into a sample training set and a sample verification set, wherein the sample target set includes images marked with disease labels;

[0075] Using the images in the sample training set as input and the corresponding disease labels as targets, the object detection model is trained until the loss function converges;

[0076] The effectiveness of the target detection model is tested on a sample validation set to obtain the optimal model parameters.

[0077] In this embodiment, a deep learning-based target detection model is used to realize the identification of diseased areas in key frames, wherein the target detection model can be any existing deep learning neural network model with target detection capabilities, such as YOLOv8, Faster R-CNN, U-Net, DeepLabv3+, etc., which can achieve the effects of this embodiment.

[0078] In one embodiment, defective areas may include water seepage, sediment accumulation, corrosion, scaling, and other defects. Multiple target detection models can be used to detect different types of defective areas. For example, YOLOv8 or Faster R-CNN can be used to locate defective areas such as water seepage and sediment accumulation. U-Net or DeepLabv3+ can be used to perform pixel-level image segmentation on the pipeline inner wall to identify complex defects such as corrosion and scaling. The specific model recognition process is well known in the art and will not be detailed here.

[0079] S400 , comparing and fusing queues obtained through multiple identifications to obtain a defect detection result form.

[0080] In order to optimize the recognition effect and avoid false detection and missed detection, this embodiment compares and fuses the queues obtained from multiple recognitions and outputs the final recognition result, that is, the defect detection result form.

[0081] In one embodiment, multiple recognitions are obtained by different target detection models. For fusion methods, please refer to Figure 3 , Figure 3 yes Figure 1 A flow chart of an implementation method corresponding to S400.

[0082] like Figure 3 As shown, the fusion methods include:

[0083] S401a, traverse the queue obtained by multiple identifications, and calculate the number of times each abnormal frame is identified.

[0084] S402a: Calculate the confidence score of each abnormal frame based on the number of times it is identified.

[0085] S403a: taking abnormal frames with confidence scores greater than a second threshold as recognition results, and outputting a defect detection result form.

[0086] In one embodiment, in order to further avoid the problem of missed detection, abnormal frames with confidence scores less than a second threshold may be collected into a set to be confirmed and fed back for manual confirmation.

[0087] In one embodiment, multiple recognitions can also be obtained by using the same target detection model with different parameters. For the fusion method, please refer to Figure 4 , Figure 4 yes Figure 1 A flow chart of another embodiment corresponding to S400.

[0088] like Figure 4 As shown, the fusion methods include:

[0089] S401b: Based on the weights of the target detection models with different parameters, perform weighted averaging or weighted voting on the queues obtained by multiple recognitions to obtain a confidence score for each abnormal frame.

[0090] S402b: taking abnormal frames with confidence scores greater than a third threshold as recognition results, and outputting a defect detection result form.

[0091] Based on the methods of the above-mentioned embodiments, key frames of the image frame sequence inside the pipeline can be automatically extracted and identified, and intelligent identification of pipeline defects can be achieved, which effectively improves the efficiency and accuracy of underground drainage pipeline defect detection.

[0092] This application also provides a CCTV pipeline disease intelligent identification device based on deep learning, please refer to Figure 5 , Figure 5 This is a structural diagram of an embodiment of the CCTV pipeline disease intelligent identification device based on deep learning in this application.

[0093] like Figure 5 As shown, the device includes an acquisition module 21, a frame extraction module 22, a target recognition module 23 and an output module 24.

[0094] The acquisition module 21 is used to acquire the image frame sequence inside the pipeline;

[0095] The frame extraction module 22 is used to extract key frames from the image frame sequence, and the key frames reflect the stable state inside the pipeline;

[0096] The target recognition module 23 is used to identify the diseased area of ​​the key frame based on the pre-trained target detection model, obtain abnormal frames marked with the diseased area and disease type, and the coordinates of the abnormal frames, and collect them into a queue;

[0097] The output module 24 is used to compare and fuse the queues obtained by multiple identifications to obtain a defect detection result form.

[0098] As above Figures 1 to 4This section describes a method for intelligently identifying CCTV pipeline defects based on deep learning, according to an embodiment of this specification. The details mentioned in the above description of the method embodiment also apply to the device for intelligently identifying CCTV pipeline defects based on deep learning, according to an embodiment of this specification. The device for intelligently identifying CCTV pipeline defects based on deep learning can be implemented using hardware, software, or a combination of hardware and software.

[0099] This application also provides an electronic device, see Figure 6 , Figure 6 This is a schematic diagram of the structure of an embodiment of the electronic device of the present application. Figure 6 As shown, the electronic device 30 may include at least one processor 31, a memory 32 (e.g., a non-volatile memory), a storage 33, and a communication interface 34, and the at least one processor 31, the storage 32, the storage 33, and the communication interface 34 are connected together via a bus 35. The at least one processor 31 executes at least one computer-readable instruction stored or encoded in the storage 32.

[0100] It should be understood that the computer executable instructions stored in the memory 32, when executed, cause at least one processor 31 to perform the above combined operations in various embodiments of this specification. Figure 1-Figure 5 Describes the various operations and functions.

[0101] In the embodiments of the present specification, the electronic device 30 may include but is not limited to: a personal computer, a server computer, a workstation, a desktop computer, a laptop computer, a notebook computer, a mobile electronic device, a smart phone, a tablet computer, a cellular phone, a personal digital assistant (PDA), a handheld device, a messaging device, a wearable electronic device, a consumer electronic device, and the like.

[0102] According to one embodiment, a program product such as a machine-readable medium is provided. The machine-readable medium may have instructions (i.e., the above-mentioned elements implemented in software form), which, when executed by a machine, causes the machine to perform the above-mentioned combined embodiments of the present specification. Figures 1-6 Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes for implementing the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can be enabled to read and execute the instructions stored in the readable storage medium.

[0103] In this case, the program code itself read from the machine-readable medium can implement the functions of any one of the above embodiments, and thus the machine-readable code and the machine-readable storage medium storing the machine-readable code constitute part of this specification.

[0104] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0105] Those skilled in the art will appreciate that the various embodiments disclosed above may be modified and altered in various ways without departing from the essence of the invention. Therefore, the scope of protection of this specification shall be defined by the appended claims.

[0106] It should be noted that not all steps and units in the above processes and system structure diagrams are required, and certain steps or units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure, that is, some units may be implemented by the same physical client, or some units may be implemented by multiple physical clients, or may be implemented by certain components in multiple independent devices.

[0107] In the above embodiments, hardware unit or module can be realized by mechanical means or electrical means. For example, a hardware unit, module or processor can include permanent dedicated circuit or logic (such as special processor, FPGA or ASIC) to complete the corresponding operation. Hardware unit or processor can also include programmable logic or circuit (such as general purpose processor or other programmable processor), can be temporarily set up to complete the corresponding operation by software. Concrete implementation (mechanical means or dedicated permanent circuit or temporary circuit) can be determined based on cost and time consideration.

[0108] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "exemplary" used throughout this specification means "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid obscuring the concepts of the described embodiments, well-known structures and devices are shown in block diagram form.

[0109] The foregoing description of the present disclosure is provided to enable any person skilled in the art to implement or use the present disclosure. Various modifications to the present disclosure will be readily apparent to those skilled in the art, and the general principles herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is intended to be consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A deep learning-based intelligent identification method for CCTV pipeline defects, characterized by: include: Get the image frame sequence inside the pipeline; Extracting key frames from the image frame sequence, wherein the key frames reflect a stable state inside the pipeline; Based on the pre-trained target detection model, the key frame is subjected to diseased area identification, abnormal frames marked with diseased areas and disease types and the coordinates of the abnormal frames are obtained, and the abnormal frames are collected into a queue; The queues obtained by multiple identifications are compared and integrated to obtain a defect detection result form.

2. The CCTV pipeline disease intelligent identification method according to claim 1 is characterized in that: The step of extracting key frames from the image frame sequence specifically includes: Calculate the optical flow vector changes of adjacent frame images based on the optical flow method and collect them into an array; The array is traversed to find the local minimum value, and the corresponding frame is extracted as the key frame.

3. The CCTV pipeline disease intelligent identification method according to claim 2 is characterized in that: The step of calculating the optical flow vector changes of adjacent frame images based on the optical flow method and collecting them into an array also includes: Extract features from each frame and calculate the feature difference between adjacent frames; Based on the feature difference values, a difference score value of each frame is calculated; The frame whose difference score value is greater than the first threshold is regarded as a mutation frame; The sudden change frame is used as a shot end frame and a next shot start frame, so as to divide the image frame sequence into a plurality of sub-image frame sequences.

4. The method for intelligently identifying CCTV pipeline defects according to claim 1, characterized in that: In the step of identifying the diseased area of ​​the key frame based on a pre-trained target detection model, the target detection model includes at least one or more combinations of YOLOv8, Faster R-CNN, U-Net and DeepLabv3+.

5. The CCTV pipeline disease intelligent identification method according to claim 1 is characterized in that: The training method of the target detection model includes: Obtaining a sample target set and dividing it into a sample training set and a sample verification set, wherein the sample target set includes images marked with disease labels; Using the images in the sample training set as input and the corresponding disease labels as targets, training the target detection model until the loss function converges; The effectiveness of the target detection model is tested using the sample validation set to obtain the optimal model parameters.

6. The method for intelligently identifying CCTV pipeline defects according to claim 1, characterized in that: In the step of comparing and fusing the queues obtained by multiple identifications, the multiple identifications are obtained by different target detection models; The step of comparing and fusing the queues obtained by multiple identifications includes: Traversing the queue obtained by multiple identifications, and calculating the number of times each abnormal frame is identified; Calculating a confidence score for each of the abnormal frames based on the number of times the abnormal frames are identified; The abnormal frames with the confidence scores greater than the second threshold are taken as recognition results, and a defect detection result form is output.

7. The intelligent identification method for CCTV pipeline defects according to claim 1 is characterized in that: In the step of comparing and fusing the queues obtained by multiple identifications, the multiple identifications are obtained by the same target detection model with different parameters; The step of comparing and fusing the queues obtained by multiple identifications includes: Based on the weights of target detection models with different parameters, weighted averaging or weighted voting is performed on the queues obtained by multiple recognitions to obtain a confidence score for each abnormal frame; The abnormal frames with the confidence scores greater than the third threshold are taken as recognition results, and a defect detection result form is output.

8. A CCTV pipeline disease intelligent identification device based on deep learning, characterized by: include: An acquisition module is used to acquire the image frame sequence inside the pipeline; a frame extraction module, configured to extract key frames from the image frame sequence, wherein the key frames reflect the stable state inside the pipeline; The target recognition module is used to identify the diseased area of ​​the key frame based on the pre-trained target detection model, obtain abnormal frames marked with the diseased area and disease type, and the coordinates of the abnormal frames, and collect them into a queue; The output module is used to compare and fuse the queues obtained by multiple identifications to obtain a defect detection result form.

9. An electronic device comprising: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to execute the CCTV pipeline disease intelligent identification method according to any one of claims 1 to 7.

10. A machine-readable storage medium storing executable instructions, wherein when the instructions are executed, the machine executes the CCTV pipeline disease intelligent identification method according to any one of claims 1 to 7.

Images