A method for detecting defects in ground drainage facilities based on dynamically monitored videos

Abstract

The present application discloses a method for detecting defects in ground drainage facilities based on dynamic monitoring videos. The method includes obtaining dynamic monitoring videos of a target monitoring area and ground drainage facility information, where the dynamic monitoring videos are collected based on a preset mobile monitoring device; performing facility defect detection and facility defect target tracking on the monitoring images of the dynamic monitoring videos based on a preset facility defect target tracking algorithm to obtain drainage facility defect detection results; obtaining the spatial information of the drainage facility defect detection results, and performing spatial matching analysis based on the spatial information of the drainage facility defect detection results and the ground drainage facility information to correspondingly obtain facility defect target detection information; the present application has the effect of improving the detection efficiency of defects in ground drainage facilities.

Description

Technical Field

[0001] This application relates to the technical field of facility defect detection, and in particular, to a method for detecting defects in ground drainage facilities based on dynamic monitoring videos. Background Art

[0002] In recent years, with the rapid development of cities in China, the number of drainage facilities has been increasing continuously, and the number of ground drainage facilities on urban roads is huge. On the one hand, the large number of drainage facilities effectively guarantees the drainage safety of the city. On the other hand, the accompanying problems of facility operation efficiency, operation benefits, and safety resilience will also restrict the high-quality development of urban infrastructure.

[0003] Although the country has gradually increased its investment in the operation and maintenance of urban drainage facilities in recent years and improved the professional operation and maintenance management mechanism of drainage facilities, the overall maintenance level of urban drainage facilities has been improved to a certain extent. However, the current operation and maintenance inspection of drainage facilities mainly still rely on traditional manual inspections. After the staff visually determine that there are defects in the ground drainage facilities, they can take pictures of the defective ground drainage facilities and obtain the corresponding geographical locations, and then send the pictures and geographical location information to the facility management platform, thus completing the defect detection and defect management work of the ground drainage facilities. The inspection efficiency is low, and the phenomena of missed inspection and mis-inspection often occur, and the drainage facilities cannot be maintained and repaired in time, resulting in the drainage facilities not being able to play their original drainage efficiency well. During heavy rain and waterlogging, abnormal operations of ground drainage facilities such as blocked rainwater inlets, lost manhole covers, and foreign object blockages at discharge ports are often found, posing potential threats to public safety, etc. Therefore, it is urgent that daily inspections can quickly and accurately detect defective facilities and promptly notify the drainage operation and maintenance personnel for handling. During the flood season, it can detect defects in ground drainage facilities in a timely manner before and during the flood season and promptly notify the flood control personnel for handling, reducing the impact of flood season waterlogging disasters. Summary of the Invention

[0004] In order to improve the detection efficiency of defects in ground drainage facilities and reduce the impact of flood season waterlogging disasters, this application provides a method for detecting defects in ground drainage facilities based on dynamic monitoring videos.

[0005] In a first aspect, the inventive object of this application is achieved by adopting the following technical solution:

[0006] A method for detecting defects in ground drainage facilities based on dynamic monitoring videos, comprising:

[0007] Obtaining a dynamic monitoring video of a target monitoring area and ground drainage facility information, where the dynamic monitoring video is collected based on a preset mobile monitoring device;

[0008] Based on a preset facility defect target tracking algorithm, perform facility defect detection and facility defect target tracking on the monitoring images of the dynamic monitoring video to obtain the detection results of drainage facility defects;

[0009] Obtain the spatial information of the detection results of the drainage facility defects, and perform spatial matching analysis based on the spatial information of the detection results of the drainage facility defects and the ground drainage facility information to correspondingly obtain the facility defect target detection information.

[0010] By adopting the above technical solutions, the present application provides a highly efficient detection method for detecting defects in ground drainage facilities based on dynamic monitoring videos. The defect detection method for ground drainage facilities is more automated and intelligent; the target monitoring area is the detection area of urban municipal drainage facilities, and the mobile monitoring device is a mobile intelligent acquisition terminal (such as a driving camera on a flood control inspection vehicle, a preset intelligent acquisition terminal, a handheld operation terminal, etc.). The ground drainage facility information includes drainage facility product information and facility installation location information; specifically, after obtaining the dynamic monitoring video and ground drainage facility information of the target monitoring area, use a preset facility defect target tracking algorithm to perform facility defect target detection on the monitoring images of the dynamic monitoring video for ground drainage facilities, and after detecting that there are facility defects in the ground drainage facilities, perform facility defect target tracking to obtain the detection results of drainage facility defects, which is beneficial to improving the detection accuracy of facility defect targets in dynamic monitoring videos. By using target detection technology to perform intelligent identification of the status of facility defect targets in ground drainage facilities, daily inspections can quickly and accurately discover facilities with defects; next, obtain the spatial information of the detection results of drainage facility defects, and perform spatial matching analysis based on the spatial information of the detection results of drainage facility defects and the ground drainage facility information. The spatial matching analysis algorithm includes the integration of the information of the monitoring images of the dynamic monitoring video (the monitoring images at this time contain the detection results of drainage facility defects) and the spatial information, and the integration of the facility defect targets of the detection results of drainage facility defects and the target tracking trajectories, so as to complete the integration of the detection of drainage facility defects and the GIS data of ground drainage facility information, that is, integrally obtain the facility defect target detection information with spatial information. Therefore, the present application provides a method for intelligent detection and management based on the defects of dynamic monitoring videos and ground drainage facilities, which is beneficial to improving the detection efficiency and detection accuracy of ground drainage facility defects, and is beneficial to saving the manual detection cost of ground drainage facility defects; during the flood season, the video of the drainage facilities (referring to the dynamic monitoring video) can be dynamically collected through the camera device before and during the flood season, the defects of the ground drainage facilities can be detected in time, and the flood control personnel can be notified to deal with them in time, which is beneficial to reducing the impact of in-city flooding disasters during the flood season.

[0011] In a preferred example of the present application: the preset facility defect target tracking algorithm includes a defect target detection algorithm and a defect target tracking algorithm; based on the preset facility defect target tracking algorithm, facility defect detection and facility defect target tracking of the drainage facility are performed on the monitoring images of the dynamic monitoring video, and a drainage facility defect detection result is obtained, including: performing frame-by-frame processing on the dynamic monitoring video to obtain multiple consecutive monitoring images to be determined;

[0012] In the preset detection-based defect target tracking model, based on the defect target detection algorithm, facility defect detection is performed on multiple frames of the monitoring images to be determined frame by frame, and an initial facility defect detection result of the corresponding consecutive multiple frames of monitoring images is obtained; the preset detection-based defect target tracking model is fused with an attention module for adaptively adjusting and optimizing the network structure of the detection-based defect target tracking model; the initial facility defect detection result includes a number of defect detection targets;

[0013] In the preset detection-based defect target tracking model, based on the defect target tracking algorithm and a number of defect detection targets of the initial facility defect detection result, defect target tracking analysis is performed to obtain a drainage facility defect detection result of the continuous video stream.

[0014] By adopting the above technical solution, first, the dynamic monitoring video is frame-divided to obtain multiple consecutive monitoring images to be determined based on the time sequence, where the monitoring images are ground road images obtained by using a preset mobile monitoring device to photograph the drainage setting area on the road surface, so as to facilitate the identification of ground drainage facilities from the monitoring images and perform facility defect detection and facility defect target tracking on the ground drainage facilities in the monitoring images; then, a defect target detection algorithm is used to perform facility defect detection on multiple frames of monitoring images to be determined frame by frame, so as to obtain an initial facility defect detection result including several defect detection targets; this application uses a preset detection-based defect target tracking model integrated with an attention module. Through the attention module, the detection performance of drainage facility defect targets can be improved. When extracting and calculating the defect feature information of facility defect targets in the monitoring images, the detection-based defect target tracking model integrated with the attention module can adaptively adjust the weights of different channels and spatial positions, which helps to improve the recognition ability of ground drainage facility defects and the model accuracy; further, in the preset detection-based defect target tracking model, based on the defect target tracking algorithm, defect target tracking analysis is performed on several defect detection targets in the initial facility defect detection result to obtain the final drainage setting defect detection result, so as to efficiently check the defect conditions of ground drainage facilities; this application provides an integrated real-time detection method for drainage facility defects, realizing real-time tracking of ground drainage facility targets. During daily inspections, defective ground drainage facilities can be quickly and accurately discovered, and drainage operation and maintenance personnel can be notified in a timely manner for handling, improving the operation efficiency of ground drainage facilities.

[0015] In a preferred example of this application: in the preset detection-based defect target tracking model, based on the defect target tracking algorithm and several defect detection targets of the initial facility defect detection result, defect target tracking analysis is performed to obtain the drainage facility defect detection result, including:

[0016] Obtain the movement trajectory of the preset mobile monitoring device corresponding to the dynamic monitoring video; if a defect detection box is obtained in multiple frames of the monitoring images to be determined, use the defect detection box as the matching basis for the next frame of defect detection box.

[0017] In a preset defect target tracking model based on detection, target detection boxes of multiple frames of the to-be-determined monitoring images are sequentially generated according to the time sequence; based on the defect target tracking algorithm, the target detection box of the to-be-determined monitoring image of the next frame is obtained by predicting and matching according to the target detection box of the to-be-determined monitoring image of the previous frame and the motion state of the corresponding moving trajectory; based on the defect target tracking algorithm, multiple target detection boxes, and a preset confidence threshold, several defect detection targets of the initial facility defect detection result are divided into multiple detection result sets with different confidences;

[0018] Perform moving trajectory matching and target detection box matching on multiple groups of the detection result sets to obtain the drainage facility defect detection result.

[0019] By adopting the above technical solution, the multi-target tracking problem of multiple drainage facility defect targets (referring to defect detection targets) in multiple frames of monitoring images is solved. Based on the target tracking framework for drainage facility defect detection, the defect target tracking model based on detection can efficiently track multiple defect detection targets when performing real-time detection tasks; specifically, through a pre-trained defect target tracking model based on detection, the defect target detection of multiple consecutive frames of monitoring images obtained from the video stream of a dynamic monitoring video is completed by using the defect target tracking algorithm, and target detection boxes matching the defect detection targets are generated. And the target detection box of the next frame is obtained by predicting and matching the target detection box of the to-be-determined monitoring image of the previous frame and the motion state of the corresponding moving trajectory, which is beneficial to efficiently and accurately realizing the matching of the moving trajectory corresponding to the mobile monitoring video and the target detection box. And by dividing several defect detection targets of the initial facility defect detection result into multiple detection result sets with different confidences, the defect detection performance of the ground drainage facilities is analyzed through the confidence of the detection results, and the detection accuracy of the defect detection targets of the target detection boxes summarized in each monitoring image is obtained, which is beneficial to meeting the speed and accuracy of the intelligent detection of the ground drainage facility status.

[0020] In a preferred example of the present application: the detection result set includes a high-confidence detection set and a low-confidence detection set; the performing moving trajectory matching and target detection box matching on multiple groups of the detection result sets to obtain the drainage facility defect detection result includes:

[0021] Extract trajectory information from the dynamic monitoring video, and construct and initialize a trajectory pool;

[0022] In the high-confidence detection set, calculate the intersection-over-union distance matrix and the appearance feature distance matrix between each trajectory in the initialized trajectory pool and several detected defect detection targets, generate a comprehensive distance matrix based on the intersection-over-union distance matrix and the appearance feature distance matrix, and apply a preset linear assignment algorithm to determine the first matching result; the first matching result includes the pairs of defect targets and trajectories with successful matches, the trajectories that were not successfully matched for the first time, and the target detection frames of the targets for which no matching trajectories were found for the first time;

[0023] For the trajectories that were not successfully matched according to the first matching result and the defect targets in the low-confidence detection set, recalculate the intersection-over-union distance matrix; and perform a second matching using the preset linear assignment algorithm and the intersection-over-union distance matrix to generate a second matching result; the second matching result includes newly added matching pairs, the trajectories that were not successfully matched for the second time, and the target detection frames of the targets for which no matching trajectories were found for the second time;

[0024] Integrate the first matching result and the second matching result to form a final matching result set, and identify all the finally unmatched trajectories;

[0025] Update the trajectory pool according to the final matching result and all the finally unmatched trajectories, and remove the finally unmatched trajectories and the trajectories that have been lost for a long time.

[0026] By adopting the above technical solution, the present application provides a customized mobile trajectory management method based on trajectory initialization, trajectory update, and trajectory deletion; specifically, a confidence threshold corresponding to the drainage facility defect detection result is set based on the target detection box generated from the monitoring image, so as to divide the detection result set into a high-confidence detection set and a low-confidence detection set. After initializing the trajectory pool, the mobile trajectory and the target detection box are preferentially matched for the high-confidence detection set. By calculating the intersection-over-union distance matrix and the appearance feature distance matrix between the mobile trajectory in the initialized trajectory pool and a plurality of corresponding defect detection targets, the defect detection target tracking of the mobile trajectory and the target detection box are obtained, so as to obtain a more accurate comprehensive distance matrix, thereby efficiently completing the defect target tracking of the ground drainage setting; through the preset linear assignment algorithm for assignment, the first matching result is obtained. After the first matching is completed, the successfully matched pairs are updated to the trajectory pool, and new mobile trajectories are created for the first time for the unmatched detection boxes; in the second matching, the remaining trajectories after the first matching and the drainage facility defect detection results in the low-confidence detection set are used to obtain the second matching result, the second matching pairs and the second unmatched trajectories are obtained. Next, the trajectories that are not successfully matched in the second matching are marked as lost, and the finally unmatched mobile trajectories are deleted, and at the same time, the mobile trajectories that are lost for a long time are removed. By executing the mobile trajectories corresponding to the drainage facility defect detection results such as trajectory update and trajectory deletion in real time and efficiently, the mobile trajectories are analyzed accurately with high efficiency, and the accuracy of the drainage facility defect detection results obtained is high.

[0027] In a preferred example of the present application: before obtaining the spatial information of the drainage facility defect detection result and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to correspondingly obtain the facility defect target detection information, it includes:

[0028] Obtain the geographical coordinates of the monitoring device of the dynamic monitoring video, and perform plane coordinate projection transformation on the geographical coordinates to obtain a plane rectangular coordinate;

[0029] Based on the plane rectangular coordinate, when performing frame-by-frame processing on the dynamic monitoring video to obtain a monitoring image, use linear interpolation to perform monitoring image coordinate positioning on each frame of the monitoring image to obtain instantaneous image positioning coordinates;

[0030] Based on the plane rectangular coordinate and the instantaneous image positioning coordinate, determine the spatial information of each frame of the monitoring image; obtain the spatial information of the drainage facility defect detection result based on the spatial information of the monitoring image;

[0031] Identify and obtain several defect detection targets of multiple drainage facility defect detection results, and perform a merging process on multiple drainage facility defect detection results identified as the same ground drainage facility by the facility defect target tracking algorithm to obtain a merged drainage facility defect detection result.

[0032] By adopting the above technical solution, before performing spatial matching analysis on the drainage facility defect detection result and the ground drainage facility information, since the positioning system carried by the mobile monitoring device for collecting dynamic monitoring videos is the global navigation satellite system (GNSS) and the coordinate system used is the WGS84 geographic coordinate system, the WGS84 coordinate system is first converted into a plane rectangular coordinate system that is the same as the data of the ground drainage facility information to obtain a plane rectangular coordinate. At the same time, when the dynamic mobile monitoring device collects video images of the ground drainage setting, the mobile monitoring device and the vehicle-mounted satellite receiver of the device equipped with the mobile monitoring device (such as a flood control inspection vehicle) receive information simultaneously, and the two cameras operate independently and have different receiving frequencies. Therefore, when obtaining the monitoring images of the dynamic monitoring video, it is necessary to use the linear interpolation method to perform monitoring image coordinate positioning on each frame of the monitoring image to obtain the instantaneous image positioning coordinates of the mobile monitoring device, so as to efficiently complete the image positioning of the monitoring images of the mobile monitoring device. At the same time, for the determined plane rectangular coordinates and the instantaneous image positioning coordinates, the spatial information of the corresponding drainage facility defect detection result is further obtained to facilitate subsequent drainage facility defect detection and facility defect target tracking.

[0033] Furthermore, in order to increase the computational amount of ground drainage facility defect detection and improve the computational detection efficiency, before performing spatial matching retrieval analysis, and since the same ground drainage facility may also have various types of facility defects, this solution also performs a merging process on multiple drainage facility defect detection results identified as the same ground drainage facility by the facility defect target tracking algorithm.

[0034] In a preferred example of the present application: obtaining the spatial information of the drainage facility defect detection result, and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to correspondingly obtain facility defect target detection information, including:

[0035] In the preset GIS database, obtain the drainage facility GIS data of the ground drainage facility information and correspondingly associate the drainage facility identifier;

[0036] Use the nearest neighbor matching algorithm with a limited radius to perform nearest neighbor coordinate matching on the spatial information of the drainage facility defect detection result and the ground drainage facility GIS data, and construct a matching relationship between the drainage facility defect detection result corresponding to the monitoring image and the ground drainage facility GIS data;

[0037] Based on the matching relationship, obtain a defect detection target corresponding to the drainage facility defect detection result, and correspondingly obtain facility defect target detection information;

[0038] In a preset GIS database, based on the drainage facility identifier, mark the drainage facilities in the facility defect target detection information for facility defects, so as to integrate the drainage facility defect detection result into the preset GIS database.

[0039] By adopting the above technical solution, from the GIS database, read the drainage facility GIS data corresponding to the ground drainage facility information, and use the nearest neighbor matching algorithm with a limited radius to perform neighboring coordinate matching on the spatial information of the drainage facility defect detection result and the ground drainage facility GIS data, so as to construct a matching association relationship (referring to the matching relationship) between the drainage facility defect detection result and the ground drainage facility GIS data, which is convenient for spatially matching the drainage facility defect detection result with spatial information and the ground drainage facility GIS data. At the same time, according to the unique drainage facility identifier of the drainage facility (where the drainage facility identifier is the product information code of the drainage facility with unique installation positioning information), update the defect detection target (referring to the new facility defect information) of the drainage facility detected in real time to the GIS database, so as to efficiently and real-time integrate the drainage facility defect detection result into the GIS database, which is convenient for completing the ground drainage facility defect safety detection and drainage facility GIS integration, and greatly improves the detection efficiency of the ground drainage facility defects.

[0040] In a preferred example of the present application: before obtaining the spatial information of the drainage facility defect detection result and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, and correspondingly obtaining facility defect target detection information, the method further includes:

[0041] Divide the drainage facility defect detection result into a first drainage facility defect detection result of the facility type and a second drainage facility defect detection result of the defect type, and the division steps include:

[0042] When performing facility defect target tracking on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm, generate a first drainage facility defect detection result with a target tracking identifier and a monitoring image frame identifier, and perform one-to-one association matching between the target tracking identifier and the monitoring image identifier;

[0043] When performing facility defect detection on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm, generate a second drainage facility defect detection result with a defect type identifier and a monitoring image frame identifier;

[0044] Based on the monitored image identifier and a preset IOU threshold, associate the defect detection target of the second drainage facility defect detection result with the defect type identifier; based on the same target tracking identifier, obtain the defect detection results of different facility defect targets in multiple frames of monitored images respectively;

[0045] Obtain the defect target tracking trajectory corresponding to the first drainage facility defect detection result, and associate and match the defect detection results of different facility defect targets with the corresponding defect target tracking trajectories to obtain the drainage facility defect detection result with spatial information.

[0046] By adopting the above technical solution, integrate the defect detection results of different facility defect targets with the corresponding defect target tracking trajectories, and complete the integration of the facility defect target and the corresponding tracking trajectory by matching the facility defect target with the defect type; specifically, when tracking the facility defect target in the monitored image of the dynamic monitoring video, associate the drainage facility defect type with the monitored image, and then based on the preset IOU threshold, associate the defect detection results of different types of facility defect targets in the monitored image with the monitored image identifier, which is beneficial to improving the detection accuracy and detection efficiency of the drainage facility defect detection result. By associating the corresponding defect detection target in the monitored image with the facility defect type, divide the drainage facility defect detection result according to the facility type and defect type respectively to obtain the corresponding first drainage facility defect detection result and second drainage facility defect detection result, so as to obtain the drainage facility defect detection result with spatial information, which is convenient to achieve the purpose of detecting and analyzing the tracking and spatial matching integration of the drainage facility defect target with high precision.

[0047] In a second aspect, the invention object of the present application is achieved by adopting the following technical solution:

[0048] A ground drainage facility defect detection system based on a dynamic monitoring video, the system includes:

[0049] A monitoring video and drainage facility information acquisition module, configured to acquire the dynamic monitoring video of the target monitoring area and the ground drainage facility information, and the dynamic monitoring video is acquired based on a preset mobile monitoring device;

[0050] A facility defect detection result acquisition module, configured to perform facility defect detection and facility defect target tracking on the drainage facilities for the monitored images of the dynamic monitoring video based on a preset facility defect target tracking algorithm to obtain the drainage facility defect detection result; a spatial information matching and analysis module, configured to acquire the spatial information of the drainage facility defect detection result, and perform spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to correspondingly obtain the facility defect target detection information.

[0051] By adopting the above technical solutions, the present application provides a high-detection-efficiency method for defect detection of ground drainage facilities based on dynamic monitoring videos, making the defect detection method of ground drainage facilities more automated and intelligent. Specifically, after obtaining the dynamic monitoring video of the target monitoring area and the information of the ground drainage facilities, a preset facility defect target tracking algorithm is used to detect the facility defect targets of the ground drainage facilities in the monitoring images of the dynamic monitoring video. After detecting that there are facility defects in the ground drainage facilities, facility defect target tracking is performed to obtain the drainage facility defect detection results, which is beneficial to improving the detection accuracy of the facility defect targets in the dynamic monitoring video. Next, the spatial information of the drainage facility defect detection results is obtained, and spatial matching analysis is performed based on the spatial information of the drainage facility defect detection results and the information of the ground drainage facilities. The spatial matching analysis algorithm includes the integration of the information of the monitoring images of the dynamic monitoring video (the monitoring images at this time contain the drainage facility defect detection results) and the spatial information, and the integration of the facility defect targets of the drainage facility defect detection results and the target tracking trajectories, so as to complete the integration of the drainage facility defect detection and the GIS data of the ground drainage facility information, that is, integrally obtain the facility defect target detection information with spatial information. Therefore, the present application provides a method for intelligent detection and management based on the defects of dynamic monitoring videos and ground drainage facilities, which is beneficial to improving the detection efficiency and detection accuracy of ground drainage facility defects, and is also beneficial to saving the manual detection cost of ground drainage facility defects.

[0052] In the third aspect, the invention object of the present application is achieved by adopting the following technical solutions:

[0053] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above method for defect detection of ground drainage facilities based on dynamic monitoring videos are implemented.

[0054] In the fourth aspect, the invention object of the present application is achieved by adopting the following technical solutions:

[0055] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method for defect detection of ground drainage facilities based on dynamic monitoring videos are implemented.

[0056] In summary, the present application includes at least one of the following beneficial technical effects:

[0057] 1. The present application provides a highly efficient method for detecting defects in ground drainage facilities based on dynamic monitoring videos, making the defect detection method of ground drainage facilities more automated and intelligent. Specifically, after obtaining the dynamic monitoring video of the target monitoring area and the information of the ground drainage facilities, a preset facility defect target tracking algorithm is used to detect the facility defect targets of the ground drainage facilities in the monitoring images of the dynamic monitoring video. After detecting that there are facility defects in the ground drainage facilities, facility defect target tracking is carried out to obtain the drainage facility defect detection results, which is conducive to improving the detection accuracy of the facility defect targets in the dynamic monitoring video. Next, the spatial information of the drainage facility defect detection results is obtained, and spatial matching analysis is carried out based on the spatial information of the drainage facility defect detection results and the information of the ground drainage facilities. The spatial matching analysis algorithm includes the integration of the information of the monitoring images of the dynamic monitoring video (the monitoring images at this time contain the drainage facility defect detection results) and the spatial information, and the integration of the facility defect targets of the drainage facility defect detection results and the target tracking trajectories, so as to complete the integration of the drainage facility defect detection and the GIS data of the ground drainage facility information, that is, integrally obtain the facility defect target detection information with spatial information. Therefore, the present application provides a method for intelligent detection and management based on the defects of dynamic monitoring videos and ground drainage facilities, which is conducive to improving the detection efficiency and accuracy of ground drainage facility defects, and is also conducive to saving the manual detection cost of ground drainage facility defects;

[0058] 2. The dynamic monitoring video is frame-processed to obtain multiple frames of consecutive monitoring images to be determined based on the time sequence. The monitoring images are ground road images obtained by using a preset mobile monitoring device to shoot the drainage setting area of the road surface, so as to facilitate the identification of ground drainage facilities from the monitoring images and conduct facility defect detection and facility defect target tracking on the ground drainage facilities in the monitoring images. Then, a defect target detection algorithm is used to detect the facility defects frame by frame for the multiple frames of monitoring images to be determined, so as to obtain an initial facility defect detection result including several defect detection targets. The present application uses a preset detection-based defect target tracking model integrated with an attention module. Through the attention module, the detection performance of the drainage facility defect targets can be improved. When extracting and calculating the defect feature information of the facility defect targets in the monitoring images, the detection-based defect target tracking model integrated with the attention module can adaptively adjust the weights of different channels and spatial positions, which helps to improve the recognition ability and model accuracy of ground drainage facility defects;

[0059] 3. Solve the multi-object tracking problem of multiple drainage facility defect targets (defect detection targets) in multi-frame monitoring images. Based on the object tracking framework for drainage facility defect detection, enable the detection-based defect object tracking model to efficiently track multiple defect detection targets when performing real-time detection tasks. Specifically, through a pre-trained detection-based defect object tracking model, use the defect object tracking algorithm to complete the defect object detection of consecutive multi-frame monitoring images obtained from the video stream of the dynamic monitoring video, and generate object detection frames that match the defect detection targets. And by predicting and matching the motion states of the object detection frames and the corresponding movement trajectories of the previous frame of the monitoring image to be determined, obtain the object detection frames of the next frame, which is beneficial to efficiently and accurately realize the matching of the movement trajectories corresponding to the mobile monitoring video and the object detection frames. And by dividing several defect detection targets of the initial facility defect detection results into multiple detection result sets with different confidence levels, analyze the ground drainage facility defect detection performance through the confidence level of the detection results, and obtain the detection accuracy of the defect detection targets of the object detection frames summarized in each monitoring image, which is beneficial to meet the speed and accuracy of the intelligent detection of the ground drainage facility status. Description of the Drawings

[0060] Figure 1 is a flowchart of a method for detecting ground drainage facility defects based on a dynamic monitoring video in an embodiment of the present application;

[0061] Figure 2 is a flowchart of step S2 in a method for detecting ground drainage facility defects based on a dynamic monitoring video in an embodiment of the present application;

[0062] Figure 3 is a flowchart of step S203 in a method for detecting ground drainage facility defects based on a dynamic monitoring video in an embodiment of the present application;

[0063] Figure 4 is a flowchart of step S2034 in a method for detecting ground drainage facility defects based on a dynamic monitoring video in an embodiment of the present application;

[0064] Figure 5 is a flowchart before step S3 in a method for detecting ground drainage facility defects based on a dynamic monitoring video in an embodiment of the present application;

[0065] Figure 6 is a schematic diagram of the coordinates of the vehicle-mounted monitoring image (vehicle-mounted picture in the figure) after interpolation of the spatial coordinates in an embodiment of the present application for detecting ground drainage facility defects based on a dynamic monitoring video;

[0066] Figure 7It is an integration result diagram of the drainage facility defect detection result and the ground drainage facility GIS data in a ground drainage facility defect detection method based on dynamic monitoring video in an embodiment of the present application;

[0067] Figure 8 It is a result diagram of the integration of defect detection and target tracking trajectory in a ground drainage facility defect detection method based on dynamic monitoring video in an embodiment of the present application;

[0068] Figure 9 It is a schematic diagram of the device in an embodiment of the present application. Detailed implementation manners

[0069] The present application will be further described in detail below with reference to the accompanying drawings.

[0070] In an embodiment, as Figure 1 shown, the present application discloses a ground drainage facility defect detection method based on dynamic monitoring video, which specifically includes the following steps:

[0071] S1: Obtain the dynamic monitoring video of the target monitoring area and the ground drainage facility information, and the dynamic monitoring video is collected based on a preset mobile monitoring device.

[0072] In this embodiment, the target monitoring area is the detection area of urban municipal drainage facilities; the mobile monitoring device is a mobile intelligent acquisition terminal, such as a driving camera on a flood control inspection vehicle, a preset intelligent acquisition terminal, a handheld operation terminal, etc.; the ground drainage facility information includes drainage facility product information and facility installation location information.

[0073] Furthermore, the dynamic monitoring video can also be collected by a fixed monitoring device pre-set in the target monitoring area. The fixed monitoring device is a monitoring device pre-set on a road or a building. When the fixed monitoring device is operating normally, it can not only play the role of monitoring road security, but also provide monitoring video analysis data for the defect detection of the ground drainage facilities in the present application.

[0074] S2: Based on a preset facility defect target tracking algorithm, perform drainage facility defect detection and facility defect target tracking on the monitoring images of the dynamic monitoring video to obtain the drainage facility defect detection result.

[0075] In this embodiment, the facility defect target tracking algorithm includes a defect target detection algorithm and a defect target tracking algorithm. The defect target detection algorithm is a YOLOV7+SE target detection algorithm that integrates the SE attention mechanism, which has both detection accuracy and detection speed in the task of detecting ground drainage facility targets, and has better comprehensive detection performance. The defect target tracking algorithm is the BoT-SORT target tracking algorithm, which can more efficiently match prediction boxes (referring to the target detection boxes of the next frame of the monitoring image based on the current processed frame); uses Global Motion Compensation (GMC) to correct image changes caused by the movement of the camera (i.e., the dynamic monitoring device); when calculating the distance between the moving trajectory and the detection box, the Jonker-Volgenant algorithm is used as the matching algorithm, and the cosine distance of the IoU distance and surface features is integrated as the cost matrix, which more efficiently and accurately realizes the matching of the trajectory and the target detection box.

[0076] Specifically, first, the dynamic monitoring video is frame-divided to obtain multiple consecutive monitoring images. After the defect target detection of the continuous single-frame images of the video stream of the dynamic monitoring video is completed by the pre-trained YOLOV7+SE target detection algorithm, the initial facility defect detection result is obtained. Then, the initial facility defect detection result is used as the input of the defect target tracking algorithm to efficiently solve the multi-target tracking problem of multiple facility defect targets.

[0077] S3: Obtain the spatial information of the drainage facility defect detection result, and perform spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, and correspondingly obtain the facility defect target detection information.

[0078] Specifically, after obtaining the spatial information of the drainage facility defect detection result, perform spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information. The spatial matching analysis algorithm includes the integration of the information of the monitoring image of the dynamic monitoring video (the monitoring image at this time contains the drainage facility defect detection result) and the spatial information, and the integration of the facility defect target of the drainage facility defect detection result and the target tracking trajectory, so as to complete the integration of the drainage facility defect detection and the GIS data of the ground drainage facility information, that is, integrally obtain the facility defect target detection information with spatial information, so as to provide a way of intelligent detection and management based on the defects of the dynamic monitoring video and the ground drainage facilities, which is beneficial to improving the detection efficiency and detection accuracy of the ground drainage facility defects.

[0079] In one embodiment, as Figure 2As shown, in step S2, the preset facility defect target tracking algorithm includes a defect target detection algorithm and a defect target tracking algorithm; based on the preset facility defect target tracking algorithm, facility defect detection and facility defect target tracking of the drainage facility are performed on the monitoring images of the dynamic monitoring video, and the drainage facility defect detection results are obtained, including:

[0080] S201: The dynamic monitoring video is frame-divided to obtain multiple consecutive monitoring images to be determined.

[0081] S202: In the preset detection-based defect target tracking model, based on the defect target detection algorithm, facility defect detection is performed on multiple frames of monitoring images to be determined frame by frame, and the initial facility defect detection results of the corresponding consecutive multiple frames of monitoring images are obtained; the preset detection-based defect target tracking model is fused with an attention module for adaptively adjusting and optimizing the network structure of the detection-based defect target tracking model; the initial facility defect detection results include several defect detection targets.

[0082] In this embodiment, the attention module is the SE attention module, whose full name is the squeeze and excitation module, which is an attention module for extracting channel features in a neural network; through the attention module, the detection performance of the drainage facility defect target can be improved. When extracting and calculating the defect feature information of the facility defect target in the monitoring image, the detection-based defect target tracking model fused with the attention module can adaptively adjust the weights of different channels and spatial positions, thus helping to improve the recognition ability and model accuracy of the ground drainage facility defects.

[0083] Specifically, the detection-based defect target tracking model uses the YOLOV7 object detector, which is fused with the SE attention module based on the SE attention mechanism. In this application, the SE attention module is fused in the backbone network of the detection-based defect target tracking model. Experimental results show that: in the feature extraction stage, the fused attention module can enable the detection-based defect target tracking model to adaptively adjust the weights of different channels and spatial positions, which helps to improve the recognition ability of the detection-based defect target tracking model for ground drainage facility defects. Compared with the method of fusing the attention module in the neck network or the prediction network, in this application, the SE attention module is fused in the backbone network, and the single lightweight designed SE attention module significantly improves the model detection accuracy while improving the model detection speed, which is the best solution for the drainage facility defect target detection task.

[0084] S203: In the preset defect target tracking model based on detection, perform defect target tracking analysis on several defect detection targets based on the defect target tracking algorithm and the initial facility defect detection results to obtain the defect detection results of the drainage facilities in the continuous video stream.

[0085] In this embodiment, perform defect target tracking analysis on several defect detection targets in the initial facility defect detection results based on the defect target tracking algorithm to obtain the final defect detection results of the drainage facilities.

[0086] In one embodiment, as Figure 3 shown, in step S203, in the preset defect target tracking model based on detection, perform defect target tracking analysis on several defect detection targets based on the defect target tracking algorithm and the initial facility defect detection results to obtain the defect detection results of the drainage facilities, including:

[0087] S2031: Obtain the movement trajectory of the preset mobile monitoring device corresponding to the dynamic monitoring video; if a defect detection frame is obtained in multiple frames of monitoring images to be determined, use the defect detection frame as the matching basis for the next frame of defect detection frame.

[0088] In this embodiment, the defect target tracking algorithm is the BoT-SORT algorithm, and the defect target image is a monitoring image containing the defect feature information of the drainage facilities. Specifically, in the BoT-SORT algorithm, the state vector of the Kalman filter is defined as: where x and y represent the coordinates of the center of the detection frame, and w and h represent the width and height of the detection frame, represents the first derivative. Therefore, the target detection frame of the next frame of monitoring image can be predicted through the motion state of the existing trajectory, and the predicted state vector can be expressed as:

[0089] The predicted covariance matrix is: where, represents the predicted state vector of the kth frame, represents the updated state vector of the (k - 1)th frame, A k is the state transition matrix, representing the motion information of the trajectory; p' k represents the predicted covariance matrix, representing the uncertainty of the prediction, Q k represents the system noise matrix; by combining the predicted vector and the observed vector, the state vector can be updated, and the vector of the kth frame prediction box after update can be expressed as:

[0090]

[0091] where K represents the Kalman gain, K kRepresents the comparison between prediction uncertainty and observation uncertainty; H is the observation matrix, R is the observation noise covariance, and Z k represents the observation value at frame k.

[0092] The updated covariance matrix is: P k = P' k - KHP' k , P k will be used to calculate the predicted covariance matrix for the next frame.

[0093] S2032: In the preset defect target tracking model based on detection, target detection boxes of multiple frames of monitoring images to be determined are sequentially generated in chronological order; based on the defect target tracking algorithm, the target detection box of the monitoring image to be determined in the next frame is obtained by predicting and matching according to the target detection box of the monitoring image to be determined in the previous frame and the motion state of the corresponding moving trajectory.

[0094] In this embodiment, the target detection box of the monitoring image to be determined in the next frame is also called the prediction box; in the inspection task of ground drainage facilities, the dynamic monitoring video is often deformed due to the shooting center of the vehicle-mounted mobile camera; at this time, the global motion compensation (GMC) module in the BoT-SORT algorithm can correct the image changes caused by camera motion and improve the accuracy of target tracking; during the target tracking process, starting from the second frame, the GMC module extracts the key point information of the previous and next frames, estimates the affine transformation matrix H, and H is represented by the matrix as where a, b, c, d represent the parameters of rotation and scaling, and are the parameters of translation. After the affine transformation of the pixel coordinates (x, y), the corrected new coordinates (x', y') can be obtained, which is expressed as

[0095] S2033: Based on the defect target tracking algorithm, multiple target detection boxes, and a preset confidence threshold, several defect detection targets of the initial facility defect detection result are divided into multiple sets of detection results with different confidence levels.

[0096] In this embodiment, the confidence threshold is the IOU distance threshold.

[0097] S2034: Perform moving trajectory matching and target detection box matching on multiple sets of detection results to obtain the drainage facility defect detection result.

[0098] In this embodiment, the defect target tracking algorithm needs to match the movement trajectories, the target detection boxes and the prediction boxes. The defect target tracking algorithm uses the Jonker-Volgenant linear matching algorithm for matching, and at the same time fuses the IoU distance and the cosine distance of the surface features as the cost matrix, which more accurately realizes the matching of the trajectory and the detection box. The fusion of the IoU distance and the cosine distance of the surface features is as follows:

[0099]

[0100] Among them, represents the IoU distance between trajectory i and detection box j, represents the cosine distance of the surface features between trajectory i and detection box j; θ emb and θ iou are preset thresholds for eliminating pairs with large differences in movement trajectories and surface features. C i,j is each element in the cost matrix finally input to the linear matching algorithm.

[0101] The Jonker-Volgenant algorithm is used in BoT-SORT to solve the matching problem between trajectories and detection boxes; the goal of the linear matching algorithm is to find a permutation matrix with the minimum total cost according to the cost matrix, that is:

[0102]

[0103] Among them, c ij is an element of the cost matrix C, and x ij is an element of the permutation matrix X. The permutation matrix X satisfies the following conditions:

[0104] In the permutation matrix, x ij = 1 indicates that the row and column match, and x ij = 0 indicates no match. Only one element in each row and each column is selected; the Jonker-Volgenant algorithm continuously optimizes the cost matrix through reduction and augmentation iterations, and uses the augmented path technique to gradually improve the matching situation until the best matching matrix X that meets the above conditions is obtained.

[0105] Specifically, by dividing several defect detection targets of the initial facility defect detection results into multiple detection result sets with different confidence levels, the defect detection performance of the ground drainage facility is analyzed through the confidence level of the detection results, and the detection accuracy of the defect detection targets of the target detection boxes summarized in each monitoring image is obtained, which is beneficial to meeting the speed and accuracy of the intelligent detection of the ground drainage facility status.

[0106] In one embodiment, as Figure 4As shown, in step S2034, the detection result set includes a high-confidence detection set and a low-confidence detection set; performing moving trajectory matching and target detection box matching on multiple groups of detection result sets to obtain drainage facility defect detection results, including:

[0107] S20341: Extract trajectory information from the dynamic monitoring video, and construct and initialize a trajectory pool.

[0108] In this embodiment, the drainage facility defect detection results are divided into a high-confidence detection set (D_High) and a low-confidence detection set (D_Low); initialize the trajectory pool (strack_pool) and unconfirmed trajectories, and predict the state of each target and perform motion camera compensation.

[0109] S20342: In the high-confidence detection set, calculate the intersection-over-union distance matrix and appearance feature distance matrix between each trajectory in the initialized trajectory pool and several detected defect detection targets, generate a comprehensive distance matrix based on the intersection-over-union distance matrix and appearance feature distance matrix, and apply a preset linear assignment algorithm to determine the first matching result; the first matching result includes the matched defect target and trajectory pairs, the trajectories that are not successfully matched for the first time, and the target detection boxes of the targets for which no matching trajectories are found for the first time.

[0110] In this embodiment, the matching of the detection result set with high confidence is preferentially performed. First, calculate the intersection-over-union distance matrix (IoU_dist) and appearance feature distance matrix (Apperance_dist) between the initialized trajectory pool and the high-confidence detection set. Then, take the minimum value between the intersection-over-union distance matrix and the appearance feature distance matrix as the comprehensive distance matrix (Dist_mat), and use the linear assignment algorithm to find the first matching pairs (matched_pair0), the first unmatched moving trajectories (u_tracks0), and the first unmatched target detection boxes (u_dets0). Update the first matching pairs (matched_pair0) to the matched trajectories, and create new trajectories for the first unmatched target detection boxes (u_dets0).

[0111] S20343: Based on the trajectories that are not successfully matched remaining in the first matching result and the defect targets in the low-confidence detection set, recalculate the intersection-over-union distance matrix; and perform secondary matching using the preset linear assignment algorithm and the intersection-over-union distance matrix to generate a secondary matching result; the secondary matching result includes the newly added matching pairs, the trajectories that are not successfully matched for the second time, and the target detection boxes of the targets for which no matching trajectories are found for the second time.

[0112] In this embodiment, after the secondary matching, use the linear assignment algorithm to find the secondary matching pairs, the secondary unmatched moving trajectories, and the secondary unmatched target detection boxes, and update the secondary matching pairs to the trajectory pool.

[0113] Specifically, the second matching is used to match the remaining tracks after the first matching with the detection results of drainage facility defects in the low-confidence detection set (D_Low). First, calculate the intersection-over-union distance matrix between the first matching pairs (matched_pair0) and the low-confidence detection set (D_Low). Then, find the second matching pairs (matched_pair1) and the second unmatched moving tracks (u_tracks1) through the linear assignment algorithm, and update the second matching results to the track pool (strack_pool).

[0114] S20344: Integrate the first matching results and the second matching results to form the final matching result set, and identify all the finally unmatched tracks.

[0115] S20345: Update the track pool according to the final matching results and all the finally unmatched tracks, and remove the finally unmatched tracks and the long-time lost tracks.

[0116] In this embodiment, the long-time lost tracks are the lost tracks whose lost time exceeds the preset lost duration threshold; the preset lost duration threshold is the maximum lost time; the moving tracks that still cannot be successfully matched after the second matching are marked as lost.

[0117] Specifically, remove the long-time lost moving tracks. For the moving tracks that have been lost since the previous frame, if the lost time of the tracks exceeds the set maximum lost time, these tracks will be removed.

[0118] In one embodiment, as Figure 5 and Figure 6 shown, before step S3, a method for detecting drainage facility defects on the ground based on dynamic monitoring videos includes:

[0119] S301: Obtain the geographical coordinates of the monitoring device of the dynamic monitoring video, and perform a planar coordinate projection transformation on the geographical coordinates to obtain a planar rectangular coordinate.

[0120] Specifically, since the coordinate system used by the Global Navigation Satellite System (GNSS) carried by the vehicle for on-site inspection of ground drainage facilities is the WGS84 geographical coordinate system; therefore, before spatially matching the drainage facility defect detection results with the ground drainage facility information, it is necessary to convert the WGS84 coordinates into the same planar rectangular coordinate system as the ground drainage facility GIS data through the Gauss-Krüger projection; the coordinate conversion relationship can be described as:

[0121]

[0122] t = tanφ

[0123]

[0124] Δλ = λ - λ 0

[0125] Wherein, represents the input longitude and latitude, a is the semi-major axis length of the ellipsoid, λ 0 is the longitude of the central meridian, N represents the radius of curvature, t is the tangent value, Δλ is the longitude difference, representing the longitude difference between the input longitude and the longitude of the central meridian, and f is the flattening ratio.

[0126] Let k 0 be the scale factor, which is set to 1.0 in this projection; η is the first eccentricity. The plane rectangular coordinates (x, y) after projection are obtained by the following formula:

[0127]

[0128] S302: Based on the plane rectangular coordinates, when performing frame division on the dynamic monitoring video to obtain monitoring images, use linear interpolation to perform monitoring image coordinate positioning on each frame of the monitoring image to obtain instantaneous image positioning coordinates.

[0129] In this embodiment, referring to Figure 6 , Figure 6 the schematic diagram of the coordinates of the vehicle-mounted monitoring image (the vehicle-mounted picture in the figure) after interpolation by spatial coordinates is the scatter plot of the vehicle-mounted picture and the plane rectangular coordinates (GNSS coordinate receiving point); when the vehicle-mounted camera collects video images of ground drainage facilities, the vehicle-mounted camera and the vehicle-mounted satellite receiver receive signals simultaneously. However, since the two devices operate independently and have different receiving frequencies; therefore, when capturing the vehicle-mounted monitoring image, it is necessary to interpolate the coordinate data to obtain the instantaneous coordinates of the vehicle-mounted camera image (referring to the monitoring image coordinate positioning).

[0130] Specifically, linear interpolation is used for monitoring image coordinate positioning. Assume that at time t 1 and t 2 (where t 2 > t 1 ), the coordinates reported by the vehicle-mounted satellite receiver are (x 1 , x 2 ), then it can be considered that for the images obtained at any time point t between them, the coordinates (x, y) can be defined as:

[0131]

[0132] S303: Based on the plane rectangular coordinates and the instantaneous image positioning coordinates, determine the spatial information of each frame of the monitoring image; obtain the spatial information of the drainage facility defect detection result based on the spatial information of the monitoring image.

[0133] Specifically, the spatial information of the drainage facility defect detection result is obtained by interpolating the instantaneous image positioning coordinates with spatial coordinates, and the correlation between the monitoring image and the space can be established as follows:

[0134]

[0135] Exemplarily, in order to effectively avoid the influence of program running time consumption on time accuracy, the interpolation program is implemented using an asynchronous thread, and the specific implementation steps are as follows:

[0136] (1) The program listens to the information returned by the GNSS receiver, records the reception time and stores it in the folder coordinates, and converts the WGS84 coordinates (geographical coordinates) into geodetic plane coordinates (plane rectangular coordinates) through the Gauss-Krüger projection; listens to the monitoring image information captured by the vehicle-mounted camera, records the capture time and stores it in the folder images.

[0137] (2) The difference timer program scans the images folder at a frequency of 2Hz, and interpolates the picture coordinates in a linear interpolation manner.

[0138] (3) If adjacent coordinate points are found and the interpolation is successful, add a row to the matrix S and remove the picture from the images folder.

[0139] (4) If the GNSS listening and the vehicle-mounted camera listening are ended, and there are no pictures to be processed in the images folder, terminate the difference timer.

[0140] S304: Identify and obtain several defect detection targets of multiple drainage facility defect detection results, and merge the multiple drainage facility defect detection results identified as the same ground drainage facility by the facility defect target tracking algorithm to obtain the merged drainage facility defect detection result.

[0141] In this embodiment, since the relationship between the drainage facility defect detection result and the ground drainage facility GIS data is many-to-many, in order to simplify the calculation model and improve the calculation efficiency, before performing the spatial matching search, the drainage facility defect detection results identified as the same ground drainage facility by the target tracking algorithm are merged.

[0142] Specifically, assume that after merging, the average center point is used for spatial matching search. If there are points identified as the same ground drainage facility by the target tracking algorithm, and their coordinates are respectively (x 1 , y 1 ), (x 2 , y 2 )......(x i , y i ), then the coordinates of the average center point C are: So far, the process of finding the nearest neighbor points can be simplified to: where the set P = {P 1 , P 2 ,..., P n}, representing the data of the drainage facility points to be searched; each point P i = (x i , y i ) in the set,

[0143] r is the set maximum search radius.

[0144] In one embodiment, in step S3, the spatial information of the drainage facility defect detection result is obtained, and spatial matching analysis is performed based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, and the facility defect target detection information is correspondingly obtained, including:

[0145] S311: In the preset GIS database, obtain the drainage facility GIS data of the ground drainage facility information and correspondingly associate the drainage facility identifier.

[0146] S312: Use the nearest neighbor matching algorithm with a limited radius to perform neighboring coordinate matching on the spatial information of the drainage facility defect detection result and the ground drainage facility GIS data, and construct the matching relationship between the drainage facility defect detection result corresponding to the monitoring image and the ground drainage facility GIS data.

[0147] Specifically, as Figure 7 shown, Figure 7 is the integrated result graph of the drainage facility defect detection result and the ground drainage facility GIS data; read the drainage facility GIS data corresponding to the ground drainage facility information from the GIS database, and use the nearest neighbor matching algorithm with a limited radius to perform neighboring coordinate matching on the spatial information of the drainage facility defect detection result and the ground drainage facility GIS data to construct the matching association relationship (referring to the matching relationship) between the drainage facility defect detection result and the ground drainage facility GIS data, so as to facilitate the spatial matching of the drainage facility defect detection result with spatial information and the ground drainage facility GIS data.

[0148] S313: Based on the matching relationship, obtain the defect detection target corresponding to the drainage facility defect detection result, and correspondingly obtain the facility defect target detection information.

[0149] S314: In the preset GIS database, based on the drainage facility identifier, mark the drainage facility with the facility defect target detection information to integrate the drainage facility defect detection result into the preset GIS database.

[0150] Specifically, according to the unique drainage facility identifier of the drainage facility (where the drainage facility identifier is the product information code of the drainage facility with unique installation positioning information), the defect detection target of the drainage facility detected in real time (referring to the new facility defect information) is updated to the GIS database, so as to efficiently and real-time integrate the drainage facility defect detection results into the GIS database, facilitating the completion of the defect detection of ground drainage facilities and the integration of drainage facilities with GIS, and greatly improving the detection efficiency of ground drainage facility defects.

[0151] In one embodiment, before step S3, the method for detecting defects of ground drainage facilities based on dynamic monitoring video further includes:

[0152] Dividing the drainage facility defect detection results into the first drainage facility defect detection results of facility types and the second drainage facility defect detection results of defect types, and the dividing steps include:

[0153] S311: When performing facility defect target tracking on the monitoring images of the dynamic monitoring video based on a preset facility defect target tracking algorithm, generate the first drainage facility defect detection results with target tracking identifiers and monitoring image frame identifiers, and perform one-to-one association matching between the target tracking identifiers and the monitoring image identifiers.

[0154] In this embodiment, the facility defect target tracking algorithm of the present application adopts a target tracking method that focuses on tracking 3 types of facility types, namely, the local parts of rainwater inlets, inspection wells, and discharge outlets. By integrating the first drainage facility defect detection results of the facility types with the target tracking trajectories, and then matching the defect targets with the corresponding facility types, the integration of the defect targets and the tracking trajectories is completed.

[0155] S312: When performing facility defect detection on the monitoring images of the dynamic monitoring video based on a preset facility defect target tracking algorithm, generate the second drainage facility defect detection results with defect type identifiers and monitoring image frame identifiers.

[0156] S313: Based on the monitoring image identifiers and a preset IOU threshold, associate the defect detection targets of the second drainage facility defect detection results with the defect type identifiers; based on the same target tracking identifier, obtain the defect detection results of different facility defect targets of multiple frames of monitoring images respectively.

[0157] Specifically, as Figure 8 shown, Figure 8 is the result diagram of the integration of defect detection and target tracking trajectories; the implementation process of the facility defect target tracking algorithm is as follows:

[0158] (1) Target detection

[0159] For each image frame of the video stream (referring to the monitoring image), based on the pre-trained defect target tracking model based on YOLOV7+SE for detection, a set of target detection results (at this time, the first drainage facility defect detection results) can be obtained, denoted as a matrix

[0160]

[0161] where classid is the classification ID of the detection result, x and y represent the center coordinates of the target detection box, w and h represent the width and height of the target detection box, and conf represents the confidence level.

[0162] (2) Detection result classification

[0163] According to classid, the target detection results (the first drainage facility defect detection results) are divided into facility categories and defect categories. The facility set is S = {0, 1, 2}, and the defect set is F = {3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17}; for each target detection box in the set D, if classid i ∈S, then d i is of the facility category, denoted as matrix D 设施 , if classid i ∈F, then it is denoted as matrix D 缺陷 .

[0164] (3) Facility tracking

[0165] The BoT-SORT target tracking algorithm is used to perform target tracking on the defect detection results of the facility category of the drainage facilities, and the tracking results are denoted as a matrix where traceid i represents the target tracking identifier, fracmeid i is the monitoring image identifier, and the monitoring image identifier is the unique identifier of each image. Through facility tracking, the matching of traceid i and fracmeid i is achieved.

[0166] (4) Defect association

[0167] After completing the facility target tracking of the video stream, it will be associated with the defects according to the IoU threshold. For each facility t j ∈T, find all the defects whose IoU value is greater than the confidence threshold and whose fracmeid i is the same. The association process can be expressed as:

[0168]

[0169] Among them, IOU(d i , t j ) represents the intersection over union of defect d i and facility t j . The associated matrix R can be expressed as:

[0170]

[0171] As shown in R, since each tracking ID corresponds to multiple images, the detection results of each image may correspond to multiple defects, that is, there are defect detection results at multiple angles of different images for each facility. To finally determine the type of defect existing in the facility, the defect detection results of multiple images need to be integrated. For the average confidence θ 2 of each type of each ground drainage facility, it can be defined as:

[0172]

[0173] Among them, n k represents the number of defects of type classid j associated with facility r k , and defect_conf i is the confidence of defect d i . Thus, the results with too low average confidence can be removed through the confidence threshold R′ = {r j |C avg (r j ) > θ 2}.

[0174] And / or,

[0175] S314: Obtain the defect target tracking trajectory corresponding to the defect detection result of the first drainage facility, and associate and match the defect detection results of different facility defect targets with the corresponding defect target tracking trajectories to obtain the defect detection result of the drainage facility with spatial information.

[0176] Specifically, through the integration of the monitoring image information and the spatial information set, as well as the integration of the defect detection result of the drainage facility and the target tracking trajectory, the defect detection result of the drainage facility with spatial information is obtained through integration.

[0177] It should be understood that the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0178] In one embodiment, a defect detection system for ground drainage facilities based on dynamic monitoring video is provided. The defect detection system for ground drainage facilities based on dynamic monitoring video corresponds to the defect detection method for ground drainage facilities based on dynamic monitoring video in the above embodiment.

[0179] A defect detection system for ground drainage facilities based on dynamic monitoring video includes a monitoring video and drainage facility information acquisition module, a facility defect detection result acquisition module, and a spatial information matching and analysis module. The detailed descriptions of each functional module are as follows:

[0180] The monitoring video and drainage facility information acquisition module is used to acquire the dynamic monitoring video of the target monitoring area and the ground drainage facility information. The dynamic monitoring video is collected based on a preset mobile monitoring device.

[0181] The facility defect detection result acquisition module is used to perform facility defect detection and facility defect target tracking on the monitoring images of the dynamic monitoring video based on a preset facility defect target tracking algorithm to obtain the drainage facility defect detection result.

[0182] The spatial information matching and analysis module is used to acquire the spatial information of the drainage facility defect detection result, and perform spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to correspondingly obtain the facility defect target detection information.

[0183] For the specific limitations of the defect detection system for ground drainage facilities based on dynamic monitoring video, reference can be made to the limitations of the defect detection method for ground drainage facilities based on dynamic monitoring video in the above text, which will not be elaborated here; each module in the above defect detection system for ground drainage facilities based on dynamic monitoring video can be implemented in whole or in part by software, hardware, and their combinations; the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above modules.

[0184] In one embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 9As shown in the figure. The computer device includes a processor, a memory, a network interface, and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store mobile trajectories, dynamic monitoring videos, drainage facility defect detection results, etc. The network interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a method for detecting defects in ground drainage facilities based on dynamic monitoring videos.

[0185] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-mentioned method for detecting defects in ground drainage facilities based on dynamic monitoring videos.

[0186] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, it implements the steps of the above-mentioned method for detecting defects in ground drainage facilities based on dynamic monitoring videos.

[0187] Those of ordinary skill in the art can understand that all or part of the processes of implementing the above-mentioned method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned method embodiments. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories.

[0188] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0189] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting defects in ground drainage facilities based on dynamic monitoring video, characterized in that: include: Obtaining dynamic monitoring video and ground drainage facility information of the target monitoring area, wherein the dynamic monitoring video is collected based on a preset mobile monitoring device; Based on a preset facility defect target tracking algorithm, the monitoring image of the dynamic monitoring video is subjected to facility defect detection and facility defect target tracking of the drainage facility to obtain a drainage facility defect detection result; Acquire the spatial information of the drainage facility defect detection result, and perform spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to obtain corresponding facility defect target detection information; The obtaining of the spatial information of the drainage facility defect detection result, and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, before obtaining the corresponding facility defect target detection information, includes: Obtaining the geographic coordinates of the monitoring device of the dynamic monitoring video, and performing plane coordinate projection transformation on the geographic coordinates to obtain plane rectangular coordinates; Based on the plane rectangular coordinates, when the dynamic monitoring video is frame-processed to obtain the monitoring image, the monitoring image coordinates of each frame of the monitoring image are positioned by linear interpolation to obtain the instantaneous image positioning coordinates; Based on the plane rectangular coordinates and the instantaneous image positioning coordinates, determine the spatial information of each frame of monitoring image; based on the spatial information of the monitoring image, obtain the spatial information of the drainage facility defect detection result; Identify and obtain a plurality of defect detection targets of the drainage facility defect detection results, merge the plurality of drainage facility defect detection results identified as the same ground drainage facility by the facility defect target tracking algorithm, and obtain a merged drainage facility defect detection result.

2. A method for detecting defects in ground drainage facilities based on dynamic monitoring video according to claim 1, characterized in that: The preset facility defect target tracking algorithm includes a defect target detection algorithm and a defect target tracking algorithm; the monitoring image of the dynamic monitoring video is subjected to facility defect detection and facility defect target tracking of the drainage facility based on the preset facility defect target tracking algorithm to obtain a drainage facility defect detection result, including: Performing frame processing on the dynamic monitoring video to obtain multiple frames of continuous monitoring images to be determined; In the preset defect target tracking model based on detection, facility defect detection is performed frame by frame on multiple frames of the monitoring images to be determined based on the defect target detection algorithm to obtain initial facility defect detection results of the corresponding continuous multiple frames of monitoring images; the preset defect target tracking model based on detection is integrated with an attention module for enabling the defect target tracking model based on detection to adaptively adjust the network structure of the optimization model; the initial facility defect detection results include a number of defect detection targets; In the preset detection-based defect target tracking model, defect target tracking analysis is performed on several defect detection targets based on the defect target tracking algorithm and the initial facility defect detection results to obtain drainage facility defect detection results of a continuous video stream.

3. A method for detecting defects in ground drainage facilities based on dynamic monitoring video according to claim 2, characterized in that: In the preset detection-based defect target tracking model, defect target tracking analysis is performed based on the defect target tracking algorithm and several defect detection targets of the initial facility defect detection result to obtain the drainage facility defect detection result, including: Obtaining a moving trajectory of a preset mobile monitoring device corresponding to the dynamic monitoring video; if a defect detection frame is obtained in multiple frames of the monitoring image to be determined, using the defect detection frame as a matching basis for the defect detection frame of the next frame; In the preset defect target tracking model based on detection, target detection frames of multiple frames of the monitoring image to be determined are generated in sequence based on the time sequence; based on the defect target tracking algorithm, the target detection frame of the monitoring image to be determined in the next frame is predicted and matched according to the target detection frame of the monitoring image to be determined in the previous frame and the motion state of the corresponding moving trajectory; Based on the defect target tracking algorithm, the plurality of target detection frames and a preset confidence threshold, the plurality of defect detection targets of the initial facility defect detection result are divided into a plurality of detection result sets with different confidence levels; Movement trajectory matching and target detection frame matching are performed on multiple groups of detection result sets to obtain the drainage facility defect detection results.

4. A method for detecting defects in ground drainage facilities based on dynamic monitoring video according to claim 3, characterized in that: The detection result set includes a high-confidence detection set and a low-confidence detection set; performing movement trajectory matching and target detection frame matching on multiple groups of the detection result sets to obtain the drainage facility defect detection results includes: Extracting trajectory information from the dynamic monitoring video, constructing and initializing a trajectory pool; In the high-confidence detection set, the intersection-over-union distance matrix and the appearance feature distance matrix between each track in the initialization track pool and the detected defect detection targets are calculated, and a comprehensive distance matrix is ​​generated based on the intersection-over-union distance matrix and the appearance feature distance matrix, and a preset linear allocation algorithm is applied to determine the first matching result; the first matching result includes the defect target and track pair that are successfully matched, the track that is not successfully matched for the first time, and the target detection frame that does not find the matching track for the first time; Recalculate the intersection-over-union distance matrix based on the remaining unsuccessfully matched trajectories of the first matching result and the defect targets in the low-confidence detection set; and apply the preset linear assignment algorithm and the intersection-over-union distance matrix to perform secondary matching to generate secondary matching results; the secondary matching results include newly added matching pairs, trajectories that were not successfully matched for the second time, and target detection boxes for which no matching trajectories were found for the second time; Integrate the first matching result and the second matching result to form a final matching result set, and identify all final unmatched tracks; According to the final matching results and all final unmatched trajectories, the trajectory pool is updated, and the final unmatched trajectories and long-term lost trajectories are removed.

5. The method for detecting defects of ground drainage facilities based on dynamic monitoring video according to claim 1 is characterized in that: The acquiring of the spatial information of the drainage facility defect detection result, and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to obtain corresponding facility defect target detection information includes: In a preset GIS database, the drainage facility GIS data of the surface drainage facility information is obtained, and the drainage facility identification is correspondingly associated; Use the nearest neighbor matching algorithm with a limited radius to perform neighbor coordinate matching on the spatial information of the drainage facility defect detection result and the ground drainage facility GIS data, and establish a matching relationship between the drainage facility defect detection result corresponding to the monitoring image and the ground drainage facility GIS data; Based on the matching relationship, a defect detection target corresponding to the drainage facility defect detection result is obtained, and facility defect target detection information is obtained accordingly; In a preset GIS database, the drainage facilities of the facility defect target detection information are marked as facility defects based on the drainage facility identification, so as to integrate the drainage facility defect detection results into the preset GIS database.

6. A method for detecting defects in ground drainage facilities based on dynamic monitoring video according to claim 1, characterized in that: The method further comprises: obtaining the spatial information of the drainage facility defect detection result, and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, before obtaining the corresponding facility defect target detection information: The drainage facility defect detection results are divided into first drainage facility defect detection results of facility type and second drainage facility defect detection results of defect type, the dividing step comprising: When performing facility defect target tracking on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm, a first drainage facility defect detection result having a target tracking identifier and a monitoring image frame identifier is generated, and the target tracking identifier is associated and matched with the monitoring image frame identifier one by one; When performing facility defect detection on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm, a second drainage facility defect detection result having a defect type identifier and a monitoring image frame identifier is generated; Based on the monitoring image frame identifier and the preset IOU threshold, the defect detection target of the second drainage facility defect detection result is associated with the defect type identifier; based on the same target tracking identifier, defect detection results of different facility defect targets of multiple frames of monitoring images are obtained respectively; Obtain a defect target tracking trajectory corresponding to the first drainage facility defect detection result, and associate and match the defect detection results of the different facility defect targets with the corresponding defect target tracking trajectory to obtain a drainage facility defect detection result with spatial information.

7. A surface drainage facility defect detection system based on dynamic monitoring video, characterized in that: The system comprises: A monitoring video and drainage facility information acquisition module is used to acquire dynamic monitoring video and ground drainage facility information of the target monitoring area, wherein the dynamic monitoring video is acquired based on a preset mobile monitoring device; A facility defect detection result acquisition module is used to perform facility defect detection and facility defect target tracking of the drainage facility on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm to obtain a drainage facility defect detection result; A spatial information matching and analysis module, used to obtain the spatial information of the drainage facility defect detection result, and perform spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, to obtain corresponding facility defect target detection information; The obtaining of the spatial information of the drainage facility defect detection result, and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, before obtaining the corresponding facility defect target detection information, includes: Obtaining the geographic coordinates of the monitoring device of the dynamic monitoring video, and performing plane coordinate projection transformation on the geographic coordinates to obtain plane rectangular coordinates; Based on the plane rectangular coordinates, when the dynamic monitoring video is frame-processed to obtain the monitoring image, the monitoring image coordinates of each frame of the monitoring image are positioned by linear interpolation to obtain the instantaneous image positioning coordinates; Based on the plane rectangular coordinates and the instantaneous image positioning coordinates, determine the spatial information of each frame of monitoring image; based on the spatial information of the monitoring image, obtain the spatial information of the drainage facility defect detection result; Identify and obtain a plurality of defect detection targets of the drainage facility defect detection results, merge the plurality of drainage facility defect detection results identified as the same ground drainage facility by the facility defect target tracking algorithm, and obtain a merged drainage facility defect detection result.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements the steps of a method for detecting defects in ground drainage facilities based on dynamic monitoring video as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a method for detecting defects in ground drainage facilities based on dynamic monitoring video as described in any one of claims 1 to 6 are implemented.

Images