Ground drainage facility defect detection method based on dynamic monitoring video
Through the combination of dynamic monitoring video and facility defect target tracking algorithm, automated and intelligent detection of ground drainage facilities is realized, and the problem of low efficiency of traditional manual inspection is solved, the accuracy and timeliness of detection are improved, and the risk of flooding disasters is reduced. .
Patent Information
- Application Number
- CN202411122887.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-15
AI Technical Summary
In the prior art, defect detection of urban ground drainage facilities depends on traditional manual inspections, which are inefficient, prone to missed inspections, and misinspection, resulting in failure to maintain and repair the facilities in a timely manner, affecting drainage efficiency, and may cause flooding disasters during heavy rains.
Using ground drainage facilities defect detection method based on dynamic monitoring video, video data is collected through mobile monitoring devices, facility defect target tracking algorithm is used for automated detection and intelligent management, and space matching analysis is carried out in combination with GIS data to achieve fast and accurate defect detection. and notification.
It improves the efficiency and accuracy of ground drainage facilities defect detection, reduces the impact of floodwater disasters during the flood season, saves manual inspection costs, and promptly notify flood control personnel for handling.
Smart Images

Figure BDA0004995568680000134 
Figure BDA0004995568680000141 
Figure BDA0004995568680000144
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of facility defect detection, and in particular to a method for detecting defects in ground drainage facilities based on dynamic monitoring videos. Background Art
[0002] In recent years, with the rapid development of cities in my country, the number of drainage facilities has continued to increase, and the number of ground drainage facilities on urban roads is huge. On the one hand, the huge number of drainage facilities has effectively guaranteed the drainage safety of the city, but on the other hand, the problems of facility operation efficiency, operation benefits, and safety resilience will also restrict the high-quality development of urban infrastructure.
[0003] Although the country has gradually increased its investment in the operation and maintenance of urban drainage facilities in recent years, and improved the professional operation and maintenance management mechanism of drainage facilities, the overall maintenance level of urban drainage facilities has been improved to a certain extent. However, the current operation and maintenance inspection of drainage facilities still mainly relies on traditional manual inspections. After the staff visually determines that the ground drainage facilities have defects, they can take photos of the defective ground drainage facilities, obtain the corresponding geographical location, and send the photos and geographical location information to the facility management platform to complete the defect detection and defect management of the ground drainage facilities. The inspection efficiency is low, often causing missed inspections and wrong inspections, and the drainage facilities are not maintained and repaired in time, resulting in the drainage facilities failing to play their original drainage efficiency. When heavy rain and waterlogging occur, it is often found that the rainwater outlets are blocked, the manhole covers are lost, and the discharge outlets are blocked by foreign objects. Abnormal operation of ground drainage facilities, etc., poses a potential threat to public safety; therefore, it is urgent to quickly and accurately detect defective facilities during daily inspections, and promptly notify drainage operation and maintenance personnel to deal with them. In the flood season, the defects of ground drainage facilities can be detected in time before and during the flood season, and flood control personnel can be notified in time to deal with them, so as to reduce the impact of flood disasters during the flood season. Summary of the invention
[0004] In order to improve the detection efficiency of defects in ground drainage facilities and reduce the impact of waterlogging disasters during flood season, the present application provides a method for detecting defects in ground drainage facilities based on dynamic monitoring videos.
[0005] In the first aspect, the invention objective of the present application is achieved by adopting the following technical solutions: A method for detecting defects of ground drainage facilities based on dynamic monitoring video, comprising: Obtaining dynamic monitoring video and ground drainage facility information of the target monitoring area, wherein the dynamic monitoring video is collected based on a preset mobile monitoring device; Based on a preset facility defect target tracking algorithm, the monitoring image of the dynamic monitoring video is subjected to facility defect detection and facility defect target tracking of the drainage facility to obtain a drainage facility defect detection result; The spatial information of the drainage facility defect detection result is obtained, and spatial matching analysis is performed based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to obtain corresponding facility defect target detection information.
[0006] By adopting the above-mentioned technical scheme, the present application provides a high detection efficiency method for detecting defects in ground drainage facilities based on dynamic monitoring videos, and the defect detection method of ground drainage facilities is more automated and intelligent; the target monitoring area is the urban municipal drainage facility detection area, and the mobile monitoring device is a mobile intelligent collection terminal (such as a driving camera on a flood prevention patrol vehicle, a preset intelligent collection terminal, a handheld operation terminal, etc.), and the ground drainage facility information includes drainage facility product information and facility installation location information; specifically, after obtaining the dynamic monitoring video and ground drainage facility information of the target monitoring area, a preset facility defect target tracking algorithm is used to perform facility defect target detection of the ground drainage facility on the monitoring image of the dynamic monitoring video, and after monitoring the existence of facility defects in the ground drainage facility, facility defect target tracking is performed to obtain the drainage facility defect detection result, which is conducive to improving the detection accuracy of facility defect targets in the dynamic monitoring video, and by using target detection technology to perform intelligent recognition of the target status of ground drainage facility defects, daily inspections can quickly and accurately detect the existence of Defective facilities; the next step is to obtain the spatial information of the drainage facility defect detection results, and perform spatial matching analysis based on the spatial information of the drainage facility defect detection results and the ground drainage facility information. The spatial matching analysis algorithm includes the integration of the monitoring image information of the dynamic monitoring video (the monitoring image at this time contains the drainage facility defect detection results) and the spatial information, and the integration of the facility defect target of the drainage facility defect detection result and the target tracking trajectory, so as to complete the integration of the drainage facility defect detection and the GIS data of the ground drainage facility information, that is, the facility defect target detection information with spatial information is integrated to obtain the facility defect target detection information, so that the present application provides a method for intelligent detection and management of defects based on dynamic monitoring videos and ground drainage facilities, which is conducive to improving the detection efficiency and detection accuracy of ground drainage facility defects, and is conducive to saving the manual detection cost of ground drainage facility defects; in the flood season, the drainage facility video (referring to the dynamic monitoring video) can be dynamically collected by the camera device before and during the flood season, the ground drainage facility defects can be detected in time, and the flood control personnel can be notified in time to deal with it, which is conducive to reducing the impact of waterlogging disasters during the flood season.
[0007] In a preferred example of the present application: the preset facility defect target tracking algorithm includes a defect target detection algorithm and a defect target tracking algorithm; the monitoring image of the dynamic monitoring video is subjected to facility defect detection and facility defect target tracking based on the preset facility defect target tracking algorithm to obtain a drainage facility defect detection result, including: performing frame processing on the dynamic monitoring video to obtain multiple frames of continuous monitoring images to be determined; In the preset defect target tracking model based on detection, facility defect detection is performed frame by frame on multiple frames of the monitoring images to be determined based on the defect target detection algorithm to obtain initial facility defect detection results of the corresponding continuous multiple frames of monitoring images; the preset defect target tracking model based on detection is integrated with an attention module for enabling the defect target tracking model based on detection to adaptively adjust the network structure of the optimization model; the initial facility defect detection results include a number of defect detection targets; In the preset detection-based defect target tracking model, defect target tracking analysis is performed on several defect detection targets based on the defect target tracking algorithm and the initial facility defect detection results to obtain drainage facility defect detection results of a continuous video stream.
[0008] By adopting the above technical scheme, the dynamic monitoring video is firstly processed by frame division to obtain multiple frames of continuous monitoring images to be judged based on time sequence, wherein the monitoring image is a ground road image obtained by shooting the drainage setting area on the road ground with a preset mobile monitoring device, so as to identify the ground drainage facilities from the monitoring image, and perform facility defect detection and facility defect target tracking on the ground drainage facilities in the monitoring image; then, a defect target detection algorithm is adopted to perform facility defect detection on multiple frames of monitoring images to be judged frame by frame to obtain an initial facility defect detection result including several defect detection targets; the present application adopts a preset detection-based defect target tracking model integrated with an attention module, and the attention module can improve the drainage facility defect target detection performance, and the defect characteristics of the facility defect target in the monitoring image can be detected. The information is extracted and settled, and the detection-based defect target tracking model integrated with the attention module can adaptively adjust the weights of different channels and spatial positions, which helps to improve the recognition ability of ground drainage facility defects and the model accuracy; further, in the preset detection-based defect target tracking model, a defect target tracking analysis is performed on several defect detection targets in the initial facility defect detection results based on the defect target tracking algorithm, and the final drainage setting defect detection result is obtained, so as to efficiently troubleshoot the defects of the ground drainage facilities; the present application provides an integrated drainage facility defect real-time detection method to realize real-time tracking of ground drainage facility targets, and daily inspections can quickly and accurately discover defective ground drainage facilities, and promptly notify drainage operation and maintenance personnel to handle them, thereby improving the operation efficiency of ground drainage facilities.
[0009] In a preferred example of the present application, in the preset detection-based defect target tracking model, defect target tracking analysis is performed based on the defect target tracking algorithm and several defect detection targets of the initial facility defect detection result to obtain a drainage facility defect detection result, including: Obtaining a moving trajectory of a preset mobile monitoring device corresponding to the dynamic monitoring video; if a defect detection frame is obtained in multiple frames of the monitoring image to be determined, using the defect detection frame as a matching basis for the defect detection frame of the next frame; In the preset detection-based defect target tracking model, multiple frames of target detection frames of the monitoring image to be determined are generated in sequence based on the time sequence; based on the defect target tracking algorithm, the target detection frame of the monitoring image to be determined in the next frame is predicted and matched according to the target detection frame of the monitoring image to be determined in the previous frame and the motion state of the corresponding moving trajectory; based on the defect target tracking algorithm, the multiple target detection frames and the preset confidence threshold, the multiple defect detection targets of the initial facility defect detection result are divided into multiple groups of detection result sets with different confidence levels; Movement trajectory matching and target detection frame matching are performed on multiple groups of the detection result sets to obtain the drainage facility defect detection results.
[0010] By adopting the above technical solution, the multi-target tracking problem of multiple drainage facility defect targets (referring to defect detection targets) in multiple frames of monitoring images is solved. Based on the target tracking framework of drainage facility defect detection, the detection-based defect target tracking model can efficiently track multiple defect detection targets when performing real-time detection tasks; specifically, through the pre-trained detection-based defect target tracking model, the defect target tracking algorithm is used to complete the defect target detection of multiple consecutive frames of monitoring images obtained from the video stream of the dynamic monitoring video, and generate a target detection frame matching the defect detection target, and the target detection frame of the next frame is obtained by predicting and matching the motion state of the target detection frame of the monitoring image to be determined in the previous frame and the corresponding moving trajectory, which is conducive to efficiently and accurately matching the moving trajectory corresponding to the mobile monitoring video with the target detection frame, and by dividing several defect detection targets of the initial facility defect detection results into multiple detection result sets with different confidences, the ground drainage facility defect detection performance is analyzed by the confidence of the detection results, and the detection accuracy of the defect detection target of the target detection frame summarized in each monitoring image is obtained, which is conducive to meeting the speed and accuracy of intelligent detection of the ground drainage facility status.
[0011] In a preferred example of the present application: the detection result set includes a high-confidence detection set and a low-confidence detection set; the movement trajectory matching and target detection frame matching are performed on the multiple groups of the detection result sets to obtain the drainage facility defect detection results, including: Extracting trajectory information from the dynamic monitoring video, constructing and initializing a trajectory pool; In the high-confidence detection set, the intersection-over-union distance matrix and the appearance feature distance matrix between each track in the initialization track pool and the detected defect detection targets are calculated, and a comprehensive distance matrix is generated based on the intersection-over-union distance matrix and the appearance feature distance matrix, and a preset linear allocation algorithm is applied to determine the first matching result; the first matching result includes the defect target and track pair that are successfully matched, the track that is not successfully matched for the first time, and the target detection frame that does not find the matching track for the first time; Recalculate the intersection-over-union distance matrix based on the remaining unsuccessfully matched trajectories of the first matching result and the defect targets in the low-confidence detection set; and apply the preset linear assignment algorithm and the intersection-over-union distance matrix to perform secondary matching to generate secondary matching results; the secondary matching results include newly added matching pairs, trajectories that were not successfully matched for the second time, and target detection boxes for which no matching trajectories were found for the second time; Integrate the first matching result and the second matching result to form a final matching result set, and identify all final unmatched tracks; According to the final matching results and all final unmatched trajectories, the trajectory pool is updated, and the final unmatched trajectories and long-term lost trajectories are removed.
[0012] By adopting the above-mentioned technical scheme, the present application provides a customized mobile trajectory management method based on trajectory initialization, trajectory updating, and trajectory deletion; specifically, a confidence threshold of the corresponding drainage facility defect detection result is set based on the target detection frame generated by the monitoring image to divide the detection result set into a high-confidence detection set and a low-confidence detection set. After initializing the trajectory pool, the mobile trajectory and the target detection frame are matched preferentially for the high-confidence detection set. By calculating the intersection-and-union distance matrix and the appearance feature distance matrix between the mobile trajectory in the initialized trajectory pool and the corresponding defect detection targets, the defect detection target tracking and the target detection frame of the mobile trajectory and the target detection frame are obtained to obtain a comprehensive distance matrix with a more accurate matching effect, thereby efficiently completing the defect target tracking of the ground drainage setting. ; The first matching result is obtained by allocating through the preset linear allocation algorithm. After the first matching is completed, the successfully matched matching pairs are updated to the trajectory pool, and a new mobile trajectory is created for the unmatched detection frame for the first time; in the second matching, the remaining trajectories remaining from the first matching and the drainage facility defect detection results in the low-confidence detection set are used to obtain the secondary matching results, and secondary matching pairs and secondary unmatched trajectories are obtained. In the next step, the trajectories that are not successfully matched in the secondary matching are marked as lost, and the final unmatched mobile trajectory is deleted. At the same time, the mobile trajectory that has been lost for a long time is removed. The mobile trajectory corresponding to the drainage facility defect detection results is executed in real-time and efficiently, such as trajectory update and trajectory deletion, so as to realize efficient and accurate analysis of the mobile trajectory, and the drainage facility defect detection result obtained is of high accuracy.
[0013] In a preferred example of the present application: the obtaining of the spatial information of the drainage facility defect detection result, and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, before obtaining the corresponding facility defect target detection information, includes: Obtaining the geographic coordinates of the monitoring device of the dynamic monitoring video, and performing plane coordinate projection transformation on the geographic coordinates to obtain plane rectangular coordinates; Based on the plane rectangular coordinates, when the dynamic monitoring video is frame-processed to obtain the monitoring image, the monitoring image coordinates of each frame of the monitoring image are positioned by linear interpolation to obtain the instantaneous image positioning coordinates; Based on the plane rectangular coordinates and the instantaneous image positioning coordinates, determine the spatial information of each frame of monitoring image; based on the spatial information of the monitoring image, obtain the spatial information of the drainage facility defect detection result; Identify and obtain a plurality of defect detection targets of the drainage facility defect detection results, merge the plurality of drainage facility defect detection results identified as the same ground drainage facility by the facility defect target tracking algorithm, and obtain a merged drainage facility defect detection result.
[0014] By adopting the above technical solution, before the drainage facility defect detection result is spatially matched and analyzed with the ground drainage facility information, since the positioning system carried by the mobile monitoring device used to collect the dynamic monitoring video is the global navigation satellite system (GNSS) and the coordinate system used is the WGS84 geographic coordinate system, the WGS84 coordinate system is first converted into a plane rectangular coordinate system that is the same as the data of the ground drainage facility information to obtain the plane rectangular coordinates; at the same time, when the dynamic mobile monitoring device collects the ground drainage setting video image, the mobile monitoring device and the vehicle-mounted satellite receiver of the equipment equipped with the mobile monitoring device (such as a flood prevention patrol vehicle) receive information at the same time, and the two cameras operate independently and have different receiving frequencies. Because when the monitoring image of the dynamic monitoring video is obtained, it is necessary to use a linear interpolation method to locate the monitoring image coordinates of each frame of the monitoring image to obtain the instantaneous image positioning coordinates of the mobile monitoring device, so as to efficiently complete the image positioning of the monitoring image of the mobile monitoring device, and at the same time, for the determined plane rectangular coordinates and the instantaneous image positioning coordinates, the spatial information of the corresponding drainage facility defect detection result is further obtained to facilitate the subsequent drainage facility defect detection and facility defect target tracking.
[0015] Furthermore, in order to increase the computational complexity of ground drainage facility defect detection and improve computational detection efficiency, before performing spatial matching retrieval analysis, and in the event that the same ground drainage facility may have multiple types of facility defects, this scheme will also merge the detection results of multiple drainage facility defects identified as the same ground drainage facility by the facility defect target tracking algorithm.
[0016] In a preferred example of the present application: the spatial information of the drainage facility defect detection result is obtained, and spatial matching analysis is performed based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to obtain corresponding facility defect target detection information, including: In a preset GIS database, the drainage facility GIS data of the surface drainage facility information is obtained, and the drainage facility identification is correspondingly associated; Use the nearest neighbor matching algorithm with a limited radius to perform neighbor coordinate matching on the spatial information of the drainage facility defect detection result and the ground drainage facility GIS data, and establish a matching relationship between the drainage facility defect detection result corresponding to the monitoring image and the ground drainage facility GIS data; Based on the matching relationship, a defect detection target corresponding to the drainage facility defect detection result is obtained, and facility defect target detection information is obtained accordingly; In a preset GIS database, the drainage facilities of the facility defect target detection information are marked as facility defects based on the drainage facility identification, so as to integrate the drainage facility defect detection results into the preset GIS database.
[0017] By adopting the above technical scheme, the drainage facility GIS data corresponding to the ground drainage facility information is read from the GIS database, and the spatial information of the drainage facility defect detection results and the ground drainage facility GIS data are matched with neighboring coordinates using the nearest neighbor matching algorithm with a limited radius, so as to construct a matching association relationship (referring to the matching relationship) between the drainage facility defect detection results and the ground drainage facility GIS data, so as to facilitate the spatial matching of the drainage facility defect detection results with spatial information and the ground drainage facility GIS data, and at the same time, according to the unique drainage facility identification of the drainage facility (where the drainage facility identification is the unique product information code of the drainage facility with installation positioning information), the defect detection target of the drainage facility detected in real time (referring to the new facility defect information) is updated to the GIS database, so as to efficiently and real-time integrate the drainage facility defect detection results into the GIS database, so as to facilitate the completion of the ground drainage facility defect detection and the drainage facility GIS integration, and greatly improve the detection efficiency of ground drainage facility defects.
[0018] In a preferred example of the present application: the spatial information of the drainage facility defect detection result is obtained, and spatial matching analysis is performed based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, and before the corresponding facility defect target detection information is obtained, the method further includes: The drainage facility defect detection results are divided into first drainage facility defect detection results of facility type and second drainage facility defect detection results of defect type, the dividing step comprising: When performing facility defect target tracking on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm, a first drainage facility defect detection result having a target tracking identifier and a monitoring image frame identifier is generated, and the target tracking identifier is associated and matched with the monitoring image identifier one by one; When performing facility defect detection on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm, a second drainage facility defect detection result having a defect type identifier and a monitoring image frame identifier is generated; Based on the monitoring image identifier and the preset IOU threshold, the defect detection target of the second drainage facility defect detection result is associated with the defect type identifier; based on the same target tracking identifier, defect detection results of different facility defect targets of multiple frames of monitoring images are obtained respectively; Obtain a defect target tracking trajectory corresponding to the first drainage facility defect detection result, and associate and match the defect detection results of the different facility defect targets with the corresponding defect target tracking trajectory to obtain a drainage facility defect detection result with spatial information.
[0019] By adopting the above technical scheme, the defect detection results of different facility defect targets are integrated with the corresponding defect target tracking trajectories, and the facility defect targets are matched with the defect types to complete the integration of the facility defect targets and the corresponding tracking trajectories; specifically, when tracking the facility defect targets in the monitoring image of the dynamic monitoring video, the drainage facility defect type is correspondingly associated with the monitoring image, and then based on the preset IOU threshold, the defect detection results of different types of facility defect targets in the monitoring image are associated with the monitoring image identification, which is beneficial to improving the detection accuracy and efficiency of the drainage facility defect detection results, and by associating the corresponding defect detection targets in the monitoring image with the facility defect type, the drainage facility defect detection results are divided according to the facility type and the defect type, respectively, to obtain the corresponding first drainage facility defect detection result and the second drainage facility defect detection result, so as to obtain the drainage facility defect detection result with spatial information, so as to achieve the purpose of drainage facility defect target tracking and spatial matching integrated detection and analysis with high precision.
[0020] In the second aspect, the invention objective of the present application is achieved by adopting the following technical solutions: A surface drainage facility defect detection system based on dynamic monitoring video, the system comprising: A monitoring video and drainage facility information acquisition module is used to acquire dynamic monitoring video and ground drainage facility information of the target monitoring area, wherein the dynamic monitoring video is acquired based on a preset mobile monitoring device; A facility defect detection result acquisition module is used to perform facility defect detection and facility defect target tracking of the drainage facilities on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm to obtain the drainage facility defect detection result; a spatial information matching analysis module is used to obtain the spatial information of the drainage facility defect detection result, and perform spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to obtain corresponding facility defect target detection information.
[0021] By adopting the above technical scheme, the present application provides a high detection efficiency method for ground drainage facility defect detection based on dynamic monitoring video, and the ground drainage facility defect detection method is more automated and intelligent; specifically, after obtaining the dynamic monitoring video and ground drainage facility information of the target monitoring area, a preset facility defect target tracking algorithm is used to perform facility defect target detection of the ground drainage facility on the monitoring image of the dynamic monitoring video, and after monitoring the existence of facility defects in the ground drainage facility, facility defect target tracking is performed to obtain the drainage facility defect detection result, which is conducive to improving the detection accuracy of the facility defect target in the dynamic monitoring video; the next step is to obtain the spatial information of the drainage facility defect detection result, and based on the drainage facility defect detection The spatial information of the detection results and the ground drainage facility information are spatially matched and analyzed. The spatial matching analysis algorithm includes the integration of the monitoring image information of the dynamic monitoring video (the monitoring image at this time contains the drainage facility defect detection result) and the spatial information, and the integration of the facility defect target of the drainage facility defect detection result and the target tracking trajectory, so as to complete the integration of the drainage facility defect detection and the GIS data of the ground drainage facility information, that is, the facility defect target detection information with spatial information is obtained through integration. Therefore, the present application provides a method for intelligent detection and management of defects based on dynamic monitoring videos and ground drainage facilities, which is beneficial to improving the detection efficiency and detection accuracy of ground drainage facility defects, and is beneficial to saving the cost of manual detection of ground drainage facility defects.
[0022] In the third aspect, the invention objective of the present application is achieved by adopting the following technical solutions: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for detecting defects in ground drainage facilities based on dynamic monitoring videos are implemented.
[0023] In a fourth aspect, the invention objective of the present application is achieved by adopting the following technical solutions: A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for detecting defects in ground drainage facilities based on dynamic monitoring videos.
[0024] In summary, the present application includes at least one of the following beneficial technical effects: 1. The present application provides a high detection efficiency method for detecting defects in ground drainage facilities based on dynamic monitoring videos, and the defect detection method for ground drainage facilities is more automated and intelligent; specifically, after obtaining the dynamic monitoring video and ground drainage facility information of the target monitoring area, a preset facility defect target tracking algorithm is used to perform facility defect target detection of the ground drainage facilities on the monitoring image of the dynamic monitoring video, and after monitoring the existence of facility defects in the ground drainage facilities, facility defect target tracking is performed to obtain drainage facility defect detection results, which is conducive to improving the detection accuracy of facility defect targets in the dynamic monitoring video; the next step is to obtain the spatial information of the drainage facility defect detection results, and based on the drainage facility defect detection results The spatial information and the ground drainage facility information are spatially matched and analyzed. The spatial matching analysis algorithm includes the integration of the monitoring image of the dynamic monitoring video (the monitoring image at this time contains the drainage facility defect detection result) information and the spatial information, and the integration of the facility defect target of the drainage facility defect detection result and the target tracking trajectory, so as to complete the integration of the drainage facility defect detection and the GIS data of the ground drainage facility information, that is, the facility defect target detection information with spatial information is integrated and obtained. Thus, the present application provides a method for intelligent detection and management of defects based on dynamic monitoring videos and ground drainage facilities, which is conducive to improving the detection efficiency and detection accuracy of ground drainage facility defects, and is conducive to saving the manual detection cost of ground drainage facility defects; 2. Perform frame processing on the dynamic monitoring video to obtain multiple frames of continuous monitoring images to be determined based on time sequence, wherein the monitoring image is a ground road image obtained by shooting the drainage setting area on the road surface using a preset mobile monitoring device, so as to identify the ground drainage facilities from the monitoring image, and perform facility defect detection and facility defect target tracking on the ground drainage facilities in the monitoring image; then adopt a defect target detection algorithm to perform facility defect detection on multiple frames of monitoring images to be determined frame by frame to obtain an initial facility defect detection result including several defect detection targets; the present application adopts a preset detection-based defect target tracking model integrated with an attention module, and the attention module can improve the detection performance of drainage facility defect targets. When extracting and settling the defect feature information of the facility defect target in the monitoring image, the detection-based defect target tracking model integrated with the attention module can adaptively adjust the weights of different channels and spatial positions, thereby helping to improve the recognition ability and model accuracy of ground drainage facility defects; 3. Solve the multi-target tracking problem of multiple drainage facility defect targets (referring to defect detection targets) in multi-frame monitoring images. The target tracking framework based on drainage facility defect detection enables the detection-based defect target tracking model to efficiently track multiple defect detection targets when performing real-time detection tasks; specifically, through the pre-trained detection-based defect target tracking model, the defect target tracking algorithm is used to complete the defect target detection of multiple consecutive frames of monitoring images obtained from the video stream of the dynamic monitoring video, and generate a target detection frame matching the defect detection target, and obtain the target detection frame of the next frame by predicting and matching the motion state of the target detection frame of the monitoring image to be determined in the previous frame and the corresponding moving trajectory, which is conducive to efficiently and accurately matching the moving trajectory corresponding to the mobile monitoring video with the target detection frame, and by dividing several defect detection targets of the initial facility defect detection results into multiple detection result sets with different confidences, the ground drainage facility defect detection performance is analyzed by the confidence of the detection results, and the detection accuracy of the defect detection target of the target detection frame summarized in each monitoring image is obtained, which is conducive to meeting the speed and accuracy of intelligent detection of the ground drainage facility status. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 It is a flow chart of a method for detecting defects of ground drainage facilities based on dynamic monitoring video in one embodiment of the present application; Figure 2 This is a flow chart of step S2 in a method for detecting defects in ground drainage facilities based on dynamic monitoring video in one embodiment of the present application; Figure 3 It is a flowchart of step S203 in a method for detecting defects of ground drainage facilities based on dynamic monitoring video in one embodiment of the present application; Figure 4 It is a flowchart of step S2034 in a method for detecting defects of ground drainage facilities based on dynamic monitoring video in one embodiment of the present application; Figure 5 It is a flow chart before step S3 in a method for detecting defects of ground drainage facilities based on dynamic monitoring video in one embodiment of the present application; Figure 6 It is a schematic diagram of the coordinates of a vehicle-mounted monitoring image (in the figure, a vehicle-mounted picture) interpolated by spatial coordinates in a method for detecting defects of ground drainage facilities based on dynamic monitoring video in one embodiment of the present application; Figure 7 It is an integrated result diagram of drainage facility defect detection results and ground drainage facility GIS data in a ground drainage facility defect detection method based on dynamic monitoring video in one embodiment of the present application; Figure 8It is a result diagram of defect detection and target tracking trajectory integration in a method for detecting defects in ground drainage facilities based on dynamic monitoring video in one embodiment of the present application; Fig. 9 It is a schematic diagram of a device in an embodiment of the present application. DETAILED DESCRIPTION
[0026] The present application is further described in detail below in conjunction with the accompanying drawings.
[0027] In one embodiment, if Figure 1 As shown, the present application discloses a method for detecting defects of ground drainage facilities based on dynamic monitoring video, which specifically includes the following steps: S1: Acquire dynamic monitoring video and ground drainage facility information of the target monitoring area, where the dynamic monitoring video is collected based on a preset mobile monitoring device.
[0028] In this embodiment, the target monitoring area is the urban municipal drainage facility inspection area; the mobile monitoring device is a mobile intelligent collection terminal, such as a driving camera on a flood prevention patrol vehicle, a preset intelligent collection terminal, a handheld operation terminal, etc.; the ground drainage facility information includes drainage facility product information and facility installation location information.
[0029] Furthermore, the dynamic monitoring video can also be collected by a pre-set fixed monitoring device in the target monitoring area. The fixed monitoring device is a monitoring device pre-set on a road or building. When the fixed monitoring device is operating normally, it can not only monitor road security, but also provide monitoring video analysis data for the defect detection of ground drainage facilities in this application. S2: Based on a preset facility defect target tracking algorithm, the monitoring image of the dynamic monitoring video is subjected to facility defect detection and facility defect target tracking of the drainage facility to obtain a drainage facility defect detection result.
[0030] In this embodiment, the facility defect target tracking algorithm includes a defect target detection algorithm and a defect target tracking algorithm, wherein the defect target detection algorithm is a target detection algorithm of YOLOV7+SE that integrates the SE attention mechanism, which has both detection accuracy and detection speed in the ground drainage facility target detection task, and has better comprehensive detection performance; the defect target tracking algorithm is a BoT-SORT target tracking algorithm, which can more efficiently match the prediction frame (referring to the target detection frame of the next frame of the monitoring image based on the current processing frame); global motion compensation (Global Motion Compensation, GMC) is used to correct the image changes caused by the movement of the camera (i.e., the dynamic monitoring device); when calculating the distance between the moving trajectory and the detection frame, the Jonker-Volgenant algorithm is used as the matching algorithm, and the IoU distance and the cosine distance of the surface feature are integrated as the cost matrix, so as to more efficiently and accurately achieve the matching of the trajectory and the target detection frame.
[0031] Specifically, the dynamic monitoring video is first divided into frames to obtain multiple frames of continuous monitoring images. The defect target detection of continuous single-frame images of the video stream of the dynamic monitoring video is completed by the pre-trained YOLOV7+SE target detection algorithm to obtain the initial facility defect detection result. The initial facility defect detection result is then used as the input of the defect target tracking algorithm to efficiently solve the multi-target tracking problem of multiple facility defect targets.
[0032] S3: Acquire the spatial information of the drainage facility defect detection results, and perform spatial matching analysis based on the spatial information of the drainage facility defect detection results and the ground drainage facility information to obtain corresponding facility defect target detection information.
[0033] Specifically, after obtaining the spatial information of the drainage facility defect detection results, spatial matching analysis is performed based on the spatial information of the drainage facility defect detection results and the ground drainage facility information. The spatial matching analysis algorithm includes the integration of the monitoring image information of the dynamic monitoring video (the monitoring image at this time contains the drainage facility defect detection results) and the spatial information, and the integration of the facility defect targets of the drainage facility defect detection results and the target tracking trajectory to complete the integration of the GIS data of the drainage facility defect detection and the ground drainage facility information, that is, the facility defect target detection information with spatial information is integrated to provide a method for intelligent detection and management of defects based on dynamic monitoring videos and ground drainage facilities, which is conducive to improving the detection efficiency and detection accuracy of ground drainage facility defects.
[0034] In one embodiment, if Figure 2As shown, in step S2, the preset facility defect target tracking algorithm includes a defect target detection algorithm and a defect target tracking algorithm; based on the preset facility defect target tracking algorithm, the monitoring image of the dynamic monitoring video is subjected to facility defect detection and facility defect target tracking of the drainage facility, and the drainage facility defect detection result is obtained, including: S201: Perform frame processing on the dynamic monitoring video to obtain multiple frames of continuous monitoring images to be determined.
[0035] S202: In the preset detection-based defect target tracking model, facility defect detection is performed frame by frame on multiple frames of monitoring images to be judged based on the defect target detection algorithm to obtain initial facility defect detection results of the corresponding continuous multiple frames of monitoring images; the preset detection-based defect target tracking model is integrated with an attention module that is used to enable the detection-based defect target tracking model to adaptively adjust the network structure of the optimization model; the initial facility defect detection results include several defect detection targets.
[0036] In this embodiment, the attention module is an SE attention module, which stands for squeeze and excitation module. It is an attention module used to extract channel features in a neural network. The attention module can improve the detection performance of drainage facility defect targets, and extract and settle the defect feature information of facility defect targets in the monitoring image. The detection-based defect target tracking model that integrates the attention module can adaptively adjust the weights of different channels and spatial positions, thereby helping to improve the recognition ability and model accuracy of ground drainage facility defects.
[0037] Specifically, the detection-based defect target tracking model adopts the target detector of YOLOV7, which integrates the SE attention module based on the SE attention mechanism. The present application integrates the SE attention module in the backbone network (Backbone) of the detection-based defect target tracking model. The experimental results show that: in the feature extraction stage, the fusion of the attention module enables the detection-based defect target tracking model to adaptively adjust the weights of different channels and spatial positions, which helps to improve the detection-based defect target tracking model's ability to identify defects in ground drainage facilities. Compared with the method of integrating the attention module in the neck network (Neck) or the prediction network (Head), the present application integrates the SE attention module in the backbone network and uses a single lightweight SE attention module, which significantly improves the model detection accuracy while improving the model detection speed, and is the best solution for the drainage facility defect target detection task.
[0038] S203: In a preset detection-based defect target tracking model, defect target tracking analysis is performed on several defect detection targets based on a defect target tracking algorithm and initial facility defect detection results to obtain drainage facility defect detection results of a continuous video stream.
[0039] In this embodiment, defect target tracking analysis is performed on several defect detection targets in the initial facility defect detection results based on a defect target tracking algorithm to obtain a final drainage setting defect detection result.
[0040] In one embodiment, if Figure 3 As shown, in step S203, in the preset detection-based defect target tracking model, defect target tracking analysis is performed on several defect detection targets based on the defect target tracking algorithm and the initial facility defect detection results to obtain drainage facility defect detection results, including: S2031: Obtain a moving trajectory of a preset mobile monitoring device corresponding to a dynamic monitoring video; if a defect detection frame is obtained in multiple frames of monitoring images to be determined, use the defect detection frame as a matching basis for a defect detection frame of the next frame.
[0041] In this embodiment, the defect target tracking algorithm is the BoT-SORT algorithm, and the defect target image is a monitoring image containing drainage facility defect feature information. Specifically, in the BoT-SORT algorithm, the state vector of the Kalman filter is defined as: Among them, x, y represent the coordinates of the center of the detection box, w, h represent the width and height of the detection box, Therefore, the target detection frame of the next monitoring image can be predicted by the motion state of the existing trajectory, and the predicted state vector can be expressed as:
[0042] The prediction covariance matrix is: in, represents the predicted state vector of the kth frame, represents the updated state vector of frame k-1, A k is the state transfer matrix, representing the motion information of the trajectory; p′ k represents the predicted covariance matrix, representing the uncertainty of the prediction, Q k Represents the system noise matrix; by combining the prediction vector with the observation vector, the state vector can be updated, and the updated vector of the prediction frame of the kth frame can be expressed as: Where K represents the Kalman gain, K k represents the comparison between prediction uncertainty and observation uncertainty; H is the observation matrix, R is the observation noise covariance, and Z k Represents the k-frame observation value.
[0043] Update the covariance matrix to: P k =P′ k -KHP′ k , P k Will be used to compute the prediction covariance matrix for the next frame.
[0044] S2032: In the preset detection-based defect target tracking model, target detection frames of multiple frames of monitoring images to be judged are generated in sequence based on time sequence; based on the defect target tracking algorithm, the target detection frame of the monitoring image to be judged in the next frame is predicted and matched according to the target detection frame of the monitoring image to be judged in the previous frame and the motion state of the corresponding moving trajectory.
[0045] In this embodiment, the target detection frame of the monitoring image to be determined in the next frame is also called the prediction frame; in the ground drainage facility inspection task, the dynamic monitoring video often causes image deformation due to the shooting center of the vehicle-mounted mobile shooting camera; at this time, the global motion compensation (GMC) module in the BoT-SORT algorithm can correct the image changes caused by camera movement and improve the accuracy of target tracking; in the target tracking process, starting from the second frame, the GMC module extracts the key point information of the previous and next two frames and estimates the affine transformation matrix H, which is expressed as a matrix: Where a, b, c, d represent the parameters of rotation and scaling, and are the parameters of translation. The coordinates (x, y) of the pixel point can be transformed into the corrected new coordinates (x', y') through affine transformation, which is expressed as
[0046] S2033: Based on the defect target tracking algorithm, multiple target detection frames and a preset confidence threshold, a number of defect detection targets of the initial facility defect detection result are divided into a plurality of detection result sets with different confidence levels.
[0047] In this embodiment, the confidence threshold is an IOU distance threshold.
[0048] S2034: Perform movement trajectory matching and target detection frame matching on multiple groups of detection result sets to obtain drainage facility defect detection results.
[0049] In this embodiment, the defect target tracking algorithm needs to match the moving trajectory, the target detection frame and the prediction frame. The defect target tracking algorithm uses the Jonker-Volgenant linear matching algorithm for matching, and combines the IoU distance and the cosine distance of the surface features as the cost matrix to more accurately achieve the matching of the trajectory and the detection frame; the fusion of the IoU distance and the cosine distance of the surface features is as follows: in, represents the IoU distance between track i and detection box j, represents the surface feature cosine distance between track i and detection box j; θ emb and θ iou is a preset threshold used to remove pairs with large differences in motion trajectories and surface features. i,j Each element in the cost matrix of the final input linear matching algorithm.
[0050] The Jonker-Volgenant algorithm is used in BoT-SORT to solve the problem of matching trajectories and detection boxes; the goal of the linear matching algorithm is to find a permutation matrix with the minimum total cost based on the cost matrix, that is: Among them, c ij is the element of the cost matrix C, x ij are the elements of the permutation matrix X. The permutation matrix X satisfies the following conditions: In the permutation matrix, x ij =1 means the row and column match, x ij = 0 means no match. Only one element is selected in each row and column; the Jonker-Volgenant algorithm continuously optimizes the cost matrix through iterations of reduction and augmentation, and gradually improves the matching status by using augmenting path technology until the best matching matrix X that meets the above conditions is obtained.
[0051] Specifically, by dividing several defect detection targets of the initial facility defect detection results into multiple detection result sets with different confidence levels, the ground drainage facility defect detection performance is analyzed through the confidence levels of the detection results, and the detection accuracy of the defect detection targets in the target detection frame summarized in each monitoring image is obtained, which is beneficial to meeting the speed and accuracy of intelligent detection of the ground drainage facility status.
[0052] In one embodiment, if Figure 4 As shown, in step S2034, the detection result set includes a high-confidence detection set and a low-confidence detection set; the multiple groups of detection result sets are matched by movement trajectories and target detection frames to obtain drainage facility defect detection results, including: S20341: Extract trajectory information from dynamic monitoring video, build and initialize the trajectory pool.
[0053] In this embodiment, the drainage facility defect detection results are divided into a high-confidence detection set (D_High) and a low-confidence detection set (D_Low); the track pool (strack_pool) and the unactivated track unconfirmed are initialized, and the state of each target is predicted and the motion camera is compensated.
[0054] S20342: In the high-confidence detection set, the intersection-over-union (IoU) distance matrix and the appearance feature distance matrix between each trajectory in the initialized trajectory pool and several detected defect detection targets are calculated, and a comprehensive distance matrix is generated based on the IoU distance matrix and the appearance feature distance matrix, and a preset linear assignment algorithm is applied to determine the first matching result; the first matching result includes the defect target and trajectory pairs that are successfully matched, the trajectories that are not successfully matched for the first time, and the target detection boxes for which no matching trajectories are found for the first time.
[0055] In this embodiment, the matching of the detection result set with high confidence is performed preferentially. The intersection over union (IoU_dist) and the appearance feature distance matrix (Apperance_dist) between the initialization trajectory pool and the high confidence detection set are first calculated. Then, the minimum value between the intersection over union (IoU) distance matrix and the appearance feature distance matrix is taken as the comprehensive distance matrix (Dist_mat). The first matching pair (matched_pair0), the first unmatched moving trajectory (u_tracks0) and the first unmatched target detection box (u_dets0) are found through the linear allocation algorithm. The matched trajectory of the first matching pair (matched_pair0) is updated, and a new trajectory is created for the first unmatched target detection box (u_dets0).
[0056] S20343: Recalculate the intersection-over-union distance matrix based on the remaining unsuccessfully matched trajectories of the first matching results and the defect targets in the low-confidence detection set; and apply the preset linear assignment algorithm and the intersection-over-union distance matrix for secondary matching to generate secondary matching results; the secondary matching results include newly added matching pairs, trajectories that were not successfully matched for the second time, and target detection boxes for which no matching trajectories were found for the second time.
[0057] In this embodiment, after the secondary matching, a linear assignment algorithm is used to find secondary matching pairs, secondary unmatched movement trajectories, and secondary unmatched target detection frames, and the secondary matching pairs are updated to the trajectory pool.
[0058] Specifically, the second matching is used to match the remaining trajectories of the first matching with the drainage facility defect detection results in the low-confidence detection set (D_Low). First, the intersection-over-union distance matrix between the first matching pair (matched_pair0) and the low-confidence detection set (D_Low) is calculated, and the secondary matching pair (matched_pair1) and the secondary unmatched mobile trajectory (u_tracks1) are found through the linear allocation algorithm, and the secondary matching results are updated to the track pool (strack_pool).
[0059] S20344: Integrate the first matching result and the second matching result to form a final matching result set, and identify all final unmatched trajectories.
[0060] S20345: Update the trajectory pool according to the final matching result and all final unmatched trajectories, and remove the final unmatched trajectories and long-term lost trajectories.
[0061] In this embodiment, a long-term lost trajectory is a lost trajectory whose loss time exceeds a preset loss time threshold; the preset loss time threshold is the maximum loss time; and a moving trajectory that is still not successfully matched after the second match is marked as lost.
[0062] Specifically, the movement trajectory that has been lost for a long time is removed. For the movement trajectory that has been lost since the previous frame, if the loss time of the trajectory exceeds the set maximum loss time, the trajectory is removed.
[0063] In one embodiment, if Figure 5 and Figure 6 As shown, before step S3, a method for detecting defects of ground drainage facilities based on dynamic monitoring video includes: S301: Acquire the geographic coordinates of the monitoring device of the dynamic monitoring video, and perform plane coordinate projection transformation on the geographic coordinates to obtain plane rectangular coordinates.
[0064] Specifically, since the coordinate system used by the global navigation satellite system (GNSS) carried by the vehicle-mounted ground drainage facility field inspection is the WGS84 geographic coordinate system, before the drainage facility defect detection results are spatially matched with the ground drainage facility information, the WGS84 coordinates need to be converted into the same plane rectangular coordinate system as the ground drainage facility GIS data through the Gauss-Kruger projection; the coordinate conversion relationship can be described as: t=tanφ Δλ=λ-λ0 in, represents the input longitude and latitude, a is the length of the major axis of the ellipsoid, λ0 is the longitude of the central meridian, N represents the radius of curvature, t is the tangent value, Δλ is the longitude difference, which means the difference between the input longitude and the longitude of the central meridian, and f is the flattening.
[0065] Let k0 be the scale factor, which is set to 1.0 for this projection; η is the first eccentricity. The rectangular coordinates (x, y) of the projected plane are obtained by the following formula: S302: Based on plane rectangular coordinates, when the dynamic monitoring video is frame-processed to obtain the monitoring image, the monitoring image coordinates of each frame of the monitoring image are positioned by linear interpolation to obtain instantaneous image positioning coordinates.
[0066] In this embodiment, reference Figure 6 , Figure 6 The schematic diagram of the coordinates of the vehicle-mounted monitoring image (the vehicle-mounted picture in the figure) interpolated by spatial coordinates is a schematic diagram of the scattering of vehicle-mounted pictures and plane rectangular coordinates (GNSS coordinate receiving points); when the vehicle-mounted camera collects video images of ground drainage facilities, the vehicle-mounted camera and the vehicle-mounted satellite receiver receive signals at the same time, but because the two devices operate independently and have different receiving frequencies; therefore, when capturing the vehicle-mounted monitoring image, the coordinate data needs to be interpolated to obtain the instantaneous coordinates of the vehicle-mounted camera image (referring to the monitoring image coordinate positioning).
[0067] Specifically, the coordinates of the monitoring image are located by linear interpolation. Assuming that between time t1 and t2 (where t2>t1), the coordinates reported by the vehicle-mounted satellite receiver are (x1, x2), then the coordinates (x, y) of the image acquired at any time point t between t1 and t2 can be defined as:
[0068] S303: Determine the spatial information of each frame of monitoring image based on the plane rectangular coordinates and the instantaneous image positioning coordinates; and obtain the spatial information of the drainage facility defect detection result based on the spatial information of the monitoring image.
[0069] Specifically, the spatial information of the drainage facility defect detection result is to interpolate the instantaneous image positioning coordinates through the spatial coordinates, and the correlation relationship between the monitoring image and the space can be established, such as: For example, in order to effectively avoid the impact of program runtime on time accuracy, the interpolation program is implemented using asynchronous threads. The specific implementation steps are as follows: (1) The program monitors the information returned by the GNSS receiver, records the reception time and stores it in the folder coordinates, converts the WGS84 coordinates (geographic coordinates) into geodetic plane coordinates (plane rectangular coordinates) through the Gauss-Krüger projection; monitors the vehicle-mounted camera to capture monitoring image information, records the capture time and stores it in the folder images.
[0070] (2) The difference timer program scans the images folder at a frequency of 2 Hz and interpolates the image coordinates using linear interpolation.
[0071] (3) If the adjacent coordinate points are found and the interpolation is successful, add a row to the matrix S and remove the image from the images folder.
[0072] (4) If the monitoring of GNSS and the monitoring of the vehicle camera are completed and there are no images to be processed in the images folder, the difference timer is terminated.
[0073] S304: Identify and obtain several defect detection targets of multiple drainage facility defect detection results, merge the multiple drainage facility defect detection results identified as the same ground drainage facility by the facility defect target tracking algorithm, and obtain a merged drainage facility defect detection result.
[0074] In this embodiment, since the drainage facility defect detection results and the ground drainage facility GIS data have a many-to-many relationship, in order to simplify the calculation model and improve the calculation efficiency, this paper merges the drainage facility defect detection results identified as the same ground drainage facility by the target tracking algorithm before performing the spatial matching search.
[0075] Specifically, assuming that after merging, its average center point Perform spatial matching search. If there are points identified as the same ground drainage facility by the target tracking algorithm, their coordinates are (x1, y1), (x2, y2)......(x i ,y i ), then the coordinates of the mean center point C are: At this point, the process of finding the nearest neighbor can be simplified to: Where the set P = {P1, P2, ..., P n}, indicating the drainage facility point data being searched; each point P in the set i =(x i ,y i ), r is the maximum search radius.
[0076] In one embodiment, in step S3, spatial information of drainage facility defect detection results is obtained, and spatial matching analysis is performed based on the spatial information of the drainage facility defect detection results and the ground drainage facility information to obtain corresponding facility defect target detection information, including: S311: In a preset GIS database, the drainage facility GIS data of the ground drainage facility information is obtained, and the corresponding drainage facility identification is associated.
[0077] S312: Use the nearest neighbor matching algorithm with a limited radius to perform neighbor coordinate matching on the spatial information of the drainage facility defect detection results and the ground drainage facility GIS data, and establish a matching relationship between the drainage facility defect detection results corresponding to the monitoring image and the ground drainage facility GIS data.
[0078] Specifically, Figure 7 As shown, Figure 7It is an integrated result diagram of the drainage facility defect detection results and the ground drainage facility GIS data; the drainage facility GIS data corresponding to the ground drainage facility information is read from the GIS database, and the spatial information of the drainage facility defect detection results and the ground drainage facility GIS data are matched with the neighboring coordinates using the nearest neighbor matching algorithm with a limited radius to construct a matching association relationship (referring to the matching relationship) between the drainage facility defect detection results and the ground drainage facility GIS data, so as to facilitate the spatial matching of the drainage facility defect detection results with spatial information and the ground drainage facility GIS data.
[0079] S313: Based on the matching relationship, the defect detection target corresponding to the drainage facility defect detection result is obtained, and the corresponding facility defect target detection information is obtained.
[0080] S314: In a preset GIS database, the drainage facilities of the facility defect target detection information are marked with facility defects based on the drainage facility identification, so as to integrate the drainage facility defect detection results into the preset GIS database.
[0081] Specifically, based on the unique drainage facility identification of the drainage facility (where the drainage facility identification is a unique product information code of the drainage facility with installation location information), the defect detection target of the drainage facility detected in real time (referring to new facility defect information) is updated to the GIS database, so as to efficiently and real-time integrate the drainage facility defect detection results into the GIS database, facilitate the completion of ground drainage facility defect detection and drainage facility GIS integration, and greatly improve the detection efficiency of ground drainage facility defects.
[0082] In one embodiment, before step S3, the method for detecting defects of ground drainage facilities based on dynamic monitoring video further includes: The drainage facility defect detection results are divided into first drainage facility defect detection results of facility type and second drainage facility defect detection results of defect type, and the dividing step includes: S311: When performing facility defect target tracking on the monitoring image of the dynamic monitoring video based on the preset facility defect target tracking algorithm, a first drainage facility defect detection result having a target tracking identifier and a monitoring image frame identifier is generated, and the target tracking identifier is associated and matched with the monitoring image identifier one by one.
[0083] In this embodiment, the facility defect target tracking algorithm of the present application adopts a target tracking method that focuses on tracking three types of local facilities: rainwater inlets, inspection wells, and discharge outlets. By integrating the first drainage facility defect detection results of the facility type with the target tracking trajectory, and then matching the defect target with the corresponding facility type, the integration of the defect target and the tracking trajectory is completed.
[0084] S312: When performing facility defect detection on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm, a second drainage facility defect detection result having a defect type identifier and a monitoring image frame identifier is generated.
[0085] S313: Based on the monitoring image identification and the preset IOU threshold, the defect detection target of the second drainage facility defect detection result is associated with the defect type identification; based on the same target tracking identification, the defect detection results of different facility defect targets of multiple frames of monitoring images are obtained respectively.
[0086] Specifically, Figure 8 As shown, Figure 8 The result diagram of the integration of defect detection and target tracking trajectory; the implementation process of the facility defect target tracking algorithm is as follows: (1) Object Detection For each image frame of the video stream (referring to the monitoring image), defect target detection is performed based on the pre-trained YOLOV7+SE-based detection-based defect target tracking model, and a set of target detection results (this is the first drainage facility defect detection result) can be obtained, which is recorded as the matrix Where classid is the detection result classification id, x and y represent the center coordinates of the target detection box, w and h represent the width and height of the target detection box, and conf represents the confidence level.
[0087] (2) Classification of test results According to classid, the target detection results (the first drainage facility defect detection results) are divided into facility class and defect class. The facility set is S = {0, 1, 2}, and the defect set is F = {3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17}; for each target detection box in set D, if classid i ∈S, then d i is the facility class, denoted by matrix D 设施 , if classid i ∈F, then it is recorded as matrix D 缺陷 .
[0088] (3) Facility tracking The BoT-SORT target tracking algorithm is used to track the defect detection results of drainage facilities of the facility type, and the tracking results are obtained, which are recorded as matrix Among them, traceid i Indicates the target tracking identifier, fracmeid iTo monitor image identification, the monitoring image identification is a unique identification for each image, and traceid is implemented through facility tracking i With fracmeid i Tracking matches.
[0089] (4) Defect association After completing the facility target tracking of the video stream, it will be associated with the defect according to the IoU threshold. j ∈T, find all the links whose IoU values are greater than the confidence threshold and whose fracmeid i The same defect. The association process can be expressed as: Among them, IOU(d i , t j ) indicates defect d i and facilities j The intersection and union ratio of . The associated matrix R can be expressed as: As shown in R, since each tracking ID corresponds to multiple pictures, the detection results of each picture may correspond to multiple defects, that is, each facility has defect detection results from different pictures and multiple angles. To finally determine the defect type of the facility, it is necessary to integrate the defect detection results of multiple pictures. The average confidence θ2 of each type of each ground drainage facility can be defined as: Among them, n k Representation and facilities j The associated type is classid k The number of defects, defect_conf i Defect d i At this point, the confidence threshold can be used to eliminate the results with low average confidence R′={r j |C avg (r j )>θ2}.
[0090] and / or, S314: Obtain the defect target tracking trajectory corresponding to the first drainage facility defect detection result, and associate and match the defect detection results of different facility defect targets with the corresponding defect target tracking trajectory to obtain the drainage facility defect detection result with spatial information.
[0091] Specifically, by integrating the monitoring image information with the spatial information set, and the drainage facility defect detection results with the target tracking trajectory, the drainage facility defect detection results with spatial information are integrated and acquired.
[0092] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0093] In one embodiment, a ground drainage facility defect detection system based on dynamic monitoring video is provided, and the ground drainage facility defect detection system based on dynamic monitoring video corresponds to the ground drainage facility defect detection method based on dynamic monitoring video in the above-mentioned embodiment.
[0094] A ground drainage facility defect detection system based on dynamic monitoring video includes a monitoring video and drainage facility information acquisition module, a facility defect detection result acquisition module and a spatial information matching analysis module. The detailed description of each functional module is as follows: The monitoring video and drainage facility information acquisition module is used to obtain dynamic monitoring videos and ground drainage facility information of the target monitoring area. The dynamic monitoring video is collected based on a preset mobile monitoring device. A facility defect detection result acquisition module is used to perform facility defect detection and facility defect target tracking of drainage facilities on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm to obtain a drainage facility defect detection result; The spatial information matching and analysis module is used to obtain the spatial information of the drainage facility defect detection results, and perform spatial matching analysis based on the spatial information of the drainage facility defect detection results and the ground drainage facility information to obtain the corresponding facility defect target detection information.
[0095] For the specific limitations of the ground drainage facility defect detection system based on dynamic monitoring video, please refer to the limitations of a ground drainage facility defect detection method based on dynamic monitoring video in the above text, which will not be repeated here; each module in the above-mentioned ground drainage facility defect detection system based on dynamic monitoring video can be implemented in whole or in part through software, hardware and a combination thereof; each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.
[0096] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store movement trajectories, dynamic monitoring videos and drainage facility defect detection results, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a ground drainage facility defect detection method based on dynamic monitoring videos is implemented.
[0097] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for detecting defects of ground drainage facilities based on dynamic monitoring video are implemented.
[0098] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for detecting defects of ground drainage facilities based on dynamic monitoring videos are implemented.
[0099] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.
[0100] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0101] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the features thereof may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for detecting defects in ground drainage facilities based on dynamic monitoring video, characterized in that: include: Obtaining dynamic monitoring video and ground drainage facility information of the target monitoring area, wherein the dynamic monitoring video is collected based on a preset mobile monitoring device; Based on a preset facility defect target tracking algorithm, the monitoring image of the dynamic monitoring video is subjected to facility defect detection and facility defect target tracking of the drainage facility to obtain a drainage facility defect detection result; The spatial information of the drainage facility defect detection result is obtained, and spatial matching analysis is performed based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to obtain corresponding facility defect target detection information.
2. A method for detecting defects in ground drainage facilities based on dynamic monitoring video according to claim 1, characterized in that: The preset facility defect target tracking algorithm includes a defect target detection algorithm and a defect target tracking algorithm; the monitoring image of the dynamic monitoring video is subjected to facility defect detection and facility defect target tracking of the drainage facility based on the preset facility defect target tracking algorithm to obtain a drainage facility defect detection result, including: Performing frame processing on the dynamic monitoring video to obtain multiple frames of continuous monitoring images to be determined; In the preset defect target tracking model based on detection, facility defect detection is performed frame by frame on multiple frames of the monitoring images to be determined based on the defect target detection algorithm to obtain initial facility defect detection results of the corresponding continuous multiple frames of monitoring images; the preset defect target tracking model based on detection is integrated with an attention module for enabling the defect target tracking model based on detection to adaptively adjust the network structure of the optimization model; the initial facility defect detection results include a number of defect detection targets; In the preset detection-based defect target tracking model, defect target tracking analysis is performed on several defect detection targets based on the defect target tracking algorithm and the initial facility defect detection results to obtain drainage facility defect detection results of a continuous video stream.
3. A method for detecting defects in ground drainage facilities based on dynamic monitoring video according to claim 2, characterized in that: In the preset detection-based defect target tracking model, defect target tracking analysis is performed based on the defect target tracking algorithm and several defect detection targets of the initial facility defect detection result to obtain the drainage facility defect detection result, including: Obtaining a moving trajectory of a preset mobile monitoring device corresponding to the dynamic monitoring video; if a defect detection frame is obtained in multiple frames of the monitoring image to be determined, using the defect detection frame as a matching basis for the defect detection frame of the next frame; In the preset defect target tracking model based on detection, target detection frames of multiple frames of the monitoring image to be determined are generated in sequence based on the time sequence; based on the defect target tracking algorithm, the target detection frame of the monitoring image to be determined in the next frame is predicted and matched according to the target detection frame of the monitoring image to be determined in the previous frame and the motion state of the corresponding moving trajectory; Based on the defect target tracking algorithm, the plurality of target detection frames and a preset confidence threshold, the plurality of defect detection targets of the initial facility defect detection result are divided into a plurality of detection result sets with different confidence levels; Movement trajectory matching and target detection frame matching are performed on multiple groups of the detection result sets to obtain the drainage facility defect detection results.
4. A method for detecting defects in ground drainage facilities based on dynamic monitoring video according to claim 3, characterized in that: The detection result set includes a high-confidence detection set and a low-confidence detection set; performing movement trajectory matching and target detection frame matching on multiple groups of the detection result sets to obtain the drainage facility defect detection results includes: Extracting trajectory information from the dynamic monitoring video, constructing and initializing a trajectory pool; In the high-confidence detection set, the intersection-over-union distance matrix and the appearance feature distance matrix between each track in the initialization track pool and the detected defect detection targets are calculated, and a comprehensive distance matrix is generated based on the intersection-over-union distance matrix and the appearance feature distance matrix, and a preset linear allocation algorithm is applied to determine the first matching result; the first matching result includes the defect target and track pair that are successfully matched, the track that is not successfully matched for the first time, and the target detection frame that does not find the matching track for the first time; Recalculate the intersection-over-union distance matrix based on the remaining unsuccessfully matched trajectories of the first matching result and the defect targets in the low-confidence detection set; and apply the preset linear assignment algorithm and the intersection-over-union distance matrix to perform secondary matching to generate secondary matching results; the secondary matching results include newly added matching pairs, trajectories that were not successfully matched for the second time, and target detection boxes for which no matching trajectories were found for the second time; Integrate the first matching result and the second matching result to form a final matching result set, and identify all final unmatched tracks; According to the final matching results and all final unmatched trajectories, the trajectory pool is updated, and the final unmatched trajectories and long-term lost trajectories are removed.
5. The method for detecting defects of ground drainage facilities based on dynamic monitoring video according to claim 1 is characterized in that: The obtaining of the spatial information of the drainage facility defect detection result, and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, before obtaining the corresponding facility defect target detection information, includes: Obtaining the geographic coordinates of the monitoring device of the dynamic monitoring video, and performing plane coordinate projection transformation on the geographic coordinates to obtain plane rectangular coordinates; Based on the plane rectangular coordinates, when the dynamic monitoring video is frame-processed to obtain the monitoring image, the monitoring image coordinates of each frame of the monitoring image are positioned by linear interpolation to obtain the instantaneous image positioning coordinates; Based on the plane rectangular coordinates and the instantaneous image positioning coordinates, determine the spatial information of each frame of monitoring image; based on the spatial information of the monitoring image, obtain the spatial information of the drainage facility defect detection result; Identify and obtain a plurality of defect detection targets of the drainage facility defect detection results, merge the plurality of drainage facility defect detection results identified as the same ground drainage facility by the facility defect target tracking algorithm, and obtain a merged drainage facility defect detection result.
6. A method for detecting defects in ground drainage facilities based on dynamic monitoring video according to claim 5, characterized in that: The acquiring of the spatial information of the drainage facility defect detection result, and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to obtain corresponding facility defect target detection information includes: In a preset GIS database, the drainage facility GIS data of the surface drainage facility information is obtained, and the drainage facility identification is correspondingly associated; Use the nearest neighbor matching algorithm with a limited radius to perform neighbor coordinate matching on the spatial information of the drainage facility defect detection result and the ground drainage facility GIS data, and establish a matching relationship between the drainage facility defect detection result corresponding to the monitoring image and the ground drainage facility GIS data; Based on the matching relationship, a defect detection target corresponding to the drainage facility defect detection result is obtained, and facility defect target detection information is obtained accordingly; In a preset GIS database, the drainage facilities of the facility defect target detection information are marked as facility defects based on the drainage facility identification, so as to integrate the drainage facility defect detection results into the preset GIS database.
7. A method for detecting defects in ground drainage facilities based on dynamic monitoring video according to claim 5, characterized in that: The method further comprises: obtaining the spatial information of the drainage facility defect detection result, and performing spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information, before obtaining the corresponding facility defect target detection information: The drainage facility defect detection results are divided into first drainage facility defect detection results of facility type and second drainage facility defect detection results of defect type, the dividing step comprising: When performing facility defect target tracking on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm, a first drainage facility defect detection result having a target tracking identifier and a monitoring image frame identifier is generated, and the target tracking identifier is associated and matched with the monitoring image identifier one by one; When performing facility defect detection on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm, a second drainage facility defect detection result having a defect type identifier and a monitoring image frame identifier is generated; Based on the monitoring image identifier and the preset IOU threshold, the defect detection target of the second drainage facility defect detection result is associated with the defect type identifier; based on the same target tracking identifier, defect detection results of different facility defect targets of multiple frames of monitoring images are obtained respectively; Obtain a defect target tracking trajectory corresponding to the first drainage facility defect detection result, and associate and match the defect detection results of the different facility defect targets with the corresponding defect target tracking trajectory to obtain a drainage facility defect detection result with spatial information.
8. A surface drainage facility defect detection system based on dynamic monitoring video, characterized in that: The system comprises: A monitoring video and drainage facility information acquisition module is used to acquire dynamic monitoring video and ground drainage facility information of the target monitoring area, wherein the dynamic monitoring video is acquired based on a preset mobile monitoring device; A facility defect detection result acquisition module is used to perform facility defect detection and facility defect target tracking of the drainage facility on the monitoring image of the dynamic monitoring video based on a preset facility defect target tracking algorithm to obtain a drainage facility defect detection result; The spatial information matching and analysis module is used to obtain the spatial information of the drainage facility defect detection result, and perform spatial matching analysis based on the spatial information of the drainage facility defect detection result and the ground drainage facility information to obtain corresponding facility defect target detection information.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a method for detecting defects in ground drainage facilities based on dynamic monitoring video as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a method for detecting defects in ground drainage facilities based on dynamic monitoring video as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Method and system for managing drainage pipe detection data and assessment result data
CN105023118A
Multi-target tracking method, equipment and medium
CN114638855A
Method and device for detecting defects of urban hardware facilities
CN114782342A
Pipeline defect detecting and tracking method and device
CN116703826A
Pipeline defect identifying and positioning method based on target detection and weak magnetic detection
CN117420196A
Cited By
Elevated facility abnormity judgment method and system based on environmental feature fusion
CN121725428A
