Intelligent embankment patrol and risk inspection method and device based on mobile unmanned aerial vehicle group and image recognition technology

Through intelligent embankment inspection methods based on mobile drone clusters and image recognition technology, the problems of low dam patrol efficiency and high false alarm rate in the existing technology are solved, and a rapid and accurate detection of multiple dam safety hazards is achieved. , improves the efficiency and economic benefits of dam inspection.

CN118968364APending Publication Date: 2024-11-15顾斌杰 +1
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411144049.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing technology relies on human sea tactics in dam inspections, and there are problems such as low detection efficiency, high false alarm rate, and the inability to detect multiple dam safety hazards at the same time. And drone technology has not yet formed an intelligent inspection method.

Method used

Using intelligent dam patrol and hazard inspection methods based on mobile drone clusters and image recognition technology, six types of dam safety hazards are identified through neural network model training, and image information is collected by drone clusters, preliminary detection and secondary screening are performed, and detection is updated. The model, combined with the ssd-vgg300 model, extract features to realize the identification and positioning of dam safety hazards.

Benefits of technology

It has achieved rapid investigation of various hidden dangers or dangers in dams, improved patrol efficiency, timely reporting of types, coordinates and time of occurrence, significantly improved detection accuracy and efficiency, and has economic and social benefits.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968364A_ABST
    Figure CN118968364A_ABST
Patent Text Reader

Abstract

The invention discloses an intelligent embankment patrol and risk inspection method and device based on a mobile unmanned aerial vehicle group and an image recognition technology. According to the method, existing dam patrol image data is trained through a neural network model to obtain a detection model capable of identifying six types of dam potential safety hazards; image information of a dam detection area is collected by means of a mobile unmanned aerial vehicle carrying system and an unmanned aerial vehicle group; inputting the acquired image data into a detection model to obtain a preliminary dam potential safety hazard type, position and confidence coefficient; carrying out secondary screening on the detection results to obtain accurate position types, positions, confidence coefficients and time of the dam potential safety hazards; finally, dam potential safety hazard image data obtained through detection serve as existing data to be subjected to self-adaptive training, a detection model is updated, and the detection capacity is further improved. According to the invention, coordinates, types, confidence and time of various hidden dangers and risks of dam safety can be rapidly checked, and the coordinates, types, confidence and time are displayed on the terminal module and reported to managers in time, so that the patrol efficiency is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of dam hazard inspection, and in particular to an intelligent dam hazard inspection method based on a mobile drone swarm and image recognition technology, and dam hazard inspection equipment constructed based on the method. Background Art

[0002] my country has a total of 260,000 kilometers of seawalls, riverbanks, and embankments above designated size, more than 90,000 reservoirs and dams of various types, and more than 5 million ponds and dams of various types, most of which are earthen embankments and earthen dams. The collapse, landslide, cracks, overtopping, water seepage, and pipe bursts of embankments threaten the safety of embankments. At present, the main reliance is on human-sea tactics to patrol and inspect embankments to find safety hazards and dangers of embankments. Not only is the investment huge, but the patrol and inspection personnel have to endure the scorching sun and mosquito bites, and the working conditions are very difficult. There is an urgent need for technological innovation to gradually replace the traditional human-sea tactics patrol and inspection methods with new technologies and new equipment.

[0003] On the basis of patrolling the dikes and checking for dangers with human sea tactics, in recent years, some units have introduced drone technology to deal with dike dangers and have made some progress, but overall it is still in the experimental and exploratory stage, and there are the following problems:

[0004] First, they mainly rely on the camera mounted on drones to observe the hidden dangers of dams from a distance or use the infrared camera mounted on drones to take images, and then manually check the image data to find seepage pipe bursts. There is no intelligent technical method for checking the hidden dangers of dam safety.

[0005] Second, it mainly detects a single type of safety hazard, namely seepage and piping in dams, and has a low detection rate and a high false alarm rate. A technical method with high accuracy and the ability to detect multiple safety hazards of dams at the same time has not yet been formed;

[0006] Third, the main detection methods used for dam safety hazards are "one person one machine" or "multiple people one machine", which requires large investment and low efficiency. A technical method with technical and economic advantages that can gradually replace traditional dam patrol and hazard inspection has not yet been formed. Summary of the invention

[0007] To this end, the present invention provides an intelligent embankment inspection method based on a mobile drone swarm and image recognition technology to solve the above problems in the prior art. In order to achieve the above purpose, the present invention provides the following technical solutions: According to a first aspect of the present invention, an intelligent embankment inspection method based on a mobile drone swarm and image recognition technology comprises the following steps:

[0008] S1: Using the existing embankment patrol image data to train a neural network model, a detection model capable of identifying six types of embankment safety hazards is obtained;

[0009] S2: Collect image information of the embankment inspection area with the help of a mobile drone-mounted system and drone swarm;

[0010] S3: Input the image data in step S2 into the detection model in step S1 to obtain the preliminary dam safety hazard type, location, and confidence level;

[0011] S4: Perform secondary screening on the detection results obtained in S3 to provide the accurate type, location, confidence level and time of the dam safety hazard;

[0012] S5: Use the detection image data obtained in step S4 to input the existing data in S1 for adaptive training and update the detection model.

[0013] Furthermore, the ssd-vgg300 model is used to extract, identify and locate the safety hazards and dangerous conditions of six types of dams. Step S1 specifically includes:

[0014] S11: Classify the image data of existing dam safety hazards into six categories: cracks, landslides, collapses, overtopping, seepage, and piping;

[0015] S12: Establish a classification database of dam safety hazards and annotate it; specifically, use the LABELIMG tool to annotate, and then further divide the six types of dam safety hazard data into training set, validation set, and test set in a ratio of 8:1:1, and convert the data set images into a size of 300*300;

[0016] S13: preprocessing the data strips of the classification database of dam safety hazards; specifically, adding six types of dam safety hazard images by clipping, rotating, scaling, translating, and scaling the training data set images, and normalizing the images;

[0017] S14: Extract the features of six types of safety hazards of safety dams, perform non-maximum suppression, and complete the training of the detection model. Specifically, the preprocessed six types of training data sets are input into the SSD-VGG300 convolutional network for deep learning, and the bright boxes of small targets are selected. After multiple convolutions, pooling, and calculation of the cross entropy of the loss function, non-maximum suppression is performed, and finally the detection results are output.

[0018] Furthermore, in step S2, the mobile UAV carrying system is a mobile airport installed on a car, and the UAV group is more than two UAV platforms, each UAV is equipped with a flight control module, a video data acquisition and sending module; the mobile airport is equipped with a detection module for six types of dam safety hazards, a dam safety hazard transmission module, a UAV platform collaborative flight planning and remote control module, a UAV charging module, and a UAV helipad; perpendicular to the axis of the dam, two UAV platforms are added for every additional 30-meter detection area, and the system can be expanded according to the size of the detection task.

[0019] Furthermore, the UAV platform collaborative flight planning and remote control module is composed of several UAV remote control devices, a UAV flight grid map, a UAV flight planning and control module, and a computer; the UAV platform collaborative flight planning and remote control module mainly completes the UAV flight altitude, flight route, detection area demarcation and execution tasks, among which:

[0020] Flight altitude: related to detection efficiency and detection accuracy. The appropriate flight altitude determined by this solution is 30 meters;

[0021] Delineation of detection area and detection grid: According to the needs of dam management, the dam safety hazard detection area is delineated on the drone flight grid map, and then the detection area is divided into several detection grids. The size of the detection grid is determined according to the detection accuracy requirements and the drone flight capabilities; the area of ​​a single detection grid is controlled within 0.15-0.3 square kilometers, and the shape of the detection grid is a rectangle, with the long side of the rectangle along the center line of the dam and the short side perpendicular to the center line of the dam. The length of the long side is controlled within 5,000 meters, and the length of the short side should be within 30 meters. The sizes of each detection grid are basically the same, and the detection task within a detection grid is completed independently by a drone flight; the number of drones required depends on the width of the short side of the detection area, and 2 drones are added for every 30 meters increase;

[0022] Flight route: Each area within a detection grid is photographed more than twice in the same flight, the interval between the two shots should be between 1 minute and 60 minutes, and the flight interval between drones should be more than 15 meters;

[0023] Execute the task: Enter the map software of the computer drone flight grid, place the digital drone platform into the flight grid on the computer screen in order and start the flight program. The drone platform will carry out the inspection and patrol mission of the dam according to the flight rules and control measures determined by the drone flight planning and flight control module, and automatically return to the drone landing pad after completing the inspection mission.

[0024] Furthermore, the image information acquisition in step S2 includes the conversion of the image data stream captured during flight and the input image:

[0025] Use the visible light camera carried by the drone to obtain visible light image data of the dam inspection area;

[0026] The visible light image data is divided into windows with 20% overlap, and a sliding window of 300*300 is used with a sliding step of 240 pixels to convert the visible light camera data into a 300*300 visible light picture; finally, for windows with a height and width less than 300 pixels, the window size is kept at 300*300 pixels by moving the upper left corner coordinates upward and leftward;

[0027] Record the coordinates (Xi, Yi) of the upper left corner of each visible light photo and the shooting time;

[0028] Input the above visible light photos into the dam hidden danger identification module.

[0029] Furthermore, the secondary screening in step S4 includes the following specific steps:

[0030] First, determine the noise threshold: determine the threshold based on the project scale, quality and importance, and the method of "online discovery, offline verification, and statistical determination";

[0031] The calculation formula for the area of ​​the detection box of each type of normalized image detection result is: |(Xa-Xb)*(Ya-Yb)|, where (Xa, Ya) and (Xb, Yb) are the coordinate values ​​of the upper left corner and the lower right corner of the detection box of the detection result, respectively; if the area is less than or equal to the threshold, the detection result is treated as noise and the detection result is not output; if the area is greater than the threshold, the latest detection result of all detection results greater than the threshold is calculated with the last detection result for IOU; when IOU is greater than 0.9, the latest detection result is regarded as an old hidden danger discovered previously, and the latest detection result is no longer output; when IOU is less than or equal to 0.9, the latest detection result is not regarded as an old hidden danger discovered previously and the detection results are output in sequence.

[0032] Furthermore, the method for determining the real position and shooting time of each patrol image in step S4 is as follows:

[0033] After the visible light image is input into the dam hidden danger identification module, if there is a dam hidden danger, the location information with the prediction box will be output; assuming that the prediction box is at the position coordinates (Xa / Ya, Xb / Yb) in the photo, the actual length of the corresponding area of ​​the photo is R*R, and the coordinates of the upper left corner of the photo are (Xi, Yi), then the coordinates of the upper left corner of the actual position of the prediction box are (Xi+Xa*R), (Yi+Ya*R), and the coordinates of the lower right corner are (Xi+Xb*R), (Yi+Yb*R);

[0034] Prediction box time: Displays the shooting time applied directly on the image.

[0035] According to the second aspect of the present invention, an intelligent dike patrol and hazard inspection device based on a mobile drone swarm and image recognition technology is a device solidified by applying any one of the intelligent dike patrol and hazard inspection methods based on a mobile drone swarm and image recognition technology according to the first aspect of the present invention.

[0036] The present invention has the following advantages: through the intelligent embankment patrol and hazard inspection method based on a mobile drone swarm and image recognition technology of the present invention, when the image information taken by the drone patrol flight is input into the intelligent system, various embankment hidden dangers or dangerous situations can be quickly checked, and the coordinates, type, confidence level and time of occurrence can be reported in time, and displayed on the terminal module and reported to the management personnel in time; the patrol efficiency is greatly improved. The invention fills the gap in this field and has significant economic and social benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A conceptual framework diagram of an intelligent dike patrol and hazard inspection method based on a mobile drone swarm and image recognition technology provided for some embodiments of the present invention.

[0038] Figure 2 A technical flow chart of an intelligent dike patrol and hazard inspection method based on a mobile drone swarm and image recognition technology is provided for some embodiments of the present invention.

[0039] Figure 3 A schematic structural diagram of an intelligent dike patrol and hazard inspection device based on a mobile drone swarm and image recognition technology provided in some embodiments of the present invention.

[0040] Figure 4 A working principle diagram of safety hazard detection in a mid-levee dam according to an intelligent levee inspection method based on a mobile drone swarm and image recognition technology provided in some embodiments of the present invention. DETAILED DESCRIPTION

[0041] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the technology can clearly understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the examples in the present invention, all other examples obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Example 1

[0043] like Figures 1 to 4 As shown, in the first aspect of the present invention, an intelligent embankment inspection method based on a mobile drone group and image recognition technology is provided, and the method comprises the following steps:

[0044] S1: Using the existing embankment patrol image data to train a neural network model, a detection model capable of identifying six types of embankment safety hazards is obtained;

[0045] S2: Collect image information of the embankment inspection area with the help of a mobile drone-mounted system and drone swarm;

[0046] S3: Input the image data in step S2 into the detection model in step S1 to obtain the preliminary dam safety hazard type, location, and confidence level;

[0047] S4: Perform secondary screening on the detection results obtained in S3 to provide the accurate type, location, confidence level and time of the dam safety hazard;

[0048] S5: Use the detection image data obtained in step S4 to input the existing data in S1 for adaptive training and update the detection model.

[0049] The technical effect achieved by the above embodiment is: through an intelligent embankment inspection method based on a mobile drone swarm and image recognition technology in this embodiment, when the image information taken by the drone patrol flight is input into the intelligent system, six types of embankment hidden dangers or dangerous situations can be quickly checked, and the type, coordinates, confidence level and time of occurrence can be reported in a timely manner. One person and one car (drone swarm and mobile airport) can inspect 20-240 kilometers of embankments every day according to different detection accuracy requirements, greatly improving inspection efficiency. This invention fills the gap in this field and has significant economic and social benefits.

[0050] Example 2

[0051] like Figures 1 to 4 As shown, an intelligent embankment inspection method based on a mobile drone swarm and image recognition technology includes all the contents of Example 1. In addition, the SSD-VGG300 model is used to complete the extraction, identification and positioning of six types of embankment safety hazards and dangerous conditions. Step S1 specifically includes:

[0052] S11: Classify the image data of existing dam safety hazards into six categories: cracks, landslides, collapses, overtopping, seepage, and piping;

[0053] S12: Establish a classification database of dam safety hazards and annotate it; specifically, use the LABELIMG tool to annotate, and then further divide the six types of dam safety hazard data into training set, validation set, and test set in a ratio of 8:1:1, and convert the data set images into a size of 300*300;

[0054] S13: Preprocessing six types of data in the dam safety hazard database; specifically, increasing the number of images of six types of dam safety hazard images, namely, cracks, landslides, collapses, overtopping, seepage, and piping, from 200 to 1,000 by clipping, rotating, scaling, translating, and scaling the images in the training data set, normalizing the images, and pre-training the model;

[0055] S14: Extract the features of six types of safety hazards of safety dams, perform non-maximum suppression, complete the training of the detection model, and conduct verification. Specifically, the six types of pre-processed training data sets are input into the SSD-VGG300 convolutional network for deep learning, and the bright-colored frames of small targets are selected for focus. After multiple convolutions, pooling, and calculation of loss functions, non-maximum suppression is performed, and finally the detection results are output; after 5,000 trainings, the six types of hidden dangers and dangerous situations feature extraction and target recognition are completed, and a relatively ideal detection model and detection effect are obtained through verification.

[0056] Example 3

[0057] like Figures 1 to 4 As shown, an intelligent dike patrol and hazard inspection method based on mobile drone swarms and image recognition technology includes all the contents of Example 2. In addition, in step S2, the mobile drone carrying system is a mobile airport installed on a car, the drone swarm is more than two drone platforms, and each drone is equipped with a flight control module and a video data acquisition and transmission module; the mobile airport is equipped with a detection module for six types of dike safety hazards, a dam safety hazard transmission module, a drone platform collaborative flight planning and remote control module, a drone charging module, and a drone apron; perpendicular to the axis of the dike, two drone platforms are added for every 30-meter detection area, and the system can be expanded according to the size of the detection task.

[0058] Optionally, the UAV platform collaborative flight planning and remote control module is composed of several UAV remote control devices, a UAV flight grid map, a UAV flight planning and control module, and a computer; the UAV platform collaborative flight planning and remote control module mainly completes the UAV flight altitude, flight route, detection area demarcation and execution tasks, among which:

[0059] Flight altitude: related to detection efficiency and detection accuracy. In order to obtain enough pixels, extract the characteristics of six types of dam safety hazards, and achieve the set detection efficiency, the appropriate flight altitude determined by this solution is 30 meters;

[0060] Delineation of detection area and detection grid: According to the needs of dam management, the dam safety hazard detection area is delineated on the drone flight grid map, and then the detection area is divided into several detection grids. The size of the detection grid is determined according to the detection accuracy requirements and the drone flight capabilities; the area of ​​a single detection grid is controlled within 0.15-0.3 square kilometers, and the shape of the detection grid is a rectangle, with the long side of the rectangle along the center line of the dam and the short side perpendicular to the center line of the dam. The length of the long side is controlled within 5,000 meters, and the length of the short side should be within 30 meters. The sizes of each detection grid are basically the same, and the detection task within a detection grid is completed independently by a drone flight; the number of drones required depends on the width of the short side of the detection area, and 2 drones are added for every 30 meters increase;

[0061] Flight route: Each area within a detection grid is photographed more than twice in the same flight, the interval between the two shots should be between 1 minute and 60 minutes, and the flight interval between drones should be more than 15 meters;

[0062] Execute the task: Enter the map software of the computer drone flight grid, place the digital drone platform into the flight grid on the computer screen in order and start the flight program. The drone platform will carry out the inspection and patrol mission of the dam according to the flight rules and control measures determined by the drone flight planning and flight control module, and automatically return to the drone landing pad after completing the inspection mission.

[0063] The airport's movement method: The first 5 kilometers from the origin along the axis of the dam in the detection area is used as the starting parking point, and a parking point is set up every 10 kilometers along the axis of the dam.

[0064] If the width of the inspection area is 90 meters, 6 drones can be equipped for inspection. The embankment inspection is calculated as one inspection every 2 hours. One person and one vehicle (mobile airport and supporting drone platform) can inspect 20km of embankment.

[0065] If the width of the detection area is 30 meters, two drone platforms can be deployed for inspection. The embankment inspection is calculated as one inspection every 24 hours. One person and one vehicle (mobile airport and supporting drone platform) can inspect 240km of embankment.

[0066] Optionally, the image information acquisition in step S2 includes the process of image data flow captured during flight and the conversion of input images: The process of image data flow:

[0067] The image information streams collected by each UAV platform are transmitted and detected through their own independent channels. After obtaining preliminary results, they are integrated during the secondary screening in step S4 to form a unified detection result and then output.

[0068] Image information flow and conversion of input images:

[0069] Use the visible light camera carried by the drone to obtain visible light image data of the dam inspection area;

[0070] The visible light image data is divided into windows with 20% overlap, and a sliding window of 300*300 is used with a sliding step of 240 pixels to convert the visible light camera data into a 300*300 visible light picture; finally, for windows with a height and width less than 300 pixels, the window size is kept at 300*300 pixels by moving the upper left corner coordinates upward and leftward;

[0071] Record the coordinates (Xi, Yi) of the upper left corner of each visible light photo and the shooting time;

[0072] Input the above visible light photos into the dam hidden danger identification module.

[0073] Example 4

[0074] like Figures 1 to 4 As shown, an intelligent embankment inspection and risk detection method based on a mobile drone swarm and image recognition technology includes all the contents of Example 3. In addition, in the actual detection of embankment safety hazards, many noises will be encountered that seriously affect the detection effect. These noises can be divided into two categories. One is the characteristics of embankment safety hazards, which are not caused by embankment safety problems and do not affect embankment safety. For example, water ripples caused by water birds and fish activities, cracks on the embankment surface, and surface water accumulation caused by non-embankment leakage. The second is the characteristics of embankment safety hazards, which may not develop into embankment safety hazards and dangerous situations. For example, small-scale collapse, small-scale landslide, etc. These noises account for a large proportion of the embankment detection results. It will take a lot of manpower to verify these detection results one by one offline, so it is necessary to perform noise removal.

[0075] In addition, referring to the regulations on manual inspection of dikes, if the drone group inspects the same dike once every 2 hours, a large number of identical inspection results may be generated. Especially in the critical period of flood control, the areas where dike dangers occur are relatively concentrated, and the newly discovered hidden dangers are intertwined with the previously discovered hidden dangers, which will increase the difficulty of managing the inspection results. It is necessary to compare the new and old inspection results to highlight the newly discovered hidden dangers and dangers. Based on the above factors, it is necessary to conduct a secondary screening.

[0076] The secondary screening in step S4 includes the following specific steps:

[0077] First, determine the noise threshold: determine the threshold based on the scale, quality and importance of the project and the method of "online discovery, offline verification, and statistical determination"; for example, the seepage area is 5 square meters, the collapse area is 2 square meters, the crack area is 1.5 square meters, the pipe burst (water ripple) area is 0.8 square meters, etc., which can be used as the noise threshold (the area of ​​the detection result detection box). If the area corresponding to each detection image is 900 square meters, the above thresholds are converted into the thresholds of the normalized image, which are 0.0055, 0.0022, 0.00166, and 0.00088 respectively.

[0078] The calculation formula for the area of ​​the detection box of each type of normalized image detection result is: |(Xa-Xb)*(Ya-Yb)|, where (Xa, Ya) and (Xb, Yb) are the coordinate values ​​of the upper left corner and the lower right corner of the detection box of the detection result, respectively; if the area is less than or equal to the threshold, the detection result is treated as noise and the detection result is not output; if the area is greater than the threshold, the latest detection result of all detection results greater than the threshold is calculated with the last detection result for IOU; when IOU is greater than 0.9, the latest detection result is regarded as an old hidden danger discovered previously, and the latest detection result is no longer output; when IOU is less than or equal to 0.9, the latest detection result is not regarded as an old hidden danger discovered previously and the detection results are output in sequence.

[0079] Example 5

[0080] like Figures 1 to 4 As shown, an intelligent dike patrol and inspection method based on a mobile drone group and image recognition technology includes all the contents of Example 4. In addition, the method for determining the real position and shooting time of each dike patrol image in step S4 is:

[0081] After the visible light image is input into the dam hidden danger identification module, if there is a dam hidden danger, the location information with the prediction box will be output; assuming that the prediction box is at the position coordinates (Xa / Ya, Xb / Yb) in the photo, the actual length of the corresponding area of ​​the photo is R*R, and the coordinates of the upper left corner of the photo are (Xi, Yi), then the coordinates of the upper left corner of the actual position of the prediction box are (Xi+Xa*R), (Yi+Ya*R), and the coordinates of the lower right corner are (Xi+Xb*R), (Yi+Yb*R);

[0082] Prediction box time: Displays the shooting time applied directly on the image.

[0083] Output of detection results: The location coordinates, type, confidence level and time of each dam safety hazard will be displayed and sent to relevant personnel.

[0084] Example 6

[0085] An intelligent dike patrol and hazard inspection device based on a mobile drone swarm and image recognition technology, applying all the contents of any one of embodiments 1 to 5.

[0086] In the description of the present invention, it is to be understood that the terms “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0087] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0088] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0089] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0090] In the description of this specification, the description with reference to the terms "embodiment one", "embodiment two", "example", etc. means that the specific method, device or feature described in conjunction with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, methods, devices or features described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0091] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An intelligent embankment inspection method based on mobile drone swarms and image recognition technology, characterized in that: The method comprises the following steps: S1: Using the existing embankment patrol image data to train a neural network model, a detection model capable of identifying six types of embankment safety hazards is obtained; S2: Collect image information of the embankment inspection area with the help of a mobile drone-mounted system and drone swarm; S3: Input the image data in step S2 into the detection model in step S1 to obtain the preliminary dam safety hazard type, location, and confidence level; S4: Perform secondary screening on the detection results obtained in S3 to provide the accurate type, location, confidence level and time of the dam safety hazard; S5: Use the detection image data obtained in step S4 to input into S1 as existing data for adaptive training and update the detection model.

2. The intelligent embankment inspection method based on mobile drone swarm and image recognition technology according to claim 1 is characterized in that: The ssd-vgg300 model is used to extract, identify and locate the safety hazards and dangerous conditions of six types of dams. Step S1 specifically includes: S11: Classify the image data of existing dam safety hazards into six categories: cracks, landslides, collapses, overtopping, seepage, and piping; S12: Establish a classification database of dam safety hazards and annotate it; specifically, use the LABELIMG tool to annotate, and then further divide the six types of dam safety hazard data into training set, validation set, and test set in a ratio of 8:1:1, and convert the data set images into a size of 300*300; S13: preprocessing the data strips of the classification database of dam safety hazards; specifically, adding six types of dam safety hazard images by clipping, rotating, scaling, translating, and scaling the training data set images, and normalizing the images; S14: Extract the features of six types of safety hazards of safety dams, perform non-maximum suppression, and complete the training of the detection model. Specifically, the preprocessed six types of training data sets are input into the SSD-VGG300 convolutional network for deep learning, and the bright boxes of small targets are selected. After multiple convolutions, pooling, and calculation of the cross entropy of the loss function, non-maximum suppression is performed, and finally the detection results are output.

3. The intelligent embankment inspection method based on mobile drone swarm and image recognition technology according to claim 1 is characterized in that: In step S2, the mobile drone carrying system is a mobile airport installed on a car. The drone group consists of more than two drone platforms, and each drone is equipped with a flight control module, a video data acquisition and transmission module; the mobile airport is equipped with a detection module for six types of dam safety hazards, a dam safety hazard transmission module, a drone platform collaborative flight planning and remote control module, a drone charging module, and a drone apron; perpendicular to the axis of the dam, two drone platforms are added for every 30-meter detection area, and the system can be expanded according to the size of the detection task.

4. The intelligent embankment inspection method based on mobile drone swarm and image recognition technology according to claim 3 is characterized in that: The UAV platform collaborative flight planning and remote control module consists of several UAV remote control devices, UAV flight grid map, UAV flight planning and control module and computer; the UAV platform collaborative flight planning and remote control module mainly completes the UAV flight altitude, flight route, detection area demarcation and execution tasks, including: Flight altitude: related to detection efficiency and detection accuracy. The appropriate flight altitude determined by this solution is 30 meters; Delineation of detection area and detection grid: According to the needs of dam management, the dam safety hazard detection area is delineated on the drone flight grid map, and then the detection area is divided into several detection grids. The size of the detection grid area is determined according to the detection accuracy requirements and the drone flight capabilities; the area of ​​a single detection grid is controlled within 0.15-0.3 square kilometers, and the shape of the detection grid is a rectangle, with the long side of the rectangle along the center line of the dam and the short side perpendicular to the center line of the dam. The length of the long side is controlled within 5,000 meters, and the length of the short side should be within 30 meters. The sizes of each detection grid are basically the same, and the detection task within a detection grid is completed independently by a drone flight; the number of drones required depends on the width of the short side of the detection area, and 2 drones are added for every 30 meters increase; Flight route: Each area within a detection grid is photographed more than twice in the same flight, the interval between the two shots should be between 1 minute and 60 minutes, and the flight interval between drones should be more than 15 meters; Execute the task: Enter the map software of the computer drone flight grid, place the digital drone platform into the flight grid on the computer screen in order and start the flight program. The drone platform will carry out the inspection and patrol mission of the dam according to the flight rules and control measures determined by the drone flight planning and flight control module, and automatically return to the drone landing pad after completing the inspection mission.

5. The intelligent embankment inspection method based on mobile drone swarm and image recognition technology according to claim 1 is characterized in that: The image information acquisition in step S2 includes the conversion of the image data stream taken during flight and the input image: Use the visible light camera carried by the drone to obtain visible light image data of the dam inspection area; The visible light image data is divided into windows with 20% overlap, and a sliding window of 300*300 is used with a sliding step of 240 pixels to convert the visible light camera data into a 300*300 visible light picture; finally, for windows with a height and width less than 300 pixels, the window size is kept at 300*300 pixels by moving the upper left corner coordinates upward and leftward; Record the coordinates (Xi, Yi) of the upper left corner of each visible light photo and the shooting time; Input the above visible light photos into the dam hidden danger identification module.

6. The intelligent embankment inspection method based on mobile drone swarm and image recognition technology according to claim 1 is characterized in that: The secondary screening in step S4 includes the following specific steps: First, determine the noise threshold: determine the threshold based on the project scale, quality and importance and the method of "online discovery, offline verification, and statistical determination"; The calculation formula for the area of ​​the detection box of each type of normalized image detection result is: |(Xa-Xb)*(Ya-Yb)|, where (Xa, Ya) and (Xb, Yb) are the coordinate values ​​of the upper left corner and the lower right corner of the detection box of the detection result, respectively; if the area is less than or equal to the threshold, the detection result is treated as noise and the detection result is not output; if the area is greater than the threshold, the latest detection result of all detection results greater than the threshold is calculated with the last detection result for IOU; when IOU is greater than 0.9, the latest detection result is regarded as an old hidden danger discovered previously, and the latest detection result is no longer output; when IOU is less than or equal to 0.9, the latest detection result is not regarded as an old hidden danger discovered previously and the detection results are output in sequence.

7. The intelligent embankment inspection method based on mobile drone swarm and image recognition technology according to claim 1 is characterized in that: The method for determining the actual position and shooting time of each patrol image in step S4 is: After the visible light image is input into the dam hidden danger identification module, if there is a dam hidden danger, the location information with the prediction box will be output; assuming that the prediction box is at the position coordinates (Xa / Ya, Xb / Yb) in the photo, the actual length of the corresponding area of ​​the photo is R*R, and the coordinates of the upper left corner of the photo are (Xi, Yi), then the coordinates of the upper left corner of the actual position of the prediction box are (Xi+Xa*R), (Yi+Ya*R), and the coordinates of the lower right corner are (Xi+Xb*R), (Yi+Yb*R); Prediction box time: Displays the shooting time applied directly on the image.

8. An intelligent embankment inspection and danger inspection device based on a mobile drone swarm and image recognition technology, characterized in that: An intelligent dike patrol and hazard inspection method based on a mobile drone swarm and image recognition technology, applying any one of claims 1 to 7.

Citation Information

Cited By

  • Quality detection method and system for water conservancy construction engineering

    CN119228804A

  • Piping dangerous case troubleshooting method and system based on image recognition

    CN121998431A

  • A pipe gushing danger investigation method and system based on image recognition

    CN121998431B