A method and system for supervising rivers and lakes based on a UAV

Abstract

The present application relates to a kind of river and lake supervision method and system based on unmanned aerial vehicle, comprising: the multiple frames target image photographed by unmanned aerial vehicle are respectively input into trained water conservancy target identification model, obtain the water conservancy target marked with detection frame and the position information of water conservancy target in target image;Using trained water conservancy target tracking network model obtains the state of water conservancy target in detection frame image;Using the reference water conservancy target quantity of line counting method according to the above state is calculated;Using spatial positioning model obtains the real geographic position of reference water conservancy target;Reference water conservancy target in the preset river management range line is judged as illegal behavior.The present application identifies water conservancy target after water conservancy target identification model, uses water conservancy target tracking network model to track counting, can calculate the real geographic coordinate information of water conservancy target by spatial positioning model, reduce the limitation of unmanned aerial vehicle in river and lake supervision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of river and lake supervision, in particular to a river and lake intelligent supervision method and system based on a UAV. BACKGROUND

[0002] A big problem of river and lake supervision is the rapid discovery of river and lake problems. The existing methods mostly rely on manual inspection, which has high cost, poor timeliness and coverage. Personnel cannot reach some severe and hidden areas, and the safety of the inspection personnel is also endangered. In the existing method, fixed cameras are used for video supervision, which can only be used for a small number of fixed key areas, and cannot well meet the overall accurate management of river and lake problems. The real-time, flexible, high-resolution and cost-effective advantages of the UAV are a powerful means to realize efficient patrol and management, and can provide strong support for reservoir area supervision. However, the current UAV has insufficient application depth and intelligence in the reservoir management in the river and lake supervision, and only exists as an information acquisition means, which has limitations. SUMMARY

[0003] The purpose of the present application is to provide a river and lake supervision method and system based on a UAV, which reduces the limitations of the UAV in river and lake supervision.

[0004] To achieve the above purpose, the present application provides the following scheme:

[0005] A river and lake supervision method based on a UAV, comprising:

[0006] obtaining multiple target images of a target supervision area; the target images are obtained by a camera on the UAV;

[0007] inputting the multiple target images into a trained water conservancy target recognition model respectively, marking water conservancy targets in the target images by using detection boxes, and determining the position information of the water conservancy targets in the target images; the trained water conservancy target recognition model is a model trained by taking a sample image as input, taking a sample water conservancy target marked by using a sample detection box and the position information of the sample water conservancy target in the sample image as output;

[0008] cutting the image in each detection box to obtain multiple detection box images;

[0009] inputting the detection box images into a trained water conservancy target tracking network model to obtain the state of the water conservancy targets in the detection box images; the state includes static, tracking and stopping tracking; the trained water conservancy target tracking network model is a model trained by taking a sample detection box image as input and taking the state of a sample water conservancy target in the sample detection box image as output;

[0010] According to the state of the water conservancy target in all the detection frame images, the number of reference water conservancy targets is calculated by using a line hitting counting method, and position information of the reference water conservancy targets in a reference image at the time of counting is obtained; the reference water conservancy target is a water conservancy target in a tracking state; the reference image is a target image of the reference water conservancy target when passing through a pre-designed number of lines;

[0011] The position information of the reference water conservancy target in the reference image and POS data corresponding to the reference image are input into a spatial positioning model to obtain a real geographical position of the reference water conservancy target; the POS data includes current time, spatial position information of the unmanned aerial vehicle, attitude information of the unmanned aerial vehicle, and attitude information of the camera; the spatial positioning model is a model trained by taking position information of a sample reference water conservancy target in a sample reference image and sample POS data corresponding to the sample reference image as input and taking a real geographical position of the sample reference water conservancy target as output;

[0012] According to the real geographical position of the reference water conservancy target and a preset river management range line, it is judged whether the reference water conservancy target is within the preset river management range line, if yes, the reference water conservancy target is determined as illegal behavior; if not, the reference water conservancy target is determined as not illegal behavior.

[0013] Optionally, further comprising:

[0014] The real geographical position of the reference water conservancy target determined as illegal behavior is displayed.

[0015] Optionally, before the plurality of target images are input into the trained water conservancy target recognition model, the method further comprises: training the water conservancy target recognition model, and the training process is as follows:

[0016] A data set is obtained; the data set includes a plurality of sample images containing sample water conservancy targets and labels corresponding to the sample images; the label is a sample water conservancy target marked by a sample detection frame and position information of the sample water conservancy target in the sample image;

[0017] The water conservancy target recognition model is trained by using the data set to obtain the trained water conservancy target recognition model.

[0018] Optionally, the inputting of the detection frame image into the trained water conservancy target tracking network model to obtain the state of the water conservancy target in the detection frame image specifically comprises:

[0019] The detection frame image is input into the trained water conservancy target tracking network model to obtain predicted position information of the water conservancy target in the detection frame image;

[0020] calculate an interaction ratio according to the predicted position information and real position information of the water conservancy target in a next frame target image;

[0021] determine a state of the water conservancy target in the bounding box image according to the interaction ratio and a preset interaction ratio.

[0022] Optionally, the determining the state of the water conservancy target in the bounding box image according to the interaction ratio and the preset interaction ratio specifically comprises:

[0023] If the interaction ratio is greater than the preset interaction ratio, the state of the water conservancy target is tracking, otherwise, the state of the water conservancy target is stopping tracking.

[0024] Optionally, before the inputting the position information of the reference water conservancy target in a reference image and POS data corresponding to the reference image into the spatial positioning model, the method further comprises:

[0025] interpolating the POS data by using an interpolation algorithm to obtain the POS data corresponding to each reference image.

[0026] Optionally, the determining whether the reference water conservancy target is in the preset river management range line according to the real geographical position of the reference water conservancy target and the preset river management range line specifically comprises:

[0027] determining whether the reference water conservancy target is in the preset river management range line according to the real geographical position of the reference water conservancy target and the preset river management range line by using a ray method.

[0028] Optionally, the trained water conservancy target tracking network model is a deepsort model.

[0029] The application further provides a river and lake supervision system based on a UAV, comprising:

[0030] a target image acquisition module, configured to acquire multiple frames of target images of a target supervision area; the target images are obtained by a camera on the UAV;

[0031] a water conservancy target identification module, configured to input the multiple frames of target images into a trained water conservancy target identification model respectively, mark water conservancy targets in the target images by using bounding boxes, and determine position information of the water conservancy targets in the target images; the trained water conservancy target identification model is a model trained by taking a sample image as input, taking a sample water conservancy target marked by using a sample bounding box and position information of the sample water conservancy target in the sample image as output;

[0032] a bounding box image acquisition module, configured to intercept images in each bounding box to obtain multiple bounding box images.

[0033] a water target tracking module, configured to input the detection frame image into a trained water target tracking network model to obtain a state of the water target in the detection frame image; the state includes being static, being tracked, and being stopped from being tracked; the trained water target tracking network model is a model trained by taking a sample detection frame image as input and taking a state of a sample water target in the sample detection frame image as output;

[0034] a counting module, configured to count a number of reference water targets according to the states of the water targets in all the detection frame images by using a photoelectric cell counting method, and obtain position information of the reference water targets in a reference image when counting; the reference water targets are water targets in the state of being tracked; the reference image is a target image of the reference water targets when passing through a pre-designed number line;

[0035] a real geographical position obtaining module, configured to input the position information of the reference water targets in the reference image and POS data corresponding to the reference image into a spatial positioning model to obtain a real geographical position of the reference water targets; the POS data includes a current time, spatial position information of a UAV, attitude information of the UAV, and attitude information of a camera; the spatial positioning model is a model trained by taking position information of a sample reference water target in a sample reference image and sample POS data corresponding to the sample reference image as input and taking a real geographical position of the sample reference water target as output;

[0036] a judgment module, configured to judge whether the reference water target is in a preset river management range line according to the real geographical position of the reference water target and the preset river management range line, and if yes, determine that the reference water target is illegal behavior, and if not, determine that the reference water target is not illegal behavior.

[0037] Optionally, the system further comprises a display module.

[0038] The display module is configured to display the real geographical position of the reference water target determined as illegal behavior.

[0039] According to the specific embodiments of the present application, the present application discloses the following technical effects: the present application provides a kind of river and lake supervision method and system based on unmanned aerial vehicle, comprising: obtaining the multiple frames target image of target supervision area;Target image is obtained by the camera on unmanned aerial vehicle;Multiple frames target image is input into the water conservancy target identification model trained respectively, and water conservancy target in target image is marked using detection frame, and the position information of water conservancy target in target image is determined;The image in each detection frame is intercepted, and multiple detection frame images are obtained;Detection frame image is input into the water conservancy target tracking network model trained, and the state of water conservancy target in detection frame image is obtained;State includes still, tracking and stop tracking;According to the state of all water conservancy targets in detection frame image, the number of reference water conservancy targets is calculated using the collision line counting method, and the position information of reference water conservancy target in reference image is obtained when counting;The position information of reference water conservancy target in reference image and the POS data corresponding to reference image are input into spatial positioning model, and the real geographical position of reference water conservancy target is obtained;POS data includes current time, unmanned aerial vehicle spatial position information, unmanned aerial vehicle attitude information, camera attitude information;According to whether the real geographical position of reference water conservancy target is in the reference water conservancy target within the preset river management range line, it is determined whether it is illegal behavior.The present application can track and count using water conservancy target tracking network model after water conservancy target identification model identifies water conservancy target, and the real geographical coordinate information of water conservancy target can be calculated by spatial positioning model, which reduces the limitations of unmanned aerial vehicle in river and lake supervision. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0041] Figure 1 A flow chart of the river and lake supervision method based on unmanned aerial vehicle provided for embodiment 1 of the present application is provided.

[0042] Figure 2 A specific implementation flow chart of the river and lake supervision method based on unmanned aerial vehicle provided for embodiment 1 of the present application is provided.

[0043] Figure 3 A structure diagram of the water conservancy target identification model provided for embodiment 1 of the present application is provided.

[0044] Figure 4 A schematic diagram of the simple house for reservoir area aquaculture identified by the water conservancy target identification model provided for embodiment 1 of the present application is provided.

[0045] Figure 5A deepsort algorithm flowchart provided for the embodiment 1 of the present application;

[0046] Figure 6 An unmanned aerial vehicle target tracking counting schematic diagram provided for the embodiment 1 of the present application;

[0047] Figure 7 A ray method principle diagram provided for the embodiment 1 of the present application;

[0048] Figure 8 A ray method intersection schematic diagram provided for the embodiment 1 of the present application;

[0049] Figure 9 A schematic diagram of several cases excluded by the ray method provided for the embodiment 1 of the present application;

[0050] Figure 10 An intelligent inspection system structure diagram provided for the embodiment 1 of the present application;

[0051] Figure 11 An intelligent inspection system large screen interface provided for the embodiment 1 of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0053] The purpose of the present application is to provide a river and lake supervision method and system based on unmanned aerial vehicles, which reduces the limitations of unmanned aerial vehicles in river and lake supervision.

[0054] In order to make the above-mentioned purposes, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0055] Embodiment 1

[0056] The embodiment provides a river and lake supervision method based on unmanned aerial vehicles, referring to Figure 1 and Figure 2 The method comprises the following steps.

[0057] S1: acquiring a plurality of target images of a target supervision area; the target images are obtained by a camera on an unmanned aerial vehicle. In the embodiment, a video of the target supervision area shot by the camera on the unmanned aerial vehicle is first acquired, and then the video is cut into one frame of image after another, and the cut one frame of image after another is taken as the target image.

[0058] S2: inputting the multiple frames of the target images into the trained water conservancy target recognition model respectively, marking the water conservancy targets in the target images by using the detection boxes, and determining the position information of the water conservancy targets in the target images; the trained water conservancy target recognition model is a model obtained by training, taking a sample image as input, taking a sample water conservancy target marked by using a sample detection box and position information of the sample water conservancy target in the sample image as output.

[0059] S3: intercepting the image in each detection box to obtain multiple detection box images.

[0060] S4: inputting the detection box images into the trained water conservancy target tracking network model to obtain the state of the water conservancy target in the detection box images; the state includes static, tracking and stopping tracking; the trained water conservancy target tracking network model is a model obtained by training, taking a sample detection box image as input, and taking the state of a sample water conservancy target in the sample detection box image as output.

[0061] S5: according to the state of the water conservancy target in all the detection box images, using the line collision counting method to calculate the number of reference water conservancy targets, and obtaining the position information of the reference water conservancy targets in the reference images when counting; the reference water conservancy target is a water conservancy target in a state of tracking; the reference image is a target image of the reference water conservancy target passing through a pre-designed number line.

[0062] S6: inputting the position information of the reference water conservancy target in the reference image and the corresponding POS data of the reference image into the spatial positioning model to obtain the real geographical position of the reference water conservancy target; the POS data includes current time, spatial position information of the unmanned aerial vehicle, attitude information of the unmanned aerial vehicle, and attitude information of the camera; the spatial positioning model is a model obtained by training, taking the position information of a sample reference water conservancy target in a sample reference image and sample POS data corresponding to the sample reference image as input, and taking the real geographical position of the sample reference water conservancy target as output.

[0063] S7: determining whether the reference water conservancy target is in the preset river management range line according to the real geographical position of the reference water conservancy target and the preset river management range line, if yes, determining that the reference water conservancy target is illegal behavior, and if no, determining that the reference water conservancy target is not illegal behavior.

[0064] In the embodiment, the construction process of the water conservancy target recognition model is also included.

[0065] The YOLO series algorithm is based on the PyTorch framework, is convenient for extension to mobile devices, and belongs to a relatively lightweight network. The YOLOv5 includes four network structures of YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x, the network widths and depths of which are different, and the parameter amounts are sequentially increased. The YOLOv5s is the preferred lightweight network, which is convenient for deployment to embedded devices, and the other three models are based on the YOLOv5s and increase the model width and depth to different degrees. The YOLOv5s model is selected in combination with the detection accuracy and detection rate. The embodiment adopts the YOLOv5 model as the water conservancy target recognition model, the network structure of which is as shown in FIG. 1, the class and accuracy of model recognition are directly related to the sample, and the data is collected and arranged by manual work to form a training data set in the early stage, and the parameters are debugged for multiple iterations to obtain a corresponding weight parameter file. The weight file and the target to be detected are input into the network model at the same time, and the output is a predicted water conservancy target recognition result. Figure 3

[0066] After the water conservancy target recognition model is constructed, the water conservancy target recognition model needs to be trained, and the training process specifically includes:

[0067] obtaining a data set; the data set includes multiple frames of sample images containing sample water conservancy targets and labels corresponding to the sample images; the label is a sample water conservancy target marked by a sample detection frame and position information of the sample water conservancy target in the sample image;

[0068] training the water conservancy target recognition model by using the data set to obtain the trained water conservancy target recognition model. Specifically:

[0069] The sample data set is randomly divided into two parts, i.e., a training set and a test set, and the ratio of the training set to the test set is 9:1. The main model parameters are set during training:

[0070] Number of learning times (epochs): the number of times of adjusting and optimizing required for model convergence, which is set to 200 by default.

[0071] Learning rate (learning_rate): the learning rate is an important hyperparameter for deep learning, which controls the speed of adjusting the neural network weight based on the loss gradient, and is involved in most optimization algorithms (SGD, RMSprop, Adam). The smaller the learning rate, the slower the loss gradient decreases, and the longer the convergence time. The default learning rate of the system is 0.001.

[0072] ​Sub-training set size (batch_size): that is, the number of data samples required for one training, the larger the parameter, the better the global optimal solution of the model, but the corresponding memory consumption also increases, and if it is set too large, it will run out of memory, causing the system to shut down and stop running, 6G memory can be set to 3-4 by default; 12G memory can be set to 8-10 by default.

[0073] Accuracy evaluation criteria: In this embodiment, the class average pixel accuracy (Mean Pixel Accuracy, MPA) and the average intersection over union (Mean Intersection over Union, MIoU) are selected as the accuracy evaluation indicators. The meaning of the class average pixel accuracy is to calculate the proportion of the number of correctly classified pixels for each class respectively, and then accumulate and average. The average intersection over union, abbreviated as mIOU, is the intersection of the predicted region and the actual region divided by the union of the predicted region and the actual region. This calculation obtains the IOU of a single class, and then repeats the algorithm to calculate the IOU of other classes, and then calculates their average. Its meaning is the ratio of the intersection and union of the predicted result and the true value of each class of the model, and then the sum is calculated and the average is calculated.

[0074] The model will automatically save the optimal training model in all iteration times according to MPA and IOU, and print the training loss, training accuracy, test loss and test accuracy in real time during each iteration process, which can be used by the user to watch the running results and accuracy of the model.

[0075] Target ground objects: In this embodiment, the water conservancy target ground objects (water conservancy targets) to be extracted include “illegal occupation, illegal mining, illegal stacking, illegal construction” and the like, wherein the classification standards of each water conservancy target are as follows:

[0076] Illegal occupation: illegal occupation of water area, beach, planting of trees and high-stem crops that hinder flood discharge without approval, etc.

[0077] Illegal mining: illegal sand mining and soil extraction in rivers and lakes, etc.

[0078] Illegal stacking: dumping and stacking garbage, dumping, landfill, storage, stacking solid waste, and abandoning and stacking objects that hinder flood discharge, etc.

[0079] Illegal construction: long-term occupation of river and lake shoreline without use, excessive occupation and less use, illegal occupation and use, illegal construction of river-related projects, and construction of buildings and structures that hinder flood discharge in the river management area, etc.

[0080] In the embodiment, the target image photographed by the unmanned aerial vehicle is input into the water conservancy target recognition model, and the target image is output after being determined by the target recognition model. If the frame target image contains a recognized water conservancy target, it is marked by using a rectangular frame (detection frame) to surround it, and the width and height of the rectangular frame are obtained. The position information of the water conservancy target in the target image is represented by the pixel coordinates of the center point of the rectangular frame. The simple house is a common illegal building. In the embodiment, the simple house is taken as the water conservancy target to train the water conservancy target recognition model. For example, Figure 4 The reservoir breeding simple house recognized by the water conservancy target recognition model.

[0081] In the embodiment, the detection frame image is input into the trained water conservancy target tracking network model to obtain the state of the water conservancy target in the detection frame image, specifically including:

[0082] The detection frame image is input into the trained water conservancy target tracking network model to obtain the predicted position information of the water conservancy target in the detection frame image;

[0083] The interaction ratio is calculated according to the predicted position information and the real position information of the water conservancy target in the next frame target image;

[0084] The state of the water conservancy target in the detection frame image is determined according to the interaction ratio and a preset interaction ratio.

[0085] The state of the water conservancy target in the detection frame image is determined according to the interaction ratio and a preset interaction ratio, specifically including:

[0086] If the interaction ratio is greater than the preset interaction ratio, the state of the water conservancy target is tracking, and if not, the state of the water conservancy target is stopping tracking. Specifically:

[0087] When the object is recognized as a water conservancy business target (i.e. a water conservancy target) by the target recognition model, it is determined as a tracking target, and the target state is recorded in the tracking list. A counting line is set in advance in the video screen, and when the center pixel point of the rectangular frame of the tracking target passes through the counting line, it is judged. If the state of the water conservancy target is in the tracking list, it is judged as a tracking target (i.e. the state of the water conservancy target is tracking), and the number of water conservancy targets in one inspection is increased by one. If the state of the water conservancy target is not in the tracking list, it is judged as not a tracking target, and the number of water conservancy business targets does not change.

[0088] In the embodiment, the state of the water conservancy target includes a static state and a motion state; the motion state includes being tracked and being stopped tracking. When the state of the water conservancy target is the static state, it indicates that the pixel coordinates of the water conservancy target in two continuous target images are not changed, that is, the two target images photographed by the unmanned aerial vehicle are repeated. The interaction ratio is calculated by the predicted position information and the real position information of the water conservancy target in the next target image, and is compared with the preset interaction ratio. If the interaction ratio is greater than the preset interaction ratio, the state of the water conservancy target is tracked, otherwise, the state of the water conservancy target is stopped tracking. In the embodiment, the preset interaction ratio is 0.6.

[0089] It should be noted that in the embodiment, if the water conservancy target v appears in the previous three frames, but the water conservancy target v does not appear in the fourth frame, it is judged whether the water conservancy target v exists in the fifth frame image. If the water conservancy target v does not exist in the fifth frame image, the state of the water conservancy target v is stopped tracking. If the state of the water conservancy target v is stopped tracking at the third frame, but the water conservancy target v appears in the fifth frame, the water conservancy target v is tracked and counted as a new water conservancy target at this time.

[0090] In the embodiment, the deepsort model is used as the water conservancy target tracking network model to determine the recognized water conservancy target between video continuous frames, to determine whether the recognized target is the same object in multiple frame images, and then use the line collision counting algorithm to count the water conservancy target that has been tracked to be the same object between multiple continuous frames, and save and record the video frame image and the water conservancy target information, including the water conservancy target position information. Before that, the water conservancy target tracking network model also needs to be constructed.

[0091] After the water conservancy target is recognized, the water conservancy target tracking and the water conservancy target quantity calculation need to be further performed, and the target tracking algorithm model is needed to realize the water conservancy target tracking. In the target tracking process, the apparent features of the water conservancy target are extracted for nearest neighbor matching, which can effectively improve the target tracking effect under the occlusion condition. As shown in the figure, the deepsort algorithm process can be divided into four parts of track processing and state estimation, correlation measurement, cascade matching and deep feature descriptor. Figure 5

[0092] ​Deepsort is to apply deep appearance features to the model, deepsort is an upgrade based on sort target tracking, and the appearance features of the target are extracted during target tracking for nearest neighbor matching, which can effectively improve the target tracking effect under occlusion. Through a standard Kalman filter based on a constant speed model and a linear observation model, the motion state of the target is predicted, if the water conservancy target cannot be matched with the existing path during target matching, it is considered that the water conservancy target may be a new water conservancy target, if the water conservancy target is continuously detected for the next 3 frames, the water conservancy target will be identified as a new water conservancy target, and a new tracking path is generated based on the starting target, otherwise no new tracking path is generated. After determining the tracking water conservancy target, the state of the water conservancy target is recorded in the tracking list, and a counting line is set in the video image, and the center pixel point of the water conservancy target is judged when it passes through the counting line, if the state of the water conservancy target is in the tracking list, it is judged as a tracking target (i.e. the state of the water conservancy target is in tracking), and the count is increased by one; if the state of the water conservancy target is not in the tracking list, it is not a tracking target, and the count is not changed, which can effectively prevent the problem of repeated counting of water conservancy targets in each frame, and achieve accurate counting. As shown in Figure 6 Figure 6 The horizontal horizontal line in the middle is the counting line, only when the center point of the water conservancy target passes through the counting line, the counting is performed, and the water conservancy targets that do not pass through and have passed through do not participate in the counting. Figure 6 In (a) of the building 1, the first simple house participating in the counting, Figure 6 In (b), building 4 is the fourth simple house participating in the counting.

[0093] In the embodiment, before the position information of the reference water conservancy target in the reference image and the POS data corresponding to the reference image are input into the spatial positioning model, the method further comprises:

[0094] The POS data is interpolated by using an interpolation algorithm to obtain the POS data corresponding to each reference image, specifically:

[0095] The target coordinate solution algorithm needs to use the POS attitude information of the unmanned aerial vehicle when shooting the video frame, and the POS data is obtained by the inspection APP and the ground control station of the called unmanned aerial vehicle. Since only 6-7 times of POS information can be returned per second, and the next second video is 24 frames, the video frame does not match the obtained POS information, and interpolation needs to be performed according to the flight attitude and speed of the aircraft to obtain the POS data of the frame. In the embodiment, the POS data corresponding to each frame of target image mainly includes the current time, the position of the unmanned aerial vehicle, the attitude of the unmanned aerial vehicle, the attitude of the camera, etc., wherein the geographic coordinates of the unmanned aerial vehicle at a certain time are: X​uva Y uva Z uva The UAV attitude includes a yaw angle yaw uva , a roll angle roll uva , and a pitch angle pitch uva . Because the camera is set perpendicular to the UAV body to reduce video distortion during inspection, the UAV camera vertically downward, so the attitude of each frame: yaw=yaw uva , roll=roll uva , pitch=pitch uva . Then the obtained UAV POS data is matched with each frame of video target image, and the video frame is analyzed in space position using the collinear equation.

[0096] Then a spatial positioning model is constructed, the position information of the reference water target in the reference image and the POS data corresponding to the reference image are input into the spatial positioning model, and the real geographical position of the reference water target is obtained. The position information of the reference water target in the reference image is the center point coordinate of the reference water target. In this embodiment, the spatial positioning model is also used to obtain the real geographical position coordinates of the image center point and the four corner points of the reference image, and the coordinate point positions and picture information are stored.

[0097] The process of spatial positioning is specifically introduced as follows:

[0098] The pixel point coordinates (I, J) of the target point A in the image are obtained, I is the pixel column number, and J is the pixel row number. The coordinates of the point in the image plane coordinate system are calculated according to formula (1):

[0099]

[0100] Where Δ is the physical size of a single pixel, m is the number of image pixel columns, and n is the number of image pixel rows.

[0101] The obtained camera parameters are: the focal length of the camera is f, and the CCD array size corresponding to the sensor is W*H. First, the spatial coordinates of the video projection center point are calculated using the collinear equation (formula (2), (3)):

[0102]

[0103]

[0104] In the formula: x, y are the image plane coordinate system coordinates of the image point; x0, y0 are the interior orientation elements of the image; X s =X uva , Y s =Y uva , ZS =Z uva X represents the object space coordinates of the camera station; A Y A Z A Let (x, y) be the object space coordinates of the ground point corresponding to (x, y); a i ,b i ,c i (i = 1, 2, 3) is a rotation matrix composed of 3 exterior orientation elements, as shown in formula (4).

[0105]

[0106] in, b1=cosωsin k; b2=cosωcosκ; b3=-sinω; In the formula: yaw=κ, roll=ω,

[0107] According to the collinearity equation, we can obtain:

[0108]

[0109]

[0110] Where (X) A Y A Let (X) be the coordinates of target point A in the ground photogrammetric coordinate system. s Y s Let (x, y) be the coordinates of the principal point in the ground photogrammetric coordinate system, and (x, y) be the coordinates of the target point A in the image plane coordinate system. Therefore, the spatial coordinates of the ground point corresponding to the image point can be calculated using formulas (5) and (6), thus achieving geographic positioning.

[0111] After the target recognition network model identifies a specific water conservancy target in the video frame, it inputs the pixel coordinates of the center point of the rectangle into the video spatial positioning model. Based on the above steps, the true geographic coordinates of the specific water conservancy target can be calculated.

[0112] In this embodiment, determining whether the reference water conservancy target is within the preset river management range based on the actual geographical location of the reference water conservancy target and the preset river management range line specifically includes:

[0113] The ray method is used to determine whether the reference water conservancy target is within the preset river management area line, based on the target's actual geographical location and the preset river management area line. Specifically:

[0114] like Figure 7As shown, determining whether a point is inside a polygon is a frequent requirement when processing spatial data. Examples include point selection in GIS software, filtering points within a polygon based on its boundaries, finding intersections, and filtering points outside the polygon. There are several different approaches to determining whether a point is inside a polygon. This project uses the ray casting method. The ray casting method involves starting from the point to be determined and drawing a ray horizontally to the right (or left). The number of intersections between the ray and each edge of the polygon is calculated. If the number of intersections is odd, the point is inside the polygon; if even, it is outside. This algorithm can also correctly determine the location of points in complex polygons. The key to the ray casting method is correctly calculating the intersections of the ray with the polygon edges. If the ray intersects an edge, the intersection count is incremented by 1. For example... Figure 8 As shown, ray a intersects the polygon at 3 points, so the endpoint of ray a is inside the polygon; ray b intersects the polygon at 4 points, so the endpoint of ray b is not inside the polygon. In this embodiment, it is also defined that a line segment overlapping a ray or a ray passing through the lower endpoint of a line segment is considered non-intersecting. First, the non-intersecting cases are excluded. Figure 9 All such cases need to be eliminated. Let's illustrate the elimination method by taking a line segment above a ray as an example. Let the coordinates of the ray be (x, y), and the coordinates of the starting and ending points of the line segment be (x, y). s y s The coordinates of the endpoint of the line segment are (x, y). e y e If y s >y,y e If >y, then the line segment is above the ray.

[0115] In this embodiment, the method further includes displaying the actual geographical location of the reference water conservancy target that has been determined to be an illegal act. Specifically:

[0116] The actual geographic coordinates of the reference water conservancy target obtained from the above calculation are uploaded to the GIS system. Then, a ray-mapping method is used to determine the position of the target by comparing it with the vector line of the inspection and management scope (preset river management scope line). This determines whether the reference water conservancy target is within the river management scope line, thereby determining whether the identified target constitutes an illegal act. If it is determined to be an illegal act, the target's actual geographic information is uploaded to the GIS system for display. In this embodiment, based on the proportional relationship between the actual geographic coordinates of each point on the river management scope line and the rectangle determined by the four corner points, the corresponding position of each point on the river management scope line in the video pixels is calculated and drawn in the corresponding reference image. The video frames are then converted into an RTMP live stream and uploaded to a streaming media server. The GIS system then retrieves the video stream for visualization.

[0117] In this embodiment, an intelligent inspection system based on unmanned aerial vehicles (UAVs) is provided, such as... Figure 10As shown, it is composed of 6 parts of unmanned aerial vehicle platform, unmanned aerial vehicle inspection APP, video stream service, intelligent interpretation service, data management system and comprehensive application system. Figure 11 It is a large screen interface of the intelligent inspection system.

[0118] (1) Unmanned aerial vehicle platform

[0119] A rotor unmanned aerial vehicle is used, and appropriate sensor equipment is selected according to the inspection task. Common sensors include optical cameras, thermal infrared cameras, multispectral imagers, laser scanners and the like.

[0120] (2) Unmanned aerial vehicle inspection APP (unmanned aerial vehicle management APP)

[0121] The APP mainly realizes the functions of inspection task receiving, unmanned flight control, flight state monitoring, flight POS information acquisition and data collection and data return. The data collection and data return function can also be realized by adding a 4G / 5G module on the unmanned aerial vehicle platform.

[0122] (3) Video stream service

[0123] The video stream service receives the video information sent back by the unmanned aerial vehicle inspection APP, transcodes while saving the data, and pushes to the intelligent analysis service in flv, hls and other protocol formats.

[0124] (4) Intelligent analysis service

[0125] The intelligent analysis service uses various technologies such as deep learning and pattern recognition, fully utilizes the powerful parallel processing capability of GPU, realizes fast analysis of video frame data or image data directly pushed by the unmanned aerial vehicle inspection APP, matches POS information, identifies problem categories, calculates spatial positions, takes screenshots for evidence retention, and calls the data management system interface to write the analysis results into the database, reminding the management user to check and make further processing. Video frame data also needs to be restored to video stream so as to be able to be synchronously displayed in the comprehensive application system.

[0126] (5) Data management system

[0127] The data management system mainly maintains and manages the entire system running data, unmanned aerial vehicle state data, intelligent identification result data and the like, so as to ensure the normal operation of the system and provide data support for the comprehensive application system.

[0128] (6) Comprehensive application system

[0129] In a multi-terminal (desktop and mobile terminal) and multi-form (two-dimensional, large screen and APP) combination mode, the functions of inspection route planning, inspection task management, flight state display, flight emergency control, inspection problem alarm, before-and-after comparison analysis and problem account management are realized.

[0130] The present application aims at the pain points of river and lake supervision problems, and constructs a water conservancy target identification model, a water conservancy target tracking network model and a spatial positioning model of a specific water conservancy object (water conservancy target); the specific water conservancy target in the unmanned aerial vehicle video image is identified through the water conservancy target identification model, and the identified specific water conservancy target is subjected to real geographic coordinate calculation processing through the spatial positioning model; after obtaining the geographic coordinate information of the specific water conservancy target, the river management range vector data is superimposed and judged, the water conservancy target in the river management range line is uploaded to the alarm platform, and the real geographic coordinates of the illegal water conservancy target and the screenshot containing the water conservancy target in the unmanned aerial vehicle are sent to the GIS system for visual display; at the same time, the river management range vector data can be inversely calculated from the geographic coordinates to the pixel coordinates in the real-time video frame, and finally superimposed to the video and pushed to the GIS system for synchronous display.

[0131] Compared with the prior art, the present application has high accuracy in identifying water conservancy targets, can track and count the water conservancy targets after identification, and can calculate the real geographic coordinate information of the water conservancy targets according to the spatial positioning model during counting, and can superimpose the geographic coordinate information with the river management vector range to accurately obtain the illegal behavior in the river management range line and push the information to the alarm, and can also be visualized and displayed to the GIS system, and can also be inversely calculated and superimposed to the video, and pushed to the GIS system for visual display.

[0132] Embodiment 2

[0133] The present embodiment provides a river and lake supervision system based on an unmanned aerial vehicle, comprising:

[0134] A target image acquisition module is configured to acquire multiple target images of a target supervision area; the target images are obtained by a camera on an unmanned aerial vehicle.

[0135] A water conservancy target identification module is configured to input the multiple target images into a trained water conservancy target identification model respectively, mark water conservancy targets in the target images by using detection boxes, and determine the position information of the water conservancy targets in the target images; the trained water conservancy target identification model is a model trained by taking a sample image as input, taking a sample water conservancy target marked by using a sample detection box and the position information of the sample water conservancy target in the sample image as output.

[0136] A detection box image acquisition module is configured to capture images in each detection box to obtain multiple detection box images.

[0137] a water conservancy target tracking module, configured to input the bounding box image into a trained water conservancy target tracking network model to obtain a state of the water conservancy target in the bounding box image; the state comprises being static, being tracked, and being stopped from being tracked; the trained water conservancy target tracking network model is a model trained by taking a sample bounding box image as input and taking a state of a sample water conservancy target in the sample bounding box image as output.

[0138] a counting module, configured to count a number of reference water conservancy targets by using a photoelectric cell counting method according to the states of the water conservancy targets in all the bounding box images, and obtain position information of the reference water conservancy targets in a reference image when counting; the reference water conservancy targets are water conservancy targets in a tracking state; the reference image is an object image of the reference water conservancy targets when passing through a pre-designed number line.

[0139] a real geographical position obtaining module, configured to input the position information of the reference water conservancy targets in the reference image and POS data corresponding to the reference image into a spatial positioning model to obtain a real geographical position of the reference water conservancy targets; the POS data comprises a current time, spatial position information of a UAV, attitude information of the UAV, and attitude information of a camera; the spatial positioning model is a model trained by taking position information of a sample reference water conservancy target in a sample reference image and sample POS data corresponding to the sample reference image as input and taking a real geographical position of the sample reference water conservancy target as output.

[0140] a judging module, configured to judge whether the reference water conservancy target is within a preset river management range line according to the real geographical position of the reference water conservancy target and the preset river management range line, and if yes, determine that the reference water conservancy target is illegal, and if not, determine that the reference water conservancy target is not illegal.

[0141] In the embodiment, a display module is further included.

[0142] The display module is configured to display the real geographical position of the reference water conservancy target determined to be illegal.

[0143] In the specification, each embodiment focuses on the difference from other embodiments, and the same or similar parts between the embodiments can be referred to each other.

[0144] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above examples are only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application. In conclusion, the content of the present specification should not be understood as the limitation of the present application.

Claims

1. A method for river and lake monitoring based on unmanned aerial vehicles (UAVs), characterized in that, include: Acquire multiple frames of target images of the target surveillance area; The target image was captured by a camera on a drone; Multiple frames of the target images are input into a trained water conservancy target recognition model. Detection boxes are used to mark the water conservancy targets in the target images, and the position information of the water conservancy targets in the target images is determined. The trained water conservancy target recognition model is a model trained by taking sample images as input and taking sample water conservancy targets marked with sample detection boxes and the position information of the sample water conservancy targets in the sample images as output. The image within each detection box is cropped to obtain multiple detection box images; The detection box image is input into the trained water conservancy target tracking network model to obtain the state of the water conservancy target in the detection box image; the state includes stationary, tracking in progress, and tracking stopped; the trained water conservancy target tracking network model is a model trained with sample detection box images as input and the state of sample water conservancy targets in the sample detection box images as output. Based on the state of the water conservancy targets in all the detection box images, the number of reference water conservancy targets is calculated using the line-hitting counting method, and the position information of the reference water conservancy targets in the reference image is obtained when counting. The reference water conservancy target is the water conservancy target in the tracking state; the reference image is the target image when the reference water conservancy target passes through the preset counting line; The location information of the reference water conservancy target in the reference image and the POS data corresponding to the reference image are input into the spatial positioning model to obtain the true geographical location of the reference water conservancy target; The POS data includes the current time, UAV spatial location information, UAV attitude information, and camera attitude information; The spatial positioning model is a model trained by taking the location information of the sample reference water conservancy target in the sample reference image and the sample POS data corresponding to the sample reference image as input, and taking the real geographical location of the sample reference water conservancy target as output. Based on the actual geographical location of the reference water conservancy target and the preset river management boundary line, it is determined whether the reference water conservancy target is within the preset river management boundary line. If it is, the reference water conservancy target is determined to be an illegal act; if not, the reference water conservancy target is determined to be a non-illegal act.

2. The method for river and lake monitoring based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, Also includes: The actual geographical location of the reference water conservancy target that was determined to be an illegal act will be displayed.

3. The method for river and lake monitoring based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, Before inputting the multiple frames of the target images into the trained water conservancy target recognition model, the method further includes: training the water conservancy target recognition model, the training process of which is as follows: Obtain a dataset; the dataset includes multiple frames of sample images containing sample water conservancy targets and corresponding labels for the sample images; the labels are the sample water conservancy targets marked with sample detection boxes and the location information of the sample water conservancy targets in the sample images; The dataset is used to train the water conservancy target recognition model to obtain the trained water conservancy target recognition model.

4. The method for river and lake monitoring based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The step of inputting the detection box image into the trained water conservancy target tracking network model to obtain the state of the water conservancy target in the detection box image specifically includes: The detection box image is input into the trained water conservancy target tracking network model to obtain the predicted location information of the water conservancy target in the detection box image; The interaction ratio is calculated based on the predicted location information and the actual location information of the water conservancy target in the next frame of the target image. The state of the water conservancy target in the detection frame image is determined based on the interaction ratio and the preset interaction ratio.

5. The method for river and lake monitoring based on unmanned aerial vehicles (UAVs) according to claim 4, characterized in that, The step of determining the state of the water conservancy target in the detection box image based on the interaction ratio and a preset interaction ratio specifically includes: If the interaction ratio is greater than the preset interaction ratio, the state of the water conservancy target is "tracking in progress"; otherwise, the state of the water conservancy target is "tracking stopped".

6. The method for river and lake monitoring based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, Before inputting the location information of the reference hydraulic target in the reference image and the POS data corresponding to the reference image into the spatial positioning model, the method further includes: An interpolation algorithm is used to interpolate the POS data to obtain the POS data corresponding to each reference image.

7. The method for river and lake monitoring based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The step of determining whether the reference water conservancy target is within the preset river management range based on the actual geographical location of the reference water conservancy target and the preset river management range line specifically includes: The ray method is used to determine whether the reference water conservancy target is within the preset river management range line based on the actual geographical location of the reference water conservancy target and the preset river management range line.

8. The method for river and lake monitoring based on unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The trained water conservancy target tracking network model is the deepsort model.

9. A river and lake monitoring system based on unmanned aerial vehicles (UAVs), characterized in that, include: The target image acquisition module is used to acquire multiple frames of target images of the target monitored area; The target image was captured by a camera on a drone; The water conservancy target recognition module is used to input multiple frames of the target images into a trained water conservancy target recognition model, use detection boxes to mark the water conservancy targets in the target images, and determine the position information of the water conservancy targets in the target images; the trained water conservancy target recognition model is a model trained by taking sample images as input and taking sample water conservancy targets marked by sample detection boxes and the position information of the sample water conservancy targets in the sample images as output. The detection box image acquisition module is used to crop the image within each detection box to obtain multiple detection box images; The water conservancy target tracking module is used to input the detection box image into the trained water conservancy target tracking network model to obtain the state of the water conservancy target in the detection box image; the state includes stationary, tracking in progress, and tracking stopped; the trained water conservancy target tracking network model is a model trained with sample detection box images as input and the state of sample water conservancy targets in the sample detection box images as output. The counting module is used to calculate the number of reference water conservancy targets based on the state of the water conservancy targets in all the detection box images using the line-crossing counting method, and to obtain the position information of the reference water conservancy targets in the reference image when counting; the reference water conservancy targets are water conservancy targets in the tracking state; the reference image is the target image when the reference water conservancy targets pass through the preset counting line; The real geographic location acquisition module is used to input the location information of the reference water conservancy target in the reference image and the POS data corresponding to the reference image into the spatial positioning model to obtain the real geographic location of the reference water conservancy target; The POS data includes the current time, UAV spatial location information, UAV attitude information, and camera attitude information; The spatial positioning model is a model trained by taking the location information of the sample reference water conservancy target in the sample reference image and the sample POS data corresponding to the sample reference image as input, and taking the real geographical location of the sample reference water conservancy target as output. The determination module is used to determine whether the reference water conservancy target is within the preset river management range based on the actual geographical location of the reference water conservancy target and the preset river management range line. If it is, the reference water conservancy target is determined to be an illegal act; if not, the reference water conservancy target is determined to be a non-illegal act.

10. The river and lake monitoring system based on unmanned aerial vehicles (UAVs) according to claim 9, characterized in that, Also includes: Display module; The display module is used to display the actual geographical location of the reference water conservancy target that is determined to be an illegal act.

Images