Intelligent river patrol method based on unmanned aerial vehicle

Through the drone, video data is collected and combined with Gauss-Kruger projection and three-dimensional camera coordinate system technology, the river management range line is identified, combined with semantic segmentation and object detection model, the problems of inefficiency and inaccurate identification in the existing technology are solved. , achieving more accurate identification of river patrol events.

CN119007052AInactive Publication Date: 2024-11-22HANGZHOU DINGCHUAN INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411488743.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-24
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing drone river patrol technology is inefficient and the river and lake management scope cannot be accurately determined, resulting in inaccurate identification of incidents such as illegal buildings and random garbage piles.

Method used

The drone collects video data, analyzes image frames, identifies and marks the river management range area, and uses Gauss-Kruger projection coordinates and three-dimensional camera coordinate system to map the river management range line to the image frame, divides the area, and combines semantic segmentation. , target detection and classification models identify targets, filter targets outside the scope of river management.

Benefits of technology

It improves the accuracy of target event recognition of intelligent river patrols, can automatically identify river management scope and filter out targets outside the range, and enhances the accuracy of identification of river patrols.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007052A_ABST
    Figure CN119007052A_ABST
Patent Text Reader

Abstract

The invention discloses an intelligent river patrol method based on an unmanned aerial vehicle. The method comprises the following steps: collecting video data of a target area through the unmanned aerial vehicle; analyzing the video data to obtain a plurality of image frames; recognizing a recognition target in each image frame; determining a river management range area in each image frame; filtering a recognition target located outside the river management range area; and marking the filtered target recognition result in each image frame to generate an unmanned aerial vehicle river patrol video. According to the intelligent river patrol method based on the unmanned aerial vehicle, the river management line is mapped to each image frame in a projection mode, so that the river management range area in the image frames is recognized, targets outside the river management range area are filtered out, and the recognition accuracy of target events of intelligent river patrol is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent river patrol, and in particular relates to an intelligent river patrol method based on unmanned aerial vehicle (UAV). Background Art

[0002] With the continuous advancement of science and technology, Unmanned Aerial Vehicle (UAV) technology has been widely used in many fields, one of which is the monitoring and management of rivers and lakes. Drone river patrol technology has gradually become an important means of river and lake patrol due to its efficiency, flexibility and cost-effectiveness. The river and lake management scope refers to the management area where non-water activities are restricted to ensure the smooth flow of rivers and lakes, the stability of river flow and lake banks, the health of the ecology and the safety of water conservancy projects. The river and lake management scope line refers to the outer boundary line of the management scope of rivers, lakes and water conservancy projects. It is composed of the geographical coordinates of the control points delineated along the outer boundary line through surveying and mapping technology.

[0003] River patrol events generally include illegal buildings, random garbage dumps, floating objects on the river surface, personnel supervision, illegal sand mining, illegal sewage discharge, etc. within the scope of river and lake management. Some current drone river patrol applications play videos, manually view frame by frame, take screenshots of river patrol events, and record geographic coordinates. Some use artificial intelligence algorithms to intelligently identify some river patrol events, such as floating objects on the river surface, illegal sand mining, illegal sewage discharge, etc. The above existing technologies have the following problems:

[0004] 1. Manual river patrols are inefficient and require comparison with remote sensing images to determine the scope of river and lake management, which is cumbersome;

[0005] 2. Artificial intelligence algorithm river patrols are also unable to determine the management boundary lines of rivers and lakes, so they cannot accurately generate river patrol events for illegal buildings, random garbage dumps, etc. Summary of the invention

[0006] The present invention provides an intelligent river patrol method based on drones to solve the above-mentioned technical problems, and specifically adopts the following technical solutions:

[0007] An intelligent river patrol method based on drones, comprising:

[0008] Collect video data of the target area through drones;

[0009] Parsing the video data to obtain a number of image frames;

[0010] Identifying an identification target in each of the image frames;

[0011] Determine the river management scope area in each of the image frames;

[0012] Filtering identified targets outside the river management area;

[0013] Annotate the filtered target recognition results in each of the image frames to regenerate the UAV river patrol video;

[0014] The specific method for determining the river management range area in each image frame is:

[0015] Acquire a river management range line, and map the river management line to each of the image frames, thereby dividing each of the image frames into the river management range area and the non-river management range area;

[0016] The specific method for obtaining the river management range line is:

[0017] The geographical coordinates of the river management range line are converted into Gauss-Krüger projection coordinates, and the converted river management range line projection coordinates are composed of a number of line segments;

[0018] The transformed projection coordinates of the river management range line are merged to obtain two line segments distributed on both sides of the river.

[0019] Furthermore, the specific method of mapping the river management line to each of the image frames is:

[0020] Obtain UAV flight data and UAV parameters;

[0021] Establish a 3D camera coordinate system;

[0022] Convert the transformed projection coordinates into the camera coordinate system coordinates;

[0023] Convert the converted camera coordinate system coordinates into pixel coordinates on the image frame;

[0024] The image frame is divided into the river management range area and the non-river management range area according to the obtained pixel coordinates on the image frame.

[0025] Furthermore, the acquired UAV flight data includes the UAV latitude and longitude coordinates, flight altitude, pitch angle and yaw angle corresponding to the image frame, and the acquired UAV parameters include camera pixel size, millimeter focal length, image principal point coordinates, image width and image height;

[0026] Establishing the three-dimensional camera coordinate system based on the latitude and longitude coordinates of the drone and the flight altitude;

[0027] Converting the converted projection coordinates into camera coordinate system coordinates based on the latitude and longitude coordinates of the drone, the flight altitude, the pitch angle, and the yaw angle;

[0028] The converted camera coordinate system coordinates are converted into pixel coordinates on an image frame based on the camera pixel size, the millimeter focal length, the image principal point coordinates, the image width and the image height.

[0029] Furthermore, before determining the river management range area in each of the image frames, the intelligent river patrol method based on a drone further comprises:

[0030] Image correction is performed on each of the image frames using distortion parameters.

[0031] Furthermore, the specific method of identifying the identification target in each of the image frames is:

[0032] The identified targets include buildings, garbage dumps, water bodies, water body colors, ships, people and floating objects;

[0033] Identify buildings, garbage dumps, and water bodies through semantic segmentation models;

[0034] Identify ships, people, and floating objects through object detection models;

[0035] Identify water color using an object classification model.

[0036] Further, for the ship and personnel detection targets, target tracking processing is performed and a first detection result is output;

[0037] For the color detection results of buildings, garbage dumps, floating objects and water bodies, based on the moving distance and moving angle of the drone, if the moving distance and angle exceed a certain threshold and there are new detection targets in the image frame, the second detection result is output.

[0038] Furthermore, for the ship and personnel detection targets, the specific method of outputting the first detection result is:

[0039] The target detection results of each image frame are traversed, and the sort multi-target tracking algorithm is used to extract the image frame with the target bounding box center point coordinates closest to the ideal display position and complete size in the target trajectory as the first detection result of the current target.

[0040] Furthermore, for the detection results of the colors of buildings, garbage dumps, floating objects and water bodies, the specific method of outputting the second detection result is:

[0041] Traverse each of the image frames. If the above-mentioned detection target is newly added to the image frame, determine whether it has a historical snapshot frame. If there is no historical snapshot frame, directly save the image frame as an event snapshot frame. If there is a historical snapshot frame, calculate the distance between the geographic coordinate position of the drone in the current image frame and the geographic coordinate position of the previous historical snapshot frame. If the moving distance is greater than 1 / 3 of the height, or the moving angle is greater than 12 degrees, save the image frame as an event snapshot frame as the second detection result of the current target.

[0042] The benefit of the present invention lies in the provided drone-based intelligent river patrol method, which maps the river management line to each of the image frames by projection, thereby identifying the river management range area in the image frame, thereby filtering out targets outside the river management range area, thereby improving the recognition accuracy of target events in intelligent river patrol. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0044] Figure 1 is a schematic diagram of an intelligent river patrol method based on a drone of the present invention;

[0045] Figure 2 It is a schematic diagram of the river management range line of the present invention. DETAILED DESCRIPTION

[0046] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0047] like Figure 1The present application shows an intelligent river patrol method based on a drone, comprising: S1: collecting video data of the target area through a drone. S2: parsing the video data to obtain a number of image frames. S3: determining the river management range area in each image frame. S4: identifying the identified targets in each image frame. S5: filtering the identified targets outside the river management range area. S6: marking the filtered target recognition results in each image frame to regenerate the drone river patrol video. The intelligent river patrol method based on a drone of the present application can automatically identify the river management range area in the image frame, and based on this, filter out targets outside the river management range area, thereby improving the recognition accuracy of target events of the intelligent river patrol. The above steps are described in detail below.

[0048] For step S1: collect video data of the target area through a drone.

[0049] Specifically, the drone is operated to collect video data at a certain height along both sides of the target river channel.

[0050] For step S2: parsing the video data to obtain a number of image frames.

[0051] The video data is decoded by a decoding algorithm to obtain a plurality of image frames. In the implementation of the present application, the video data is specifically decoded by an opencv decoding algorithm to decode the video data into images.

[0052] For step S3: determine the river management area in each image frame.

[0053] In the implementation manner of the present application, the specific method for determining the river management range area in each image frame is:

[0054] The river management range line is obtained, and the river management line is mapped to each image frame, so as to divide each image frame into a river management range area and a non-river management range area.

[0055] The river management range line is composed of two or more line segments, which are distributed on both sides of the river management range boundary. Each line segment is connected by multiple geographic coordinate control points.

[0056] In the present application, the geographic coordinates of the river management range line are first converted into Gauss-Krüger projection coordinates, and the converted river management range line projection coordinates are composed of a number of line segments.

[0057] Next, the transformed river management range line projection coordinates are merged to obtain two line segments distributed on both sides of the river. The result of merging the line segments is as follows: Figure 2 shown.

[0058] In an embodiment of the present application, the steps of merging line segments are as follows:

[0059] A: Traverse the starting and ending points of all line segments, and determine the starting and ending points of line segment A and line segment B which are the farthest apart.

[0060] B: Traverse the start and end points of all line segments, determine the closest start and end points of line segment C and line segment D, and the start and end points of the closest line segments C and D do not overlap with the start and end points of the farthest line segments A and B. Connect the found start and end points of line segment C and line segment D.

[0061] C: Repeat steps A and B until the number of line segments obtained is 2, then exit. At this time, the river management range line obtained is two line segments along both sides of the river.

[0062] Furthermore, the specific method of mapping the river management line to each image frame is as follows:

[0063] Obtain UAV flight data (SRT file) and UAV parameters. The obtained UAV flight data includes the UAV latitude and longitude coordinates, flight altitude, pitch angle and yaw angle corresponding to the image frame, and the obtained UAV parameters include camera pixel size, millimeter focal length, image principal point coordinates (cx, cy), image width and image height.

[0064] Establish a 3D camera coordinate system. Establish a 3D camera coordinate system based on the longitude and latitude coordinates and flight altitude of the drone. Specifically, the center of the 3D coordinate system (0, 0, 0) is obtained by combining the GPS coordinates of the drone position (x_c, y_c) and its flight altitude. The camera lens is vertically downward, with the lens optical axis direction as the Z axis, the drone flight direction as the Y axis, and the vertical drone flight direction as the X axis.

[0065] The converted projection coordinates are converted into camera coordinate system coordinates. In the implementation of the present application, the converted projection coordinates are converted into camera coordinate system coordinates based on the longitude and latitude coordinates, flight altitude, pitch angle and yaw angle of the drone. Specifically, taking the projection coordinates (x_1, y_1) as an example, the point coordinates in the three-dimensional coordinate system are (x', y', z'), where the x' value is x_1 -x_c, the y' value is y_1 - y_c, and the z' value is the flight altitude. The yaw angle and pitch angle are then rotated using the following formula.

[0066] Yaw rotation (rotation around the Z axis):

[0067]

[0068] Pitch rotation (rotation around the x-axis):

[0069]

[0070] The converted camera coordinate system coordinates are converted into pixel coordinates on the image frame. In the present application, the converted camera coordinate system coordinates are converted into pixel coordinates on the image frame based on the camera pixel size, millimeter focal length, image principal point coordinates, image width and image height.

[0071] Specifically, let fx and fy be the focal length of the camera pixel, and fx = fy = focal length in millimeters / camera pixel. img_width and img_height are the image width and height of the drone video, and capture_width and capture_height are the original width and height of the image taken by the drone. It is understandable that the resolution of the video recorded by the camera is different from the resolution of the camera photo. For example, the video recorded by the camera has a resolution of 1920*1080, but the camera photo resolution is 5280*3956. The entire three-dimensional space coordinate system is converted to a two-dimensional image coordinate system, which corresponds to the image coordinate system taken by the camera, and it also needs to be converted into the coordinate system of the video image. The conversion formula is,

[0072] pixel_x = (x * fx / z + cx) / capture_width * img_width

[0073] pixel_y = (y * fy / z + cy) / capture_height * img_height

[0074] The x, y, and z in the formula refer to the three-dimensional space coordinates (x', y', and z'), which are the x, y, and z values ​​obtained after the yaw angle and pitch angle are converted.

[0075] The image frame is divided into the river management area and the non-river management area according to the pixel coordinates on the image frame. The image frame is on both sides of the river, and the line formed by the projected pixel coordinates divides the image into two parts. The part containing the water area is the river management area. The part not containing the water area is the non-river management area.

[0076] Specifically, according to the pixel coordinates of the two river management lines on the obtained image frame, they are recorded as the first line segment and the second line segment, each line segment has endpoints 1 and 2, and the following steps are used to connect the pixel coordinates of the two line segments into a quadrilateral:

[0077] A: connect endpoint 1 of the first line segment and endpoint 1 of the second line segment, record it as line segment line1, connect endpoint 2 of the first line segment and endpoint 2 of the second line segment, record it as line segment line2;

[0078] B: Calculate the intersection of line1 and line2. If there is no intersection, the connection of the quadrilateral is first line segment endpoint 1, second line segment endpoint 1, second line segment endpoint 2, first line segment endpoint 2, a total of four vertices. If there is an intersection, the connection of the quadrilateral is first line segment endpoint 1, second line segment endpoint 2, second line segment endpoint 1, first line segment endpoint 2, a total of four vertices.

[0079] According to the quadrilateral obtained by the above strategy, for each pixel point of the image frame, it can be judged whether it is within the quadrilateral. If it is within the quadrilateral, it belongs to the river management area, and if it is outside the quadrilateral, it does not belong to the river management area.

[0080] It can be understood that in the implementation manner of the present application, before determining the river management range area in each image frame, the drone-based intelligent river patrol method also includes: performing image correction on each image frame through distortion parameters.

[0081] For step S4: identify the identification target in each image frame.

[0082] In the implementation of the present application, the recognition targets include buildings, garbage dumps, water bodies, water body colors, ships, people and floating objects. The specific method of identifying the recognition targets in each image frame is:

[0083] Buildings, garbage dumps and water bodies are identified by a semantic segmentation model. In an embodiment of the present application, the semantic segmentation model adopts a segformer model.

[0084] Ships, personnel and floating objects are identified through the target detection model. Specifically, the target detection model is the yolov5 model.

[0085] The water body color is identified by the target classification model. Specifically, the target classification model adopts the Resnet model.

[0086] For step S5: filter the identified targets outside the river management area.

[0087] The identified target identified in step S4 may be located within the river management area or outside the river management area. Further, the identified targets outside the river management area are filtered out, and only the part within the river management area is retained. In this process, it is preferred to further filter out the misdetected targets by using the rule that ships and floating objects on the river surface are at the edge or center of the water body.

[0088] In the drone video image, for the detection targets of ships and personnel, a frame of image needs to be saved as an event record. In the implementation mode of the present application, for the detection targets of ships and personnel, target tracking processing is performed and the first detection result is output. Specifically, the target detection results of each image frame are traversed, and the sort multi-target tracking algorithm for target tracking is used to calculate the picture frame name and pixel bounding box of each tracked target. The pixel bounding box includes the pixel coordinates of the upper left vertex, the pixel width and height of the bounding box, and the image width is recorded as width and the image height is recorded as height. In the target trajectory, the image frame whose coordinates of the center point of the target bounding box are closest to the ideal display position (the pixel coordinates are defined as (height / 4, width / 2) as the ideal position in this application) and whose size is complete (the width and height of the bounding box of the target defined in this application are both greater than 80% of the average width and height in the target trajectory as the size is complete) is extracted as the first detection result of the current target.

[0089] In drone video images, for buildings, garbage dumps, floating objects, and water bodies with abnormal colors (black, brown), one frame of image needs to be saved for each target as an event record. For the color detection results of buildings, garbage dumps, floating objects, and water bodies, according to the moving distance and moving angle of the drone, if the moving distance and angle exceed a certain threshold and there are new detection targets in the image frame, the second detection result is output.

[0090] Specifically, each image frame is traversed. If the above-mentioned detection target is newly added in the image frame, it is determined whether there is a historical snapshot frame. If there is no historical snapshot frame, this frame is directly saved as an event snapshot frame. If there is a historical snapshot frame, the distance between the geographic coordinate position of the drone in the current image frame and the geographic coordinate position of the previous snapshot frame is calculated. If the moving distance is greater than 1 / 3 of the height, or the moving angle is greater than 12 degrees, this frame is saved as an event snapshot frame as the second detection result of the current target.

[0091] Height H = camera pixel / mm focal length * flight altitude * original altitude of the drone image / 1000, in meters. The moving angle of the drone is calculated from the flight data by calculating the pitch angle difference and yaw angle difference of the two frames of images, and taking the maximum difference, in degrees.

[0092] The first detection result and the second detection result mentioned above are both river patrol events.

[0093] For step S6: annotate the filtered target recognition results in each image frame to regenerate the UAV river patrol video.

[0094] Specifically, the detection results of each frame are superimposed and displayed in the image, and finally a drone river patrol video is generated.

[0095] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. An intelligent river patrol method based on drones, characterized in that: Include: Collect video data of the target area through drones; Parsing the video data to obtain a number of image frames; Identifying an identification target in each of the image frames; Determine the river management scope area in each of the image frames; Filtering identified targets outside the river management area; Annotate the filtered target recognition results in each of the image frames to regenerate the UAV river patrol video; The specific method for determining the river management range area in each image frame is: Acquire a river management range line, and map the river management line to each of the image frames, thereby dividing each of the image frames into the river management range area and the non-river management range area; The specific method for obtaining the river management range line is: The geographical coordinates of the river management range line are converted into Gauss-Krüger projection coordinates, and the converted river management range line projection coordinates are composed of a number of line segments; The transformed projection coordinates of the river management range line are merged to obtain two line segments distributed on both sides of the river.

2. The intelligent river patrol method based on drone according to claim 1 is characterized in that: The specific method of mapping the river management line to each of the image frames is: Obtain UAV flight data and UAV parameters; Establish a three-dimensional camera coordinate system; Convert the transformed projection coordinates into the camera coordinate system coordinates; Convert the converted camera coordinate system coordinates into pixel coordinates on the image frame; The image frame is divided into the river management range area and the non-river management range area according to the obtained pixel coordinates on the image frame.

3. The intelligent river patrol method based on drone according to claim 2 is characterized in that: The acquired UAV flight data includes the UAV latitude and longitude coordinates, flight altitude, pitch angle and yaw angle corresponding to the image frame, and the acquired UAV parameters include camera pixel size, millimeter focal length, image principal point coordinates, image width and image height; Establishing the three-dimensional camera coordinate system based on the latitude and longitude coordinates of the drone and the flight altitude; Converting the converted projection coordinates into camera coordinate system coordinates based on the longitude and latitude coordinates of the drone, the flight altitude, the pitch angle, and the yaw angle; The converted camera coordinate system coordinates are converted into pixel coordinates on an image frame based on the camera pixel size, the millimeter focal length, the image principal point coordinates, the image width and the image height.

4. The intelligent river patrol method based on drone according to claim 1 is characterized in that: Before determining the river management range area in each of the image frames, the intelligent river patrol method based on a drone further comprises: Image correction is performed on each of the image frames using distortion parameters.

5. The intelligent river patrol method based on drone according to claim 1 is characterized in that: The specific method of identifying the identification target in each of the image frames is: The identified targets include buildings, garbage dumps, water bodies, water body colors, ships, people and floating objects; Identify buildings, garbage dumps, and water bodies through semantic segmentation models; Identify ships, people, and floating objects through object detection models; Identify water color using an object classification model.

6. The intelligent river patrol method based on drone according to claim 5 is characterized in that: For ship and personnel detection targets, target tracking processing is performed and a first detection result is output; For the color detection results of buildings, garbage dumps, floating objects and water bodies, based on the moving distance and moving angle of the drone, if the moving distance and angle exceed a certain threshold and there are new detection targets in the image frame, the second detection result is output.

7. The intelligent river patrol method based on drone according to claim 6 is characterized in that: For ship and personnel detection targets, the specific method of outputting the first detection result is: The target detection results of each image frame are traversed, and the sort multi-target tracking algorithm is used to extract the image frame with the target bounding box center point coordinates closest to the ideal display position and complete size in the target trajectory as the first detection result of the current target.

8. The intelligent river patrol method based on drone according to claim 6 is characterized in that: For the building, garbage dump, floating object and water color detection results, the specific method of outputting the second detection result is: Traverse each of the image frames. If the above-mentioned detection target is newly added to the image frame, determine whether it has a historical snapshot frame. If there is no historical snapshot frame, directly save the image frame as an event snapshot frame. If there is a historical snapshot frame, calculate the distance between the geographic coordinate position of the drone in the current image frame and the geographic coordinate position of the previous historical snapshot frame. If the moving distance is greater than 1 / 3 of the height, or the moving angle is greater than 12 degrees, save the image frame as an event snapshot frame as the second detection result of the current target.

Citation Information

Patent Citations

  • River and lake supervision method and system based on unmanned aerial vehicle

    CN115690628A

  • Lofting method, device and equipment and storage medium

    CN116399310A

  • Attention mechanism-based river channel illegal building detection method

    CN116883843A

  • River and lake disorder automatic identification method and device, electronic equipment and storage medium

    CN118500357A

Cited By

  • Unmanned aerial vehicle-based river channel management method and computer readable storage medium

    CN120726493A