UAV water and shoreline inspection method and computer-readable storage medium

Abstract

The present invention discloses a method for inspecting water shorelines using a drone and a computer-readable storage medium. The method comprises: controlling a drone to fly along the water shoreline to be inspected; obtaining a video stream captured by the drone during flight, identifying video frames in the video stream, and recording the drone's position offset and yaw angle offset between the capture times of adjacent identified video frames; performing image recognition on each identified video frame according to a preset recognition task to obtain a recognition result, wherein the recognition result includes a target point and a target area; merging recognition results belonging to the same event and recording the drone's positioning position and yaw angle corresponding to each recognition result in the same event; and mapping each event onto a map based on each recognition result in each event and its corresponding drone positioning position and yaw angle. The present invention can automatically complete river patrols according to the recognition task, improving the versatility of drone inspections and the accuracy and efficiency of recognition.

Description

Technical Field

[0001] The present invention relates to the field of patrol technology, and in particular to a method for patrolling waters and shorelines with an unmanned aerial vehicle (UAV) and a computer-readable storage medium. Background Art

[0002] In water and shoreline monitoring scenarios, riverside video surveillance stations and drone patrols are commonly used to obtain image and video data. Image processing techniques are then used to identify events requiring early warning. Video surveillance stations offer the advantage of continuous monitoring over long periods of time, but their spatial coverage is limited. Drone patrols, on the other hand, offer the advantage of truly extensive spatial coverage, but their duration is very short. Therefore, these two monitoring methods complement each other. Training models based on sample sets for specific recognition tasks is considered standard practice for these image recognition tasks. Common tasks include object detection and image segmentation, which process single frames to identify key objects as the basis for event determination. The perspective of video surveillance footage is similar to that of ordinary human observation, making image samples relatively easy to collect online, from everyday life, or using readily available open source datasets. However, drones offer a wide range of viewing angles. The features of images viewed from above differ significantly from those from a normal viewing angle. Models trained on sample sets from a normal viewing angle cannot be directly applied to scenarios viewed from above. Furthermore, while drones' high altitudes can expand coverage, the features of target objects are often too small, making recognition less accurate. Such drone perspective samples are not easy to obtain and need to be specially produced, so the types of tasks that can be monitored are very limited. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method for patrolling water and shorelines with a drone and a computer-readable storage medium, which can automatically complete river patrols according to identification tasks, thereby improving the versatility of drone inspections and the accuracy and efficiency of identification.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for patrolling waters and coastlines using a drone, comprising:

[0005] Control the drone to fly along the coastline of the water area to be inspected;

[0006] Obtaining a video stream captured by the drone during flight, determining identification video frames in the video stream, and recording the position offset and yaw angle offset of the drone between the shooting moments of adjacent identification video frames;

[0007] According to the preset recognition task, image recognition is performed on each recognition video frame to obtain a recognition result, which includes a target point and a target area;

[0008] Merge the recognition results belonging to the same event and record the drone positioning position and yaw angle corresponding to each recognition result in the same event;

[0009] Each event is mapped onto a map based on the recognition results in each event and its corresponding UAV positioning position and yaw angle.

[0010] The present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method when the program is executed by a processor.

[0011] The beneficial effects of the present invention are that it can automatically complete river patrols according to recognition tasks. By performing recognition tasks on only part of the video frames in the video stream and automatically merging the recognition results, it can skip repeated coverage areas, reduce computing power consumption, and reduce the complexity of manual operations, thereby improving recognition accuracy and efficiency, and better completing the drone shore patrol mission; at the same time, the recognition results can also be mapped to a map for easy access to a GIS system or a digital twin system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a flow chart of a method for inspecting waters and shorelines using a drone according to the present invention;

[0013] Figure 2 Schematic diagram of the inverse mapping transformation according to the first embodiment of the present invention;

[0014] Figure 3 This is a flow chart of the method for the first stage of embodiment 1 of the present invention;

[0015] Figure 4 This is a schematic diagram of yaw angle adjustment when flying along a water coastline according to the first embodiment of the present invention;

[0016] Figure 5 This is a schematic diagram of yaw angle adjustment when flying to the opposite bank according to the first embodiment of the present invention;

[0017] Figure 6 This is a flow chart of the method for the second stage of embodiment 1 of the present invention;

[0018] Figure 7 Schematic diagram of the shooting screen and monitoring coverage area of ​​the first embodiment of the present invention;

[0019] Figure 8 Schematic diagram of monitoring coverage areas at different times in the first embodiment of the present invention;

[0020] Figure 9 This is a schematic diagram of the location where a UAV flight event occurs according to the first embodiment of the present invention;

[0021] Figure 10This is a schematic diagram of mapping events to a map according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0022] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the embodiments and accompanying drawings.

[0023] See also Figure 1 , a method for inspecting water and shoreline using a drone, comprising:

[0024] Control the drone to fly along the coastline of the water area to be inspected;

[0025] Obtaining a video stream captured by the drone during flight, determining identification video frames in the video stream, and recording the position offset and yaw angle offset of the drone between the shooting moments of adjacent identification video frames;

[0026] According to the preset recognition task, image recognition is performed on each recognition video frame to obtain a recognition result, which includes a target point and a target area;

[0027] Merge the recognition results belonging to the same event and record the drone positioning position and yaw angle corresponding to each recognition result in the same event;

[0028] Each event is mapped onto a map based on the recognition results in each event and its corresponding UAV positioning position and yaw angle.

[0029] From the above description, it can be seen that the beneficial effects of the present invention are: river patrol can be automatically completed according to the identification task, and the versatility of drone inspection as well as the accuracy and efficiency of identification can be improved.

[0030] Furthermore, it also includes:

[0031] At a moment in the drone's flight, the world coordinate system corresponding to the moment is constructed, with the vertical projection position of the drone on the plane where the water surface is located as the origin, the plane where the water surface is located as the XOY plane, the direction of the drone's camera as the Y-axis direction, and the vertical direction as the Z-axis direction.

[0032] As can be seen from the above description, the world coordinate system corresponding to different moments will change as the drone's position on the horizontal plane and the camera's orientation change.

[0033] Furthermore, the determining and identifying video frames in the video stream is specifically:

[0034] Sequentially acquiring a video frame from the video stream as a current video frame;

[0035] If the current video frame is the first video frame, then the current video frame is used as the recognition video frame, the shooting time of the current video frame is used as the recognition time, and the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to its shooting time is determined. Otherwise, based on the speed sensor information of the UAV, the position offset of the UAV between the shooting time of the current video frame and the latest recognition time is calculated, and the yaw angle offset of the UAV between the shooting time of the current video frame and the latest recognition time is obtained;

[0036] Determine the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the shooting time of the current video frame, and determine the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the latest recognition time based on the position offset and the yaw angle offset;

[0037] Calculate the intersection area of ​​the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the latest recognition time and the monitoring coverage area of ​​the latest recognized video frame in the world coordinate system corresponding to the latest recognition time;

[0038] If the intersection area is smaller than a preset area threshold, the current video frame is used as the recognition video frame, and the shooting time of the current video frame is used as the recognition time.

[0039] From the above description, it can be seen that the recognition video frame used to perform the recognition task is determined according to the intersection area of ​​the monitoring coverage areas between different video frames, avoiding recognition of each video frame, obtaining more repeated recognition results, and saving computing resources.

[0040] Furthermore, the determination of the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the shooting time thereof is specifically as follows:

[0041] The two vertices of the bottom edge of the current video frame are respectively used as the first vertex and the second vertex, and the world coordinates of the first vertex and the second vertex in the world coordinate system corresponding to the shooting time of the current video frame are respectively calculated using the inverse perspective mapping formula;

[0042] Determine, based on the minimum value of the farthest recognition distances for each recognition task, a third vertex and a fourth vertex on the left and right sides of the current video frame, respectively, where, in the world coordinate system corresponding to the capture moment of the current video frame, the distance between the world coordinates of the third vertex and the fourth vertex and the origin is the minimum value;

[0043] According to the world coordinates of the first vertex, the second vertex, the third vertex and the fourth vertex in the world coordinate system corresponding to the shooting moment of the current video frame, the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the shooting moment of the current video frame is determined.

[0044] As can be seen from the above description, at a certain moment during a drone's stable flight, the actual coverage area of ​​its captured image on the XOY plane of the world coordinate system is an inverted trapezoid. If the horizon is in the frame, the long side of this inverted trapezoid is at infinity. In this case, it is necessary to truncate it using the maximum recognition distance of each recognition task.

[0045] Furthermore, the method further includes: performing image recognition on each recognition video frame according to the preset recognition task, and obtaining the recognition result;

[0046] Calculating, based on the image coordinates of a target point in a recognition result of a recognition video frame, the world coordinates of the target point in a world coordinate system corresponding to the time when the recognition video frame was shot using an inverse perspective mapping formula, and calculating, in the world coordinate system corresponding to the time when the recognition video frame was shot, the true distance between the target point and the origin and the recognition angle of the drone corresponding to the target point;

[0047] If the actual distance is greater than the farthest recognition distance corresponding to the recognition task, or the recognition angle is not within the recognition angle range corresponding to the recognition task, the recognition result of the recognition video frame is discarded.

[0048] As can be seen from the above description, by discarding recognition results whose recognition distance and recognition angle do not meet the requirements, the number of recognition results is simplified and the processing efficiency of subsequent recognition results is improved.

[0049] Furthermore, the identification results belonging to the same event are merged, and the drone positioning position and yaw angle corresponding to each identification result in the same event are recorded, including:

[0050] Sequentially obtaining recognition results of the recognition video frames according to the shooting time of each recognition video frame, and comparing the recognition results of the recognition video frames with the existing recognition results corresponding to each event to determine whether they belong to the same event;

[0051] If the recognition result of the identified video frame belongs to an event, the recognition result of the identified video frame and the positioning position and yaw angle of the drone at the time of shooting the identified video frame are combined and recorded in the event; otherwise, the recognition result of the identified video frame is taken as a new event, and the positioning position and yaw angle of the drone at the time of shooting the identified video frame are recorded.

[0052] As can be seen from the above description, by performing event attribution analysis on the recognition results, the recognition results belonging to the same event are merged.

[0053] Furthermore, the recognition result of the identified video frame is compared with the recognition results corresponding to the existing events to determine whether they belong to the same event, specifically:

[0054] Calculating, according to the image coordinates of the recognition result of a recognition video frame, the world coordinates of the recognition result of the recognition video frame in the world coordinate system corresponding to the shooting moment of the recognition video frame by using an inverse perspective mapping formula;

[0055] Obtaining a recognition result of a newly recorded recognition video frame of an existing event as the latest recognition result of the event, and calculating the world coordinates of the latest recognition result in the world coordinate system corresponding to the shooting time using an inverse perspective mapping formula;

[0056] Determine, based on a position offset and a yaw angle offset of the drone between the shooting time of the recognition video frame and the latest recognition video frame recorded in the event, the world coordinates in the world coordinate system corresponding to the recognition result of the recognition video frame and the latest recognition result at the same time;

[0057] Calculating, based on the world coordinates of the world coordinate system corresponding to the recognition result of the first recognition video frame and the latest recognition result at the same time, an offset distance between a target point in the recognition result of the first recognition video frame and a target point in the latest recognition result, or calculating an intersection-over-union ratio between a target area in the recognition result of the first recognition video frame and a target area in the latest recognition result;

[0058] If the offset distance is smaller than a preset distance threshold, or the intersection-over-union ratio is larger than a preset ratio threshold, it is determined that the recognition result of the recognized video frame belongs to the event.

[0059] From the above description, it can be seen that the event attribution analysis of the recognition results is achieved based on the distance or intersection-over-union ratio of different recognition results in the same world coordinate system.

[0060] Furthermore, the step of merging the recognition results belonging to the same event and recording the drone positioning position and yaw angle corresponding to each recognition result in the same event also includes:

[0061] If the recognition result of the identified video frame belongs to an existing event, then the reliability of the event is increased according to the reliability change value corresponding to the event, and the reliability of other existing events is reduced according to the reliability change values ​​corresponding to the other existing events; otherwise, the reliability of each existing event is reduced according to the reliability change value corresponding to each event;

[0062] The reliability change value corresponding to an event is the reciprocal of the distance in the world coordinate system corresponding to the vertical projection position of the latest recognition result in the event and the vertical projection position when the drone shoots the recognition video frame at the same time.

[0063] From the above description, we can see that the closer the distance between the drone and the location of the incident, the clearer the image captured and the higher its reliability.

[0064] Furthermore, each event is mapped onto a map according to each recognition result in each event and its corresponding drone positioning position and yaw angle, specifically:

[0065] If the latest recognition result in an event exceeds the visual coverage area of ​​the current drone and the reliability of the event is greater than zero, the event is mapped onto the map according to each recognition result in the event.

[0066] From the above description, we can see that when the drone flies out of the area where the event occurs, the event will no longer be updated. It will be mapped to the map in time to avoid excessive computational complexity when traversing existing events later.

[0067] Furthermore, mapping the event onto a map according to each recognition result in the event is specifically as follows:

[0068] The map coordinates of each recognition result on the map are calculated according to the world coordinates of each recognition result in the event in the world coordinate system corresponding to the shooting time and the drone positioning position and yaw angle corresponding to each recognition result.

[0069] From the above description, it can be seen that the results can be accurately projected onto the overhead plane, making it convenient to connect the results to the GIS system or digital twin system.

[0070] Furthermore, the controlling of the drone to fly along the coastline of the water area to be inspected includes:

[0071] Determine the flight altitude of the drone based on the preset maximum recognition distance of each recognition task;

[0072] Determining an initial flight position of the drone according to the flight altitude, a preset pitch angle, and a preset initial orientation, and controlling the drone to fly forward from the initial flight position according to a preset first flight speed;

[0073] According to a preset first adjustment period, the real-time image captured by the UAV is subjected to water surface recognition, and the yaw angle is adjusted according to the water surface recognition result so that the UAV flies along the shoreline of the water area to be inspected;

[0074] When the flight time reaches a preset time threshold, the real-time image captured by the drone is used for water surface recognition, and the yaw angle is adjusted according to the water surface recognition result so that the drone flies to the opposite bank and flies along the shoreline of the water on the opposite bank.

[0075] From the above description, it can be seen that automatic flight control of the drone can be achieved without the need for manual route setting. It is more convenient in new environments or temporary patrol scenarios, and is also more suitable for situations where the water surface coastline changes greatly.

[0076] Furthermore, the flight altitude of the UAV is determined according to the preset maximum recognition distance of each recognition task, specifically:

[0077] Get the minimum value of the farthest recognition distance of each recognition task;

[0078] Presetting an effective area for image recognition in the captured image, and determining a pixel point on the top edge of the effective area;

[0079] The flight altitude is calculated based on the image coordinates of the pixel point in the captured image, the minimum value, a preset set of equations and a pitch angle. The set of equations includes an inverse perspective mapping formula and a distance calculation formula. The inverse perspective mapping formula is used to map the pixel point in the captured image to the world coordinate system corresponding to the shooting moment of the captured image. The distance calculation formula is constructed based on the world coordinates of the pixel point in the world coordinate system corresponding to the shooting moment of the captured image and the minimum value.

[0080] From the above description, it can be seen that the minimum value of the longest recognition distances of all recognition tasks is used as a reference for calculating the flight altitude of the drone, so that a part of the imaging screen (such as the lower half) becomes the effective recognition area.

[0081] Furthermore, according to the preset first adjustment period, the real-time image captured by the drone is subjected to water surface recognition, and the yaw angle is adjusted according to the water surface recognition result so that the drone flies along the coastline of the water area to be inspected, specifically:

[0082] According to a preset first adjustment period, the latest captured image is obtained as the current image, and the world coordinate system corresponding to the capturing moment of the current image is used as the current world coordinate system;

[0083] Performing image segmentation on the current image to obtain a water surface area, and performing contour extraction on the water surface area to obtain a contour point set of the water surface area;

[0084] Traversing the contour point set, performing inverse perspective transformation on each contour point in turn, and obtaining the world coordinates of each contour point in the current world coordinate system;

[0085] Calculate the distance between each contour point and the origin of the current world coordinate system respectively, and use the contour point whose distance from the origin of the current world coordinate system is a preset first distance as the direction anchor point;

[0086] Calculate the angle between the line between the origin of the current world coordinate system and the direction anchor point and the Y axis of the current world coordinate system;

[0087] The yaw angle is adjusted in a current first adjustment period according to the included angle.

[0088] From the above description, it can be seen that by using the water surface segmentation result in the image to control the flight direction of the drone, only the flight action is controlled without requiring the drone to have a precise state, which has a high tolerance.

[0089] Furthermore, the image segmentation is performed on the current image to obtain the water surface area, and the contour extraction is performed on the water surface area to obtain the contour point set of the water surface area, specifically:

[0090] Performing image segmentation on the current image to obtain at least one water surface area;

[0091] Extract the contours of each water surface area and calculate the area of ​​each contour;

[0092] Get the contour point set of the contour with the largest area.

[0093] From the above description, it can be seen that by obtaining the water surface area with the largest contour area, interference areas such as small puddles can be filtered out.

[0094] Furthermore, the adjusting of the yaw angle in the current first adjustment period according to the included angle is specifically as follows:

[0095] Determining an adjustment frequency and a unit adjustment angle according to the included angle, the first adjustment period, and a preset unit time;

[0096] The yaw angle is adjusted according to the adjustment frequency and the unit adjustment angle.

[0097] It can be seen from the above description that the stability of the angle adjustment can be maintained.

[0098] Furthermore, when the flight time reaches a preset time threshold, the real-time image captured by the drone is subjected to water surface recognition, and the yaw angle is adjusted according to the water surface recognition result so that the drone flies to the opposite bank and flies along the shoreline of the water on the opposite bank, specifically:

[0099] When the flight time reaches a preset time threshold, the water surface is identified on the latest captured image to obtain the water surface area;

[0100] Determining the orientation of the water surface area relative to the drone, and determining a rotation target side based on the orientation;

[0101] Adjusting the yaw angle according to the rotation target side and the preset rotation angle so that the drone camera faces the opposite bank, and controlling the drone to fly forward according to a preset second flight speed so that the drone flies toward the opposite bank;

[0102] When the UAV flies to the other side, according to the preset recognition cycle, the latest captured image is used as the current image, and the world coordinate system corresponding to the shooting time of the current image is used as the current world coordinate system;

[0103] Perform water surface recognition and contour extraction on the current image to obtain the water surface contour;

[0104] Obtaining an intersection point of the water surface contour and a midline of the image along the width direction as a first reference point;

[0105] If the vertical coordinate value of the image coordinates of the first reference point is less than a preset height value, then calculating the world coordinates of each contour point of the water surface contour in the current world coordinate system by using an inverse perspective mapping formula, and determining a second reference point from the contour points of the water surface contour, where the second reference point is located on the rotation target side of the first reference point, and the distance between the first reference point and the second reference point in the current world coordinate system is a preset second distance;

[0106] Calculating an adjustment angle based on the world coordinates of the second reference point in the current world coordinate system, and adjusting the yaw angle based on a preset second adjustment period and the adjustment angle until an angle between a line connecting the first reference point and the second reference point in the world coordinate system corresponding to the same moment and the Y-axis of the world coordinate system corresponding to the same moment is less than a preset angle;

[0107] According to the preset first flight speed, the drone is controlled to fly forward. At the same time, according to the preset first adjustment period, the real-time captured image of the drone is subjected to water surface recognition, and the yaw angle is adjusted according to the water surface recognition result so that the drone flies along the shoreline of the water on the opposite bank.

[0108] From the above description, it can be seen that the drone can fly along one bank of the river for a period of time, then fly to the other bank and return along the other bank, thereby making full use of the drone and allowing the drone to inspect both sides of a certain section of the river in one round trip.

[0109] Furthermore, the method further includes: adjusting the yaw angle according to the rotation target side and the preset rotation angle so that the drone camera faces the opposite bank;

[0110] Obtain the latest preset number of yaw angle adjustment angles in history and accumulate them to obtain the cumulative adjustment angle;

[0111] The yaw angle is corrected according to the accumulated adjustment angle.

[0112] It can be seen from the above description that the adjustment of the yaw angle is made relative to a section of the coast rather than relative to a certain point.

[0113] The present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method when the program is executed by a processor.

[0114] Example 1

[0115] Please refer to Figure 2-10 , Embodiment 1 of the present invention is: a UAV water shoreline inspection method, which can be applied to UAV shore patrol tasks and can automatically complete river patrols according to identification tasks.

[0116] There are various patrol tasks along the river. Those based on ordinary RGB images include floating objects on the lake surface, water hyacinth and blue algae monitoring, swimming and fishing monitoring, illegal construction along the shore, garbage and slag dumping, etc. Those based on thermal imaging image channels include personnel and engineering vehicle monitoring, sand mining ship operation monitoring, etc. These image recognition tasks are all achieved by identifying the target objects in the picture, and can be classified into target detection tasks and image segmentation tasks. In this embodiment, the target detection and image segmentation methods used are both based on neural network models trained with image sample data sets. For example, yolov8 can be selected for target detection, and mask2former can be selected for image segmentation, both of which have very good results. Among them, the construction of the data set only requires the use of samples from conventional perspectives. Each recognition task has a maximum recognition distance d task and recognition angle range (θ 1 task ,θ 2 task ), the recognition angle range is within 0°~90°, and the maximum recognition distance and recognition angle range can be set based on experience.

[0117] The principle of drone image imaging is no different from that of ordinary surveillance cameras. At a certain moment, the imaging range of the water surface is determined by the flight altitude h, the pitch angle θ, and the camera field of view FOV. The pitch angle and yaw angle of the camera on the drone are determined by the drone's flight attitude and the gimbal. The target pitch angle is achieved by controlling the gimbal during stable cruising. In this embodiment, the pitch angle θ is the pitch angle of the camera relative to the ground. If the gimbal is fixed facing straight ahead, the yaw angle is determined by the orientation of the drone, and the compass direction can be directly obtained from the sensor as the yaw angle φ. The roll angle caused by shaking will cause the image to rotate, which is offset by rotating the center of the picture and will not be discussed in detail.

[0118] like Figure 2As shown, in this embodiment, at a certain moment in the drone's flight, the drone's world coordinate system corresponding to that moment is constructed, with the drone's vertical projection position on the plane where the water surface lies as the origin O, the plane where the water surface lies as the XOY plane, the drone's camera's orientation (i.e., yaw angle direction) as the Y-axis direction, and the vertical upward direction as the positive Z-axis direction. In other words, in this embodiment, the world coordinate system changes with changes in the drone's position (i.e., position in the horizontal plane) and the camera's orientation.

[0119] In addition, for the images taken by the drone, an image coordinate system is constructed with the upper left corner of the image as the origin, the right direction as the positive direction of the X axis, and the downward direction as the positive direction of the Y axis.

[0120] Using the inverse perspective mapping formula, a point (x, y) in an image captured at a given moment can be mapped to the XOY plane of the world coordinate system corresponding to that moment. The specific formula is as follows. The water surface is a flat surface, making it suitable for inverse perspective transformation. Although the ground near the shore is not strictly flush with the water surface, considering the limited detection range of the method in this embodiment, the ground can be approximated to zero height.

[0121]

[0122]

[0123] Among them, (x, y) represents the image coordinates of a pixel point in the captured image in the image coordinate system, (X p ,Y p ) represents the world coordinates corresponding to the point (i.e., the coordinates in the world coordinate system corresponding to the moment the image was taken), h represents the height of the drone camera relative to the water surface (i.e., the flight altitude of the drone), and θ represents the pitch angle of the drone camera; width image and height image represents the resolution of the captured image, that is, the width and height of the captured image; α and β are the internal parameters of the drone camera, representing the horizontal and vertical viewing angles, which are determined by the focal length of the camera.

[0124] This embodiment consists of two main phases: the first involves controlling the drone to fly along the shoreline of the water area to be inspected, and the second phase involves performing recognition tasks based on the video stream captured by the drone during flight and processing the recognition results. This second phase can be performed simultaneously with the first phase, performing the recognition task in real time based on the captured video stream. It can also be performed offline, collecting the captured video stream and sensor flight data during the first phase of flight, and then performing the second phase offline.

[0125] For the first stage, in some optional embodiments, the route can be pre-planned based on map data and terrain data, so that the drone can fly based on the preset route. In this embodiment, autonomous flight control of the drone is achieved based on real-time image recognition and analysis. Specifically, Figure 3 As shown, the following steps are included:

[0126] S101: Determine the flight altitude of the UAV based on the preset maximum recognition distance of each recognition task.

[0127] Specifically, find the minimum value d from the farthest recognition distances of all recognition tasks min In this embodiment, the lower half of the drone imaging screen is used as the effective area for image recognition. Therefore, at the pixel point x=0, y=(1 / 2)height image At, through the above inverse perspective mapping formula, its world coordinates (X p ,Y p ), and the distance between the world coordinate and the drone projection position (i.e., the origin) satisfies d min 2 =Y P 2 +X P 2 , substituting into the above inverse perspective mapping formula, the value of h can be solved by the simultaneous equations, that is, the constant flight altitude of the subsequent drone cruise. In this embodiment, the pitch angle θ is fixed at 30°.

[0128] S102: Determine the initial flight position of the UAV based on the flight altitude, the preset pitch angle, and the preset initial orientation (i.e., the direction of the shoreline of the water area to be inspected), and control the UAV to fly forward with the initial flight position as the inspection starting point based on the preset first flight speed.

[0129] This step is flight initialization. First, place the drone near the water's edge and point it toward the shoreline of the water to be inspected (facing the water, there are two possible directions: left and right). Then, control the drone to take off to the flight altitude h calculated in step S101 and adjust the pitch angle θ to 30°, which is close to the imaging angle of most samples. Finally, control the drone to fly forward at a certain speed.

[0130] S103: According to a preset first adjustment cycle, water surface recognition is performed on the real-time captured image of the UAV, and the yaw angle is adjusted according to the water surface recognition result so that the UAV flies along the coastline of the water area to be inspected.

[0131] Specifically, the steps include:

[0132] S1031: According to a preset first adjustment cycle, the latest captured image is obtained as the current image, and the world coordinate system corresponding to the capturing moment of the current image is used as the current world coordinate system.

[0133] S1032: performing image segmentation on the current image to obtain a water surface area, and performing contour extraction on the water surface area to obtain a contour point set of the water surface area.

[0134] That is, the water surface area of ​​the current image is segmented to obtain at least one water surface area; then the contours of each water surface area are extracted respectively, and the contour area is calculated, and the contour with the largest area C = {(x, y)} is selected as the analysis object, where (x, y) is the image coordinate of the contour point.

[0135] S1033: Traverse the set of contour points and perform inverse perspective transformation on each contour point in turn to obtain the world coordinates of each contour point in the current world coordinate system. That is, the world coordinates (X p ,Y p ).

[0136] S1034: Calculate the distance between each contour point and the origin of the current world coordinate system respectively, and use the contour point whose distance from the origin of the current world coordinate system is a preset first distance as a direction anchor point.

[0137] like Figure 4 As shown, through the formula Calculate the distance between the world coordinates of each contour point in the current world coordinate system and the origin O of the current world coordinate system (i.e., the vertical projection position point of the drone). In this embodiment, the preset first distance is 50 meters, that is, the contour point with d = 50 meters is selected as the direction anchor point P.

[0138] S1035: Calculate the angle between the line between the origin of the current world coordinate system and the direction anchor point and the Y axis of the current world coordinate system.

[0139] like Figure 4 As shown, the angle between OP and Y axis is calculated as ρ = arctan (X p / Y p ).

[0140] S1036: Adjust the yaw angle in the current first adjustment period according to the included angle.

[0141] Specifically, the adjustment frequency and the unit adjustment angle are determined according to the angle, the first adjustment period and the preset unit time; and the yaw angle is adjusted according to the adjustment frequency and the unit adjustment angle so that the camera of the drone tends to the line between the origin of the current world coordinate system and the direction anchor point.

[0142] For example, in this embodiment, the first adjustment period is 5 seconds, and the unit time is 1s, that is, the yaw angle is corrected by Δρ=ρ / 5 every second, so as to maintain the stability of the angle adjustment and allow the drone to be roughly at the junction of the water and the shore and always face the water shore directly in front.

[0143] In this embodiment, the positive direction of the yaw angle is clockwise. Therefore, if the yaw angle is to be adjusted to the right of the image center line, the yaw angle is increased, and vice versa.

[0144] S104: When the flight time reaches a preset time threshold, the real-time captured image of the UAV is subjected to water surface recognition, and the yaw angle is adjusted according to the water surface recognition result so that the UAV flies to the opposite bank and flies along the water shoreline of the opposite bank.

[0145] That is, set a time threshold t threshold , when the drone's flight time reaches this threshold, it will search for the opposite bank. If the application scenario is to patrol a large lake, it is necessary to search the opposite bank. If the application scenario is a small reservoir, it is not necessary to patrol the opposite bank, and it is sufficient to patrol along the direction of the water bank. For these two cases, the time threshold t threshold Set it to infinity, so that the drone will patrol along the waterfront until the battery runs out and it starts to return.

[0146] If the actual scenario is to patrol a river, the drone can fly along one bank of the river for a while, then fly to the other bank and return along the other bank. This allows the drone to be fully utilized, allowing the drone to patrol both sides of a certain section of the river in one round trip.

[0147] Specifically, the steps include:

[0148] S1041: When the flight time reaches a preset time threshold, water surface recognition is performed on the latest captured image to obtain a water surface area.

[0149] S1042: Determine the orientation of the water surface area relative to the UAV, and determine the rotation target side based on the orientation.

[0150] S1043: According to the rotation target side and the preset rotation angle, the yaw angle is adjusted so that the drone camera faces the opposite bank, and according to the preset second flight speed, the drone is controlled to fly forward so that the drone flies toward the opposite bank.

[0151] In this embodiment, the preset rotation angle is 90°. This means that the current captured image is subjected to water surface recognition. By analyzing the water surface area, it is determined whether the water area is on the left or right side of the drone, and that side is used as the rotation target. If the water area is on the right side of the drone, the yaw angle is adjusted 90° clockwise. If the water area is on the left side of the drone, the yaw angle is adjusted 90° counterclockwise, thereby adjusting the drone's heading to face the opposite bank.

[0152] Furthermore, before performing the rotation adjustment, the yaw angle adjustment angles of the most recent preset number of times (eg, 10 times) can be obtained and accumulated to obtain the cumulative adjustment angle Δρ total , first perform Δρ total / 2 angle correction, so the adjusted yaw angle is adjusted relative to a section of the coast rather than relative to a certain point.

[0153] S1044: When the UAV is flying toward the other bank, according to a preset recognition cycle, the latest captured image is used as the current image, and the world coordinate system corresponding to the capturing time of the current image is used as the current world coordinate system.

[0154] S1045: Perform water surface recognition and contour extraction on the current image to obtain the water surface contour.

[0155] S1046: Obtain the intersection of the water surface contour and the center line of the image along the width direction as the first reference point.

[0156] In this embodiment, the recognition cycle is 2 seconds, that is, the water surface is recognized and the contour is extracted every two seconds, and the water surface contour is compared with the straight line x=(1 / 2)width image The intersection point is taken as the first reference point A.

[0157] S1047: When the vertical coordinate value in the image coordinates of the first reference point is less than the preset height value, the world coordinates of each contour point of the water surface contour in the current world coordinate system are calculated respectively through the inverse perspective mapping formula, and the second reference point is determined from the contour points of the water surface contour. The second reference point is located on the rotation target side of the first reference point, and the distance between the first reference point and the second reference point in the current world coordinate system is a preset second distance.

[0158] In this embodiment, the preset height value is half of the image height, and the preset second distance is 50 meters.

[0159] like Figure 5 As shown, when the vertical coordinate y in the image coordinate of the first reference point A A <height image / 2, calculate the world coordinates of each contour point of the water surface contour in the current world coordinate system through the inverse perspective mapping formula, on the rotation target side of the first reference point A (for example, if it is on the left side, x<width image / 2) find the second reference point B, the distance between the first reference point A and the second reference point B in the current world coordinate system is 50 meters.

[0160] S1048: Calculate the adjustment angle based on the world coordinates of the second reference point in the current world coordinate system, and adjust the yaw angle based on the preset second adjustment period and the adjustment angle until the angle between the line connecting the first reference point and the second reference point in the world coordinate system corresponding to the same moment and the Y-axis of the world coordinate system corresponding to the same moment is less than the preset angle.

[0161] Through the inverse perspective mapping formula, the world coordinates (X B ,Y B ), the yaw angle is adjusted with the second reference point B as the anchor point while the UAV flies forward, and the adjustment angle Δρ'=arctan(X B / Y B ). This correction is performed once per second until |Y B -Y A |>|X B -X A |, that is, in the world coordinate system corresponding to the same moment, the angle between the line connecting points AB and the Y axis is less than 45°.

[0162] S1049: According to a preset first flight speed, the UAV is controlled to fly forward. At the same time, according to a preset first adjustment period, the real-time captured image of the UAV is subjected to water surface recognition. According to the water surface recognition result, the yaw angle is adjusted so that the UAV flies along the shoreline of the water on the opposite bank.

[0163] Through the adjustment in step S1048, the drone has adjusted to the direction of travel toward the opposite bank. At this time, the drone is controlled to fly forward, and referring to step S103, the yaw angle is adjusted in real time during the flight so that the drone flies along the shoreline of the water on the other side.

[0164] This embodiment controls the flight direction of the drone by utilizing the water surface segmentation results in the image. It only controls the flight action without requiring the drone to have a precise state, and has a high tolerance. It can also make the forward process smoother without frequent turns that would cause a decrease in image quality and affect subsequent image recognition tasks.

[0165] For the second stage, based on the video stream captured by the UAV during the shore patrol flight and the sensor flight data, the image recognition task is performed and the recognition results are processed.

[0166] At a certain point during stable flight, the drone's imagery, captured on the XOY plane of the world coordinate system, forms an inverted trapezoidal shape. If the horizon is within the frame, the long side of this inverted trapezoid is at infinity. In this case, it is necessary to truncate the image using the maximum recognition distance for each recognition task. Typically, video frame rates are above 24 frames per second, allowing for continuous detection of a target event. In scenarios involving multiple recognition tasks, computing resources are scarce, so sampling based on coverage is more efficient, avoiding the need to perform recognition for all tasks in every frame. The following example uses a single recognition task as an example; in actual applications, multiple recognition tasks can be processed simultaneously.

[0167] like Figure 6 As shown, the following steps are included:

[0168] S201: Obtain a video stream captured by a drone during flight, identify video frames in the video stream, and record the position offset and yaw angle offset of the drone between the shooting moments of adjacent identified video frames.

[0169] Specifically, the steps include:

[0170] S2011: Sequentially obtain a video frame from the video stream as the current video frame.

[0171] S2012: Determine whether the current video frame is the first video frame. If so, execute step S2013; if not, execute step S2014.

[0172] S2013: Using the current video frame as the recognition video frame, using the shooting time of the current video frame as the recognition time, and determining the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the shooting time.

[0173] like Figure 7 As shown, Figure 7 The left side is the shooting picture, and the right side is the actual coverage area on the XOY plane of the corresponding world coordinate system. The two vertices at the bottom edge of the current video frame are respectively taken as the first vertex P1 and the second vertex P2. The image coordinates of P1 and P2 are (0, height image ) and (width image , height image ), through the above inverse perspective mapping formula, its world coordinates in the current world coordinate system can be calculated, that is, Figure 7 P1' and P2' on the right. Then find the third vertex P3 and the fourth vertex P4 on the left and right sides of the current video frame, that is, the horizontal coordinate of P3 in the image coordinate is 0, and the horizontal coordinate of P4 in the image coordinate is width image, the third vertex P3 and the fourth vertex P4 correspond to P3' and P4' in the current world coordinate system, and the distance between P3' and P4' and the origin O of the current world coordinate system is the minimum value of the farthest recognition distance of each recognition task, that is, OP3'=OP4'=d min , by solving the simultaneous equations, we can calculate the vertical coordinates of P3 and P4 in the image coordinates. With a general camera focal length, when the pitch angle is 30°, the horizon will appear in the picture, so the vertical coordinates of the two points on the top edge of the inverted trapezoid correspond to the positions in the picture in the range of (0, height image / 2].

[0174] Finally, based on P1', P2', P3' and P4', the monitoring coverage area in the world coordinate system corresponding to the current video frame at the shooting moment can be determined.

[0175] Then continue to obtain the next video frame, that is, continue to execute step S2011.

[0176] S2014: Calculate the position offset of the drone between the shooting time of the current video frame and the latest recognition time based on the speed sensor information of the drone, and obtain the yaw angle offset of the drone between the shooting time of the current video frame and the latest recognition time.

[0177] For example, assuming that the shooting time of the latest recognized video frame (i.e. the video frame that was previously determined to be a recognized video frame) is t, and the shooting time of the current video frame is t+n, then calculate the position offset dX and dY of the drone's projection position between time t and time t+n, and obtain the yaw angle offset between time t and time t+n.

[0178] S2015: Determine the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the shooting moment, and determine the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the latest recognition moment based on the position offset and yaw angle offset.

[0179] Referring to the above step S2013, the monitoring coverage area in the world coordinate system corresponding to the shooting time of the current video frame (i.e. time t+n) is determined. Figure 8 As shown, the world coordinate system (O t ) after dX, dY translation and The world coordinate system (O t+n ), therefore, the world coordinate system corresponding to the time t+n is calculated by -dX, -dY, By performing a transformation (i.e., a translation plus a rotation), we can obtain the position of the current monitoring coverage area in the world coordinate system corresponding to the most recent identification time (i.e., time t). In other words, the monitoring coverage areas at different times are converted into the world coordinate system corresponding to the same time, so that the intersection area of ​​the two monitoring coverage areas can be calculated later.

[0180] S2016: Calculate the intersection area of ​​the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the latest recognition time and the monitoring coverage area of ​​the latest recognized video frame in the world coordinate system corresponding to the latest recognition time.

[0181] That is, the intersection area of ​​the monitoring coverage areas at two different times is calculated in the world coordinate system corresponding to the same time.

[0182] S2017: Determine whether the intersection area is smaller than a preset area threshold. If so, execute step S2018; if not, continue to obtain the next video frame, that is, continue to execute step S2011.

[0183] S2018: Using the current video frame as the recognition video frame and the shooting time of the current video frame as the recognition time.

[0184] Due to the continuity of the captured video stream, an event will appear in a series of consecutive video frames. To save computing power, image recognition can be performed on every video frame. Considering the coverage area of ​​the drone's field of view, which only changes significantly after flying a certain distance, and this is related to the drone's speed and video frame rate, this embodiment directly compares the overlap area ratio at two moments to determine the video frame for image recognition.

[0185] In this embodiment, the area threshold is 70% of the monitoring coverage area. That is, if the intersection area is less than 70% of the inverted trapezoid area, the next step is executed, otherwise the video frame is searched backwards for recognition.

[0186] S202: performing image recognition on each recognition video frame according to a preset recognition task to obtain a recognition result, wherein the recognition result includes a target point and a target area.

[0187] Specifically, after determining that the current video frame is a recognition video frame in steps S2013 and S2018, image recognition can be performed on the current video frame according to the recognition task.

[0188] The recognition task may include target detection and image segmentation, so the recognition result may include target detection result and image segmentation result. The target detection result can be expressed as (x c ,y c ,w box ,h box )t , where (x c ,y c ) represents the coordinate of the midpoint of the bottom edge of the target detection frame (that is, the midpoint of the bottom edge of the target detection frame is taken as the location where the event occurs), w box and h box Represent the width and height of the target detection box respectively; the image segmentation result can be expressed as (x c ,y c ,C) t , where C represents the contour point set of the target area, (x c ,y c ) represents the centroid coordinates of each contour point in C (i.e., the contour centroid is taken as the location where the event occurs), and the subscript t represents the shooting time of the identified video frame.

[0189] Furthermore, assuming that the shooting time of a recognition video frame is time t, the world coordinates of the target point in the recognition result of the recognition video frame in the world coordinate system corresponding to time t are calculated by the inverse perspective mapping formula, and the real distance d between the target point and the vertical projection position of the drone at time t (that is, the origin of the world coordinate system corresponding to time t) is calculated. even And the recognition angle θ of the drone corresponding to the target point even , identification angle θ even The calculation formula is:

[0190]

[0191] The calculated true distance and recognition angle need to be within the maximum recognition distance and recognition angle range of the current recognition task, that is, d even ≤d task And θ 1 task ≤θ even ≤θ 2 task If the condition is not met, that is, the actual distance is greater than the farthest recognition distance corresponding to the recognition task, or the recognition angle is not within the recognition angle range corresponding to the recognition task, the recognition result of the recognition video frame is discarded.

[0192] S203: Merge the recognition results belonging to the same event, and record the drone positioning position and yaw angle corresponding to each recognition result in the same event.

[0193] Specifically, first create an event set S for temporarily storing events.

[0194] Then, according to the shooting time of each recognition video frame, the recognition result of a recognition video frame is obtained in sequence. The recognition result can also be obtained in real time after a recognition video frame is recognized in step S202, as the current recognition result. Then, the current recognition result is compared with each event in the event set S to determine whether it belongs to the same event. If it belongs to the same event as a certain event, the current recognition result and the positioning position and yaw angle of the drone at the shooting time of the recognition video frame are merged and recorded into the event. If it does not belong to the same event as each event in the event set S, the current recognition result is saved as a new event in the event set S, and the positioning position and yaw angle of the drone at the shooting time of the recognition video frame are recorded. In this embodiment, the drone positioning position (GPS position) and yaw angle are only used to calculate the final output result of the event. The information recorded by the speed sensor is more accurate in the intermediate judgment process.

[0195] It is assumed that the shooting time of the current recognition video frame is t+n, and the shooting time of the latest recorded recognition result (recorded as the latest recognition result) in an existing event is t.

[0196] For the target detection result (i.e. the recognition result when the recognition task is target detection), the world coordinates of the target point in the current recognition result at time t+n in the world coordinate system and the world coordinates of the target point in the latest recognition result at time t can be calculated by the inverse perspective mapping formula. Then, according to the position offset dX, dY and yaw angle offset of the drone between time t and time t+n, the world coordinates of the target point in the latest recognition result at time t can be calculated. Perform -dX, -dY, The world coordinates of the target point in the current recognition result at the time t+n in the world coordinate system can be obtained by translation and rotation, so that the world coordinates of the target points at two times in the world coordinate system at the same time can be obtained. In another embodiment, the target point in the latest recognition result can also be subjected to dX, dY, The translation and rotation of the object are performed to obtain its world coordinates in the world coordinate system corresponding to time t. Then, in the world coordinate system corresponding to the same time, the offset distance between the target point in the current recognition result and the target point in the latest recognition result is calculated. If the offset distance is less than a preset distance threshold (such as 2 meters), the current recognition result and the latest recognition result are considered to belong to the same event, that is, the current recognition result belongs to that event.

[0197] For the image segmentation result (i.e. the recognition result when the recognition task is image segmentation), similarly, the inverse perspective mapping formula can be used to calculate the coverage area of ​​the target area in the current recognition result in the world coordinate system corresponding to time t+n, and the coverage area of ​​the target area in the latest recognition result in the world coordinate system corresponding to time t, and the coverage area of ​​the target area in the latest recognition result in the world coordinate system corresponding to time t, and the coverage area of ​​the target area in the world coordinate system corresponding to time t, respectively, by -dX, -dY, or dX, dY, The target area in the current recognition result and the target area in the latest recognition result are transformed to obtain the coverage area in the world coordinate system corresponding to the same time. Then, in the world coordinate system corresponding to the same time, the intersection-and-union ratio of the coverage areas corresponding to the two target areas is calculated. If the intersection-and-union ratio is greater than a preset ratio threshold (such as 50%), it is considered that the current recognition result and the latest recognition result belong to the same event, that is, the current recognition result belongs to that event.

[0198] Furthermore, in another optional embodiment, for the target detection results, it is also possible to determine whether they belong to the same event by judging whether the intersection-and-union ratio of the target area in the current recognition result and the target area in the latest recognition result in the world coordinate system corresponding to the same moment is greater than a preset ratio threshold.

[0199] Furthermore, after each event attribution analysis of the current recognition result, each event in the event set S is traversed and checked. If an event is updated (i.e., the current recognition result belongs to this event), the reliability Vr of this event is increased. The closer the distance, the clearer the image of the event, and the higher its reliability. Therefore, the reliability change value ΔVr = 1 / d even , where d even That is, the real distance between the world coordinates of the current recognition result in the current world coordinate system and the vertical projection position of the drone (that is, the origin of the current world coordinate system). If no update is performed (that is, the current recognition result does not belong to this event), the reliability of this event is reduced by ΔVr'=1 / d even ', where d even ' is the distance between the latest recognition result of the event and the vertical projection position of the current drone in the world coordinate system at the same time. In order to obtain the positions of the two in the world coordinate system at the same time, it is necessary to convert the latest recognition result of the event or the vertical projection position of the current drone based on the position offset and yaw angle offset of the drone between the two shooting moments (i.e., the shooting moment of the latest recognition result at this time and the shooting moment of the current recognition result).

[0200] S204: Map each event onto a map based on each recognition result in each event and its corresponding drone positioning position and yaw angle.

[0201] Since the drone is always flying forward, the recognition result of an event will not be updated once the drone's field of view exceeds the location where an event occurs. Therefore, when the latest recognition result of an event exceeds the current drone's field of view, the event is removed from the event set S to avoid excessive computational complexity when performing event attribution analysis on subsequent recognition results.

[0202] Specifically, after each event attribution analysis is performed on the current recognition result, each event in the event set S is traversed and checked. If an event has not been updated (i.e., the current recognition result does not belong to this event), the Y value of the latest recognition result in this event (i.e., the latest recorded recognition result) in the world coordinate system corresponding to the shooting time is obtained (as the latest Y value of this event), such as Figure 9 As shown, if the bottom edge of the current video frame (ie y = height image ) The Y value in the current world coordinate system is corrected by -dY according to the Y-axis offset dY between the shooting time of the current recognition video frame of the drone and the shooting time of the latest recognition result in the event. If it exceeds the latest Y value of the event, it can be considered that the drone has flown out of the location where the event occurred, that is, the event can be moved out of the event set S.

[0203] For events removed from the event set S, if their reliability Vr>0, they can be regarded as correct events and used for subsequent result push display; otherwise, they are directly discarded.

[0204] Then the correct event is mapped onto a map. In this embodiment, it is mapped onto a GIS map. Specifically, Figure 10 As shown in the figure, the map coordinates of each recognition result on the GIS map are calculated based on the world coordinates of each recognition result in the event at the time of shooting, the UAV positioning position (i.e., the GPS coordinates of the vertical projection point O of the UAV) and the yaw angle (i.e., the angle of the UAV camera direction relative to the north direction) corresponding to each recognition result.

[0205] Specifically, for each recognition result, the yaw angle is first corrected for the world coordinate system corresponding to the shooting time, and then the offset distance relative to point O is calculated and converted into the offset in longitude and latitude. It should be noted that the conversion between longitude and meters must take the latitude value into account. Assume that the world coordinates of a recognition result in the world coordinate system corresponding to the shooting time are (X, Y), and the corresponding yaw angle is The calculation formula for its latitude and longitude on the map is:

[0206]

[0207] Y'=Ycos(φ)–X sin(φ)

[0208]

[0209] latitude=latitude O +Y'*0.00000898

[0210] Among them, (lonitude O , latitude O ) represents the GPS coordinates of the drone’s location corresponding to the recognition result.

[0211] Furthermore, if it is a target detection task, the map coordinates of the target points of each recognition result of the event are averaged; if it is an image segmentation task, the target areas of each recognition result are merged.

[0212] This embodiment can actively control the flight process of the drone according to the recognition task, so that it can generate image pictures that meet the task requirements, thereby directly migrating the model trained with ordinary perspective to the drone patrol scene, enriching the types of recognition tasks while ensuring monitoring coverage.

[0213] By combining drone sensor information with image recognition results, drones can automatically complete water and shoreline inspections without manually setting routes. Compared to designated fixed patrol routes, this is much more convenient in new environments or temporary inspection scenarios, and is also suitable for situations where the water surface and shoreline are subject to significant changes. The image recognition process leverages flight sensor information to automatically perform appropriate recognition tasks, disambiguating and merging recognition results, and skipping overlapping areas to reduce computing power consumption. Furthermore, the results can be accurately projected onto a bird's-eye view, facilitating integration into GIS systems or digital twin systems.

[0214] Example 2

[0215] This embodiment is a computer-readable storage medium corresponding to the above embodiment, on which a computer program is stored. When the program is executed by the processor, it implements the various steps of a drone water and shoreline inspection method in the above embodiment, and can achieve the same technical effect, which will not be repeated here.

[0216] In summary, the present invention provides a method for patrolling water shorelines with a drone and a computer-readable storage medium, which can automatically complete river patrols according to recognition tasks. By performing recognition tasks on only some video frames in the video stream and automatically merging the recognition results, repeated coverage areas can be skipped, computing power consumption can be reduced, and the complexity of manual operations can be reduced, thereby improving recognition accuracy and efficiency and better completing drone shore patrol tasks. At the same time, the recognition results can also be mapped to a map for easy access to a GIS system or a digital twin system.

[0217] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for patrolling waters and coastlines using a drone, characterized in that: include: Control the drone to fly along the coastline of the water area to be inspected; Obtaining a video stream captured by the drone during flight, determining identification video frames in the video stream, and recording the position offset and yaw angle offset of the drone between the shooting moments of adjacent identification video frames; According to the preset recognition task, image recognition is performed on each recognition video frame to obtain a recognition result, which includes a target point and a target area; Merge the recognition results belonging to the same event and record the drone positioning position and yaw angle corresponding to each recognition result in the same event; Map each event onto a map based on the recognition results and the corresponding drone positioning and yaw angle. The determining and identifying the video frame in the video stream is specifically: Sequentially acquiring a video frame from the video stream as a current video frame; If the current video frame is the first video frame, then the current video frame is used as the recognition video frame, the shooting time of the current video frame is used as the recognition time, and the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to its shooting time is determined. Otherwise, based on the speed sensor information of the UAV, the position offset of the UAV between the shooting time of the current video frame and the latest recognition time is calculated, and the yaw angle offset of the UAV between the shooting time of the current video frame and the latest recognition time is obtained; Determine the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the shooting time of the current video frame, and determine the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the latest recognition time based on the position offset and the yaw angle offset; Calculate the intersection area of ​​the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the latest recognition time and the monitoring coverage area of ​​the latest recognized video frame in the world coordinate system corresponding to the latest recognition time; If the intersection area is smaller than a preset area threshold, the current video frame is used as the recognition video frame, and the shooting time of the current video frame is used as the recognition time.

2. The method for inspecting waters and coastlines using a drone according to claim 1, characterized in that: Also includes: At a moment in the drone's flight, the world coordinate system corresponding to the moment is constructed, with the vertical projection position of the drone on the plane where the water surface is located as the origin, the plane where the water surface is located as the XOY plane, the direction of the drone's camera as the Y-axis direction, and the vertical direction as the Z-axis direction.

3. The method for inspecting waters and coastlines using a drone according to claim 1, characterized in that: The determination of the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the shooting time is specifically as follows: The two vertices of the bottom edge of the current video frame are respectively used as the first vertex and the second vertex, and the world coordinates of the first vertex and the second vertex in the world coordinate system corresponding to the shooting time of the current video frame are respectively calculated using the inverse perspective mapping formula; Determine, based on the minimum value of the longest recognition distances for each recognition task, a third vertex and a fourth vertex on the left and right sides of the current video frame, respectively, wherein, in the world coordinate system corresponding to the capture moment of the current video frame, the distance between the world coordinates of the third vertex and the fourth vertex and the origin is the minimum value; According to the world coordinates of the first vertex, the second vertex, the third vertex and the fourth vertex in the world coordinate system corresponding to the shooting moment of the current video frame, the monitoring coverage area of ​​the current video frame in the world coordinate system corresponding to the shooting moment of the current video frame is determined.

4. The method for inspecting waters and coastlines using a drone according to claim 2, characterized in that: The method further includes: performing image recognition on each recognition video frame according to the preset recognition task, and obtaining the recognition result; Calculating, based on the image coordinates of a target point in a recognition result of a recognition video frame, the world coordinates of the target point in a world coordinate system corresponding to the time when the recognition video frame was shot using an inverse perspective mapping formula, and calculating, in the world coordinate system corresponding to the time when the recognition video frame was shot, the true distance between the target point and the origin and the recognition angle of the drone corresponding to the target point; If the actual distance is greater than the farthest recognition distance corresponding to the recognition task, or the recognition angle is not within the recognition angle range corresponding to the recognition task, the recognition result of the recognition video frame is discarded.

5. The method for inspecting waters and coastlines using a drone according to claim 2, characterized in that: The identification results belonging to the same event are merged, and the drone positioning position and yaw angle corresponding to each identification result in the same event are recorded, including: Sequentially obtaining recognition results of the recognition video frames according to the shooting time of each recognition video frame, and comparing the recognition results of the recognition video frames with the existing recognition results corresponding to each event to determine whether they belong to the same event; If the recognition result of the identified video frame belongs to an event, the recognition result of the identified video frame and the positioning position and yaw angle of the drone at the time of shooting the identified video frame are combined and recorded in the event; otherwise, the recognition result of the identified video frame is taken as a new event, and the positioning position and yaw angle of the drone at the time of shooting the identified video frame are recorded.

6. The method for inspecting waters and coastlines using a drone according to claim 5, characterized in that: The recognition result of the identified video frame is compared with the existing recognition results corresponding to each event to determine whether they belong to the same event, specifically: Calculating, according to the image coordinates of the recognition result of a recognition video frame, the world coordinates of the recognition result of the recognition video frame in the world coordinate system corresponding to the shooting moment of the recognition video frame by using an inverse perspective mapping formula; Obtaining a recognition result of a newly recorded recognition video frame of an existing event as the latest recognition result of the event, and calculating the world coordinates of the latest recognition result in the world coordinate system corresponding to the shooting time using an inverse perspective mapping formula; Determine, based on a position offset and a yaw angle offset of the drone between the shooting time of the recognition video frame and the latest recognition video frame recorded in the event, the world coordinates in the world coordinate system corresponding to the recognition result of the recognition video frame and the latest recognition result at the same time; Calculating, based on the world coordinates of the world coordinate system corresponding to the recognition result of the first recognition video frame and the latest recognition result at the same time, an offset distance between a target point in the recognition result of the first recognition video frame and a target point in the latest recognition result, or calculating an intersection-over-union ratio between a target area in the recognition result of the first recognition video frame and a target area in the latest recognition result; If the offset distance is smaller than a preset distance threshold, or the intersection-over-union ratio is larger than a preset ratio threshold, it is determined that the recognition result of the recognized video frame belongs to the event.

7. The method for inspecting waters and coastlines using a drone according to claim 5, characterized in that: The method of merging the recognition results belonging to the same event and recording the drone positioning position and yaw angle corresponding to each recognition result in the same event also includes: If the recognition result of the identified video frame belongs to an existing event, then the reliability of the event is increased according to the reliability change value corresponding to the event, and the reliability of other existing events is reduced according to the reliability change values ​​corresponding to the other existing events; otherwise, the reliability of each existing event is reduced according to the reliability change value corresponding to each event; The reliability change value corresponding to an event is the reciprocal of the distance in the world coordinate system corresponding to the vertical projection position of the latest recognition result in the event and the vertical projection position when the drone shoots the recognition video frame at the same time.

8. The method for inspecting waters and coastlines using a drone according to claim 7, characterized in that: The events are mapped onto the map according to the recognition results of each event and its corresponding drone positioning position and yaw angle, specifically: If the latest recognition result in an event exceeds the visual coverage area of ​​the current drone and the reliability of the event is greater than zero, the event is mapped onto the map according to each recognition result in the event.

9. The method for inspecting waters and coastlines using a drone according to claim 8, characterized in that: Mapping the event onto a map according to each recognition result in the event is specifically as follows: The map coordinates of each recognition result on the map are calculated according to the world coordinates of each recognition result in the event in the world coordinate system corresponding to the shooting time and the drone positioning position and yaw angle corresponding to each recognition result.

10. The method for inspecting waters and coastlines using a drone according to claim 1, characterized in that: The controlling the drone to fly along the coastline of the water area to be inspected includes: Determine the flight altitude of the drone based on the preset maximum recognition distance of each recognition task; Determining an initial flight position of the drone according to the flight altitude, a preset pitch angle, and a preset initial orientation, and controlling the drone to fly forward from the initial flight position according to a preset first flight speed; According to a preset first adjustment period, the real-time image captured by the UAV is subjected to water surface recognition, and the yaw angle is adjusted according to the water surface recognition result so that the UAV flies along the shoreline of the water area to be inspected; When the flight time reaches a preset time threshold, the real-time image captured by the drone is used for water surface recognition, and the yaw angle is adjusted according to the water surface recognition result so that the drone flies to the opposite bank and flies along the shoreline of the water on the opposite bank.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images