Unmanned aerial vehicle bank protection patrol obstacle avoidance method and computer readable storage medium

Abstract

The invention discloses an unmanned aerial vehicle bank protection patrol obstacle avoidance method and a computer readable storage medium. The method comprises the steps that the initial flight position of an unmanned aerial vehicle is determined, and the unmanned aerial vehicle is controlled to fly forwards from the initial flight position; according to a preset period, performing water surface identification on the real-time shot image of the unmanned aerial vehicle, and according to a water surface identification result, determining a flight path point so that the unmanned aerial vehicle flies along a shoreline; according to the real-time depth map and the water surface recognition result, whether an obstacle exists in front is judged; if the obstacle exists and the depth value of the obstacle is smaller than a preset depth threshold value, newly adding a flight path point according to the height of the obstacle; and according to the real-time positioning position of the unmanned aerial vehicle, calculating map coordinates of each flight path point on the map, and according to a preset number of flight path points closest to the unmanned aerial vehicle, updating the flight path. According to the invention, autonomous shore patrol and obstacle avoidance of the unmanned aerial vehicle can be realized.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicle control technology, and in particular to an unmanned aerial vehicle shore protection patrol obstacle avoidance method and a computer-readable storage medium. Background Art

[0002] River bank protection is an important dividing line between the water environment and the human activity area, and is the focus of water shoreline space control and ecological governance. Arranging personnel to inspect the health of the bank protection is time-consuming and laborious, and the coverage of the bank protection by video monitoring stations is extremely limited. Using drones to fly along the river surface can clearly observe the front of the bank protection, and can cover the entire river section, so as to achieve true no-dead-angle monitoring, which is the best choice for this inspection task. Drones conduct bank protection inspections along the river, and the targets of attention include whether there is bank protection damage, weed growth on the slope, and the health of the bank protection, such as the accumulation of debris, and can also include the monitoring content of water surface pollution and personnel violations. Thanks to the rapid development of computer image processing and artificial intelligence technology, these monitoring tasks can be completed by computer algorithms instead of human eyes, and the identified event information such as location coordinates, severity, and photo evidence can be reported to river managers in real time to achieve efficient supervision and governance. The environment of small rivers, especially urban rivers, is complex. Drones must not only ensure that they move safely and steadily in the predetermined direction, but also ensure that the images of the bank protection taken are clear and stable. There are certain angle and distance requirements, and it is not easy to operate. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a method for unmanned aerial vehicle (UAV) bank protection patrol and obstacle avoidance and a computer-readable storage medium, which can realize autonomous bank patrol and obstacle avoidance of the UAV without pre-setting the route, saving labor costs and ensuring the flight safety of the UAV.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for patrolling and avoiding obstacles by using an unmanned aerial vehicle for bank protection, comprising:

[0005] Determine the initial flight position of the UAV according to the preset flight altitude, pitch angle and initial orientation, and control the UAV to fly forward from the initial flight position according to the preset flight speed;

[0006] According to the preset cycle, the real-time shooting images of the UAV are subjected to water surface recognition, and the flight path points are determined based on the water surface recognition results so that the UAV flies along the coastline;

[0007] Obtain a real-time depth map through the real-time images captured by the binocular camera of the drone, and determine whether there is an obstacle ahead based on the real-time depth map and the water surface recognition result;

[0008] If the obstacle exists and the depth value of the obstacle is less than the preset depth threshold, a new flight path point is added according to the flight path point and the height of the obstacle;

[0009] According to the real-time positioning position of the UAV, the map coordinates of each flight path point on the map are calculated respectively, and the flight path is updated according to the preset number of flight path points closest to the UAV.

[0010] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0011] The beneficial effects of the present invention are as follows: by determining subsequent flight path points in real time according to the water surface recognition results of the captured images during the flight, the UAV can automatically fly along the coastline of the water area without the need to manually set the route. Compared with the designated fixed cruise route, it can be more convenient in new environments or temporary patrol scenarios; at the same time, by real-time detection of whether there are obstacles ahead, and when approaching the obstacles, new flight path points are added according to the current flight path points and the height of the obstacles, so that the UAV can fly over the obstacles along the current flight path, thereby achieving automatic obstacle avoidance and effectively ensuring the safety of the UAV. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a flow chart of a method for patrolling and avoiding obstacles for unmanned aerial vehicles (UAVs) according to the present invention;

[0013] Figure 2 Schematic diagram of the inverse mapping transformation according to the first embodiment of the present invention;

[0014] Figure 3 This is a flow chart of a method according to Embodiment 1 of the present invention;

[0015] Figure 4 This is a schematic diagram of offshore distance correction according to Embodiment 1 of the present invention;

[0016] Figure 5 This is a schematic diagram of the farthest nearshore contour points in different water surface areas according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the implementation methods and the accompanying drawings.

[0018] See also Figure 1 , a method for avoiding obstacles during bank protection inspection by a UAV, comprising:

[0019] Determine the initial flight position of the UAV according to the preset flight altitude, pitch angle and initial orientation, and control the UAV to fly forward from the initial flight position according to the preset flight speed;

[0020] According to the preset cycle, the real-time shooting images of the UAV are subjected to water surface recognition, and the flight path points are determined based on the water surface recognition results so that the UAV flies along the coastline;

[0021] Obtain a real-time depth map through the real-time images captured by the binocular camera of the drone, and determine whether there is an obstacle ahead based on the real-time depth map and the water surface recognition result;

[0022] If the obstacle exists and the depth value of the obstacle is less than the preset depth threshold, a new flight path point is added according to the flight path point and the height of the obstacle;

[0023] According to the real-time positioning position of the UAV, the map coordinates of each flight path point on the map are calculated respectively, and the flight path is updated according to the preset number of flight path points closest to the UAV.

[0024] From the above description, it can be seen that the beneficial effects of the present invention are: autonomous shore patrol and obstacle avoidance of the UAV can be realized without pre-setting the route, saving labor costs, and ensuring the flight safety of the UAV.

[0025] Furthermore, at a moment in the flight of the drone, the vertical projection position of the drone on the plane where the water surface is located is taken as the origin, the plane where the water surface is located is taken as the XOY plane, the direction of the drone camera is taken as the Y-axis direction, and the vertical direction is taken as the Z-axis direction, to construct a world coordinate system corresponding to the moment.

[0026] From the above description, it can be seen that the world coordinate system corresponding to different moments will change as the position of the drone on the horizontal plane and the direction of the camera change.

[0027] Furthermore, the method of performing water surface recognition on the real-time captured image of the drone according to a preset period, and determining a flight path point according to the water surface recognition result so that the drone flies along the coastline includes:

[0028] According to a preset cycle, the latest captured image is obtained as the current image, and the world coordinate system corresponding to the shooting time of the current image is used as the current world coordinate system;

[0029] Performing image segmentation on the current image to obtain a water surface area, and performing contour extraction on the water surface area to obtain a contour point set of the water surface area, wherein the contour points in the contour point set take the contour point close to the upper left corner of the image as a starting point and are arranged in a counterclockwise direction;

[0030] Eliminate contour points at the boundary of the current image from the contour point set;

[0031] According to the contour point set, a nearshore point set and an opposite shore point set are obtained;

[0032] According to the contour points in the nearshore set, corresponding flight path points are calculated so that the UAV flies along the nearshore line.

[0033] From the above description, it can be seen that by identifying the contour points of the water surface area and separating the contour points on both sides of the river, a one-to-one corresponding flight path point is calculated based on the contour points located near the shore, so that the drone can move along the direction of the river and maintain a certain distance from the river bank.

[0034] Furthermore, before obtaining the nearshore point set and the opposite shore point set according to the contour point set, the method further includes:

[0035] Traversing the contour point set, sequentially acquiring a contour point as a current contour point, and using the first contour point acquired as a reference point;

[0036] Perform inverse perspective transformation on the current contour point to obtain the world coordinates of the current contour point in the current world coordinate system;

[0037] Calculate the distance between the current contour point and the latest reference point according to their world coordinates in the current world coordinate system;

[0038] If the distance between the current contour point and the latest reference point is greater than the preset distance threshold, the current contour point is used as the new reference point;

[0039] After traversing the contour point set, contour points that are not reference points are removed from the contour point set.

[0040] From the above description, it can be seen that by screening the contour points and retaining the contour points with moderate adjacent distances, the number of contour points can be reduced, the efficiency of subsequent path point calculation can be improved, and the uniformity of the flight path points can be ensured.

[0041] Furthermore, if the initial orientation is the right direction after the drone takes off towards the river center, the nearshore point set is the right bank point set, and the opposite bank point set is the left bank point set; if the initial orientation is the left direction after the drone takes off towards the river center, the nearshore point set is the left bank point set, and the opposite bank point set is the right bank point set;

[0042] The step of obtaining a nearshore point set and an opposite shore point set according to the contour point set is specifically as follows:

[0043] If a contour point meets the preset right bank point condition, the contour point is recorded in the right bank point set, and the right bank point condition is Y i-1 <Y i <Y i+1 And |Yi+1 -Y i-1 |>|X i+1 -X i-1 |;

[0044] If a contour point meets the preset left bank point condition, the contour point is recorded in the left bank point set, and the left bank point condition is Y i-1 >Y i >Y i+1 And |Y i+1 -Y i-1 |>|X i+1 -X i-1 |;

[0045] Among them, (X i , Y i ) represents the world coordinates of the contour point in the current world coordinate system, (X i-1 , Y i-1 ) represents the world coordinates of the previous contour point of the contour point in the contour point set in the current world coordinate system, (X i+1 , Y i+1 ) represents the world coordinates of the next contour point of the contour point in the contour point set in the current world coordinate system.

[0046] Furthermore, the world coordinate T of the flight path point corresponding to a contour point in the nearshore point set in the current world coordinate system is i =(X Ti , Y Ti , h0),

[0047]

[0048] Among them, (X Pi , Y Pi ) represents the world coordinates of a contour point in the nearshore point set in the current world coordinate system, (X Pi-1 , Y Pi-1 ) and (X Pi+1 , Y Pi+1 ) respectively represent the world coordinates of the previous contour point and the next contour point of the contour point in the nearshore point set in the current world coordinate system; d represents the preset offshore distance, and h0 represents the preset flight altitude.

[0049] From the above description, it can be seen that the UAV can move along the river bank and maintain a certain distance from the river bank.

[0050] Furthermore, before adding a new flight path point according to the flight path point and the height of the obstacle, the method further includes:

[0051] Reduce the flight speed of the drone.

[0052] From the above description, it can be seen that when an obstacle is detected in front and close to the drone, the flight speed is reduced to ensure the flight safety of the drone.

[0053] Further, judging whether there is an obstacle ahead according to the real-time depth map and the water surface recognition result includes:

[0054] Acquire, from the nearshore point set, a contour point that belongs to the water surface area closest to the drone and is farthest from the drone as a first contour point, and acquire, from the opposite shore point set, a contour point that is closest to the first contour point as a second contour point;

[0055] If a pixel point in the current depth map satisfies a preset blocking point condition, the pixel point is marked as a blocking point, otherwise the pixel point is marked as a non-blocking point, wherein the blocking point condition includes being located between the first contour point and the second contour point in the X-axis direction of the image coordinate system of the current depth map, not being located in the water surface area, and having a depth value less than the sum of the maximum value of the depth value of the first contour point and the depth value of the second contour point and a preset allowable value;

[0056] If in the current depth map, there are blocking points greater than a preset first proportion among the pixel points with the same x value, and there are blocking points greater than a preset second proportion among the pixel points with the same y value, then it is determined that there is an obstacle across the water surface, and the minimum value in the y value range corresponding to the obstacle is obtained as the top y value of the obstacle.

[0057] Further, the adding of a new flight path point according to the flight path point and the height of the obstacle includes:

[0058] estimating the height of the obstacle according to the world coordinates of the first contour point in the current world coordinate system, the top y value of the obstacle, and the current flight altitude and pitch angle;

[0059] If the height of the obstacle is less than the preset height threshold, two new flight path points are determined according to the last two flight path points between the UAV and the obstacle.

[0060] The world coordinates of the two new flight path points in the current world coordinate system are:

[0061] and

[0062]

[0063] in, and They represent the world coordinates of the last two flight path points between the UAV and the obstacle in the current world coordinate system, h 阻挡 represents the height of the obstacle, and L represents the preset flight distance;

[0064] If the height of the obstacle is greater than or equal to a preset height threshold, the drone is controlled to return.

[0065] From the above description, it can be seen that by adding a flight path point at the location of the corresponding obstacle, the drone can fly over the obstacle and continue to patrol the bank.

[0066] Furthermore, before estimating the height of the obstacle according to the world coordinates of the first contour point in the current world coordinate system, the top y value of the obstacle, and the current flight altitude and pitch angle, the method further includes:

[0067] Determine whether there is a contour point in the nearshore point set, the depth value of the contour point being greater than the sum of the depth value of the obstacle and a preset allowable value;

[0068] If so, the step of calculating the map coordinates of each flight path point on the map according to the real-time positioning position of the drone and updating the flight path according to the preset number of flight path points closest to the drone is executed;

[0069] If not, the step of estimating the height of the obstacle according to the world coordinates of the first contour point in the current world coordinate system, the top y value of the obstacle, and the current flight altitude and pitch angle is executed.

[0070] From the above description, it can be seen that if there is a contour point that is farther away from the drone than the obstacle, it is considered that the drone can directly pass through from the bottom, or fly over from the top by the obstacle avoidance algorithm, so there is no need to add a new flight path point corresponding to the obstacle.

[0071] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described above is implemented.

[0072] Embodiment 1

[0073] Please refer to Figure 2-5 , Embodiment 1 of the present invention is: a method for avoiding obstacles in bank protection inspection by a UAV, which can be applied to the task of bank patrol by a UAV.

[0074] In this embodiment, a binocular camera is installed at the front of the drone, and the forward depth map D can be calculated by using the disparity map of the two cameras and the camera parameters. 前向The camera is off center, but the offset is small and the error can be ignored, so you can choose any camera image. 前向 The screen is used for image analysis.

[0075] The drone is equipped with a gimbal and a high-definition camera. The high-definition camera is controlled by the gimbal to maintain a certain pitch and yaw angle with the fuselage, so that the revetment surface on one side can be clearly photographed while the fuselage keeps moving forward, which is convenient for subsequent algorithms to monitor the revetment situation.

[0076] The sonar device is mounted on the bottom of the fuselage to measure the height h of the drone from the water surface in real time. Considering that the water surface height of the inland river will change, this height value is more accurate than the relative height value obtained by the drone's own sensor, and can protect the drone at a safe distance h above the water surface. 安全 activities within the scope of.

[0077] like Figure 2 As shown, in this embodiment, at a certain moment in the flight of the drone, the vertical projection position point of the drone on the plane where the water surface is located is taken as the origin O, the plane where the water surface is located is taken as the XOY plane, the direction of the drone camera (i.e., the yaw angle direction) is taken as the Y-axis direction, and the vertical upward direction is taken as the positive direction of the Z-axis to construct the world coordinate system of the drone corresponding to that moment. That is to say, in this embodiment, the world coordinate system will change with the change of the position of the drone (the change of position in the horizontal plane) and the change of the camera direction.

[0078] In addition, for the images taken by the drone, an image coordinate system is constructed with the upper left corner of the image as the origin, the right direction as the positive direction of the X-axis, and the downward direction as the positive direction of the Y-axis.

[0079] Through the inverse perspective mapping formula, a point (x, y) on an image taken at a certain moment can be mapped to the XOY plane of the world coordinate system corresponding to that moment. The specific formula is as follows. Although the binocular camera is a fixed forward-facing camera, there is an inclination during flight, so the sensor installed on the fuselage needs to provide the pitch angle θ value during calculation.

[0080]

[0081] Among them, (x, y) represents the image coordinates of a pixel point in the captured image in the image coordinate system, (X, Y) represents the world coordinates corresponding to the point (that is, the coordinates in the world coordinate system corresponding to the time when the image was captured), h represents the height of the drone camera relative to the water surface (that is, the flight height of the drone), and θ represents the pitch angle of the drone camera; width image and height imageIt represents the resolution of the captured image, that is, the width and height of the captured image; α and β are the internal parameters of the drone camera, representing the horizontal and vertical viewing angles, which are determined by the focal length of the camera.

[0082] like Figure 3 As shown, the method of this embodiment includes the following steps:

[0083] S1: Determine the initial flight position of the UAV according to a preset flight altitude, pitch angle and initial orientation, and control the UAV to fly forward from the initial flight position according to a preset flight speed.

[0084] Specifically, the drone takes off from the river bank, flies a certain distance (e.g., about 10 meters) toward the center of the river to reach above the water surface, and then rotates 90° to the right so that the front of the drone is roughly facing the river inspection direction. In this embodiment, the right side of the take-off direction is used as the inspection direction. In other embodiments, the left side of the take-off direction can also be used as the inspection direction. In this embodiment, the drone rotates 90° to the left after reaching above the water surface.

[0085] In this embodiment, the distance between the drone and the shoreline (i.e., the initial distance from the shore) is set to d = 10m, the flight speed is v = 5m / s, the initial flight altitude is h0 = 5-10m, and the initial pitch angle is θ0 = 45°. Furthermore, the yaw angle at this moment is recorded.

[0086] S2: According to a preset period, the real-time image captured by the drone is subjected to water surface recognition, and the flight path points are determined according to the water surface recognition result. In this embodiment, the preset period is 3s.

[0087] Specifically, this step includes the following steps:

[0088] S201: According to a preset cycle, the latest captured image is obtained as the current image, and the world coordinate system corresponding to the capturing time of the current image is used as the current world coordinate system.

[0089] S202: performing image segmentation on the current image to obtain a water surface area, and performing contour extraction on the water surface area to obtain a contour point set of the water surface area.

[0090] That is, the current image is segmented into water surface areas to obtain at least one water surface area, wherein the deeplabv3 network based on the convolutional neural network can be used for image segmentation; then contours of each water surface area are extracted to obtain a contour point set of each water surface area. In this embodiment, for each water surface area, the contour point at its upper left corner is taken as the starting point, contour points are obtained counterclockwise and stored in the corresponding contour point set in sequence.

[0091] Furthermore, the segmented water surface area can be screened and filtered to retain only the water surface area belonging to the river channel to exclude interference factors such as small puddles.

[0092] S203: screening contour points in the contour point set.

[0093] Specifically, the contour points at the boundary of the current image are removed from the contour point set, and these points do not belong to the points on the water shoreline. That is, if the image coordinates of a contour point in the image coordinate system of the current image satisfy x=0, x=width image , y=0 or y=height image , then the contour point is considered to be at the boundary of the current image and is removed from the contour point set.

[0094] In this embodiment, contour points are also screened according to the distance between contour points. Specifically, each contour point in the contour point set is subjected to an inverse perspective transformation to obtain the world coordinates in the current world coordinate system. Then, each contour point in the contour point set is traversed in sequence, and the first contour point in the contour point set is directly used as a reference point. For the contour points traversed thereafter, the distance between the contour point and the latest reference point is calculated based on the world coordinates of the contour point and the latest reference point in the current world coordinate system. If the distance is greater than a preset distance threshold (such as 5 meters), the contour point is used as a new reference point. After the contour point set is traversed, the contour points that are not reference points are removed from the contour point set, and only the reference points are retained.

[0095] S204: Acquire a nearshore point set and an opposite shore point set according to the contour point set.

[0096] Specifically, each contour point in the contour point set is traversed in order (the first and last two contour points can be ignored). If a contour point meets the preset right bank point condition, the contour point is recorded in the right bank point set; if the contour point meets the preset left bank point condition, the contour point is recorded in the left bank point set. i-1 <Y i <Y i+1 And |Y i+1 -Y i-1 |>|X i+1 -X i-1 |; The left bank point condition is Y i-1 >Y i >Y i+1 And |Y i+1 -Y i-1 |>|X i+1 -X i-1 |;(X i , Y i ) represents the world coordinates of the contour point in the current world coordinate system, (Xi-1 , Y i-1 ) and (X i+1 , Y i+1 ) respectively represent the world coordinates of the previous contour point and the next contour point of the contour point in the contour point set in the current world coordinate system.

[0097] If there are multiple contour point sets, the above operation is performed on each contour point respectively, and the finally obtained nearshore point set and opposite shore point set respectively include the nearshore points and opposite shore points of multiple water surface areas.

[0098] In this embodiment, since the right side of the take-off direction is used as the patrol direction, the near-shore point set is the right-shore point set, and the opposite-shore point set is the left-shore point set.

[0099] Further, the contour points in the nearshore point set and the opposite shore point set are arranged in ascending order according to the Y values ​​of the world coordinates in the current world coordinate system.

[0100] S205: Calculate corresponding flight path points according to the contour points in the nearshore point set, so that the UAV flies along the nearshore line.

[0101] Specifically, the riverbank orientation at a contour point in the nearshore point set can be approximated as a vector (X Pi+1 -X Pi-1 , Y Pi+1 -Y Pi-1 ), and the direction toward the center of the river can be approximated by the vector (-(Y Pi+1 -Y Pi-1 ), X Pi+1 -X Pi-1 ), then the world coordinate T of the flight path point corresponding to the contour point in the current world coordinate system i =(X Ti , Y Ti , h0),

[0102]

[0103] Among them, (X Pi , Y Pi ) represents the world coordinates of the contour point in the current world coordinate system, (X Pi-1 , Y Pi-1 ) and (X Pi+1 , Y Pi+1 ) respectively represent the world coordinates of the previous contour point and the next contour point of the contour point in the nearshore point set in the current world coordinate system; d represents the distance from the shore, and h0 represents the preset flight altitude.

[0104] That is to say, in the nearshore point set, except for the first and last two contour points, other contour points can calculate a corresponding flight path point, so that the drone can move along the river channel and maintain a distance of d meters from the river bank and a height of h0 from the water surface.

[0105] Furthermore, if the calculated Y Ti If it is less than 0, it is considered to be the point behind the drone and the flight path point is discarded.

[0106] S206: Adjust the flight path points according to the contour points in the opposite bank point set so that the UAV flies above the water surface area.

[0107] In actual scenarios, if the drone is flying at a low altitude and the river is narrow, it is safer to fly the drone in the river area. Therefore, in this embodiment, according to the distance from the opposite bank point to the path point projected on the horizontal plane, the distance from the path point to the right bank is adjusted as a whole, that is, the offshore distance is adjusted to ensure the flight safety of the drone.

[0108] Specifically, the opposite bank point set is traversed, and a contour point in the opposite bank point set is sequentially acquired as the current contour point.

[0109] The two flight path points closest to the current contour point are obtained, and the distance between the current contour point and the flight path is calculated according to the world coordinates of the current contour point and the two flight path points in the current world coordinate system.

[0110] For example, suppose the current contour point is P 对 , the world coordinates in the current world coordinate system are (X p对 , Y p对 ), the two flight path points closest to the contour point are P1 and P2, and their world coordinates in the current world coordinate system are (X p1 , Y p1 ) and (X p2 , Y p2 ), where Y p1 <Y p2 , then the distance between the contour point and the flight path is:

[0111]

[0112] After traversing the opposite bank point set, the distance between each contour point in the opposite bank point set and the flight path can be obtained, and the minimum value can be found as the corrected distance d 修正 ,like Figure 4 shown.

[0113] If d 修正<0, it means that using the current offshore distance d to calculate the flight path point will cause the path point to be outside the water area, so the offshore distance d is corrected, that is, d = d + d 修正 Then, the flight path points are recalculated according to the corrected offshore distance, that is, step S205 is re-executed.

[0114] If d 修正 ≥0, it means that the flight path points are all within the river area and no correction is required.

[0115] S3: According to the real-time depth map and the water surface recognition result, determine whether there is an obstacle ahead. If so, execute step S4; if not, execute step S6.

[0116] Since small and medium-sized bridges and sluices are very common in urban rivers, and in order to clearly capture the embankment, the flight altitude of drones is usually low, it is necessary to determine whether there are such obstacles on the drone's flight path before deciding on the next action.

[0117] In this embodiment, a binocular camera is installed on the front of the fuselage of the drone. Therefore, the depth map at the corresponding moment can be calculated through the disparity map of the real-time image captured by the binocular camera and combined with the camera parameters.

[0118] Specifically, this step includes the following steps:

[0119] S301: Obtain a contour point that belongs to the water surface area closest to the drone and is farthest from the drone from the nearshore point set as a first contour point, and obtain a contour point that is closest to the first contour point from the opposite shore point set as a second contour point.

[0120] That is, the contour point farthest from the shore can be found in the contour point set of each water surface area, such as Figure 5 As shown, the bridge divides the water surface area of ​​the river into two parts. At this time, two water surface areas can be obtained, one of which is closer to the drone and the other is farther away from the drone. In this step, the farthest nearshore contour point of the water surface area closer to the drone is obtained as the first contour point.

[0121] Then, the set of opposite bank points is traversed, and the distance between each opposite bank point and the first contour point is calculated, and the contour point closest to the first contour point is found as the second contour point. Generally speaking, the second contour point also belongs to the water surface area closest to the drone.

[0122] Furthermore, if the UAV is very close to the obstacle or the water surface is very wide, the second contour point cannot be found, and the x value in the image coordinates of the second contour point is set to 0.

[0123] S302: Obtain pixel points that meet a preset blocking point condition in the current depth map, and mark them as blocking points.

[0124] In this step, the current depth map is the latest depth map, that is, the depth map at the same time as the latest captured image in step S201.

[0125] The blocking point condition includes being located between the first contour point and the second contour point in the X-axis direction of the image coordinate system of the current depth map (i.e., x2<x<x1), not being located in the water surface area, and having a depth value less than the sum of the maximum value of the depth value of the first contour point and the depth value of the second contour point and a preset allowable value. In this embodiment, the preset allowable value is 5, i.e., D<max(D1, D2)+5. x1 and x2 respectively represent the x values ​​of the first contour point and the second contour point in the image coordinate system of the current depth map, and D1 and D2 respectively represent the depth values ​​of the first contour point and the second contour point in the current depth map.

[0126] Furthermore, in step S202, after the image is segmented to obtain the water surface area, the pixels in the water surface area may be marked as 1, and the other pixels may be marked as 0. In this step, it may be directly determined whether the image is in the water surface area based on the mark.

[0127] S303: Determine whether there is an obstacle across the water surface according to the image coordinates of each blocking point in the image coordinate system of the current depth map.

[0128] Specifically, in the current depth map, if there are blocking points of a preset first ratio or more in the pixel points of the same x value, and there are blocking points of a preset second ratio or more in the pixel points of the same y value, it is considered that there is an obstacle across the water surface. In this embodiment, the first ratio is 80% and the second ratio is 70%. That is, for a certain x value, the proportion of blocking points is greater than 80%, and for a certain y value, the proportion of blocking points is greater than 70%.

[0129] Furthermore, the minimum value of the y values ​​that meet the requirements is recorded as y 阻挡 That is, if there are blocking points with a preset second ratio or more in the pixel points with the same y value, it is considered that the same y value is within the y value range corresponding to the obstacle, and the minimum value in the y value range corresponding to the obstacle is obtained as the top y value of the obstacle.

[0130] S4: Determine whether the depth value of the obstacle is less than a preset depth threshold, if so, execute step S5, if not, execute step S6.

[0131] In this embodiment, the preset depth threshold is 20 meters. This step is equivalent to determining whether the distance between the drone and the obstacle is less than 20 meters. If so, different path strategies are implemented according to different situations. Otherwise, it is considered safe and unobstructed.

[0132] S5: reducing the flight speed, and adding a new flight path point according to the flight path point and the height of the obstacle.

[0133] Specifically, after determining that there is an obstacle across the water surface ahead, the flight speed is first reduced. In this embodiment, the flight speed is reduced to half of the original speed.

[0134] Then search in the nearshore point set. If there is a contour point in the nearshore point set, the depth value of the contour point is greater than the sum of the depth value of the obstacle and the preset allowable value (i.e. D>D 阻挡 +5), that is, there is a contour point farther from the drone than the obstacle, that is, the water surface area behind the obstacle can be seen in the image taken by the drone, then it is considered that the drone can directly pass through from the bottom, or fly over from the top by the obstacle avoidance algorithm, which generally corresponds to the case of high piers and simple bridges. At this time, directly execute step S6.

[0135] If there is no contour point that meets the above conditions in the nearshore point set, then obtain the last contour point in the nearshore point set. Since this is equivalent to the obstacle blocking the water surface area behind, the last contour point in the nearshore point set at this time is the farthest nearshore contour point of the water surface area close to the drone, which is the first contour point mentioned above. Then estimate the height of the obstacle based on the world coordinates of the first contour point in the current world coordinate system, the top y value of the obstacle, and the current flight altitude and pitch angle. Generally speaking, the bridge sluice is in the straight line of the river channel. When the obstacle appears 20 meters in front of the drone, it is usually located directly in front. Therefore, it can be considered that the Y value of the first contour point in the current world coordinate system is similar to the Y value of the obstacle in the current world coordinate system. That is, the Y value of the first contour point in the world coordinate system can be regarded as the Y value of the obstacle, recorded as Y 终 , substitute into the following formula to estimate the height h of the obstacle 阻挡 :

[0136]

[0137] Among them, y 阻挡 represents the top y value of the obstacle; h represents the current flight altitude, which is measured by the sonar device mounted on the bottom of the drone fuselage; θ represents the current pitch angle. If the pitch angle has not changed, it is consistent with the initial pitch angle; height image Indicates the height at which the image was captured.

[0138] If the height of the obstacle is less than the preset height threshold, in this embodiment, the height threshold is 10 meters, that is, h 阻挡<10, then according to the last two flight path points between the UAV and the obstacle (since only the water surface area between the UAV and the obstacle can be photographed at this time, correspondingly only the flight path points between the UAV and the obstacle can be determined, so this is also equivalent to the last two path points in the flight path points determined in step S2), two new flight path points are determined, and the world coordinates of the two new flight path points in the current world coordinate system are respectively:

[0139] and

[0140]

[0141] in, and They respectively represent the world coordinates of the last two flight path points in the current world coordinate system, and L represents the preset flight distance.

[0142] In this embodiment, L = 10 meters. That is, first increase the flight altitude of the drone according to the height of the obstacle, and then fly forward 10 meters along the original path direction to fly over the obstacle. When flying forward along the newly added flight path point, the water surface area behind the obstacle can be photographed, and the path points can be calculated according to the above steps. If the water surface area of ​​the river is not found after crossing the obstacle, the search is terminated and the drone is controlled to return.

[0143] If the height of the obstacle is greater than or equal to the preset height threshold, i.e. h 阻挡 If the value is ≥10, it means that the obstacle is a tall building, such as a large sluice gate. The situation is more complicated at this time. From a safety perspective, the search is terminated directly and the drone is controlled to return.

[0144] S6: According to the real-time positioning position of the UAV, the map coordinates of each flight path point on the map are calculated respectively, and the flight path is updated according to a preset number of flight path points closest to the UAV.

[0145] Specifically, obtain the GPS positioning information (longitude) of the drone at the current time (i.e., the time when the current image is taken in step S201). O , latitude O ), first correct the yaw angle of the current world coordinate system to make it face due north, then convert it into the offset of longitude and latitude based on the offset distance of each flight path point equivalent to the origin of the current world coordinate system. Assuming that the world coordinates of a flight path point in the current world coordinate system are (X, Y), the corresponding yaw angle is The calculation formula for its longitude and latitude on the map is:

[0146]

[0147] Y′=Ycos(φ)–X sin(φ)

[0148]

[0149] latitude=latitude O +Y′*0.00000898

[0150] Then the historical path points in the drone control system (i.e., the flight path points calculated in the previous cycle) are cleared, and the drone path points are updated to the flight path points calculated in the current cycle. In this embodiment, only the five flight path points closest to the drone are retained. In the current cycle, the drone is controlled to fly forward according to these five flight path points.

[0151] Furthermore, each time a flight path point is completed, including the position at the moment of starting to execute a new path, its map coordinates on the map are recorded, which can be used for subsequent recognition modeling.

[0152] The drone can repeat the above process when returning, so that it can patrol the other side during the return journey.

[0153] The drone should calculate the obstacle avoidance path throughout the process of moving forward to avoid protruding branches, cables, etc. Usually, the binocular camera can provide a relatively fine obstacle perception capability within a distance of 30 meters. A typical obstacle avoidance algorithm is 3DVFH. This obstacle avoidance method is a local planning method suitable for processing forward binocular depth images. It can solve the problem of drones avoiding obstacles and reaching the target path point in 3D space. You can also directly use the obstacle avoidance function of the drone platform.

[0154] This embodiment can realize autonomous bank patrol and obstacle avoidance of the UAV, and realize the full process automation of river bank protection and surrounding environment inspection.

[0155] Embodiment 2

[0156] This embodiment is a computer-readable storage medium corresponding to the above embodiment, on which a computer program is stored. When the program is executed by the processor, it implements the various steps of a drone shore protection patrol and obstacle avoidance method in the above embodiment, and can achieve the same technical effect, which will not be repeated here.

[0157] In summary, the present invention provides a method for UAV shore protection patrol and obstacle avoidance and a computer-readable storage medium, which determines subsequent flight path points based on the water surface recognition results of the captured images in real time during the flight, so that the UAV can automatically fly along the shoreline of the water area without manually setting the route. Compared with the designated fixed cruise route, it can be more convenient in new environments or temporary patrol scenarios; at the same time, by real-time detection of whether there are obstacles ahead, and when approaching the obstacles, new flight path points are added according to the current flight path points and the height of the obstacles, so that the UAV can fly over the obstacles along the current flight path, thereby achieving automatic obstacle avoidance and effectively ensuring the safety of the UAV.

[0158] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for avoiding obstacles during bank protection inspection by a drone, characterized in that: include: Determine the initial flight position of the UAV according to the preset flight altitude, pitch angle and initial orientation, and control the UAV to fly forward from the initial flight position according to the preset flight speed; According to the preset cycle, the real-time shooting images of the UAV are subjected to water surface recognition, and the flight path points are determined based on the water surface recognition results so that the UAV flies along the coastline; Obtain a real-time depth map through the real-time images captured by the binocular camera of the drone, and determine whether there is an obstacle ahead based on the real-time depth map and the water surface recognition result; If the obstacle exists and the depth value of the obstacle is less than the preset depth threshold, a new flight path point is added according to the flight path point and the height of the obstacle; According to the real-time positioning position of the UAV, the map coordinates of each flight path point on the map are calculated respectively, and the flight path is updated according to the preset number of flight path points closest to the UAV.

2. The method for avoiding obstacles while patrolling with a UAV bank protection according to claim 1 is characterized in that: At a moment in the drone's flight, the world coordinate system corresponding to the moment is constructed with the vertical projection position of the drone on the plane where the water surface is located as the origin, the plane where the water surface is located as the XOY plane, the direction of the drone's camera as the Y-axis direction, and the vertical direction as the Z-axis direction.

3. The method for avoiding obstacles while patrolling with a UAV bank protection according to claim 2 is characterized in that: The method of performing water surface recognition on the real-time image captured by the drone according to a preset period, and determining a flight path point according to the water surface recognition result so that the drone flies along the coastline, includes: According to a preset cycle, the latest captured image is obtained as the current image, and the world coordinate system corresponding to the shooting time of the current image is used as the current world coordinate system; Performing image segmentation on the current image to obtain a water surface area, and performing contour extraction on the water surface area to obtain a contour point set of the water surface area, wherein the contour points in the contour point set take the contour point close to the upper left corner of the image as a starting point and are arranged in a counterclockwise direction; Eliminate contour points at the boundary of the current image from the contour point set; According to the contour point set, a nearshore point set and an opposite shore point set are obtained; According to the contour points in the nearshore set, corresponding flight path points are calculated so that the UAV flies along the nearshore line.

4. The method for avoiding obstacles while patrolling with a UAV bank protection according to claim 3 is characterized in that: If the initial orientation is the right direction after the drone takes off towards the river center, the nearshore point set is the right bank point set, and the opposite bank point set is the left bank point set; if the initial orientation is the left direction after the drone takes off towards the river center, the nearshore point set is the left bank point set, and the opposite bank point set is the right bank point set; The step of obtaining a nearshore point set and an opposite shore point set according to the contour point set is specifically as follows: If a contour point meets the preset right bank point condition, the contour point is recorded in the right bank point set, and the right bank point condition is Y i-1 <Y i <Y i+1 And |Y i+1 -Y i-1 |>|X i+1 -X i-1 |; If a contour point meets the preset left bank point condition, the contour point is recorded in the left bank point set, and the left bank point condition is Y i-1 >Y i >Y i+1 And |Y i+1 -Y i-1 |>|X i+1 -X i-1 |; Among them, (X i , Y i ) represents the world coordinates of the contour point in the current world coordinate system, (X i-1 , Y i-1 ) represents the world coordinates of the previous contour point of the contour point in the contour point set in the current world coordinate system, (X i+1 , Y i+1 ) represents the world coordinates of the next contour point of the contour point in the contour point set in the current world coordinate system.

5. The method for avoiding obstacles while patrolling with a UAV bank protection according to claim 3 is characterized in that: The world coordinate T of the flight path point corresponding to a contour point in the nearshore point set in the current world coordinate system i =(X Ti , Y Ti , h0), Among them, (X Pi , Y Pi ) represents the world coordinates of a contour point in the nearshore point set in the current world coordinate system, (X Pi-1 , Y Pi-1 ) and (X Pi+1 , Y Pi+1 ) respectively represent the world coordinates of the previous contour point and the next contour point of the contour point in the nearshore point set in the current world coordinate system; d represents the preset offshore distance, and h0 represents the preset flight altitude.

6. The method for avoiding obstacles while patrolling with a UAV bank protection according to claim 1 is characterized in that: Before adding a new flight path point according to the flight path point and the height of the obstacle, the method further includes: Reduce the flight speed of the drone.

7. The method for avoiding obstacles while patrolling with a UAV bank protection according to claim 3 is characterized in that: The determining whether there is an obstacle ahead according to the real-time depth map and the water surface recognition result includes: Acquire, from the nearshore point set, a contour point that belongs to the water surface area closest to the drone and is farthest from the drone as a first contour point, and acquire, from the opposite shore point set, a contour point that is closest to the first contour point as a second contour point; If a pixel point in the current depth map satisfies a preset blocking point condition, the pixel point is marked as a blocking point, wherein the blocking point condition includes being located between the first contour point and the second contour point in the X-axis direction of the image coordinate system of the current depth map, not being located in the water surface area, and having a depth value less than the sum of the maximum value of the depth value of the first contour point and the depth value of the second contour point and a preset allowable value; If in the current depth map, there are blocking points greater than a preset first proportion among the pixel points with the same x value, and there are blocking points greater than a preset second proportion among the pixel points with the same y value, then it is determined that there is an obstacle across the water surface, and the minimum value in the y value range corresponding to the obstacle is obtained as the top y value of the obstacle.

8. The method for avoiding obstacles while patrolling with a UAV bank protection according to claim 7 is characterized in that: The adding of a new flight path point according to the flight path point and the height of the obstacle includes: estimating the height of the obstacle according to the world coordinates of the first contour point in the current world coordinate system, the top y value of the obstacle, and the current flight altitude and pitch angle; If the height of the obstacle is less than the preset height threshold, two new flight path points are determined according to the last two flight path points between the UAV and the obstacle. The world coordinates of the two new flight path points in the current world coordinate system are: and in, and They represent the world coordinates of the last two flight path points between the UAV and the obstacle in the current world coordinate system, h 阻挡 represents the height of the obstacle, and L represents the preset flight distance.

9. The method for avoiding obstacles while patrolling with a UAV bank protection according to claim 8 is characterized in that: Before estimating the height of the obstacle according to the world coordinates of the first contour point in the current world coordinate system, the top y value of the obstacle, and the current flight altitude and pitch angle, the method further includes: Determine whether there is a contour point in the nearshore point set, the depth value of the contour point being greater than the sum of the depth value of the obstacle and a preset allowable value; If so, the step of calculating the map coordinates of each flight path point on the map according to the real-time positioning position of the drone and updating the flight path according to a preset number of flight path points closest to the drone is executed; If not, the step of estimating the height of the obstacle is performed according to the world coordinates of the first contour point in the current world coordinate system, the top y value of the obstacle, and the current flight altitude and pitch angle.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images