Real-time obstacle avoidance method and device for unmanned aerial vehicle

Abstract

The invention relates to the technical field of obstacle avoidance, and provides a real-time obstacle avoidance method and device for an unmanned aerial vehicle. The method comprises the following steps: acquiring a real-time area video when the unmanned aerial vehicle flies according to a preset flight line; inputting the real-time area video into an obstacle recognition model to obtain an obstacle target; converting the pixel position error of the obstacle target in the real-time area video into an actual position error; correcting the actual position based on the actual position error to obtain a corrected position of the obstacle target; and controlling the unmanned aerial vehicle to avoid the obstacle in real time based on the corrected position of the obstacle target. The obstacle recognition model is obtained by training a historical area video of the unmanned aerial vehicle and an obstacle label in the historical area video on the basis of the generative adversarial network. The obstacle can be accurately identified in the low airspace, and the positioning error of the obstacle is reduced, so that the obstacle is timely avoided, and the flight safety of the unmanned aerial vehicle and the smooth completion of the monitoring task are ensured.

Description

Technical Field

[0001] This application relates to the technical field of obstacle avoidance, and particularly to a method and device for real-time obstacle avoidance of an unmanned aerial vehicle (UAV). Background Art

[0002] Low-altitude monitoring of UAVs is a technology that uses UAVs as flight platforms, carrying high-resolution digital remote sensing equipment or other sensors to collect data and monitor in the low-altitude area. This technology has advantages such as mobility, rapidity, and economy, and can achieve rapid and accurate monitoring of target areas. It is widely used in urban management, public safety, agriculture and forestry, and power inspection fields, and has broad application prospects and important social value.

[0003] There are usually many obstacles in the low-altitude area. Therefore, when a UAV conducts low-altitude monitoring, it is necessary to effectively identify and locate obstacles in a timely manner to avoid obstacles and ensure the flight safety of the UAV and the smooth completion of the monitoring task.

[0004] However, the existing UAV obstacle avoidance methods, on the one hand, cannot accurately identify obstacles in the case of variable climate and complex terrain and landforms in the low-altitude area; on the other hand, there is a large error in the positioning of obstacles, and the positioning error will further increase when the obstacle identification is inaccurate. In summary, the existing UAV obstacle avoidance methods cannot accurately identify obstacles in the low-altitude area and have a large error in the positioning of obstacles. Therefore, they cannot avoid obstacles in a timely manner, seriously affecting the flight safety of the UAV and the smooth completion of the monitoring task. Summary of the Invention

[0005] The embodiments of this application provide a method and device for real-time obstacle avoidance of a UAV to solve the technical problem that the existing UAV obstacle avoidance methods cannot accurately identify obstacles in the low-altitude area and have a large error in the positioning of obstacles. Therefore, they cannot avoid obstacles in a timely manner, seriously affecting the flight safety of the UAV and the smooth completion of the monitoring task.

[0006] In the first aspect, the embodiments of this application provide a method for real-time obstacle avoidance of a UAV, including: Collecting real-time regional video when the UAV flies along a preset flight route; Inputting the real-time regional video into an obstacle recognition model to obtain an obstacle target output by the obstacle recognition model; Based on the mapping relationship between the pixel position of the obstacle target in the real-time regional video and the actual position of the obstacle target, obtaining the actual position of the obstacle target; Converting the pixel position error of the obstacle target in the real-time regional video into an actual position error; Correct the actual position based on the actual position error to obtain the corrected position of the obstacle target; Control the UAV to avoid obstacles in real time based on the corrected position of the obstacle target; The obstacle recognition model is trained based on the historical area video of the UAV and the obstacle labels therein on the basis of a generative adversarial network.

[0007] In one embodiment, the conversion of the pixel position error of the obstacle target in the real-time area video into the actual position error includes: Obtain the picture frame in the real-time area video where the obstacle target exists; Convert the pixel position error of the obstacle target in the picture frame into the actual position error.

[0008] In one embodiment, the conversion of the pixel position error of the obstacle target in the picture frame into the actual position error includes: Obtain the number of pixels of the picture frame in the horizontal and vertical directions to obtain the picture pixel width and the picture pixel height; Obtain the vertical distance between the horizontal plane where the obstacle target is located and the horizontal plane where the acquisition camera of the real-time area video is located; Obtain the attribute parameters of the acquisition camera; Based on the picture pixel width, the vertical distance, and the attribute parameters, calculate the actual horizontal position error corresponding to the pixel horizontal position error of the obstacle target in the picture frame; Based on the picture pixel height, the vertical distance, and the attribute parameters, calculate the actual vertical position error corresponding to the pixel vertical position error of the obstacle target in the picture frame.

[0009] In one embodiment, the control of the UAV to avoid obstacles in real time based on the corrected position of the obstacle target includes: Randomly generate a first set of random points at the current position of the UAV; Randomly generate a second set of random points at the end position of the preset flight route; Randomly generate a third random point and a fourth random point on and near the preset flight route; Obtain the first random point closest to the third random point in the first set of random points to obtain a first target point; Obtain the second random point closest to the fourth random point in the second set of random points to obtain a second target point; Move the first target point towards the third random point by a first distance to obtain a third target point; Move the second target point towards the fourth random point by a second distance to obtain a fourth target point; If the corrected position of the obstacle target exists on the movement path from the first target point to the third target point or on the movement path between the second target point and the fourth target point, then return to the step of randomly generating a third random point and a fourth random point on and near the preset flight route until the corrected position of the obstacle target does not exist on the movement path from the first target point to the third target point and on the movement path between the second target point and the fourth target point; Add the current third target point to the first random point set and add the current fourth target point to the second random point set; If the distance between the third target point and the fourth target point is greater than or equal to the distance threshold, then return to the step of randomly generating a third random point and a fourth random point on and near the preset flight route until the distance between the third target point and the fourth target point is less than the distance threshold; Generate an obstacle avoidance route for the drone based on the current first target point, second target point, third target point, and fourth target point; Control the drone to avoid obstacles in real time based on the obstacle avoidance route.

[0010] In one embodiment, the controlling the drone to avoid obstacles in real time based on the obstacle avoidance route includes: Prune and interpolate the obstacle avoidance route to obtain an optimized route; Set a speed constraint space for the drone based on the current flight state of the drone; Generate a plurality of feasible routes between the current position of the drone and the end position of the preset flight route based on the speed constraint space; Select the feasible route with the highest similarity to the optimized route from the plurality of feasible routes to obtain a target feasible route; Control the drone to fly along the optimized route based on the speed constraint corresponding to the target feasible route.

[0011] In one embodiment, after controlling the drone to fly along the optimized route based on the speed constraint corresponding to the target feasible route, it includes: Extract key nodes in the optimized route; Track the flight of the drone based on the key nodes and detect whether there is an obstacle target in the front route of the drone during the tracking process; When there is an obstacle target, control the drone to avoid obstacles in real time based on a preset obstacle avoidance strategy.

[0012] Second aspect, an embodiment of the present application provides a real-time obstacle avoidance device for a drone, including: A video acquisition module, configured to: acquire a real-time area video when the drone flies along a preset flight route; An obstacle recognition module, configured to: input the real-time area video into an obstacle recognition model to obtain an obstacle target output by the obstacle recognition model; An obstacle position acquisition module, configured to: based on the mapping relationship between the pixel position of the obstacle target in the real-time area video and the actual position of the obstacle target, acquire the actual position of the obstacle target; An obstacle error conversion module, configured to: convert the pixel position error of the obstacle target in the real-time area video into an actual position error; An obstacle position correction module, configured to: based on the actual position error, correct the actual position to obtain a corrected position of the obstacle target; A real-time obstacle avoidance module for the drone, configured to: based on the corrected position of the obstacle target, control the drone to avoid obstacles in real time; The obstacle recognition model is trained based on the historical area video of the drone and the obstacle labels therein on the basis of a generative adversarial network.

[0013] Third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory storing a computer program, and when the processor executes the program, the steps of the drone real-time obstacle avoidance method described in the first aspect are implemented.

[0014] Fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the drone real-time obstacle avoidance method described in the first aspect are implemented.

[0015] Fifth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, including a computer program, and when the computer program is executed by a processor, the steps of the drone real-time obstacle avoidance method described in the first aspect are implemented.

[0016] The real-time obstacle avoidance method and device provided by this application collect the real-time regional video when the drone flies along a preset flight route, input the real-time regional video into an obstacle recognition model to obtain the obstacle targets output by the obstacle recognition model, and based on the mapping relationship between the pixel positions of the obstacle targets in the real-time regional video and the actual positions of the obstacle targets, obtain the actual positions of the obstacle targets, convert the pixel position error of the obstacle targets in the real-time regional video into an actual position error, correct the actual position based on the actual position error to obtain the corrected positions of the obstacle targets, and control the drone to avoid obstacles in real time based on the corrected positions of the obstacle targets. On the one hand, since the obstacle recognition model is trained through the historical regional videos of the drone and the obstacle labels therein based on a generative adversarial network, when the obstacle recognition model recognizes obstacles, it can utilize the characteristics of the generative adversarial network to accurately extract obstacle features from the rich and complex features presented by the changeable climate and complex terrain and landforms in the low-altitude region in the real-time regional video, thereby realizing the accurate recognition of obstacle targets; on the other hand, since the pixel position error of the obstacle targets is converted into an actual position error and the actual position is corrected, the positioning error of the obstacle targets can be reduced. When the recognition accuracy of the obstacle targets is improved, the positioning error can be further reduced. In summary, this application can accurately identify obstacles in the low-altitude region and reduce the positioning error of the obstacles, so as to avoid obstacles in time and ensure the flight safety of the drone and the smooth completion of the monitoring task. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in this application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 is one of the flow diagrams of the real-time obstacle avoidance method for drones provided by the embodiments of this application; Figure 2 is the second flow diagram of the real-time obstacle avoidance method for drones provided by the embodiments of this application; Figure 3 is the third flow diagram of the real-time obstacle avoidance method for drones provided by the embodiments of this application; Figure 4 is the fourth flow diagram of the real-time obstacle avoidance method for drones provided by the embodiments of this application; Figure 5 is the fifth flow diagram of the real-time obstacle avoidance method for drones provided by the embodiments of this application; Figure 6It is a schematic structural diagram of the real-time obstacle avoidance device for drones provided by the embodiments of the present application; Figure 7 It is a schematic structural diagram of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0019] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some but not all of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without making creative efforts shall fall within the scope of protection of the present application.

[0020] It should be noted that in the description of the embodiments of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device including a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element. The orientation or positional relationship indicated by terms such as "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present application. Unless otherwise expressly specified and limited, the terms "mounted", "connected" and "coupled" should be construed broadly, for example, it may be a fixed connection, a detachable connection or an integral connection; it may be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the internal communication of two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0021] The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same category, and do not limit the number of objects. For example, the first object may be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0022] Figure 1 This is one of the flow charts of the real-time obstacle avoidance method for a drone provided in an embodiment of the present application. Figure 1 , the embodiment of the present application provides a real-time obstacle avoidance method for a drone, which may include: 101. Collect real-time regional video when the drone flies along a preset flight route; 102. Input the real-time regional video into the obstacle recognition model to obtain the obstacle target output by the obstacle recognition model; 103. Acquire the actual position of the obstacle target based on the mapping relationship between the pixel position of the obstacle target in the real-time regional video and the actual position of the obstacle target; 104. Convert the pixel position error of the obstacle target in the real-time area video into an actual position error; 105. Correct the actual position based on the actual position error to obtain a corrected position of the obstacle target; 106. Based on the corrected position of the obstacle target, control the drone to avoid obstacles in real time.

[0023] The obstacle recognition model is trained based on the generative adversarial network through the historical area videos of the drone and the obstacle labels therein.

[0024] In step 101, the preset flight route of the drone is pre-planned according to the requirements of the monitoring task, combined with the model of the drone and the equipment it carries. The real-time regional video is a video of the area in front of the drone collected in real time by a collection camera while the drone is flying. After the video is collected, it can be uploaded to a cloud data center or a ground data center for analysis and storage.

[0025] In step 104 to step 105, since there is a one-to-one mapping relationship between the pixel position and the actual position, the pixel position error will lead to the actual position error, and there is also a corresponding relationship between the two errors. The pixel position error can be converted into the actual position error and the actual position can be corrected. While reducing the positioning error and facilitating subsequent real-time obstacle avoidance, it can also facilitate the operation and maintenance personnel to reach the corrected position in time to repair the drone when a malfunction occurs.

[0026] The real-time obstacle avoidance method for drones provided in this embodiment collects the real-time regional video when the drone flies along a preset flight route, inputs the real-time regional video into an obstacle recognition model to obtain the obstacle targets output by the obstacle recognition model, and based on the mapping relationship between the pixel positions of the obstacle targets in the real-time regional video and the actual positions of the obstacle targets, obtains the actual positions of the obstacle targets, converts the pixel position error of the obstacle targets in the real-time regional video into an actual position error, corrects the actual position based on the actual position error to obtain the corrected positions of the obstacle targets, and controls the drone to avoid obstacles in real time based on the corrected positions of the obstacle targets. On the one hand, in this embodiment, since the obstacle recognition model is trained through the historical regional video of the drone and the obstacle labels therein based on a generative adversarial network, when the obstacle recognition model recognizes obstacles, it can utilize the characteristics of the generative adversarial network to accurately extract obstacle features from the rich and complex features presented by the changing climate and complex terrain and landforms in the low-altitude region in the real-time regional video, thereby achieving accurate recognition of obstacle targets. On the other hand, since the pixel position error of the obstacle targets is converted into an actual position error and the actual position is corrected, the positioning error of the obstacle targets can be reduced. When the recognition accuracy of the obstacle targets is improved, the positioning error can be further reduced. In summary, this embodiment can accurately recognize obstacles in the low-altitude region and reduce the positioning error of the obstacles, so as to avoid obstacles in time and ensure the flight safety of the drone and the smooth completion of the monitoring task.

[0027] Figure 2 It is the second flowchart of the real-time obstacle avoidance method for drones provided by an embodiment of this application. Refer to Figure 2 , in one embodiment, converting the pixel position error of the obstacle target in the real-time regional video into an actual position error may include: 201. Obtain the picture frame with obstacle targets in the real-time regional video; 202. Convert the pixel position error of the obstacle target in the picture frame into an actual position error.

[0028] In step 202, for each picture frame, the pixel position error of the obstacle target therein may be converted into an actual position error according to the following method: 202a. Obtain the number of pixels in the horizontal and vertical directions of the picture frame to obtain the picture pixel width and the picture pixel height; 202b. Obtain the vertical distance between the horizontal plane where the obstacle target is located and the horizontal plane where the acquisition camera of the real-time regional video is located; 202c. Obtain the attribute parameters of the acquisition camera; 202d. Calculate the actual horizontal position error corresponding to the pixel horizontal position error of the obstacle target in the image frame based on the pixel width, vertical distance, and attribute parameters of the image. 202e. Calculate the actual vertical position error corresponding to the pixel vertical position error of the obstacle target in the image frame based on the pixel height, vertical distance, and attribute parameters of the image.

[0029] In practical applications, there is no strict timing relationship among steps 202a, 202b, and 202c; that is, they can be executed simultaneously, or any one of the steps can be executed first, depending on the actual requirements, and it is not limited here; in addition, there is also no strict timing relationship between steps 202d and 202e; that is, they can be executed simultaneously, or any one of the steps can be executed first, depending on the actual requirements, and it is not limited here either.

[0030] In step 202c, the attribute parameters of the acquisition camera may include the height of the sensor in the acquisition camera (i.e., the length in the vertical direction), the width of the sensor (i.e., the length in the horizontal direction), the minimum focal length of the acquisition camera, the zoom ratio of the acquisition camera, etc.

[0031] In steps 202d to 202e, the actual horizontal position error can be calculated according to the following formula and the actual vertical position error : ; ; where is the pixel width of the image, is the pixel horizontal position error, that is, the distance between the center position of the image frame and the center position of the obstacle target in the horizontal direction, is the width of the sensor in the acquisition camera, is the pixel height of the image, is the pixel vertical position error, that is, the distance between the center position of the image frame and the center position of the obstacle target in the vertical direction, is the height of the sensor in the acquisition camera, is the vertical distance, is the minimum focal length of the acquisition camera, is the zoom ratio of the acquisition camera.

[0032] For any obstacle target, all picture frames from the time point when the obstacle target starts to appear to the time point when it completely appears can be obtained from the real-time area video. After converting the pixel position errors of the obstacle target in these picture frames into actual position errors, all the actual position errors are fused to obtain the actual position fusion error of the obstacle target. Subsequently, the actual position of the obstacle target is corrected based on the actual position fusion error to obtain the corrected position. For any obstacle target, multiple picture frames in which the obstacle target completely appears within a preset time period can also be obtained from the real-time area video. After converting the pixel position errors of the obstacle target in these picture frames into actual position errors, all the actual position errors are used to correct each other to obtain the actual position correction error of the obstacle target. Subsequently, the actual position of the obstacle target is corrected based on the actual position correction error to obtain the corrected position. It should be noted that in order to ensure the real-time performance of obstacle avoidance, the preset time period in this method should be as short as possible.

[0033] Based on the picture pixel width, vertical distance, and the attribute parameters of the acquisition camera, this embodiment can accurately convert the pixel horizontal position error of the obstacle target in the picture frame into the actual horizontal position error, and based on the picture pixel height, vertical distance, and the attribute parameters of the acquisition camera, can accurately convert the pixel vertical position error of the obstacle target in the picture frame into the actual vertical position error, and provides multiple ways to use these actual position errors to correct the actual position. Whether the obstacle target is complete in the picture frame or not, the actual position of the obstacle target can be accurately corrected.

[0034] Figure 3 is the third flow diagram of the real-time obstacle avoidance method for drones provided by the embodiments of this application. Refer to Figure 3 , in one embodiment, based on the corrected position of the obstacle target, controlling the drone for real-time obstacle avoidance may include: 301. Randomly generate a first set of random points at the current position of the drone; 302. Randomly generate a second set of random points at the end position of the preset flight route; 303. Randomly generate a third random point and a fourth random point on the preset flight route and its vicinity; 304. Obtain the first random point closest to the third random point in the first set of random points to get the first target point; 305. Obtain the second random point closest to the fourth random point in the second set of random points to get the second target point; 306. Move the first target point towards the third random point by a first distance to get the third target point; 307. Move the second target point towards the fourth random point by a second distance to get the fourth target point; 308. If the corrected position of the obstacle target exists on the movement path from the first target point to the third target point or on the movement path between the second target point and the fourth target point, return to step 303; 309. If the corrected position of the obstacle target does not exist on the movement path from the first target point to the third target point and on the movement path between the second target point and the fourth target point, add the current third target point to the first random point set and add the current fourth target point to the second random point set; 310. If the distance between the third target point and the fourth target point is greater than or equal to the distance threshold, return to step 303; 311. If the distance between the third target point and the fourth target point is less than the distance threshold, generate an obstacle avoidance route for the drone based on the current first target point, second target point, third target point, and fourth target point; 312. Control the drone to avoid obstacles in real time based on the obstacle avoidance route.

[0035] In steps 306 to 307, the first distance and the second distance may be the same or different, which is not limited here. Move the first target point to the third random point, that is, start from the first target point to search for a path to the end position of the preset flight route, and move the second target point to the fourth random point, that is, start from the second target point to search for a path to the current position of the drone.

[0036] In steps 308 to 309, that is, during the two-way path search process, as long as the movement path of the target point in one direction collides with the obstacle target, it is determined that the current two-way movement path is not credible, discard the third target point and the fourth target point, and randomly generate a third random point and a fourth random point near the preset flight route until the movement paths of the target points in both directions do not collide with the obstacle target, determine that the current two-way movement path is feasible, and add the third target point and the fourth target point to the corresponding random point sets.

[0037] In steps 310 to 311, that is, after determining that the bidirectional movement path is feasible, further determine whether the distance between the third target point and the fourth target point is close enough. If the distance is far and it is difficult to ensure whether there are obstacle targets on the movement path between the third target point and the fourth target point, randomly generate a third random point and a fourth random point again near the preset flight route and its vicinity. At this time, the third target point will participate in the screening of the first target point as a member of the first random point set, and the fourth target point will return to the second random point set as a member to participate in the screening of the second target point until the distance between the newly obtained third target point and the fourth target point is close enough to minimize the possibility of there being obstacle targets on the movement path between the two target points. At this time, an obstacle avoidance route is formed from the first target point to the third target point and from the third target point to the second target point.

[0038] This embodiment continuously searches for an obstacle-free route between the current position of the drone and the end position based on random points near the preset flight route and its vicinity. By introducing random factors, multiple movement paths with a high similarity to the preset flight route can be generated for screening, and the global optimal solution of the movement path between the current position and the end position can be reached as soon as possible, thereby generating an obstacle avoidance route to achieve real-time obstacle avoidance of the drone.

[0039] Figure 4 It is the fourth flowchart of the drone real-time obstacle avoidance method provided by the embodiments of the present application. Refer to Figure 4 , in one embodiment, based on the obstacle avoidance route, controlling the drone for real-time obstacle avoidance may include: 401. Prune and interpolate the obstacle avoidance route to obtain an optimized route; 402. Set the speed constraint space of the drone based on the current flight state of the drone; 403. Generate multiple feasible routes between the current position of the drone and the end position of the preset flight route based on the speed constraint space; 404. Select the feasible route with the highest similarity to the optimized route from multiple feasible routes to obtain a target feasible route; 405. Control the drone to fly along the optimized route based on the speed constraint corresponding to the target feasible route.

[0040] In step 401, when pruning the obstacle avoidance route, redundant points in the route can be removed. When interpolating the obstacle avoidance route, the route can be made smoother, which is beneficial to the safe and stable flight of the drone.

[0041] In step 402, the drone is restricted by various aspects such as its own performance, task requirements, environmental conditions, and safety rules, and its flight speed forms a speed constraint space between the maximum value and the minimum value.

[0042] In steps 403 to 404, although the optimized route was obtained previously and the UAV can avoid obstacles in real time to the greatest extent when flying along this optimized route, under the speed constraint, it may not be sufficient to support the UAV to fly along this optimized route. Therefore, first generate all feasible routes based on the speed constraint, and then screen out the target feasible route that is closest to the optimized route, and the corresponding speed constraint can support the UAV to fly along this optimized route to the greatest extent.

[0043] In this embodiment, first optimize the obstacle avoidance route to make it smoother, and then introduce the speed constraint of the UAV. Screen out the target feasible route that is closest to the optimized route from all feasible routes under the speed constraint, and use the speed constraint corresponding to this target feasible route to control the UAV to fly along this optimized route, so as to achieve real-time obstacle avoidance of the UAV under the speed constraint.

[0044] Figure 5 It is the fifth flow diagram of the UAV real-time obstacle avoidance method provided by the embodiments of the present application. Refer to Figure 5 , in one embodiment, after controlling the UAV to fly along the optimized route based on the speed constraint corresponding to the target feasible route, it may include: 501. Extract key nodes in the optimized route; 502. Track the flight of the UAV based on the key nodes, and detect whether there are obstacle targets on the front route of the UAV during the tracking process; 503. When there are obstacle targets, control the UAV to avoid obstacles in real time based on the preset obstacle avoidance strategy.

[0045] In step 501, the key nodes can be the starting point, ending point, points near the obstacle target, turning points, speed adjustment points, etc. in the optimized route, which are not limited here.

[0046] In steps 502 to 503, due to the speed constraint, there may be a certain deviation when the UAV flies along the optimized route. Therefore, further perform real-time detection of obstacles on the front route of the UAV, and control the UAV to avoid obstacles in real time based on the preset obstacle avoidance strategy to avoid obstacle avoidance failure caused by flight route deviation.

[0047] Furthermore, when the UAV performs real-time obstacle avoidance, if an abnormal situation occurs, an abnormal report will be generated, and while pushing this abnormal report to the administrator, an abnormal alarm will be sent to the administrator, so that the administrator can handle the abnormal situation in time, including improving the obstacle avoidance strategy, etc.

[0048] When the UAV flies along the optimized route in this embodiment, it detects in real time whether there are obstacle targets on the front route. On the premise of controlling the UAV to perform real-time obstacle avoidance at the front area level, it further controls the UAV to perform real-time obstacle avoidance at the front route level to avoid the failure of obstacle avoidance when the flight route deviates due to speed constraints.

[0049] The UAV real-time obstacle avoidance device provided by the embodiments of the present application will be described below. The UAV real-time obstacle avoidance device described below can be correspondingly referred to the UAV real-time obstacle avoidance method described above.

[0050] Figure 6 It is a schematic structural diagram of the UAV real-time obstacle avoidance device provided by the embodiments of the present application. Refer to Figure 6 The embodiments of the present application provide a UAV real-time obstacle avoidance device, which may include: A video acquisition module 601, configured to: acquire a real-time area video when the UAV flies along a preset flight route; An obstacle recognition module 602, configured to: input the real-time area video into an obstacle recognition model to obtain obstacle targets output by the obstacle recognition model; An obstacle position acquisition module 603, configured to: acquire the actual position of the obstacle target based on the mapping relationship between the pixel position of the obstacle target in the real-time area video and the actual position of the obstacle target; An obstacle error conversion module 604, configured to: convert the pixel position error of the obstacle target in the real-time area video into an actual position error; An obstacle position correction module 605, configured to: correct the actual position based on the actual position error to obtain the corrected position of the obstacle target; A UAV real-time obstacle avoidance module 606, configured to: control the UAV to perform real-time obstacle avoidance based on the corrected position of the obstacle target; The obstacle recognition model is trained based on the historical area video of the UAV and the obstacle labels therein on the basis of a generative adversarial network.

[0051] The real-time obstacle avoidance device for the drone provided in this embodiment collects the real-time regional video when the drone flies along a preset flight route, inputs the real-time regional video into an obstacle recognition model to obtain an obstacle target output by the obstacle recognition model, and based on the mapping relationship between the pixel position of the obstacle target in the real-time regional video and the actual position of the obstacle target, obtains the actual position of the obstacle target, converts the pixel position error of the obstacle target in the real-time regional video into an actual position error, corrects the actual position based on the actual position error to obtain the corrected position of the obstacle target, and controls the drone to avoid obstacles in real time based on the corrected position of the obstacle target. On the one hand, in this embodiment, since the obstacle recognition model is trained by using the historical regional video of the drone and the obstacle labels therein based on a generative adversarial network, when the obstacle recognition model recognizes obstacles, it can utilize the characteristics of the generative adversarial network to accurately extract obstacle features from the rich and complex features presented by the changeable climate and complex terrain and landforms in the low-altitude domain in the real-time regional video, so as to achieve accurate recognition of the obstacle target; on the other hand, since the pixel position error of the obstacle target is converted into an actual position error and the actual position is corrected, the positioning error of the obstacle target can be reduced. When the recognition accuracy of the obstacle target is improved, the positioning error can be further reduced. In summary, this embodiment can accurately recognize obstacles in the low-altitude domain and reduce the positioning error of the obstacles, so as to avoid obstacles in time and ensure the flight safety of the drone and the smooth completion of the monitoring task.

[0052] In one embodiment, the obstacle error conversion module 604 is specifically configured to: Obtain the picture frame in the real-time regional video where the obstacle target exists; Convert the pixel position error of the obstacle target in the picture frame into an actual position error.

[0053] In one embodiment, the obstacle error conversion module 604 is specifically configured to: Obtain the number of pixels of the picture frame in the horizontal direction and the vertical direction to obtain the picture pixel width and the picture pixel height; Obtain the vertical distance between the horizontal plane where the obstacle target is located and the horizontal plane where the acquisition camera of the real-time regional video is located; Obtain the attribute parameters of the acquisition camera; Based on the picture pixel width, the vertical distance, and the attribute parameters, calculate the actual horizontal position error corresponding to the pixel horizontal position error of the obstacle target in the picture frame; Based on the picture pixel height, the vertical distance, and the attribute parameters, calculate the actual vertical position error corresponding to the pixel vertical position error of the obstacle target in the picture frame.

[0054] In one embodiment, the real-time obstacle avoidance module 606 of the UAV is specifically configured to: Randomly generate a first set of random points at the current position of the UAV; Randomly generate a second set of random points at the end position of the preset flight route; Randomly generate a third random point and a fourth random point on and near the preset flight route; Obtain the first random point closest to the third random point in the first set of random points to obtain a first target point; Obtain the second random point closest to the fourth random point in the second set of random points to obtain a second target point; Move the first target point towards the third random point by a first distance to obtain a third target point; Move the second target point towards the fourth random point by a second distance to obtain a fourth target point; If the corrected position of the obstacle target exists on the movement path from the first target point to the third target point or between the movement paths from the second target point to the fourth target point, then return to the step of randomly generating a third random point and a fourth random point on and near the preset flight route until the corrected position of the obstacle target does not exist on the movement path from the first target point to the third target point and between the movement paths from the second target point to the fourth target point; Add the current third target point to the first set of random points and add the current fourth target point to the second set of random points; If the distance between the third target point and the fourth target point is greater than or equal to the distance threshold, then return to the step of randomly generating a third random point and a fourth random point on and near the preset flight route until the distance between the third target point and the fourth target point is less than the distance threshold; Generate an obstacle avoidance route for the UAV based on the current first target point, second target point, third target point, and fourth target point; Control the UAV to avoid obstacles in real time based on the obstacle avoidance route.

[0055] In one embodiment, the real-time obstacle avoidance module 606 of the UAV is specifically configured to: Trim and interpolate the obstacle avoidance route to obtain an optimized route; Set the speed constraint space of the UAV based on the current flight state of the UAV; Generate multiple feasible routes between the current position of the UAV and the end position of the preset flight route based on the speed constraint space; Select the feasible route with the highest similarity to the optimized route from the multiple feasible routes to obtain the target feasible route; Based on the speed constraint corresponding to the target feasible route, control the UAV to fly along the optimized route.

[0056] In one embodiment, the UAV real-time obstacle avoidance module 606 is specifically configured to: Extract the key nodes in the optimized route; Track the flight of the UAV based on the key nodes, and detect whether there is an obstacle target in the front route of the UAV during the tracking process; When there is an obstacle target, control the UAV to avoid obstacles in real time based on a preset obstacle avoidance strategy.

[0057] FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 7 shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communication interface 720, and the memory 730 complete mutual communication through the communication bus 740. The processor 710 can call a computer program in the memory 730 to execute the steps of the UAV real-time obstacle avoidance method, for example, including: Collect the real-time area video when the UAV flies according to a preset flight route; Input the real-time area video into an obstacle recognition model to obtain the obstacle target output by the obstacle recognition model; Based on the mapping relationship between the pixel position of the obstacle target in the real-time area video and the actual position of the obstacle target, obtain the actual position of the obstacle target; Convert the pixel position error of the obstacle target in the real-time area video into an actual position error; Based on the actual position error, correct the actual position to obtain the corrected position of the obstacle target; Based on the corrected position of the obstacle target, control the UAV to avoid obstacles in real time; The obstacle recognition model is trained based on the historical area video of the UAV and the obstacle labels therein on the basis of a generative adversarial network.

[0058] In addition, when the logical instructions in the above-mentioned memory 730 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0059] On the other hand, an embodiment of this application also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the steps of the real-time obstacle avoidance method for unmanned aerial vehicles provided in the above-mentioned various embodiments. For example, it includes: Collect the real-time area video when the unmanned aerial vehicle flies along a preset flight route; Input the real-time area video into an obstacle recognition model to obtain the obstacle targets output by the obstacle recognition model; Based on the mapping relationship between the pixel position of the obstacle target in the real-time area video and the actual position of the obstacle target, obtain the actual position of the obstacle target; Convert the pixel position error of the obstacle target in the real-time area video into an actual position error; Based on the actual position error, correct the actual position to obtain the corrected position of the obstacle target; Based on the corrected position of the obstacle target, control the real-time obstacle avoidance of the unmanned aerial vehicle; The obstacle recognition model is trained based on the historical area video of the unmanned aerial vehicle and the obstacle labels therein on the basis of a generative adversarial network.

[0060] On the other hand, an embodiment of this application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. The computer program is used to cause a processor to execute the steps of the real-time obstacle avoidance method for unmanned aerial vehicles provided in the above-mentioned various embodiments. For example, it includes: Collect the real-time area video when the unmanned aerial vehicle flies along a preset flight route; Input the real-time regional video into the obstacle recognition model to obtain the obstacle targets output by the obstacle recognition model; Based on the mapping relationship between the pixel positions of the obstacle targets in the real-time regional video and their actual positions, obtain the actual positions of the obstacle targets; Convert the pixel position error of the obstacle targets in the real-time regional video into an actual position error; Based on the actual position error, correct the actual position to obtain the corrected positions of the obstacle targets; Based on the corrected positions of the obstacle targets, control the UAV to avoid obstacles in real time; The obstacle recognition model is trained based on the historical regional video of the UAV and the obstacle labels therein on the basis of a generative adversarial network.

[0061] The non-transitory computer-readable storage medium can be any available medium or data storage device accessible by a processor, including but not limited to magnetic memories (such as floppy disks, hard disks, magnetic tapes, magneto-optical discs (MO), etc.), optical memories (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor memories (such as ROM, EPROM, EEPROM, non-volatile memories (NAND FLASH), solid-state drives (SSD)), etc.

[0062] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0063] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, also by hardware. Based on this understanding, the above technical solutions, in essence, or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disks, optical discs, etc., including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A real-time obstacle avoidance method for a drone, characterized in that: include: Collect real-time area video as the drone flies along a preset flight path; Inputting the real-time regional video into an obstacle recognition model to obtain an obstacle target output by the obstacle recognition model; Acquire the actual position of the obstacle target based on a mapping relationship between a pixel position of the obstacle target in the real-time area video and an actual position of the obstacle target; Converting the pixel position error of the obstacle target in the real-time area video into an actual position error; Correcting the actual position based on the actual position error to obtain a corrected position of the obstacle target; Based on the corrected position of the obstacle target, controlling the drone to avoid obstacles in real time; The obstacle recognition model is obtained by training the historical area video of the UAV and the obstacle labels therein on the basis of a generative adversarial network.

2. The real-time obstacle avoidance method for unmanned aerial vehicles according to claim 1, characterized in that: The step of converting the pixel position error of the obstacle target in the real-time area video into the actual position error comprises: Acquire a picture frame in which the obstacle target exists in the real-time area video; The pixel position error of the obstacle target in the picture frame is converted into an actual position error.

3. The real-time obstacle avoidance method for unmanned aerial vehicles according to claim 2, characterized in that: The step of converting the pixel position error of the obstacle target in the picture frame into the actual position error comprises: Obtain the number of pixels of the picture frame in the horizontal direction and the vertical direction to obtain the picture pixel width and the picture pixel height; Obtaining the vertical distance between the horizontal plane where the obstacle target is located and the horizontal plane where the acquisition camera of the real-time area video is located; Obtaining attribute parameters of the acquisition camera; Calculate the actual horizontal position error of the obstacle target corresponding to the pixel horizontal position error in the picture frame based on the picture pixel width, the vertical distance and the attribute parameter; An actual vertical position error corresponding to the pixel vertical position error of the obstacle target in the picture frame is calculated based on the picture pixel height, the vertical distance and the attribute parameter.

4. The real-time obstacle avoidance method for unmanned aerial vehicles according to claim 1, characterized in that: The method of controlling the drone to avoid obstacles in real time based on the corrected position of the obstacle target includes: Randomly generating a first set of random points at the current position of the drone; Randomly generating a second random point set at the end point of the preset flight route; Randomly generate a third random point and a fourth random point on the preset flight route and its vicinity; Obtain a first random point in the first random point set that is closest to the third random point to obtain a first target point; Obtain a second random point in the second random point set that is closest to the fourth random point to obtain a second target point; Move the first target point toward the third random point by a first distance to obtain a third target point; Move the second target point toward the fourth random point by a second distance to obtain a fourth target point; If the corrected position of the obstacle target exists on the moving path from the first target point to the third target point or the moving path from the second target point to the fourth target point, return to the step of randomly generating a third random point and a fourth random point on the preset flight line and its vicinity until the corrected position of the obstacle target does not exist on the moving path from the first target point to the third target point or the moving path from the second target point to the fourth target point; Adding the third target point at this time to the first random point set, and adding the fourth target point at this time to the second random point set; If the distance between the third target point and the fourth target point is greater than or equal to the distance threshold, return to the step of randomly generating a third random point and a fourth random point on the preset flight route and its vicinity until the distance between the third target point and the fourth target point is less than the distance threshold; Based on the first target point, the second target point, the third target point and the fourth target point at this time, an obstacle avoidance route of the drone is generated; Based on the obstacle avoidance route, the drone is controlled to avoid obstacles in real time.

5. The real-time obstacle avoidance method for unmanned aerial vehicles according to claim 4, characterized in that: The controlling the UAV to avoid obstacles in real time based on the obstacle avoidance route includes: Pruning and interpolating the obstacle avoidance route to obtain an optimized route; Setting a speed constraint space of the UAV based on a current flight state of the UAV; Based on the speed constraint space, generating a plurality of feasible routes between the current position of the drone and the end position of the preset flight route; Selecting a feasible route with the highest similarity to the optimized route from the multiple feasible routes to obtain a target feasible route; Based on the speed constraint corresponding to the target feasible route, the UAV is controlled to fly along the optimized route.

6. The real-time obstacle avoidance method for a UAV according to claim 5, characterized in that: After controlling the UAV to fly along the optimized route based on the speed constraint corresponding to the target feasible route, the method includes: Extracting key nodes in the optimization circuit; Tracking the flight of the UAV based on the key nodes, and detecting whether there are obstacles or targets in the front route of the UAV during the tracking process; When there is an obstacle target, the drone is controlled to avoid the obstacle in real time based on a preset obstacle avoidance strategy.

7. A real-time obstacle avoidance device for a drone, characterized in that: include: The video acquisition module is used to: collect real-time regional video when the drone flies along a preset flight route; The obstacle recognition module is used to: input the real-time regional video into an obstacle recognition model to obtain an obstacle target output by the obstacle recognition model; The obstacle position acquisition module is used to: acquire the actual position of the obstacle target based on the mapping relationship between the pixel position of the obstacle target in the real-time area video and the actual position of the obstacle target; An obstacle error conversion module is used to: convert the pixel position error of the obstacle target in the real-time area video into an actual position error; An obstacle position correction module is used to correct the actual position based on the actual position error to obtain a corrected position of the obstacle target; The UAV real-time obstacle avoidance module is used to: control the UAV to avoid obstacles in real time based on the corrected position of the obstacle target; The obstacle recognition model is obtained by training the historical area video of the UAV and the obstacle labels therein on the basis of a generative adversarial network.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, the steps of the real-time obstacle avoidance method for a drone according to any one of claims 1 to 6 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the real-time obstacle avoidance method for a drone according to any one of claims 1 to 6 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the real-time obstacle avoidance method for a drone according to any one of claims 1 to 6 are implemented.

Images