Obstacle avoidance processing method and device of unmanned aerial vehicle, equipment and storage medium

Abstract

The invention provides an obstacle avoidance processing method and device for an unmanned aerial vehicle, equipment and a storage medium. The method comprises the following steps: acquiring flight speed and flight attitude data of the unmanned aerial vehicle; according to the flight speed and the flight attitude data, a flight channel area under a target camera corresponding to the flight direction of the unmanned aerial vehicle is determined, and the flight channel area comprises boundary areas in the vertical direction and the horizontal direction; acquiring an environment image of the unmanned aerial vehicle in the flight process through the target camera; target segmentation is carried out on the environment image to segment the environment image into a plurality of target segmentation areas, and each target segmentation area comprises at least one target object in the environment image; and determining an obstacle avoidance region of the unmanned aerial vehicle according to the plurality of target segmentation regions and the flight channel region, and performing obstacle avoidance processing on the unmanned aerial vehicle based on the obstacle avoidance region. According to the invention, consumption of computing resources during obstacle avoidance of the unmanned aerial vehicle can be reduced, accurate obstacle avoidance of the unmanned aerial vehicle is realized, and the obstacle avoidance cost of the unmanned aerial vehicle can be reduced.

Description

Technical Field

[0001] The present application relates to the field of visual obstacle avoidance technology, and in particular to an obstacle avoidance processing method, device, equipment and storage medium for an unmanned aerial vehicle. Background Art

[0002] Obstacle avoidance technology is the key to ensuring the safety of sports equipment and achieving autonomous navigation. Taking sports equipment such as drones as an example, mainstream drone obstacle avoidance solutions can be roughly divided into sensor-based solutions and vision-based solutions. Sensor-based solutions: including laser radar (LiDAR), sonar, infrared sensors, etc. These solutions can provide direct distance measurement and have high obstacle avoidance accuracy and reliability. Vision-based solutions: mainly use cameras to capture images and achieve obstacle avoidance through image processing technology. Compared with sensor solutions, vision solutions are low-cost, light-weight, and more suitable for small drones.

[0003] However, high-precision sensors are expensive and increase the load of drones, affecting flight time and flexibility. Vision solutions are sensitive to lighting conditions and environmental complexity, require complex algorithms to ensure accuracy, and have high demands on computing resources. Summary of the invention

[0004] In view of this, the present application provides an obstacle avoidance processing method, device, equipment and storage medium for a drone, which can reduce the consumption of computing resources of the drone when avoiding obstacles, realize accurate obstacle avoidance of the drone, and reduce the obstacle avoidance cost of the drone.

[0005] An embodiment of the present application provides an obstacle avoidance method for an unmanned aerial vehicle, comprising: obtaining flight speed and flight attitude data of the unmanned aerial vehicle; determining a flight channel area under a target camera corresponding to the flight direction of the unmanned aerial vehicle based on the flight speed and the flight attitude data, wherein the flight channel area includes boundary areas in vertical and horizontal directions; collecting an environmental image of the unmanned aerial vehicle during flight through the target camera; performing target segmentation on the environmental image to obtain a plurality of target segmentation areas, wherein each of the target segmentation areas includes at least one target object in the current environment; determining an obstacle avoidance area of ​​the unmanned aerial vehicle based on the plurality of target segmentation areas and the flight channel area, and performing obstacle avoidance processing for the unmanned aerial vehicle based on the obstacle avoidance area.

[0006] Compared with the related art, the embodiment of the present application has at least the following advantages: since the speed and flight performance of the drone limit the range of the area that the drone can reach, not all areas captured by the drone camera will affect the safety performance of the drone. The present application obtains the flight speed and flight attitude data of the drone to obtain the flight channel area of ​​the drone, and then performs target segmentation on the environmental image captured by the target camera to segment the environmental image into multiple target segmentation areas. Since each target segmentation area includes at least one target object in the environmental image, the obstacle avoidance area that the drone actually needs to avoid obstacles can be determined based on the multiple target segmentation areas and the flight channel area, so that data processing can be performed only based on the obstacle avoidance area, while reducing the amount of data and thus reducing the consumption of computing data, the data processing accuracy is improved, and accurate obstacle avoidance of the drone is achieved. In addition, during the movement of the drone, it is only necessary to collect environmental images through the target camera, without the need to use high-precision sensors, which reduces the obstacle avoidance cost of the drone.

[0007] In some possible implementations, performing target segmentation on the environmental image to divide the environmental image into multiple target segmentation areas includes: inputting the environmental image into a pre-trained image segmentation network model, and determining the multiple target segmentation areas based on the output result of the image segmentation network model; wherein the image segmentation network model is used to output labeling information of the target object in each of the target segmentation areas, and the labeling information includes at least size information of the target object and position information in the environmental image.

[0008] By adopting the above technical solution, the accuracy of target segmentation of environmental images can be improved, ensuring that each target segmentation area includes the target object, thereby improving the accuracy of the subsequent obstacle avoidance area obtained based on the target segmentation area, and further improving the obstacle avoidance accuracy of the drone.

[0009] In some possible implementations, the image segmentation network model is trained according to the following steps: acquiring historical environment images of the drone during historical flights; annotating historical target objects in the historical environment images to obtain training images with real labels, wherein the real labels include at least the size of the historical target objects and the position in the historical environment images; and training the initial image segmentation network model using the training images to obtain the image segmentation network model.

[0010] By adopting the above technical solution, the model accuracy of the image segmentation network model can be improved, thereby improving the accuracy of the image segmentation network model in performing target segmentation on environmental images.

[0011] In some possible implementations, the flight speed includes at least a flight speed value and a speed direction, and the flight attitude data includes at least a lateral acceleration critical upper limit value and a pitch angle; determining the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and the flight attitude data includes: determining the flight channel boundary in the horizontal direction according to the flight speed value, the speed direction, the lateral acceleration critical upper limit value, and the pixel parameters of the target camera; determining the flight channel boundary in the vertical direction according to the flight speed value, the speed direction, the lateral acceleration critical upper limit value, the pitch angle, and the pixel parameters of the target camera.

[0012] By adopting the above technical solution, the flight channel boundary of the UAV can be accurately determined, thereby improving the accuracy of the obstacle avoidance area subsequently obtained based on the flight channel boundary, and further improving the obstacle avoidance accuracy of the UAV.

[0013] In some possible implementations, determining the obstacle avoidance area of ​​the drone based on the multiple target segmentation areas and the flight channel area includes: determining obstacles within the flight channel area based on the multiple target segmentation areas; and using the area with obstacles within the flight channel area as the obstacle avoidance area.

[0014] By adopting the above technical solution, the accuracy of the obstacle avoidance area can be improved, so that after the obstacle avoidance processing of the drone is subsequently performed based on the obstacle avoidance area, accurate obstacle avoidance of the drone can be achieved.

[0015] In some possible implementations, before determining the obstacles in the flight channel area based on the multiple target segmentation areas, it also includes: obtaining the priority of each of the target segmentation areas, wherein the larger the priority of the target segmentation area, the smaller the shortest straight-line distance between the drone and the target segmentation area; taking the target segmentation area whose priority satisfies a preset priority screening rule as the final segmentation area; determining the obstacles in the flight channel area based on the multiple target segmentation areas includes: determining the obstacles in the flight channel area based on the final segmentation area.

[0016] By adopting the above technical solution, it is possible to improve the accuracy of the obstacle avoidance area subsequently obtained based on the target object while reducing the flight path of the UAV in the obstacle avoidance area, thereby improving the flight efficiency of the UAV.

[0017] In some possible implementations, the target segmentation area whose priority satisfies a preset priority screening rule is taken as the final segmentation area, including: sorting the priority of each target segmentation area in order from large to small to obtain a priority queue; and taking the target segmentation areas corresponding to the first N priorities in the priority queue as the final segmentation area, where N is an integer greater than or equal to 1.

[0018] By adopting the above technical solution, the flight path of the UAV in the obstacle avoidance area can be further reduced, thereby improving the flight efficiency of the UAV.

[0019] The second aspect of the present application discloses an obstacle avoidance processing device for a drone, comprising: an acquisition module, a first determination module, a collection module, a segmentation module, a second determination module and a processing module; the acquisition module is used to acquire the flight speed and flight attitude data of the drone; the first determination module is used to determine the flight channel area under the target camera corresponding to the flight direction of the drone according to the flight speed and the flight attitude data, wherein the flight channel area includes boundary areas in the vertical and horizontal directions; the acquisition module is used to acquire the environmental image of the drone during flight; the segmentation module is used to perform target segmentation on the environmental image to divide the environmental image into multiple target segmentation areas, wherein each of the target segmentation areas includes at least one target object in the environmental image; the second determination module is used to determine the obstacle avoidance area of ​​the drone according to the multiple target segmentation areas and the flight channel area; the processing module is used to perform obstacle avoidance processing on the drone based on the obstacle avoidance area.

[0020] The third aspect of the present application discloses an electronic device, which includes a processor and a memory, wherein the memory is used to store instructions, and the processor is used to call the instructions in the memory so that the electronic device executes the above-mentioned obstacle avoidance method for the drone.

[0021] The fourth aspect of the present application discloses a storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned obstacle avoidance method for a drone.

[0022] It can be understood that the device of the second aspect, the electronic device of the third aspect and the storage medium of the fourth aspect provided above all correspond to the method of the first aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 14 is a flow chart of an obstacle avoidance method for a drone according to an embodiment of the present invention.

[0024] Figure 2 4 is a flow chart of an obstacle avoidance method for a drone according to an embodiment of the present invention.

[0025] Figure 3 This is an application scenario diagram of image segmentation according to an embodiment of the present invention.

[0026] Figure 4 4 is a flow chart of an obstacle avoidance method for a drone according to an embodiment of the present invention.

[0027] Figure 5 4 is a flow chart of an obstacle avoidance method for a drone according to an embodiment of the present invention.

[0028] Figure 6 1 is a schematic diagram of functional modules of an obstacle avoidance processing device for a drone according to an embodiment of the present invention.

[0029] Figure 7 Schematic diagram of functional modules of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the implementation methods of the present application and the features in the implementation methods can be combined with each other without conflict.

[0031] In the following description, many specific details are set forth to facilitate a full understanding of the present application. The described implementations are only part of the implementations of the present application, rather than all the implementations.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0033] It should be further noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0034] In this application, "at least one" means one or more, and "more" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0035] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0036] Please refer to Figure 1 , is a flow chart of the obstacle avoidance method of the drone provided in the embodiment of the present application. Figure 1 As shown, this embodiment is applied to a drone and includes the following steps: Step 101: Obtain the flight speed and flight attitude data of the drone.

[0037] In some embodiments, the flight speed is not only the specific value of the speed, but also the determination of the direction of the speed. The flight attitude data is the state of the three axes of the drone in the air relative to a reference line or a reference plane, or a fixed coordinate system. The drone has various flight attitudes, and the flight attitude determines the direction of the aircraft, affecting both the flight altitude and the direction of flight. The maneuverability of the drone is related to the flight performance and flight direction of the drone. The drone can capture images with a larger field of view in the flight direction during flight, but the speed, flight performance and other factors of the drone restrict the range of the area that the drone can reach. Therefore, not all areas in the image will affect the safety performance of the drone.

[0038] Step 102: Determine the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and flight attitude data.

[0039] Specifically, the flight channel area includes boundary areas in vertical and horizontal directions.

[0040] In some embodiments, the flight speed includes at least a flight speed value and a speed direction, and the flight attitude data includes at least a lateral acceleration critical upper limit value and a pitch angle; the flight channel area under the target camera corresponding to the flight direction of the drone is determined in the following manner: the flight channel boundary in the horizontal direction is determined according to the flight speed value, the speed direction, the lateral acceleration critical upper limit value and the pixel parameters of the target camera; the flight channel boundary in the vertical direction is determined according to the flight speed value, the speed direction, the lateral acceleration critical upper limit value, the pitch angle and the pixel parameters of the target camera.

[0041] In some embodiments, the flight channel boundary in the horizontal direction is determined according to the flight speed value, speed direction, lateral acceleration critical upper limit value and pixel parameters of the target camera, including: determining a first lateral displacement of the UAV in the horizontal direction of the world coordinate system according to the flight speed value, speed direction, lateral acceleration critical upper limit value and preset time; determining a first forward displacement of the UAV in the horizontal direction of the world coordinate system according to the speed value and the preset time; determining a direction angle of the UAV in the horizontal direction according to the first lateral displacement and the first forward displacement; determining the flight channel boundary in the horizontal direction according to the direction angle in the horizontal direction, the position of the UAV on the camera pixel interface, and the pixel parameters of the target camera.

[0042] It is worth noting that there is no limitation on how to determine the above-mentioned flight speed value, speed direction, lateral acceleration critical upper limit value and preset time in this embodiment. The main purpose is to establish the angle and pixel relationship based on the pixel parameters of the drone camera and the area captured by the drone camera according to the coordinate system, where the angle can be determined based on the flight speed value and speed direction of the drone. At this time, according to the angle and pixel relationship, the left and right boundary ranges that the drone can actually reach can be determined.

[0043] In some embodiments, the flight channel boundary in the vertical direction is determined according to the flight speed value, speed direction, lateral acceleration critical upper limit value, pitch angle and pixel parameters of the target camera, including: determining the second lateral displacement of the UAV in the vertical direction of the world coordinate system according to the flight speed value, speed direction, lateral acceleration critical upper limit value and preset time; determining the second forward displacement of the UAV in the vertical direction of the world coordinate system according to the flight speed value and the preset time; determining the direction angle of the UAV in the vertical direction according to the second lateral displacement and the second forward displacement; determining the flight channel boundary in the vertical direction according to the direction angle in the vertical direction, the pitch angle, the position of the UAV on the camera pixel interface, and the pixel parameters of the target camera.

[0044] It should be noted that the process of determining the flight channel boundary in the vertical direction is similar to the process of determining the flight channel boundary in the horizontal direction. Since the pitch angle parameter is introduced in the vertical direction in the world coordinate system, and the pitch angle is related to the speed, the algorithm for determining the flight channel boundary in the horizontal direction in the above embodiment is based on the relationship between the angle corresponding to the pitch angle and the pixel, that is, the upper and lower boundary ranges that the drone can actually reach are determined.

[0045] Step 103: Collect environmental images of the UAV during flight through the target camera.

[0046] In some embodiments, the target camera is preferably a monocular camera, which has the advantages of small size, light weight, low energy consumption, and wide field of view, and is more suitable for obstacle avoidance perception of small drones.

[0047] Step 104: performing object segmentation on the environment image to segment the environment image into a plurality of object segmentation regions.

[0048] It should be noted that how to perform target segmentation on the environment image is described in detail in subsequent embodiments, and will not be described again here to avoid repetition.

[0049] Step 105: Determine an obstacle avoidance area for the UAV according to the multiple target segmentation areas and the flight channel area, and perform obstacle avoidance processing on the UAV based on the obstacle avoidance area.

[0050] In some embodiments, there is no specific limitation on how to perform obstacle avoidance processing on the drone based on the obstacle avoidance area, and it can be set according to actual needs.

[0051] It should be noted that how to determine the obstacle avoidance area of ​​the drone based on multiple target segmentation areas and flight channel areas is described in detail in subsequent embodiments, and will not be repeated here to avoid repetition.

[0052] For ease of understanding, the obstacle avoidance process of the drone of this embodiment is specifically described below: The drone includes a monocular camera, a CPU (Central Processing Unit) and an NPU (Network Processor Unit). The NPU is used to perform target segmentation on the environmental image. During the flight of the drone, the monocular camera captures the environmental image, the CPU inputs the environmental image into the NPU, and the NPU outputs the environmental image segmented into multiple target segmentation areas. The NPU sends the environmental image segmented into multiple target segmentation areas to the CPU. The CPU obtains the flight channel area based on the flight speed and flight attitude data of the drone, and then determines the obstacle avoidance area based on the flight channel area and multiple target segmentation areas. Finally, the drone performs obstacle avoidance processing based on the obstacle avoidance area.

[0053] It is worth mentioning that from the automatic obstacle avoidance process of the above-mentioned drone, it can be seen that the segmentation of the environmental image is performed in the NPU, which greatly reduces the CPU resource usage.

[0054] Compared with the related art, the embodiment of the present application has at least the following advantages: since the speed and flight performance of the drone limit the range of the area that the drone can reach, not all areas captured by the drone camera will affect the safety performance of the drone. The present application obtains the flight speed and flight attitude data of the drone to obtain the flight channel area of ​​the drone, and then performs target segmentation on the environmental image captured by the target camera to segment the environmental image into multiple target segmentation areas. Since each target segmentation area includes at least one target object in the environmental image, the obstacle avoidance area that the drone actually needs to avoid obstacles can be determined based on the multiple target segmentation areas and the flight channel area, so that data processing can be performed only based on the obstacle avoidance area, while reducing the amount of data and thus reducing the consumption of computing data, the data processing accuracy is improved, and accurate obstacle avoidance of the drone is achieved. In addition, during the movement of the drone, it is only necessary to collect environmental images through the target camera, without the need to use high-precision sensors, which reduces the obstacle avoidance cost of the drone.

[0055] Please refer to Figure 2 , which is a flow chart of the obstacle avoidance processing method for a drone provided in an embodiment of the present application. This embodiment is a specific description of the above-mentioned embodiment, and mainly describes how to perform target segmentation on an environment image to segment the environment image into multiple target segmentation areas.

[0056] The specific process of this embodiment is as follows Figure 2 As shown, the following steps are included: Step 201: Obtain the flight speed and flight attitude data of the UAV.

[0057] Step 202: Determine the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and flight attitude data.

[0058] Step 203: Collect environmental images of the UAV during flight through the target camera.

[0059] Step 204: Input the environment image into a pre-trained image segmentation network model, and determine multiple target segmentation areas based on the output results of the image segmentation network model.

[0060] Specifically, the image segmentation network model is used to output the annotation information of the target object in each target segmentation area, and the annotation information at least includes the size information of the target object and the position information in the environment image. It is worth noting that the target objects in the environment image include obstacles and non-obstacles. For example, the target objects in the environment image can be tree crowns and the sky, the tree crown is an obstacle, and the sky is a non-obstacle. By outputting the annotation information of the object through the image segmentation network model, it is possible to determine whether the target object is an obstacle based on the annotation information.

[0061] For ease of understanding, the following Figure 3 How to obtain multiple target segmentation regions in this embodiment is specifically described: Please refer to Figure 3 , is an application scenario diagram of image segmentation provided in the embodiment of the present application. Figure 3 As shown in (a), the environment image is captured by the target camera. The CPU transmits the environment image to the image segmentation network model of the NPU. The image segmentation network model segments the target object in the environment image and obtains Figure 3 The environment image is segmented into multiple target segmentation regions as shown in (b).

[0062] In some embodiments, the image segmentation network model is trained according to the following steps: obtaining historical environment images of the drone during historical flights; annotating historical target objects in the historical environment images to obtain training images with real labels, wherein the real labels include at least the size of the historical target objects and the position in the historical environment images; and training the initial image segmentation network model using the training images to obtain an image segmentation network model.

[0063] In some embodiments, since the computing power of the embedded chip selected by the drone platform is relatively weak, a lightweight neural network model needs to be selected to ensure the real-time requirements of image segmentation. Therefore, the image segmentation network model can be MobileNetV2 combined with U-Net. MobileNetV2 can efficiently extract image features, and combined with the U-Net structure, image segmentation can be performed quickly.

[0064] In some embodiments, the size of the image segmentation network model is further reduced and the running speed is improved through model quantization, pruning and other technologies. Taking model quantization as an example, the model parameters of the image segmentation network model are converted into fixed-point representation according to a preset quantization calibration table, wherein the quantization calibration table includes at least the number of quantization bits, quantization method and quantization range. By quantizing the image segmentation network model, the storage requirements of the image segmentation network model can be reduced, while the inference running speed on the NPU embedded side can be improved: model quantization converts floating-point model parameters into fixed-point representation, thereby reducing the space occupied by the image segmentation network model on the storage medium, memory bandwidth and memory overhead; at the same time, the quantized model can take advantage of the higher computational efficiency of integer operations on the NPU compared to floating-point operations, thereby achieving more efficient calculations on embedded hardware.

[0065] In some embodiments, the trained image segmentation network model is deployed on the NPU chip of the drone, and the NPU is used to perform image segmentation processing.

[0066] For ease of understanding, the training process of the image segmentation network model of this embodiment is specifically described below: (1) Collect a large number of target object images during the flight of the UAV, mark the precise location of the target object in the target object image, and construct a training set.

[0067] (2) Perform noise processing such as distortion, occlusion, motion blur, and defect on the above-mentioned images of the labeled target objects to supplement the training set.

[0068] (3) Use the above training set to train the image segmentation network model to obtain the first weight of the image segmentation network model.

[0069] (4) Repeat steps (1) to (3) until the image segmentation network model converges.

[0070] Step 205: Determine an obstacle avoidance area for the UAV according to the multiple target segmentation areas and the flight channel area, and perform obstacle avoidance processing on the UAV based on the obstacle avoidance area.

[0071] In some embodiments, as can be seen from the foregoing description, the image segmentation network model is used to output the labeling information of the target object in each target segmentation area. The labeling information includes at least the size information of the target object and the position information in the environmental image. Therefore, obstacles in the flight channel area can be determined through the labeling information, and then the area with obstacles can be used as an obstacle avoidance area.

[0072] Steps 201 to 203 of this embodiment are similar to steps 101 to 103 of the aforementioned embodiment, and are not described again here to avoid repetition.

[0073] Compared with the related art, the embodiment of the present application has at least the following advantages: since the speed and flight performance of the drone limit the range of the area that the drone can reach, not all areas captured by the drone camera will affect the safety performance of the drone. The present application obtains the flight speed and flight attitude data of the drone to obtain the flight channel area of ​​the drone, and then performs target segmentation on the environmental image captured by the target camera to segment the environmental image into multiple target segmentation areas. Since each target segmentation area includes at least one target object in the environmental image, the obstacle avoidance area that the drone actually needs to avoid obstacles can be determined based on the multiple target segmentation areas and the flight channel area, so that data processing can be performed only based on the obstacle avoidance area, while reducing the amount of data and thus reducing the consumption of computing data, the data processing accuracy is improved, and accurate obstacle avoidance of the drone is achieved. In addition, during the movement of the drone, it is only necessary to collect environmental images through the target camera, without the need to use high-precision sensors, which reduces the obstacle avoidance cost of the drone.

[0074] Please refer to Figure 4 , is a flow chart of the obstacle avoidance processing method for a drone provided in an embodiment of the present application. This embodiment is a specific description of the above embodiment, and further explains how to determine the obstacle avoidance area of ​​a drone based on multiple target segmentation areas and flight channel areas.

[0075] The specific process of this embodiment is as follows Figure 4 As shown, the following steps are included: Step 301: Obtain the flight speed and flight attitude data of the UAV.

[0076] Step 302: Determine the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and flight attitude data.

[0077] Step 303: Collect environmental images of the UAV during flight through the target camera.

[0078] Step 304: Perform object segmentation on the environment image to segment the environment image into a plurality of object segmentation regions.

[0079] Steps 301 to 304 of this embodiment are similar to steps 101 to 104 of the above-mentioned embodiment, and are not described again here to avoid repetition.

[0080] Step 305: Determine obstacles in the flight channel area according to the multiple target segmentation areas, and use the area with obstacles in the flight channel area as an obstacle avoidance area.

[0081] Step 306: Perform obstacle avoidance processing on the UAV based on the obstacle avoidance area.

[0082] Compared with the related art, the embodiment of the present application has at least the following advantages: since the speed and flight performance of the drone limit the range of the area that the drone can reach, not all areas captured by the drone camera will affect the safety performance of the drone. The present application obtains the flight speed and flight attitude data of the drone to obtain the flight channel area of ​​the drone, and then performs target segmentation on the environmental image captured by the target camera to segment the environmental image into multiple target segmentation areas. Since each target segmentation area includes at least one target object in the environmental image, the obstacle avoidance area that the drone actually needs to avoid obstacles can be determined based on the multiple target segmentation areas and the flight channel area, so that data processing can be performed only based on the obstacle avoidance area, while reducing the amount of data and thus reducing the consumption of computing data, the data processing accuracy is improved, and accurate obstacle avoidance of the drone is achieved. In addition, during the movement of the drone, it is only necessary to collect environmental images through the target camera, without the need to use high-precision sensors, which reduces the obstacle avoidance cost of the drone.

[0083] Please refer to Figure 5 , which is a flow chart of the obstacle avoidance processing method for the drone provided in the embodiment of the present application. This embodiment is a further improvement on the aforementioned embodiment, and the main improvement is that: in this embodiment, the priority of each target segmentation area is also obtained, and the target segmentation area whose priority meets the preset priority screening rule is used as the final segmentation area, and then the obstacles in the flight channel area are determined based on the final segmentation area. In this way, while improving the accuracy of the obstacle avoidance area obtained based on the target object, the flight path of the drone in the obstacle avoidance area can be reduced, thereby improving the flight efficiency of the drone.

[0084] The specific process of this embodiment is as follows Figure 5 As shown, the following steps are included: Step 401: Obtain the flight speed and flight attitude data of the UAV.

[0085] Step 402: Determine the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and flight attitude data.

[0086] Step 403: Collect environmental images of the UAV during flight through the target camera.

[0087] Step 404: Perform object segmentation on the environment image to segment the environment image into a plurality of object segmentation regions.

[0088] Step 405: Obtain the priority of each target segmentation region, wherein the greater the priority of the target segmentation region, the smaller the shortest straight-line distance between the drone and the target segmentation region.

[0089] In some embodiments, the correspondence between priority and distance can be stored in the drone in advance. For example, the priorities are set from high to low: priority 1 to priority 5, the distance corresponding to priority 1 is 0 to 2 meters, the distance corresponding to priority 2 is 2 meters to 5 meters, the distance corresponding to priority 3 is 5 meters to 10 meters, the distance corresponding to priority 4 is 10 meters to 20 meters, and the distance corresponding to priority 5 is more than 20 meters. After obtaining multiple target segmentation areas, the drone calculates the shortest straight-line distance between the drone and each target segmentation area, and determines the priority corresponding to each target segmentation area based on the calculated shortest straight-line distance.

[0090] Step 406: The target segmentation region whose priority satisfies the preset priority screening rule is taken as the final segmentation region.

[0091] In some embodiments, the priority of each target segmentation area is sorted from large to small to obtain a priority queue; the target segmentation areas corresponding to the first N priorities in the priority queue are used as the final segmentation areas, where N is an integer greater than or equal to 1.

[0092] For ease of understanding, the following takes the case where the target segmentation regions are 6 and the value of N is 3 as an example to specifically describe how to obtain the final segmentation regions in this embodiment: 1. Preset the correspondence between priority and distance, and store the correspondence in the drone. For example, the correspondence can be set as follows: the priority is from high to low, from priority 1 to priority 5, the distance corresponding to priority 1 is greater than 0 and less than 2 meters; the distance corresponding to priority 2 is greater than or equal to 2 meters and less than 5 meters; the distance corresponding to priority 3 is greater than or equal to 5 meters and less than 10 meters; the distance corresponding to priority 4 is greater than or equal to 10 meters and less than 20 meters; the distance corresponding to priority 5 is greater than or equal to 20 meters.

[0093] 2. After the drone divides the environment image into 6 target segmentation areas, the shortest straight-line distance between each target segmentation area and the drone is calculated. Assume that the calculated shortest straight-line distances of the 6 target segmentation areas are: the shortest straight-line distance between target segmentation area 1 and the drone is 5 meters, the shortest straight-line distance between target segmentation area 2 and the drone is 15 meters, the shortest straight-line distance between target segmentation area 3 and the drone is 2 meters, the shortest straight-line distance between target segmentation area 4 and the drone is 6 meters, the shortest straight-line distance between target segmentation area 5 and the drone is 12 meters, and the shortest straight-line distance between target segmentation area 6 and the drone is 20 meters.

[0094] 3. Based on the preset correspondence between priority and distance, the drone obtains that the priority of target segmentation area 1 is priority 2, the priority of target segmentation area 2 is priority 4, the priority of target segmentation area 3 is priority 1, the priority of target segmentation area 4 is priority 2, the priority of target segmentation area 5 is priority 4, and the priority of target segmentation area 6 is priority 5.

[0095] 4. The UAV determines that the target segmentation areas corresponding to the first three priorities are: target segmentation area 1, target segmentation area 3 and target segmentation area 4. The UAV uses target segmentation area 1, target segmentation area 3 and target segmentation area 4 as the final segmentation areas.

[0096] Step 407: According to the final segmented area, obstacles in the flight channel area are determined, and the area with obstacles in the flight channel area is used as an obstacle avoidance area.

[0097] Step 408: Perform obstacle avoidance processing on the UAV based on the obstacle avoidance area.

[0098] Steps 401 to 404, 407 and 408 of this embodiment are similar to steps 301 to 306 of the above-mentioned embodiment, and will not be described again to avoid repetition.

[0099] Compared with the related art, the embodiment of the present application has at least the following advantages: since the speed and flight performance of the drone limit the range of the area that the drone can reach, not all areas captured by the drone camera will affect the safety performance of the drone. The present application obtains the flight speed and flight attitude data of the drone to obtain the flight channel area of ​​the drone, and then performs target segmentation on the environmental image captured by the target camera to segment the environmental image into multiple target segmentation areas. Since each target segmentation area includes at least one target object in the environmental image, the obstacle avoidance area that the drone actually needs to avoid obstacles can be determined based on the multiple target segmentation areas and the flight channel area, so that data processing can be performed only based on the obstacle avoidance area, while reducing the amount of data and thus reducing the consumption of computing data, the data processing accuracy is improved, and accurate obstacle avoidance of the drone is achieved. In addition, during the movement of the drone, it is only necessary to collect environmental images through the target camera, without the need to use high-precision sensors, which reduces the obstacle avoidance cost of the drone.

[0100] Please refer to Figure 6 , is a functional module diagram of the obstacle avoidance processing device 100 for a drone provided in an embodiment of the present application. Figure 6 As shown, the obstacle avoidance processing device 100 of the drone includes: an acquisition module 1, a first determination module 2, a collection module 3, a segmentation module 4, a second determination module 5 and a processing module 6.

[0101] The acquisition module 1 is used to obtain the flight speed and flight attitude data of the UAV; the first determination module 2 is used to determine the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and flight attitude data, wherein the flight channel area includes the boundary areas in the vertical direction and the horizontal direction; the acquisition module 3 is used to collect the environmental image of the UAV during the flight; the segmentation module 4 is used to perform target segmentation on the environmental image to divide the environmental image into multiple target segmentation areas, wherein each target segmentation area includes at least one target object in the environmental image; the second determination module 5 is used to determine the obstacle avoidance area of ​​the UAV according to the multiple target segmentation areas and the flight channel area; the processing module 6 is used to perform obstacle avoidance processing on the UAV based on the obstacle avoidance area.

[0102] Please refer to Figure 7 , is a schematic diagram of the hardware structure of the electronic device 1000 provided in the embodiment of the present application. Figure 7 As shown, the electronic device 1000 may include a processor 1001 and a memory 1002. The memory 1002 is used to store one or more computer programs 1003. The one or more computer programs 1003 are configured to be executed by the processor 1001. The one or more computer programs 1003 include instructions, and the above instructions can be used to implement the above-mentioned obstacle avoidance processing method for the drone in the electronic device 1000.

[0103] It is understandable that the structure shown in this embodiment does not constitute a specific limitation on the electronic device 1000. In other embodiments, the electronic device 1000 may include more or fewer components than shown, or combine or separate some components, or arrange the components differently.

[0104] The processor 1001 may include one or more processing units, for example, the processor 1001 may include an application processor (AP), a modem, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0105] The processor 1001 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1001 is a cache memory. The memory may store instructions or data that the processor 1001 has just used or circulated. If the processor 1001 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 1001, and thus improves the efficiency of the system.

[0106] In some embodiments, the processor 1001 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0107] In some embodiments, processor 1001 is used to execute acceleration schemes such as single instruction multiple data (SIMD) and very long instruction word (VLIW).

[0108] In some embodiments, memory 1002 may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0109] This embodiment also provides a computer-readable storage medium, which stores computer instructions. When the instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the obstacle avoidance processing method for the drone in the above-mentioned embodiment.

[0110] Among them, the electronic device and computer storage medium provided in this embodiment are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0111] In practical applications, the above functions can be distributed to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0112] In several embodiments provided in the present application, the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are schematic. For example, the division of the modules or units is a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0113] The unit described as a separate component may or may not be physically separated, and the component shown as a unit may be one physical unit or multiple physical units, that is, it may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment.

[0114] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0115] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0116] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application.

Claims

1. An obstacle avoidance method for a drone, characterized in that: include: Obtain the flight speed and flight attitude data of the drone; Determine, according to the flight speed and the flight attitude data, a flight channel area under the target camera corresponding to the flight direction of the UAV, wherein the flight channel area includes boundary areas in vertical and horizontal directions; Collecting environmental images of the UAV during flight by the target camera; Performing target segmentation on the environment image to segment the environment image into a plurality of target segmentation regions, wherein each of the target segmentation regions includes at least one target object in the environment image; An obstacle avoidance area of ​​the UAV is determined according to the plurality of target segmentation areas and the flight channel area, and obstacle avoidance processing is performed on the UAV based on the obstacle avoidance area.

2. The obstacle avoidance method for a drone according to claim 1, characterized in that: The performing target segmentation on the environment image to segment the environment image into a plurality of target segmentation areas includes: Inputting the environment image into a pre-trained image segmentation network model, and determining a plurality of target segmentation regions based on an output result of the image segmentation network model; The image segmentation network model is used to output the labeling information of the target object in each target segmentation area, and the labeling information at least includes the size information of the target object and the position information in the environment image.

3. The obstacle avoidance method for a drone according to claim 2, characterized in that: The image segmentation network model is trained according to the following steps: Obtain historical environmental images of drones during historical flights; Labeling the historical target object in the historical environment image to obtain a training image with a real label, wherein the real label at least includes the size of the historical target object and the position in the historical environment image; The initial image segmentation network model is trained using the training images to obtain the image segmentation network model.

4. The obstacle avoidance method for a drone according to claim 1, characterized in that: The flight speed at least includes a flight speed value and a speed direction, and the flight attitude data at least includes a lateral acceleration critical upper limit value and a pitch angle; The step of determining the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and the flight attitude data includes: Determining the flight channel boundary in the horizontal direction according to the flight speed value, the speed direction, the lateral acceleration critical upper limit value and the pixel parameter of the target camera; The flight channel boundary in the vertical direction is determined according to the flight speed value, the speed direction, the critical upper limit value of the lateral acceleration, the pitch angle and the pixel parameters of the target camera.

5. The obstacle avoidance method for a drone according to any one of claims 1 to 4, characterized in that: The determining the obstacle avoidance area of ​​the UAV according to the plurality of target segmentation areas and the flight channel area comprises: Determining obstacles within the flight path area according to the plurality of target segmented areas; The area with obstacles in the flight channel area is used as the obstacle avoidance area.

6. The obstacle avoidance method for a drone according to claim 5, characterized in that: Before determining obstacles in the flight channel area according to the plurality of target segmentation areas, the method further includes: Obtaining the priority of each target segmentation area, wherein the greater the priority of the target segmentation area, the smaller the shortest straight-line distance between the UAV and the target segmentation area; The target segmentation area whose priority satisfies a preset priority screening rule is taken as a final segmentation area; The step of determining the obstacle in the flight channel area according to the plurality of target segmentation areas comprises: The obstacles in the flight channel area are determined according to the final segmented area.

7. The obstacle avoidance method for a drone according to claim 6, characterized in that: The step of taking the target segmentation area whose priority satisfies a preset priority screening rule as the final segmentation area includes: Sort the priority of each target segmentation area in descending order to obtain a priority queue; The target segmentation areas corresponding to the first N priorities in the priority queue are used as the final segmentation areas, where N is an integer greater than or equal to 1.

8. An obstacle avoidance device for a drone, characterized in that: include: An acquisition module, a first determination module, a collection module, a segmentation module, a second determination module, and a processing module; The acquisition module is used to acquire the flight speed and flight attitude data of the UAV; The first determination module is used to determine the flight channel area under the target camera corresponding to the flight direction of the UAV according to the flight speed and the flight attitude data, wherein the flight channel area includes boundary areas in the vertical direction and the horizontal direction; The acquisition module is used to acquire environmental images of the UAV during flight; The segmentation module is used to perform target segmentation on the environment image, so as to segment the environment image into a plurality of target segmentation regions, wherein each of the target segmentation regions includes at least one target object in the environment image; The second determination module is used to determine the obstacle avoidance area of ​​the UAV according to the plurality of target segmentation areas and the flight channel area; The processing module is used to perform obstacle avoidance processing on the drone based on the obstacle avoidance area.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store instructions, and the processor is used to call the instructions in the memory, so that the electronic device executes the obstacle avoidance processing method for a drone according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the obstacle avoidance processing method for a drone as claimed in any one of claims 1 to 7.

Images