Obstacle avoidance control method and device of unmanned aerial vehicle and storage medium

Abstract

The invention provides an obstacle avoidance control method and device of an unmanned aerial vehicle and a storage medium, and relates to the technical field of unmanned aerial vehicle obstacle avoidance. The method comprises the following steps: inputting a current frame image in a video stream into an obstacle detection model to obtain an obstacle detection result; based on the obstacle detection result, calculating to obtain an area expansion rate and an obstacle risk degree corresponding to each obstacle; determining a dangerous obstacle vector and an obstacle risk degree of the dangerous obstacle pointing to the two-dimensional optical imaging system, calculating to obtain a repulsive force vector corresponding to each dangerous obstacle, and performing weighted summation to obtain a total repulsive force vector; performing weighted summation on the avoidance direction corresponding to each dangerous obstacle through the obstacle risk degree to obtain an optimal avoidance direction; a component, perpendicular to the total repulsive force vector, of the optimal avoidance direction is calculated as a real avoidance direction, and the unmanned aerial vehicle is controlled to perform obstacle avoidance; the problems that the system of the unmanned aerial vehicle is complex and the operation and maintenance cost is high can be solved, and the complexity and the maintenance cost of the system of the unmanned aerial vehicle are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of obstacle avoidance for unmanned aerial vehicles, and in particular to an obstacle avoidance control method, device and storage medium for an unmanned aerial vehicle. Background Art

[0002] Unmanned aerial vehicles (UAVs) have been widely used in modern society, especially in the fields of logistics and distribution, agricultural monitoring, environmental monitoring, infrastructure inspection, etc. In order to ensure that UAVs can fly efficiently and safely, obstacle avoidance technology has become a key factor. Especially in complex three-dimensional environments, the flight path of UAVs may be affected by various obstacles, including static obstacles such as buildings, trees, and wires, as well as dynamic obstacles such as moving objects such as birds, aircraft, and vehicles. If the UAV fails to avoid these obstacles in time during flight, a collision may occur, resulting in equipment damage or even casualties. Therefore, the development of an efficient and real-time dynamic obstacle avoidance algorithm is a necessary condition to ensure the safe flight of UAVs.

[0003] At present, the traditional obstacle avoidance control method of unmanned aerial vehicles includes: equipping the unmanned aerial vehicle with a variety of sensors, such as laser radar (LiDAR), depth camera and event camera; obtaining real-time depth information of the environment and the position of moving objects through sensors; and dynamically calculating the safe path of the unmanned aerial vehicle by analyzing the data collected by the sensors and the obstacle avoidance algorithm.

[0004] However, although obstacle avoidance technology based on lidar and depth cameras can provide real-time environmental perception, these sensors are costly, expensive and bulky. At the same time, the deployment and maintenance requirements of high-precision sensors are also high. In harsh conditions such as strong light, rain, snow or haze, the accuracy of the sensors will decrease, resulting in complex unmanned aerial vehicle systems and high operation and maintenance costs. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide an obstacle avoidance control method, device and storage medium for an unmanned aerial vehicle to eliminate or improve one or more defects in the prior art, and solve the problem of complex systems and high operation and maintenance costs of unmanned aerial vehicles.

[0006] One aspect of the present invention provides an obstacle avoidance control method for an unmanned aerial vehicle, which is applied to an unmanned aerial vehicle; the unmanned aerial vehicle is provided with a two-dimensional optical imaging system for continuously shooting a video stream; the method comprises the following steps:

[0007] Input the current frame image in the video stream into the pre-trained obstacle detection model, and the obstacle detection model outputs the obstacle detection result; the obstacle detection result includes the obstacle category of each obstacle in the current frame image and the obstacle detection frame corresponding to each obstacle; the obstacle detection frame is used to indicate the current pixel area and current pixel coordinates of the obstacle in the current frame image;

[0008] Compare the obstacle detection result with the detection result corresponding to the historical frame image to obtain a comparison result; the comparison result is used to indicate whether each obstacle is the same as the obstacle in the historical frame image;

[0009] When the comparison result indicates that each obstacle is the same as the obstacle in the historical frame image, the area expansion rate corresponding to each obstacle is calculated based on the current pixel area and the historical pixel area of ​​each obstacle in the historical frame image;

[0010] The area expansion rate is multiplied by the expansion rate coefficient to obtain the obstacle hazard level corresponding to each obstacle; the expansion rate coefficient is obtained by taking the square root or logarithm of the current pixel area;

[0011] Compare the obstacle hazard level corresponding to each obstacle with a preset hazard level threshold, and determine the obstacles whose obstacle hazard level is greater than the preset hazard level threshold as dangerous obstacles;

[0012] Determine the dangerous obstacle vector of each dangerous obstacle pointing to the two-dimensional optical imaging system based on the pixel width and pixel height of the current frame image, the real width and real height corresponding to the current frame image, the distance from the optical center of the two-dimensional optical imaging system to the center of the virtual imaging plane, and the current pixel coordinates;

[0013] Based on the dangerous obstacle vectors and the obstacle hazard degree corresponding to each dangerous obstacle, the repulsive force vector corresponding to each dangerous obstacle is calculated, and the repulsive force vector is weighted and summed by the obstacle hazard degree corresponding to each dangerous obstacle to obtain the overall repulsive force vector;

[0014] The vectors corresponding to the avoidance directions of each dangerous obstacle are weighted and summed using the corresponding obstacle danger degree, and the direction pointed by the summed vector is taken as the optimal avoidance direction; wherein the avoidance direction is obtained based on a preset path planning algorithm;

[0015] The component of the optimal avoidance direction perpendicular to the overall repulsive force vector is calculated as the true avoidance direction, and the unmanned aerial vehicle is controlled to avoid obstacles according to the true avoidance direction.

[0016] In some embodiments of the present invention, before calculating the area expansion rate corresponding to each obstacle based on the current pixel area of ​​each obstacle in the current frame image and the historical pixel area of ​​each obstacle in the historical frame image, the method further includes:

[0017] The current pixel area and current pixel coordinates corresponding to each obstacle are smoothed by using the Kalman filter and the historical pixel coordinates and historical pixel areas of each obstacle in the historical frame image.

[0018] In some embodiments of the present invention, when the comparison result indicates that the obstacle in the current frame image is different from the obstacle in the historical frame image and there is a missing obstacle, the method further includes:

[0019] Based on the preset hazard decay rate and the historical obstacle hazard corresponding to the lost obstacle, the predicted obstacle hazard is obtained;

[0020] When the predicted obstacle hazard level is greater than the preset hazard level threshold, the lost obstacle is predicted through the Kalman filter to obtain the predicted pixel coordinates corresponding to the lost obstacle; the predicted pixel coordinates are used to calculate the dangerous obstacle vector corresponding to the lost obstacle;

[0021] The lost obstacles are determined as dangerous obstacles, and the step of determining dangerous obstacle vectors pointing to the two-dimensional optical imaging system of each dangerous obstacle is performed.

[0022] In some embodiments of the present invention, when the video frame processing speed of the obstacle detection model is lower than the frame rate of the two-dimensional optical imaging system, the video stream is converted into video frames and stored in a preset cache;

[0023] Input the current frame image in the video stream into the pre-trained obstacle detection model, including:

[0024] According to the preset calculation window, the latest video frame is obtained from the preset cache as the current frame image, and input into the obstacle detection model;

[0025] Delete the video frame before the current frame image in the preset cache.

[0026] In some embodiments of the present invention, the training process of the obstacle detection model includes:

[0027] Input the current frame image of the sample into the teacher network and the student network in the initial obstacle detection model respectively, and obtain the output results of the teacher network and the student network;

[0028] By simplifying the optimal transport allocation method, the output of the teacher network and the true value label corresponding to the current frame image of the sample are matched and divided to obtain the teacher positive sample and the teacher negative sample;

[0029] By simplifying the optimal transport allocation method, the teacher's positive samples, teacher's negative samples and student output results are matched and divided to obtain the student's positive samples;

[0030] The first distillation loss between the student positive sample and the teacher positive sample, the second distillation loss between the student positive sample and the teacher negative sample, and the true value loss between the student output result and the true value label are calculated through the preset loss function; the preset loss function includes the category loss function, the regression loss function or the foreground-background loss function; the ratio between the true value loss, the first distillation loss and the second distillation loss is set to 20:2:1;

[0031] The first distillation loss is weighted by the first weight corresponding to the teacher's positive sample, and the second distillation loss is weighted by the second weight corresponding to the teacher's negative sample;

[0032] The student network is iteratively trained through true value loss, first distillation loss and second distillation loss until the student network converges to obtain the obstacle detection model.

[0033] In some embodiments of the present invention, the detection result corresponding to the historical frame image is obtained by querying the historical obstacle information table and the obstacle information queue corresponding to each historical obstacle; the historical obstacle information table includes an area expansion rate field, an obstacle danger field, a motion direction field, and an update timestamp field; the obstacle information queue includes a historical obstacle size queue, a frame interval queue, and a historical obstacle position queue;

[0034] After multiplying the area expansion rate by the expansion rate coefficient to obtain the obstacle hazard level corresponding to each obstacle, it also includes:

[0035] Add the current pixel area corresponding to each obstacle to the historical obstacle size queue corresponding to each obstacle;

[0036] Update the timestamp of the current frame image after processing in the frame interval queue;

[0037] Add the current pixel coordinates of each obstacle to the historical obstacle position queue corresponding to each obstacle;

[0038] The historical obstacle information table is updated based on the area expansion rate, obstacle danger, movement direction and update timestamp corresponding to each obstacle; the movement direction is calculated based on the pixel coordinates of each obstacle in the first frame image and the pixel coordinates in the last frame image in the current frame image.

[0039] In some embodiments of the present invention, before inputting the current frame image in the video stream into the pre-trained obstacle detection model, the method further includes:

[0040] Based on the current time and the update timestamps corresponding to each obstacle in the historical obstacle information queue, determine the update time difference corresponding to each obstacle in the historical obstacle information queue;

[0041] Obstacles whose update time difference is greater than the preset time length are removed from the historical obstacle queue.

[0042] In some embodiments of the present invention, after the obstacle detection model outputs the obstacle detection result, the following further includes:

[0043] The category confidence is compared with the preset confidence threshold, and the obstacle categories whose category confidence is less than the preset confidence threshold are deleted.

[0044] On the other hand, the present invention provides an obstacle avoidance control device for an unmanned aerial vehicle, comprising a two-dimensional optical imaging system for continuously acquiring a video stream; a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above-mentioned obstacle avoidance control method for the unmanned aerial vehicle.

[0045] Another aspect of the present invention provides a computer-readable storage medium, in which a program is stored. When the program is executed by a processor, it is used to implement the steps of the obstacle avoidance control method of the unmanned aerial vehicle.

[0046] The obstacle avoidance control method and device for an unmanned aerial vehicle of the present invention can solve the problems of the complexity of the unmanned aerial vehicle system and the high operation and maintenance cost; obstacle detection is performed on the video stream obtained by the two-dimensional optical imaging system to obtain obstacle information and update it, and then, based on the relative approach speed and relative position between the obstacle and the unmanned aerial vehicle, the danger posed by each obstacle to the unmanned aerial vehicle is evaluated and a corresponding weight is assigned, and then, using this weight and the relative position of the obstacle, the current repulsion vector of the obstacle is calculated, and finally, combined with the environmental safety information provided by the path planning system and the specific position of the obstacle, the most suitable avoidance direction and speed are determined in the plane perpendicular to the repulsion vector, thereby guiding the unmanned aerial vehicle to perform effective avoidance actions, and the unmanned aerial vehicle can be controlled to perform dynamic obstacle avoidance without the need for dynamic environment modeling and real-time depth information. Therefore, it is possible to avoid equipping the unmanned aerial vehicle with a sensor for obtaining depth information, thereby reducing the complexity and maintenance cost of the unmanned aerial vehicle system.

[0047] In addition, in order to address the problem that the operation of neural networks requires more computing resources, the present invention optimizes the obstacle detection model through the knowledge distillation method to reduce the algorithm's demand for computing resources and improve the obstacle avoidance efficiency of the unmanned aerial vehicle.

[0048] In addition, the obstacle collision time is used instead of the actual depth information at this moment as indirect depth information to model the obstacle three-dimensional space. At the same time, the obstacle expansion rate at a certain moment is used instead of the obstacle collision time at this moment, and then the Kalman filter is used to track the dynamic obstacle trajectory in the obstacle three-dimensional space. The Kalman filter's prediction ability for the obstacle trajectory is combined with other methods to make obstacle avoidance decisions. In this way, when the obstacle detection model misses the obstacle or the obstacle trajectory can be predicted in advance in the early stage of the obstacle, the obstacle avoidance decision can be made, which can avoid redundant obstacle avoidance actions and make the dynamic obstacle avoidance function of the unmanned aerial vehicle more efficient and stable. At the same time, combined with the reactive obstacle avoidance strategy, when the error between the predicted trajectory and the observed trajectory is less than the preset threshold and the initial stage of observing the obstacle is out of the observation, and when entering the stable stage, the obstacle avoidance action is planned according to the motion trajectory predicted by the Kalman filter. When the error between the predicted trajectory and the observed trajectory exceeds the preset threshold, the reactive obstacle avoidance method is used until the error between the predicted trajectory and the observed trajectory is less than the preset threshold again, which can improve the success rate of obstacle avoidance.

[0049] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0050] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0052] Figure 1 A flowchart of an obstacle avoidance control method for an unmanned aerial vehicle provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of the training process of an obstacle detection model provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of three-dimensional space modeling provided by an embodiment of the present invention;

[0055] Figure 4 A position relationship diagram of obstacles in a video frame image provided by an embodiment of the present invention;

[0056] Figure 5A flowchart of an obstacle avoidance control method for an unmanned aerial vehicle provided by another embodiment of the present invention;

[0057] Figure 6 A flowchart of an obstacle avoidance control method for an unmanned aerial vehicle provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0059] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0060] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0061] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection with an intermediate.

[0062] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0063] The obstacle avoidance control method for the unmanned aerial vehicle provided in this application is introduced in detail below.

[0064] like Figure 1 As shown, an embodiment of the present application provides an obstacle avoidance control method for an unmanned aerial vehicle, which is used in an unmanned aerial vehicle. The unmanned aerial vehicle is provided with a two-dimensional optical imaging system for continuously shooting a video stream, such as a monocular camera or an infrared camera. The method includes at least the following steps:

[0065] Step S101: input the current frame image in the video stream into a pre-trained obstacle detection model, and the obstacle detection model outputs an obstacle detection result.

[0066] In some embodiments of the present invention, a real-time video stream is acquired by a two-dimensional optical imaging system equipped on an unmanned aerial vehicle, and a frame is intercepted from the acquired video stream as a current frame image input into an obstacle detection model to obtain an obstacle detection result. The current frame image refers to the latest series of continuous video frame images in the real-time acquired video stream, and the number of frames is determined by a pre-set calculation window.

[0067] In actual implementation, due to the limitation of hardware equipment, there is a possibility that the processing speed of the obstacle detection model is lower than the frame rate of the two-dimensional optical imaging system. At this time, it is impossible to detect obstacles for each frame image in the video stream. Based on this, the real-time acquired video stream can be converted into video frames and cached in the preset cache of the unmanned aerial vehicle. After the obstacle detection model completes the obstacle detection of the previous frame image, the latest video frame is taken from the preset cache as the current frame image to input the obstacle detection model for obstacle detection, and the undetected video frame images between the current frame image and the previous frame image are discarded from the preset cache.

[0068] Specifically, when the video frame processing speed of the obstacle detection model is lower than the frame rate of the two-dimensional optical imaging system, the video stream is converted into video frames and stored in a preset cache; accordingly, the current frame image in the video stream is input into the pre-trained obstacle detection model, including: obtaining the latest video frame from the preset cache as the current frame image according to the preset calculation window, and inputting it into the obstacle detection model; and deleting the video frame before the current frame image in the preset cache.

[0069] Since the factor that affects the performance of the UAV in avoiding dynamic obstacles is mainly the size of the obstacle collision time (TTC) when the system detects an obstacle, the longer the collision time is, the easier it is for the UAV to avoid the obstacle. When the collision time of the same obstacle is longer, the area of ​​the obstacle in the current frame image is smaller. Therefore, improving the detection accuracy of the obstacle detection model for small targets can improve the performance of the UAV in avoiding dynamic obstacles to a certain extent.

[0070] Based on this, in some embodiments of the present invention, a real-time obstacle detection model is obtained by combining deep learning and knowledge distillation training. The YOLO (You Only Look Once)-Faetest network is improved in an anchor-free manner to increase the reasoning speed; at the same time, the improved model is distilled using knowledge distillation technology, which can improve the detection accuracy without reducing the real-time performance of the model and reduce the algorithm's demand for computing resources.

[0071] Specifically, the training process of the obstacle detection model includes at least steps S1010 to S1015:

[0072] Step S1010, input the sample current frame image into the teacher network and the student network in the initial obstacle detection model respectively, and obtain the teacher network output result and the student network output result.

[0073] refer to Figure 2 In some embodiments of the present invention, the initial obstacle detection model includes a teacher network and a student network. The model structure of the teacher network is the same as that of the student network, including a backbone network (Backbone) and a detection head (Detection). The main task of the backbone network is to extract a feature map (Feature Map) from the input sample current frame image. Usually, the backbone network uses a convolutional neural network (CNN) structure, and extracts high-level features layer by layer from the original image through a series of operations such as convolution layers and pooling layers. These feature maps contain various information of the image, such as the shape, color, texture, edge, etc. of the object. The task of the detection head is to perform a series of processing on the feature map based on the feature map extracted by the backbone network, and finally generate prediction information related to each object as the detection result output, including: obstacle category (Class labels), obstacle detection box (Bounding Box Regression) and target confidence.

[0074] The obstacle category is used to indicate the category of the detected obstacle. Since unmanned aerial vehicles generally do not fly at too low an altitude, the more common obstacle categories during their flight include birds, other unmanned aerial vehicles or helicopters.

[0075] The obstacle detection box is used to indicate the location and size of the obstacle (usually the pixel coordinates of the upper left corner and the lower right corner, or the center coordinates and the width and height). The target confidence is used to indicate the confidence of the obstacle detection model in the existence of the obstacle.

[0076] Step S1011, by simplifying the optimal transport allocation method, the teacher network output result and the true value label corresponding to the current frame image of the sample are matched and divided to obtain the teacher positive sample and the teacher negative sample.

[0077] After inputting the current frame image of the sample into the teacher network to obtain the teacher network output result, first, the teacher network output result is roughly screened according to the target confidence, and samples with target confidence lower than the preset confidence threshold are discarded. By filtering out low-confidence detection results, the amount of calculation is reduced, training efficiency is improved, false alarms are reduced, label noise is reduced, etc., making subsequent processing more efficient and accurate. This approach not only improves the overall performance of the system, but also helps improve the learning quality and consistency between the teacher network and the student network.

[0078] After coarse screening of the teacher network output results, the remaining teacher network output results and the true value labels corresponding to the current frame image of the sample are divided into teacher positive samples and teacher negative samples through the simplified optimal transport assignment method (Similarity-based Optimal Transport Assignment, SimOTA).

[0079] Among them, the actual meaning of the teacher's positive sample is the part that is similar to the true value. The loss (loss) generated by each sample in the teacher's positive sample in the subsequent distillation will be weighted by its intersection over Union (IoU) with the corresponding true value label.

[0080] The teacher negative sample includes two cases. The first case is the low-quality output caused by the misdetection of the teacher network. The second case refers to the high-quality output that is not labeled in the current frame image of the sample. Under the dataset size of the current frame image of the sample, the quality of the teacher negative sample is basically positively correlated with its target confidence. Based on this, the loss of each sample in the teacher negative sample in the subsequent distillation will be weighted by its target confidence. The loss of the teacher positive sample, the teacher negative sample, and the output result of the student network is Figure 2 Distillation losses.

[0081] Step S1012, matching and dividing the teacher positive samples, teacher negative samples and student output results by simplifying the optimal transport allocation method to obtain the student positive samples.

[0082] In some embodiments of the present invention, by simplifying the optimal transport allocation method, the teacher positive samples, teacher negative samples and student output results are matched and divided to obtain student positive samples and student negative samples under the guidance of the teacher, discard the student negative samples and retain the student positive samples.

[0083] Step S1013, calculating the first distillation loss between the student positive sample and the teacher positive sample, the second distillation loss between the student positive sample and the teacher negative sample, and the true value loss between the student output result and the true value label through a preset loss function.

[0084] In some embodiments of the present invention, the preset loss function includes a category loss function, a regression loss function or a foreground-background loss function; wherein the category loss function includes the KL divergence (Kullback-Leibler Divergence, KLDivergence) loss; the regression loss function includes the EioU loss (Efficient-IoU) function; the foreground-background loss function includes the cross-entropy (Cross-Entropy) loss function.

[0085] In some embodiments of the present invention, in the calculation of distillation loss, adaptive sample matching and weight calculation are achieved through two SimOTA methods.

[0086] Among them, the purpose of using the SimOTA method for the first time is to distinguish high-quality positive samples in the teacher's output results that are similar to the true value labels, and negative samples that are not recorded in the true value labels but may contain effective guidance information, and at the same time determine the first weight corresponding to the teacher's positive samples and the second weight corresponding to the teacher's negative samples.

[0087] The purpose of using the SimOTA method for the second time is to match the samples in the student output that are similar to the teacher's positive samples and the samples that are similar to the teacher's negative samples, and use these matched sample pairs to calculate the loss value.

[0088] Figure 2 The true value loss is the training loss in a general training scenario. In some embodiments of the present invention, the class loss, regression loss, and foreground-background loss are calculated after the student output result and the true value are matched by the SimOTA method. At this time, the class loss is composed of cross entropy loss.

[0089] Step S1014, weighting the first distillation loss by the first weight corresponding to the teacher's positive sample, and weighting the second distillation loss by the second weight corresponding to the teacher's negative sample.

[0090] Among them, the first weight refers to the intersection and union ratio of each sample in the teacher's positive sample and the corresponding true value label, and the second weight refers to the target confidence corresponding to each sample in the teacher's negative sample.

[0091] Step S1015, iteratively training the student network through the true value loss, the first distillation loss and the second distillation loss until the student network converges to obtain the obstacle detection model.

[0092] Among them, the set ratio between the true value loss, the first distillation loss and the second distillation loss is 20:2:1, and the true value loss plays a major role in model training to avoid increasing the convergence difficulty of the student network.

[0093] In addition, the obstacle detection model should pay more attention to the dynamic objects in the current frame image. Figure 2 As shown in the figure, the present invention uses Fourier transform to extract dynamic features between video frames as a feature map attention gain information. Dynamic features are irrelevant to the knowledge distillation process and are mainly used as additional information in the model training and reasoning process. Possible dynamic object parts in the current image are extracted through continuous video frames as a kind of attention information to enhance the feature map predicted by the model.

[0094] In addition, Figure 2In the above method, feature loss is optional. Usually, the feature map output by the student network is input into an adaptive scale transformation layer composed of a convolution or MLP layer to obtain the transformed features with the same scale as the feature map of the teacher network, and the feature loss of the transformed features and the feature map output by the teacher is calculated.

[0095] After training, the obstacle detection model is deployed in the unmanned aerial vehicle. During the flight of the unmanned aerial vehicle, the obstacle detection model dynamically detects surrounding obstacles and outputs obstacle detection results. The obstacle detection results output by the obstacle detection model include the obstacle category of each obstacle in the current frame image and the obstacle detection frame corresponding to each obstacle.

[0096] In some embodiments of the present invention, the present invention pre-sets a series of target categories as potential dynamic obstacle categories. These categories are objects in the real environment that are prone to change position in a short period of time, which makes the real environment different from the environment recorded by the system, resulting in the path planning algorithm being unable to take them into consideration in a timely manner. Based on this, the present invention focuses on identifying target categories and performing subsequent obstacle avoidance operations. For the obstacle categories in the detection results output by the obstacle detection model, they will be filtered according to the set potential dynamic obstacle categories to filter out the key processing objects.

[0097] In the obstacle detection result, the obstacle detection frame is used to indicate the current pixel area and current pixel coordinates of the obstacle in the current frame image, usually the pixel coordinates of the upper left corner and lower right corner of the obstacle in the current frame image, or the center coordinates and width and height.

[0098] In actual implementation, the obstacle detection results also include the category confidence corresponding to each obstacle category. The three contents are stored in three lists of the same length (obstacle category list, obstacle detection box list, and category confidence list). The corresponding positions are the detection results of the same obstacle and are associated through the same obstacle identifier, which is used to uniquely indicate the obstacle. A judgment is made based on the obstacle category list in the detection results of the obstacle detection model. If it is a pre-set dynamic obstacle category (such as birds, unmanned aerial vehicles) and the category confidence is higher than the preset confidence threshold, the detection result will be retained, otherwise the detection result will be discarded.

[0099] Specifically, after the obstacle detection model outputs the obstacle detection result, the method further includes: comparing the category confidence with a preset confidence threshold, and deleting obstacle categories whose category confidence is less than the preset confidence threshold.

[0100] Step S102, compare the obstacle detection result with the detection result corresponding to the historical frame image to obtain a comparison result, wherein the comparison result is used to indicate whether each obstacle is the same as the obstacle in the historical frame image.

[0101] In some embodiments of the present invention, the historical frame image refers to the previous video frame image of the current frame image in the video stream; or, the historical frame image refers to the previous video frame image in a preset cache or in the video stream with a frame interval between the current frame image and the previous video frame image.

[0102] The detection results corresponding to the historical frame images are recorded in the current dynamic obstacle log, including the historical obstacle information table and the obstacle information queue corresponding to each historical obstacle in the current dynamic obstacle log. The historical obstacle information table includes an obstacle identification field, an obstacle category field, an area expansion rate field, an obstacle hazard field, a movement direction field, and an update timestamp field; the obstacle information queue includes a historical obstacle size queue, a frame interval queue, and a historical obstacle position queue.

[0103] In some embodiments of the present invention, the obstacle detection result is compared with the detection result corresponding to the historical frame image, including the comparison between the obstacle category, obstacle size and obstacle position, and the comparison result is the obstacle difference obtained based on the obstacle category comparison, obstacle size comparison and obstacle position comparison.

[0104] Among them, the contribution of obstacle category to obstacle difference is a constant value, which is 1 when the obstacle category is the same and 0 when the obstacle category is different. The obstacle category difference takes into account the possibility of category misdetection, so when the obstacle category is different, it does not directly exclude the possibility that they belong to the same obstacle. The obstacle position difference is the Euclidean distance of the information pixel coordinates in the obstacle multiplied by the balance coefficient. The obstacle size difference is calculated by the following formula:

[0105]

[0106] In the formula, s query Indicates the pixel area of ​​the query obstacle; s target Represents the pixel area of ​​the source obstacle; α is the balance coefficient.

[0107] For an obstacle detection result output by the obstacle detection model, first, calculate its difference with all obstacles in the current dynamic obstacle log; then, assign the detection result to the obstacle with the smallest difference with it and less than a preset difference threshold, indicating that the detection result and the obstacle to which it is assigned are the same obstacle.

[0108] When the difference between this detection result and all obstacles is greater than the preset difference threshold, the obstacle corresponding to the detection result is identified as a new obstacle, and this detection result is used to add a new obstacle object in the current dynamic obstacle log.

[0109] Step S103 , when the comparison result indicates that each obstacle is the same as the obstacle in the historical frame image, the area expansion rate corresponding to each obstacle is calculated based on the current pixel area and the historical pixel area of ​​each obstacle in the historical frame image.

[0110] In some embodiments of the present invention, the degree of danger of an obstacle is determined by the area expansion rate of the obstacle in n consecutive video frames. Based on this, it is necessary to first calculate the area expansion rate corresponding to each obstacle, specifically including the following two cases:

[0111] In the first case, for any obstacle, the area expansion rate calculation method considering the frame interval can be expressed by the following formula:

[0112]

[0113] Among them, v r represents the area expansion rate, n represents the nth frame of the video image, s i represents the pixel area of ​​obstacle r in the i-th frame video image, s i+1 represents the pixel area of ​​obstacle r in the i+1th frame video image; t i Indicates the frame interval between the i-th and i+1-th frames.

[0114] In the second case, for a spherical obstacle, the area expansion rate calculation method considering the radius of the spherical obstacle can be expressed by the following formula:

[0115]

[0116] In the formula, represents the derivative of the function R(t), where function R(t) represents the area expansion rate of the obstacle within the time range of t to t+Δt, and Δt represents the frame interval time; a represents the radius of the spherical obstacle; v represents the relative speed at which the spherical obstacle approaches the optical center of the two-dimensional optical imaging system; x represents the distance from the spherical obstacle to the optical center of the two-dimensional optical imaging system. At the initial moment, the distance between the spherical obstacle and the optical center is x′, then x=x′-vt.

[0117] Among them, the function R(t) can be expressed by the following formula:

[0118]

[0119] In the formula, Ω represents the spatial solid angle occupied by the spherical obstacle relative to the optical center, which can be expressed as Among them, x represents the distance from the spherical obstacle to the optical center of the two-dimensional optical imaging system, a represents the radius of the spherical obstacle; Δt represents the frame interval time; Ω(t) is the solid angle occupied by the spherical obstacle relative to the optical center at time t. According to the conditions, the distance from the spherical obstacle to the optical center x = x′-vt. That is, Ω(t) = Ω(x(t)). The derivative of the area expansion rate with time is expressed as:

[0120]

[0121] Wherein, Ω represents the spatial solid angle occupied by the spherical obstacle relative to the optical center; Δt represents the frame interval time; a represents the radius of the spherical obstacle; v represents the relative speed at which the spherical obstacle approaches the optical center of the two-dimensional optical imaging system; and x represents the distance from the spherical obstacle to the optical center of the two-dimensional optical imaging system.

[0122] If the spherical obstacle is far away, the v r The expression shown is approximate.

[0123] In addition, in order to reduce the impact of the output error of the obstacle detection model on the obstacle avoidance effect of the unmanned aerial vehicle, in some embodiments of the present invention, for each obstacle, a Kalman filter is used to smooth the current pixel area and the current pixel coordinates. Specifically, the Kalman filter corrects the output result of the obstacle detection model based on the historical pixel coordinates and historical pixel areas of each obstacle in the historical frame image and the actual observation data of the current frame.

[0124] Specifically, based on the current pixel area of ​​each obstacle in the current frame image and the historical pixel area of ​​each obstacle in the historical frame image, before calculating the area expansion rate corresponding to each obstacle, it also includes: smoothing the current pixel area and current pixel coordinates corresponding to each obstacle through a Kalman filter and the historical pixel coordinates and historical pixel areas of each obstacle in the historical frame image.

[0125] Step S104: multiply the area expansion rate by the expansion rate coefficient to obtain the obstacle hazard level corresponding to each obstacle.

[0126] Specifically, the obstacle hazard α is expressed as:

[0127] α=μv r

[0128] In the formula, v r Represents the area expansion rate corresponding to the obstacle; μ represents the expansion rate coefficient.

[0129] In some embodiments of the present invention, the expansion rate coefficient is obtained by taking the square root or logarithm of the current pixel area corresponding to the obstacle. For example, taking the logarithm of the current pixel area as an example, the expansion rate coefficient can be expressed as:

[0130] μ=α t (v r )=∈v r log(s)

[0131] Where s represents the current pixel area; ∈ is a constant; v r Represents the area expansion rate corresponding to the obstacle.

[0132] In addition, since the detection results corresponding to the historical frame images are obtained by querying the historical obstacle information table and the obstacle information queue corresponding to each historical obstacle, based on this, after calculating the obstacle hazard level corresponding to each obstacle, in order to ensure the next calculation can proceed smoothly, the historical obstacle information table and the obstacle information queue need to be updated.

[0133] In some embodiments of the present invention, the historical obstacle information table includes an obstacle identification field, an obstacle category field, an area expansion rate field, an obstacle hazard field, a movement direction field and an update timestamp field; the obstacle information queue includes a historical obstacle size queue, a frame interval queue and a historical obstacle position queue.

[0134] Among them, the obstacle identification field is an integer type field, which is used to indicate the unique identification of the obstacle; the obstacle category field is an integer type field, which is used to indicate the category of the obstacle; the area expansion rate field is a floating point type field, which is used to record the area expansion rate of the obstacle per unit time in the current frame image; the obstacle danger field is a floating point type field, which is used to indicate the threat level of the obstacle to the flight safety of the unmanned aerial vehicle calculated by factors such as the obstacle area expansion rate; the movement direction field is a one-dimensional array field, which is used to indicate the movement direction trend of the obstacle in the image. In some embodiments of the present invention, the movement direction is calculated based on the pixel coordinates of each obstacle in the first frame image and the pixel coordinates in the last frame image in the current frame image; the update timestamp field is a floating point type field, which is used to indicate the timestamp of the last update of the obstacle information.

[0135] The historical obstacle size queue is a bidirectional queue used to record the pixel area of ​​obstacles in each frame of the image; the frame interval queue is a bidirectional queue used to record the timestamp of each frame of the image after processing; the historical obstacle position queue is a bidirectional queue used to record the pixel coordinates of obstacles in each frame of the image.

[0136] When updating obstacle information, add the current obstacle size value to the historical obstacle size queue, update the frame interval queue, and add the current obstacle pixel coordinate value to the historical obstacle position queue; select the area expansion rate calculated by the latest frame in the above three queues according to the calculation window, and update the area expansion rate field; update the obstacle hazard degree by multiplying the area expansion rate by the expansion rate coefficient; update the movement direction field by the movement direction calculated by the head and tail positions of the historical obstacle position queue in the calculation window; and update the update timestamp corresponding to this update according to the current actual time.

[0137] Specifically, after the area expansion rate is multiplied by the expansion rate coefficient to obtain the obstacle hazard level corresponding to each obstacle, the method further includes: adding the current pixel area corresponding to each obstacle to the historical obstacle size queue corresponding to each obstacle; updating the timestamp after the current frame image is processed in the frame interval queue; adding the current pixel coordinates corresponding to each obstacle to the historical obstacle position queue corresponding to each obstacle; and updating the historical obstacle information table according to the obstacle identification field, the area expansion rate corresponding to each obstacle, the obstacle hazard level, the movement direction and the update timestamp corresponding to this update.

[0138] In some embodiments of the present invention, the last time that an obstacle was actually detected by the obstacle detection model is recorded by updating the timestamp. If the obstacle cannot be detected, there is a possibility that the obstacle has left the range that threatens the unmanned aerial vehicle. At this time, in order to prevent the obstacle from staying indefinitely in the current dynamic obstacle log and interfering with the system's decision, the system will check the update timestamps of all obstacles before performing obstacle detection on the current frame image. If the update time difference between the update timestamp of an obstacle and the current actual time is greater than the preset time length, that is, the obstacle has not been detected for a certain period of time, then the obstacle will be removed from the current dynamic obstacle log.

[0139] Specifically, before inputting the current frame image in the video stream into the pre-trained obstacle detection model, it also includes: determining the update time difference corresponding to each obstacle in the historical obstacle information queue based on the current time and the update timestamp corresponding to each obstacle in the historical obstacle information queue; and removing obstacles with an update time difference greater than a preset time length from the historical obstacle queue.

[0140] When the update time difference between the update timestamp of an obstacle and the current actual time is less than or equal to the preset time length, in the obstacle detection of a frame image, if an obstacle recorded in the current dynamic obstacle log is not detected, the system will predict its motion state and execute the empty update method. The empty update method is used to track the obstacle within a period of time when the obstacle detection model does not detect the obstacle or the obstacle exceeds the image boundary. The empty update method does not update the update timestamp.

[0141] In some embodiments of the present invention, the Kalman filter is used to predict the obstacle size, obstacle position and obstacle movement direction. The predicted obstacle size is added to the historical obstacle size queue corresponding to the obstacle, and the frame interval queue is updated synchronously.

[0142] In some embodiments of the present invention, due to the field of view limitation of the two-dimensional optical imaging system, after the obstacle moves beyond the field of view, the program will also perform the same prediction and empty update operations as described above to maintain vigilance against such obstacles for a certain period of time.

[0143] Step S105 : comparing the obstacle hazard level corresponding to each obstacle with a preset hazard level threshold, and determining the obstacle whose obstacle hazard level is greater than the preset hazard level threshold as a dangerous obstacle.

[0144] Step S106, based on the pixel width and pixel height of the current frame image, the real width and real height corresponding to the current frame image, the distance from the optical center of the two-dimensional optical imaging system to the center of the virtual imaging plane and the current pixel coordinates, determine the dangerous obstacle vectors of each dangerous obstacle pointing to the two-dimensional optical imaging system.

[0145] refer to Figure 3 In some embodiments of the present invention, the image pixel width is w p , the image pixel height is h p , the image origin is the upper left corner, and the distance from the optical center O of the two-dimensional optical imaging system to the center O' of the virtual imaging plane is d.

[0146] By specifying the value of d, the real scales of the image pixel width and height w and h can be calculated according to the field of view (FOV) of the two-dimensional optical imaging system, thereby determining the real position of the obstacle relative to the two-dimensional optical imaging system. o ,y o ) is used as an example to illustrate. For simplicity, the aspect ratio of the captured image is set to 1:1, the horizontal field of view angle is f, and the true scale is expressed as:

[0147] Then in the X of the two-dimensional optical imaging systemc Y c Z c In the coordinate system, the dangerous obstacle vector pointing from the obstacle to the two-dimensional optical imaging system is expressed as:

[0148]

[0149] In the formula, w p h represents the image pixel width of the two-dimensional optical imaging system; p Represents the image pixel height of the two-dimensional optical imaging system; (x o ,y o ) represents the pixel coordinates of the obstacle in the image; d represents the distance from the optical center O of the two-dimensional optical imaging system to the center O' of the virtual imaging plane; w represents the real scale width corresponding to the image pixel width; h represents the real scale height corresponding to the image pixel height.

[0150] In some embodiments of the present invention, dangerous obstacles also include obstacles whose predicted obstacle danger level is greater than a preset danger level threshold value among lost obstacles. The predicted obstacle danger level is calculated based on a preset danger level decay rate and a historical obstacle danger level corresponding to the lost obstacle. The preset danger level decay rate refers to a preset danger level decay rate, including 5%, 10% or 15%, etc. This embodiment does not limit the value of the preset danger level decay rate.

[0151] When the predicted obstacle hazard level corresponding to the lost obstacle is greater than the preset hazard level threshold, the Kalman filter is used to predict the lost obstacle, obtain the predicted pixel coordinates corresponding to the lost obstacle, and calculate the dangerous obstacle vector corresponding to the lost obstacle.

[0152] Specifically, when the comparison result indicates that the obstacle in the current frame image is different from the obstacle in the historical frame image and there is a lost obstacle, the method also includes: obtaining a predicted obstacle hazard based on a preset hazard attenuation rate and the historical obstacle hazard corresponding to the lost obstacle; when the predicted obstacle hazard is greater than a preset hazard threshold, predicting the lost obstacle through a Kalman filter to obtain predicted pixel coordinates corresponding to the lost obstacle; the predicted pixel coordinates are used to calculate the dangerous obstacle vector corresponding to the lost obstacle; the lost obstacle is determined as a dangerous obstacle, and the step of determining the dangerous obstacle vector pointing to the two-dimensional optical imaging system of each dangerous obstacle is executed.

[0153] Step S107, based on the dangerous obstacle vectors and obstacle hazard levels corresponding to each dangerous obstacle, the repulsion vectors corresponding to each dangerous obstacle are calculated, and the repulsion vectors are weighted summed by the obstacle hazard levels corresponding to each dangerous obstacle to obtain the overall repulsion vector.

[0154] Among them, the repulsive force vector of the obstacle on the UAV is expressed as:

[0155]

[0156] In the formula, Represents the dangerous obstacle vector pointing to the two-dimensional optical imaging system; α t Indicates the obstacle hazard level corresponding to the obstacle at time t.

[0157] Accordingly, the overall repulsive force vector can be expressed as follows:

[0158]

[0159] In the formula, Represents the dangerous obstacle vector pointing to the two-dimensional optical imaging system; α t represents the obstacle hazard level corresponding to the obstacle at time t; α represents the preset obstacle hazard level threshold.

[0160] Take the maximum obstacle hazard among the dangerous obstacles as the overall hazard α f , expressed as:

[0161]

[0162] Where s represents a dangerous obstacle; C a represents the set of dangerous obstacles; α(s) represents the obstacle hazard level corresponding to the dangerous obstacle s.

[0163] Step S108: For the vectors corresponding to the avoidance directions of the dangerous obstacles, weighted sum is performed using the corresponding obstacle danger levels, and the direction pointed by the summed vectors is taken as the optimal avoidance direction.

[0164] Among them, the avoidance direction is obtained based on a preset path planning algorithm.

[0165] In some embodiments of the present invention, the best avoidance direction is the avoidance direction of the UAV calculated on the image plane. The calculation method can be selected in different ways according to the needs, for example, the direction with the sparsest obstacles on the image can be selected as the avoidance direction, or the avoidance direction can be calculated by combining the environmental safety direction given by the path planning system with the obstacle position.

[0166] In actual implementation, when an obstacle is on one side close to the center of the image and moves to the other side at a higher tangential speed relative to the UAV, the opposite direction of the obstacle can be selected as the avoidance direction.

[0167] For example: Reference Figure 4When the obstacle is located in the central area shown by the dotted box and moves at a large tangential speed, choosing to avoid in the opposite direction will help you escape from the obstacle danger zone more quickly; when the obstacle is located outside the central area, choose the direction of the line connecting the center of the obstacle and the center of the image as the avoidance direction, and the size is inversely proportional to the distance between the obstacle and the center of the image.

[0168] After determining the avoidance direction corresponding to each obstacle, the avoidance directions are weighted and summed according to the obstacle hazard degree corresponding to each obstacle, and the vector obtained by the weighted sum is determined as the optimal avoidance direction.

[0169] Step S109, calculating the component of the optimal avoiding direction perpendicular to the overall repulsive force vector as the real avoiding direction, and controlling the unmanned aerial vehicle to avoid obstacles according to the real avoiding direction.

[0170] In summary, the obstacle avoidance control method for an unmanned aerial vehicle provided in the present application continuously shoots a video stream through a two-dimensional optical imaging system, inputs the current frame image in the video stream into a pre-trained obstacle detection model, and the obstacle detection model outputs an obstacle detection result; based on the current pixel area and the historical pixel area of ​​each obstacle in the historical frame image, the area expansion rate and obstacle hazard degree corresponding to each obstacle are calculated; the dangerous obstacle vector pointing to the two-dimensional optical imaging system of each dangerous obstacle is determined; based on the dangerous obstacle vector and the obstacle hazard degree corresponding to each dangerous obstacle, the repulsion vector corresponding to each dangerous obstacle is calculated and weighted summed to obtain the overall repulsion vector; the avoidance direction corresponding to each dangerous obstacle is weighted summed according to the obstacle hazard degree corresponding to each dangerous obstacle to obtain the optimal avoidance direction; the component of the optimal avoidance direction perpendicular to the overall repulsion vector is calculated as the true avoidance direction, and the unmanned aerial vehicle is controlled to follow the true avoidance direction. The method can avoid obstacles by following the actual avoidance direction; it can solve the problems of complex systems and high operation and maintenance costs of unmanned aerial vehicles; it detects obstacles on the video stream obtained by the two-dimensional optical imaging system, obtains obstacle information and updates it, and then evaluates the danger posed by each obstacle to the unmanned aerial vehicle based on the relative approach speed and relative position between the obstacle and the unmanned aerial vehicle, and assigns corresponding weights; then, using this weight and the relative position of the obstacle, it calculates the current repulsion vector of the obstacle; finally, in combination with the environmental safety information provided by the path planning system and the specific position of the obstacle, it determines the most suitable avoidance direction and speed in the plane perpendicular to the repulsion vector, and then guides the unmanned aerial vehicle to perform effective avoidance actions; it can control the unmanned aerial vehicle to perform dynamic obstacle avoidance without dynamic environment modeling and real-time depth information; therefore, it can avoid equipping the unmanned aerial vehicle with sensors for obtaining depth information, thereby reducing the complexity and maintenance cost of the unmanned aerial vehicle system.

[0171] In addition, in order to address the problem that the operation of neural networks requires more computing resources, the present invention optimizes the obstacle detection model through the knowledge distillation method to reduce the algorithm's demand for computing resources and improve the obstacle avoidance efficiency of the unmanned aerial vehicle.

[0172] The obstacle avoidance control method for the unmanned aerial vehicle provided by the present invention is described below with reference to another example. Figure 5 The method at least includes the following steps S501 to S506:

[0173] Step S501: extract obstacle information from the required video frame to calculate the information required for avoiding the obstacle, including the obstacle danger level.

[0174] The required video frames refer to the latest series of continuous video frame images in the real-time video stream acquired by the two-dimensional optical imaging system equipped on the unmanned aerial vehicle, and the number of frames is determined by a pre-set calculation window.

[0175] The required video frames are input into a pre-trained obstacle detection model, and the obstacle detection model outputs the obstacle detection results as the extracted obstacle information, including the obstacle category, the obstacle detection frame, and the category confidence corresponding to the obstacle category.

[0176] The obstacle detection frame is used to indicate the current pixel area and current pixel coordinates of the obstacle in the current frame image, which are usually the pixel coordinates of the upper left corner and the lower right corner of the obstacle in the current frame image, or the center coordinates and width and height. The training method of the obstacle detection model refers to the above-mentioned obstacle avoidance control method embodiment of the unmanned aerial vehicle, which will not be repeated here.

[0177] Based on the current pixel area and the historical pixel area of ​​each obstacle in the historical frame image, the area expansion rate corresponding to each obstacle is calculated; the area expansion rate is multiplied by the expansion rate coefficient to obtain the obstacle hazard degree corresponding to each obstacle.

[0178] Step S502, determining whether the obstacle hazard level is greater than a preset hazard level threshold, if the obstacle hazard level is greater than the preset hazard level threshold, executing step S503; otherwise, executing step S501;

[0179] Step S503, obtaining control of the UAV to perform evasive action.

[0180] Step S504, determining whether the current obstacle hazard level is less than a preset hazard level threshold. If the current obstacle hazard level is less than the preset hazard level threshold, executing step S505; otherwise, executing step S501.

[0181] Step S505, return to the original path and return the control of the UAV.

[0182] Step S506, determining whether the flight mission is completed, if the flight mission is completed, the current flight is terminated, otherwise, executing step S501.

[0183] In summary, the obstacle avoidance control method for an unmanned aerial vehicle provided in the present application can solve the problems of the complexity of the system and high operation and maintenance costs of the unmanned aerial vehicle; obstacle detection is performed on the video stream obtained by the two-dimensional optical imaging system to obtain obstacle information and update it, and then, based on the relative approach speed and relative position between the obstacle and the unmanned aerial vehicle, the degree of danger posed by each obstacle to the unmanned aerial vehicle is evaluated and a corresponding weight is assigned, and then, using this weight and the relative position of the obstacle, the current repulsion vector of the obstacle is calculated, and finally, combined with the environmental safety information provided by the path planning system and the specific position of the obstacle, the most suitable avoidance direction and speed are determined in the plane perpendicular to the repulsion vector, thereby guiding the unmanned aerial vehicle to perform effective avoidance actions, and the unmanned aerial vehicle can be controlled to perform dynamic obstacle avoidance without dynamic environment modeling and real-time depth information. Therefore, it is possible to avoid equipping the unmanned aerial vehicle with a sensor for obtaining depth information, thereby reducing the complexity and maintenance cost of the unmanned aerial vehicle system;

[0184] In the existing unmanned aerial vehicle obstacle avoidance method, the image depth information is first obtained through the imaging system, and the dynamic obstacle is modeled in three-dimensional space based on the image depth information to obtain the xyz coordinates of the obstacle in three-dimensional space. Figure 3 , the horizontal direction of the image is the x-axis, the vertical direction is the y-axis, and the z-axis is the depth information axis; then, the Kalman filter is used to track the trajectory of dynamic obstacles in three-dimensional space, and the Kalman filter's ability to predict obstacle trajectories is combined with other methods to make obstacle avoidance decisions.

[0185] The ultimate goal of using depth information to model obstacles in three dimensions is to determine whether they will collide with the UAV at some point in the future based on their future trajectory, which is similar to the concept of obstacle collision time. The obstacle collision time can be expressed as:

[0186] Where x represents the distance between the obstacle and the optical center of the imaging system in the UAV; v represents the component of the obstacle's velocity on the line connecting the obstacle and the optical center.

[0187] Obviously, the obstacle collision time t is proportional to the distance x. Based on this, in some embodiments of the present invention, the obstacle collision time can be used instead of the actual depth information at this moment as indirect depth information to model the coordinates of the obstacle in three-dimensional space. The indirect coordinates established by the indirect depth information can also be input into the Kalman filter to track and predict the obstacle trajectory. The final obstacle prediction trajectory is the same shape as the actual obstacle trajectory and is in a certain proportion.

[0188] For example, a fast-approaching obstacle from a distance and a slow-approaching obstacle from a nearby location may collide with the UAV at the same time. The trajectories of the two obstacles are the same under this modeling form.

[0189] At the same time, from the area expansion rate formula of the second case above, it can be seen that when the obstacle is far away, the area expansion rate Combination Obstacle collision time can be obtained Since the area expansion rate v r It can be measured, therefore, the obstacle expansion rate at a certain moment can be used to replace the obstacle collision time at this moment.

[0190] It is worth noting that the depth information in the traditional sense is the Figure 3 The distance in the z-axis direction in , that is, the distance to the imaging plane xy, and the above-mentioned alternative method is to replace the distance between the obstacle and the optical center of the two-dimensional imaging optical system. Both methods can describe the spatial position of the obstacle, and the rules followed when the Kalman filter is updated are slightly different, which will not be repeated here.

[0191] In some embodiments of the present invention, as a supplementary obstacle avoidance method to the obstacle avoidance control method of the unmanned aerial vehicle in the above-mentioned steps S101 to S109, the obstacle collision time can be used instead of the actual depth information at this moment, and the obstacle three-dimensional space can be modeled as indirect depth information. At the same time, the obstacle expansion rate at a certain moment is used to replace the obstacle collision time at this moment, and then a Kalman filter is used to track the dynamic obstacle trajectory in the obstacle three-dimensional space. The Kalman filter's ability to predict the obstacle trajectory is combined with other methods (such as a path planning algorithm or an obstacle avoidance algorithm based on machine learning, etc.) to make obstacle avoidance decisions.

[0192] In this way, the obstacle trajectory can be predicted in advance and then the obstacle avoidance decision can be made. Compared with the obstacle avoidance control method of the unmanned aerial vehicle in the above steps S101 to S109, redundant obstacle avoidance actions can be avoided, making the dynamic obstacle avoidance function of the unmanned aerial vehicle more efficient and stable. However, the disadvantage is that certain observation data is required to predict obstacles, which requires more reaction time.

[0193] Based on this, in some embodiments of the present invention, the above-mentioned supplementary obstacle avoidance method is applied simultaneously with the obstacle avoidance control method of the unmanned aerial vehicle in the above-mentioned steps S101 to S109 (hereinafter referred to as the reactive obstacle avoidance strategy), such as Figure 6 As shown, the method at least includes steps S601 to S612:

[0194] Step S601, determine whether an obstacle is detected, and if an obstacle is detected, execute step S602; otherwise, execute step S608.

[0195] Step S602: Update corresponding obstacle detection information.

[0196] The updating method refers to the obstacle avoidance control method of the UAV in the above steps S101 to S109, which will not be described in detail here.

[0197] Step S603: The Kalman filter updates data and predicts the obstacle motion trajectory.

[0198] Step S604, determine whether the current state is in the initial stage. If the current state is in the initial stage, execute step S607; otherwise, execute step S605.

[0199] When an obstacle is detected in the early stage and the Kalman filter cannot accurately track and predict the obstacle's trajectory, a reactive obstacle avoidance method is used; otherwise, the obstacle avoidance action can be planned based on the motion trajectory predicted by the Kalman filter.

[0200] The initial stage can be set to a certain obstacle information is actually updated less than a certain number of times, for example, 15 times. In actual situations, the time to leave the initial stage will fluctuate according to the performance of the obstacle detection model and the operating platform.

[0201] The Kalman filter used in this embodiment is different from the Kalman filter used in the above embodiment. The Kalman filter in this embodiment can be used to replace the Kalman filter in the above embodiment, or the Kalman filter in the above embodiment can be retained.

[0202] Step S605, determining whether the prediction and observation error of the Kalman filter is less than a preset threshold. If so, executing step S606; otherwise, executing step S607.

[0203] When a certain amount of observation data is accumulated for a certain obstacle, the error between the predicted trajectory and the observed trajectory is less than the preset threshold, and it leaves the initial stage of observing the obstacle and enters the stable stage, the obstacle avoidance action can be planned according to the motion trajectory predicted by the Kalman filter.

[0204] In the stable stage, the motion state of the obstacle may suddenly change. At this time, the error between the predicted trajectory and the observed trajectory may exceed the preset threshold. At this time, the reactive obstacle avoidance method for this obstacle will be restarted until the error between the predicted trajectory and the observed trajectory is less than the preset threshold again.

[0205] Step S606, planning obstacle avoidance actions according to the predicted motion trajectory.

[0206] Step S607: Plan obstacle avoidance actions using a reactive obstacle avoidance strategy.

[0207] Step S608: The Kalman filter updates data and predicts the obstacle's motion trajectory.

[0208] When the obstacle detection model misses an obstacle, there is no obstacle information update at this time, and the empty update method is performed. The empty update method refers to the obstacle avoidance control method of the unmanned aerial vehicle in the above steps S101 to S109, which will not be repeated here.

[0209] If the obstacle is in the initial stage, reactive obstacle avoidance is used. If the obstacle is in the stable stage, the error judgment between the predicted trajectory and the observed trajectory is no longer performed, and the obstacle avoidance action is planned directly according to the motion trajectory predicted by the Kalman filter. The prediction ability of the Kalman filter can combat missed detection to a certain extent.

[0210] Step S609, determine whether the current state is in the initial stage. If the current state is in the initial stage, execute step S610; otherwise, execute step S611.

[0211] Step S610: Plan obstacle avoidance actions using a reactive obstacle avoidance strategy.

[0212] Step S611, planning obstacle avoidance actions according to the predicted motion trajectory.

[0213] Step S612, determining whether the obstacle avoidance task is completed, if the obstacle avoidance task is completed, the task is completed; otherwise, executing step S601.

[0214] In summary, the obstacle avoidance control method for the unmanned aerial vehicle provided in this embodiment uses the obstacle collision time instead of the actual depth information at this moment as indirect depth information to model the obstacle three-dimensional space. At the same time, the obstacle expansion rate at a certain moment is used to replace the obstacle collision time at this moment, and then the Kalman filter is used to track the dynamic obstacle trajectory in the obstacle three-dimensional space. The Kalman filter is used to predict the obstacle trajectory in combination with other methods to make obstacle avoidance decisions. In this way, when the obstacle detection model misses the obstacle or the obstacle trajectory can be predicted in advance in the early stage of the obstacle, the obstacle avoidance decision is made, which can avoid redundant obstacle avoidance actions and make the dynamic obstacle avoidance function of the unmanned aerial vehicle more efficient and stable. At the same time, combined with the reactive obstacle avoidance strategy, when the error between the predicted trajectory and the observed trajectory is less than the preset threshold and the initial stage of observing the obstacle is separated and the stable stage is entered, the obstacle avoidance action is planned according to the motion trajectory predicted by the Kalman filter. When the error between the predicted trajectory and the observed trajectory exceeds the preset threshold, the reactive obstacle avoidance method is used until the error between the predicted trajectory and the observed trajectory is less than the preset threshold again, which can improve the success rate of obstacle avoidance.

[0215] Corresponding to the above method, the present invention also provides an obstacle avoidance control method device for an unmanned aerial vehicle, comprising a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the above-mentioned obstacle avoidance control method for the unmanned aerial vehicle.

[0216] The present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the aforementioned obstacle avoidance control method for an unmanned aerial vehicle.

[0217] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0218] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0219] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0220] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An obstacle avoidance control method for an unmanned aerial vehicle, characterized in that: The unmanned aerial vehicle is provided with a two-dimensional optical imaging system for continuously shooting a video stream; the method comprises the following steps: Inputting the current frame image in the video stream into a pre-trained obstacle detection model, and having the obstacle detection model output an obstacle detection result; the obstacle detection result includes an obstacle category of each obstacle in the current frame image and an obstacle detection frame corresponding to each obstacle; the obstacle detection frame is used to indicate a current pixel area and a current pixel coordinate of the obstacle in the current frame image; Comparing the obstacle detection result with the detection result corresponding to the historical frame image to obtain a comparison result; the comparison result is used to indicate whether each obstacle is the same as the obstacle in the historical frame image; When the comparison result indicates that each obstacle is the same as the obstacle in the historical frame image, based on the current pixel area and the historical pixel area of ​​each obstacle in the historical frame image, calculate the area expansion rate corresponding to each obstacle; The area expansion rate is multiplied by the expansion rate coefficient to obtain the obstacle hazard degree corresponding to each obstacle; the expansion rate coefficient is obtained by taking the square root or logarithm of the current pixel area; Comparing the obstacle hazard level corresponding to each obstacle with a preset hazard level threshold, and determining an obstacle among the obstacles whose obstacle hazard level is greater than the preset hazard level threshold as a dangerous obstacle; Determine a dangerous obstacle vector pointing to the two-dimensional optical imaging system from each of the dangerous obstacles based on the pixel width and pixel height of the current frame image, the real width and real height corresponding to the current frame image, the distance from the optical center of the two-dimensional optical imaging system to the center of the virtual imaging plane, and the current pixel coordinates; Based on the dangerous obstacle vectors and the obstacle hazard levels corresponding to the dangerous obstacles, the repulsive force vectors corresponding to the dangerous obstacles are calculated, and the repulsive force vectors are weighted and summed according to the obstacle hazard levels corresponding to the dangerous obstacles to obtain the overall repulsive force vector; For the vectors corresponding to the avoidance directions of the dangerous obstacles, the corresponding obstacle danger degrees are used to perform weighted summation, and the direction pointed by the summed vectors is used as the optimal avoidance direction; wherein the avoidance direction is obtained based on a preset path planning algorithm; The component of the optimal avoiding direction perpendicular to the overall repulsive force vector is calculated as the real avoiding direction, and the unmanned aerial vehicle is controlled to avoid obstacles according to the real avoiding direction.

2. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 1, characterized in that: Before calculating the area expansion rate corresponding to each obstacle based on the current pixel area and the historical pixel area of ​​each obstacle in the historical frame image, the method further includes: The current pixel area and the current pixel coordinates corresponding to each obstacle are smoothed by using a Kalman filter and the historical pixel coordinates and the historical pixel areas of each obstacle in the historical frame image.

3. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 2, characterized in that: When the comparison result indicates that the obstacle in the current frame image is different from the obstacle in the historical frame image and there is a missing obstacle, the method further includes: Obtaining a predicted obstacle hazard based on a preset hazard attenuation rate and a historical obstacle hazard corresponding to the lost obstacle; When the predicted obstacle hazard level is greater than the preset hazard level threshold, the lost obstacle is predicted by the Kalman filter to obtain predicted pixel coordinates corresponding to the lost obstacle; the predicted pixel coordinates are used to calculate the dangerous obstacle vector corresponding to the lost obstacle; The lost obstacle is determined as a dangerous obstacle, and the step of determining a dangerous obstacle vector pointing to the two-dimensional optical imaging system for each of the dangerous obstacles is performed.

4. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 1, characterized in that: When the video frame processing speed of the obstacle detection model is lower than the frame rate of the two-dimensional optical imaging system, the video stream is converted into video frames and stored in a preset cache; The step of inputting the current frame image in the video stream into a pre-trained obstacle detection model comprises: Obtaining the latest video frame from the preset cache as the current frame image according to the preset calculation window, and inputting it into the obstacle detection model; The video frame before the current frame image in the preset cache is deleted.

5. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 1, characterized in that: The training process of the obstacle detection model includes: Input the current frame image of the sample into the teacher network and the student network in the initial obstacle detection model respectively, and obtain the output results of the teacher network and the student network; By simplifying the optimal transport allocation method, the output result of the teacher network and the true value label corresponding to the current frame image of the sample are matched and divided to obtain a teacher positive sample and a teacher negative sample; Matching and dividing the teacher positive sample, the teacher negative sample and the student output result by the simplified optimal transport allocation method to obtain the student positive sample; Calculate the first distillation loss between the student positive sample and the teacher positive sample, the second distillation loss between the student positive sample and the teacher negative sample, and the true value loss between the student output result and the true value label by a preset loss function; the preset loss function includes a category loss function, a regression loss function or a foreground-background loss function; the set ratio between the true value loss, the first distillation loss and the second distillation loss is 20:2:1; The first distillation loss is weighted by a first weight corresponding to the teacher's positive sample, and the second distillation loss is weighted by a second weight corresponding to the teacher's negative sample; The student network is iteratively trained through the true value loss, the first distillation loss, and the second distillation loss until the student network converges to obtain the obstacle detection model.

6. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 1, characterized in that: The detection result corresponding to the historical frame image is obtained by querying the historical obstacle information table and the obstacle information queue corresponding to each historical obstacle; the historical obstacle information table includes an obstacle identification field, an obstacle category field, an area expansion rate field, an obstacle danger field, a movement direction field and an update timestamp field; the obstacle information queue includes a historical obstacle size queue, a frame interval queue and a historical obstacle position queue; After the area expansion rate is multiplied by the expansion rate coefficient to obtain the obstacle hazard level corresponding to each obstacle, the method further includes: Adding the current pixel area corresponding to each obstacle to the historical obstacle size queue corresponding to each obstacle; Updating the timestamp of the current frame image after processing in the frame interval queue; Adding the current pixel coordinates corresponding to each obstacle to the historical obstacle position queue corresponding to each obstacle; The historical obstacle information table is updated according to the obstacle identification field, the area expansion rate, the obstacle danger level, the movement direction and the update timestamp corresponding to the current update of each obstacle; the movement direction is calculated based on the pixel coordinates of each obstacle in the first frame image and the pixel coordinates in the last frame image in the current frame image.

7. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 6, characterized in that: Before inputting the current frame image in the video stream into the pre-trained obstacle detection model, the method further includes: Determine an update time difference corresponding to each obstacle in the historical obstacle information queue based on the current time and the update timestamp corresponding to each obstacle in the historical obstacle information queue; Obstacles whose update time difference is greater than a preset time length are removed from the historical obstacle queue.

8. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 1, characterized in that: The obstacle detection result also includes the category confidence corresponding to each obstacle category; after the obstacle detection model outputs the obstacle detection result, it also includes: The category confidence is compared with a preset confidence threshold, and obstacle categories whose category confidence is less than the preset confidence threshold are deleted.

9. An obstacle avoidance control device for an unmanned aerial vehicle, characterized in that: include: A two-dimensional optical imaging system for continuous acquisition of video streams; A processor, a memory and a computer program / instruction stored in the memory, wherein the processor is used to execute the computer program / instruction, and when the computer program / instruction is executed, the device implements the steps of the obstacle avoidance control method for an unmanned aerial vehicle as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, which, when executed by a processor, is used to implement the steps of the obstacle avoidance control method for an unmanned aerial vehicle as described in any one of claims 1 to 8.

Images