Obstacle avoidance method of unmanned aerial vehicle, storage medium, task machine and unmanned aerial vehicle system

Abstract

The invention discloses an obstacle avoidance method of an unmanned aerial vehicle, a storage medium, a task machine and an unmanned aerial vehicle system, and relates to the technical field of unmanned aerial vehicles. The method comprises the following steps: acquiring front-view images and flight attitude data of an unmanned aerial vehicle in real time, and performing target detection on each front-view image by using a pre-trained target recognition model to obtain a target detection result; when the target detection result is that the obstacle exists, tracking the obstacle by using a preset target tracking algorithm to obtain a tracking result of the obstacle; and judging whether the obstacle is a target obstacle based on the tracking result of the obstacle, the center point coordinate of the front view image and the area proportion of the obstacle in the front view image, and determining an obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data when the obstacle is the target obstacle. And the unmanned aerial vehicle performs obstacle avoidance according to the obstacle avoidance path. Therefore, the problems of poor real-time performance, inaccurate obstacle avoidance and the like in the prior art are solved.

Description

Technical Field

[0001] This application relates to the technical field of unmanned aerial vehicles, and particularly to an obstacle avoidance method, a storage medium, a mission machine and an unmanned aerial vehicle system for an unmanned aerial vehicle. Background Art

[0002] With the rapid development of unmanned aerial vehicle technology, its applications in the fields of logistics, inspection, agriculture, etc. are becoming increasingly widespread. However, the autonomous obstacle avoidance ability of unmanned aerial vehicles still faces many challenges in complex environments. Existing obstacle avoidance technologies mainly rely on multi-sensor fusion (such as vision, lidar, ultrasonic waves) and obstacle avoidance strategies based on rules or machine learning.

[0003] However, at present, the obstacle avoidance strategies based on machine learning have relatively high false detection rates and missed detection rates, and the obstacle avoidance strategies are single, resulting in inaccurate and inefficient obstacle avoidance for unmanned aerial vehicles. Summary of the Invention

[0004] The main purpose of this application is to provide an obstacle avoidance method, a storage medium, a mission machine and an unmanned aerial vehicle system for an unmanned aerial vehicle, so as to solve the problems of poor real-time performance and inaccurate obstacle avoidance in the prior art.

[0005] To achieve the above purpose, this application provides an obstacle avoidance method for an unmanned aerial vehicle, including: Real-time obtain the front view image and flight attitude data of the unmanned aerial vehicle, and use a pre-trained target recognition model to perform target detection on each of the front view images to obtain a target detection result; In the case that the target detection result indicates the existence of an obstacle, use a preset target tracking algorithm to track the obstacle to obtain a tracking result of the obstacle; Based on the tracking result of the obstacle, the center point coordinates of the front view image, and the area ratio of the obstacle in the front view image, determine whether the obstacle is a target obstacle, and in the case that the obstacle is the target obstacle, determine an obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data, so that the unmanned aerial vehicle performs obstacle avoidance according to the obstacle avoidance path.

[0006] Optionally, the tracking result of the obstacle includes the coordinate data of the obstacle in each frame of the front view image; based on the tracking result of the obstacle, the center point coordinates of the front view image, and the area ratio of the obstacle in the front view image, determining whether the obstacle is a target obstacle includes: based on the coordinate data of the obstacle in each frame of the front view image and the center point coordinates of the front view image, determine whether the obstacle is moving towards the center point of the front view image; if it is determined that the obstacle is moving towards the center point of the front view image or the area ratio of the obstacle in the front view image is greater than a preset ratio, then use the obstacle as the target obstacle.

[0007] Optionally, determining whether the obstacle moves towards the center point of the forward-looking image based on the coordinate data of the obstacle in each frame of the forward-looking image and the center point coordinates of the forward-looking image includes: determining a plurality of displacement amounts of the obstacle based on the coordinate data of the obstacle in each frame of the forward-looking image, where the displacement amount is the displacement amount of the obstacle between the forward-looking image of the current frame and the forward-looking image of the next frame; determining the moving speed of the obstacle based on each of the displacement amounts; and using a preset formula and based on the moving speed of the obstacle and the center point coordinates of the forward-looking image, determining whether the obstacle moves towards the center point of the forward-looking image.

[0008] Optionally, the preset formula is:

[0009] In the formula, is the angle between the moving speed and the center point vector; is the moving speed of the obstacle; is the center point vector; is the center point coordinates of the forward-looking image; a and c are the speeds in the x and y directions respectively.

[0010] Optionally, the moving speed of the obstacle is calculated by the following formula:

[0011] In the formula, is the moving speed of the obstacle; are the coordinates of the obstacle in the forward-looking image of the t-th frame; are the coordinates of the obstacle in the forward-looking image of the (t + 1)-th frame; T is the preset total number of frames of the forward-looking image.

[0012] Optionally, the training steps of the target recognition model include: obtaining a training data set, where the training data set includes a plurality of sample images with annotation frames and annotation information, and performing processing on each of the sample images by using Mosaic data augmentation and adaptive image scaling; obtaining target anchor boxes based on the annotation frames in each of the sample images and using a preset adaptive anchor box algorithm; inputting each of the processed sample images into a pre-constructed target recognition model, where the target recognition model obtains a target recognition result based on the target anchor boxes, and calculates a loss value by using a CIOU loss function based on the target recognition result and the annotation information, and updating the parameters of the target recognition model based on the loss value until the target recognition model converges, to obtain the trained target recognition model.

[0013] Optionally, obtaining the target anchor box based on the annotation boxes in each of the sample images and using a preset adaptive anchor box algorithm includes: when the recall rate of the initial anchor box of the target recognition model is lower than a preset threshold, randomly initializing N anchor boxes, where N is a positive integer; using a genetic algorithm, a clustering algorithm, each of the annotation boxes in each of the sample images, and each of the anchor boxes to update each of the anchor boxes until each of the anchor boxes meets a preset condition, and then taking each of the anchor boxes as the target anchor box, where the preset condition is that the recall rate of each of the anchor boxes is higher than the preset threshold.

[0014] In addition, to achieve the above object, the present application also provides a mission aircraft of an unmanned aerial vehicle, including: an acquisition module, configured to acquire the forward-looking image and flight attitude data of the unmanned aerial vehicle in real time; a target recognition module, configured to perform target detection on each of the forward-looking images by using a pre-trained target recognition model to obtain a target detection result; a target tracking module, configured to, when the target detection result indicates the presence of an obstacle, use a preset target tracking algorithm to track the obstacle to obtain a tracking result of the obstacle; an obstacle avoidance module, configured to determine whether the obstacle is a target obstacle based on the tracking result of the obstacle, the central point coordinates of the forward-looking image, and the area ratio of the obstacle in the forward-looking image, and when the obstacle is the target obstacle, determine an obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data, so that the unmanned aerial vehicle performs obstacle avoidance according to the obstacle avoidance path.

[0015] The present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the main distribution network collaborative voltage regulation method as described in any one of the above.

[0016] The present application also provides a drone system, including the mission aircraft, the camera and the flight control chip as described above. The mission aircraft, the camera and the flight control chip use ros1 to achieve data interaction. The camera is used to obtain the front view image of the drone in real time and send the front view image to the mission aircraft. The flight control chip is used to obtain the flight attitude data of the drone and send the flight attitude data to the mission aircraft. The mission aircraft is used to receive the front view image and the flight attitude data of the drone in real time, and use the pre-trained target recognition model to perform target detection on each front view image to obtain the target detection result. In the case that the target detection result indicates the presence of an obstacle, use a preset target tracking algorithm to track the obstacle to obtain the tracking result of the obstacle. Based on the tracking result of the obstacle, the center point coordinates of the front view image, and the area ratio of the obstacle in the front view image, determine whether the obstacle is a target obstacle. In the case that the obstacle is the target obstacle, determine the obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data, so that the drone performs obstacle avoidance according to the obstacle avoidance path.

[0017] The obstacle avoidance method of the drone of the present application obtains the front view image and the flight attitude data of the drone, and uses the pre-trained target recognition model and the preset target tracking algorithm based on the front view image to find the obstacle in the front view image and its tracking result. Finally, based on the tracking result of the obstacle, the center point coordinates of the front view image, and the area ratio of the obstacle in the front view image, determine whether the obstacle is a target obstacle, and determine the obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data, so that the drone performs obstacle avoidance according to the obstacle avoidance path, thereby being able to effectively screen and identify the target obstacles that may cause obstacles to the drone, reducing the missed detection rate and the false detection rate, and further improving the obstacle avoidance accuracy of the drone. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of the drone system according to an embodiment of the present application; Figure 2 is one of the flowcharts of the obstacle avoidance method according to an embodiment of the present application; Figure 3 is a framework diagram of the target recognition model according to an embodiment of the present application; Figure 4 is the second flowchart of the obstacle avoidance method according to an embodiment of the present application; Figure 5 is a schematic structural diagram of the mission aircraft according to an embodiment of the present application; In the figure, 500 is the mission aircraft; 510 is the acquisition module; 520 is the target recognition module; 530 is the target tracking module; 540 is the obstacle avoidance module.

[0019] The realization, functional features, and advantages of the present application will be further described in conjunction with embodiments and with reference to the accompanying drawings. Detailed Embodiments

[0020] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions in the present application will be clearly and completely described below in conjunction with the accompanying drawings in the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0021] For ease of understanding, the drone system of the embodiments of the present application will be introduced in detail below. Figure 1 is a schematic diagram of the drone system of the embodiments of the present application.

[0022] As Figure 1 shown, the drone system may include a mission aircraft, a camera, and a flight control chip. Among them, the mission aircraft, the camera, and the flight control chip use ros1 to achieve data interaction. The camera is used to obtain the forward-looking image of the drone in real time and send the forward-looking image to the mission aircraft; the flight control chip is used to obtain the flight attitude data of the drone and send the flight attitude data to the mission aircraft; the mission aircraft is used to receive the forward-looking image and flight attitude data of the drone in real time, and use a pre-trained target recognition model to perform target detection on each forward-looking image to obtain a target detection result. In the case where the target detection result indicates the presence of an obstacle, a preset target tracking algorithm is used to track the obstacle to obtain an obstacle tracking result. Based on the obstacle tracking result, the center point coordinates of the forward-looking image, and the area ratio of the obstacle in the forward-looking image, it is determined whether the obstacle is a target obstacle. In the case where the obstacle is a target obstacle, an obstacle avoidance path is determined based on the target obstacle tracking result and the flight attitude data, so that the drone avoids obstacles according to the obstacle avoidance path.

[0023] In this embodiment, the drone system framework can be constructed based on three core frameworks: ROS1 (Robot Operating System version 1, robot operating system), TensorRT, and PX4 flight control framework. Among them, the mission aircraft may include an acquisition module, a target recognition module, a target tracking module, and an obstacle avoidance module. The internal communication of the mission aircraft is implemented in the form of ROS1 nodes, thereby realizing forward-looking image acquisition, target recognition, target tracking, and obstacle avoidance. Each module of the mission aircraft adopts the architecture of ROS1 nodes. It should be noted that ROS1 supports three types of communication methods: topics, services, and actions.

[0024] Furthermore, the deep learning part in the mission computer is implemented by the TensorRT framework. During the actual application process, the pre-trained object recognition model can be converted into an Engine file that can be accelerated by the mission computer hardware. The object recognition module then obtains each forward-looking image through the ROS1 node, converts each forward-looking image into a Tensor tensor recognizable by TensorRT, and inputs it into the object recognition model to obtain the object recognition result. After obtaining the object recognition result, the object recognition module then converts the object recognition result into a ROS1 topic for publishing, so that the nodes of each module can call and read it.

[0025] In this embodiment, the camera used can be a finished FPV monocular camera, which provides corresponding interfaces to obtain real-time forward-looking images. The PX4 flight control framework can be deployed on a flight control chip that supports PX4, and then the flight control chip can be deployed in the drone.

[0026] In addition, the communication between the ROS1 node in the mission computer and the PX4 flight control chip can be implemented using the mavros / mavlink protocol. Mavros converts the ROS1 topic into mavlink protocol information, and then transmits the mavlink protocol information to the mavlink service in the flight control chip through a physical connection for parsing by this service. Similarly, the flight control chip can also transmit data to the mavros of the mission computer through the mavlink protocol, and then convert it into a ros1 topic for calling and reading by the nodes of each module.

[0027] Exemplarily, the hardware environment of the mission computer can be: Nvidia Nano 0.5Tops, and the software environment can be: ubuntu20.04. The software environment of the flight control chip can be: Arm system.

[0028] The drone system of this embodiment adopts a modular design concept. After integrating the visual sensor module, radar sensor module, and mission computer through a bracket, they are installed at the corresponding positions of the drone and physically connected to the flight control chip to achieve the effect of perception and obstacle avoidance. It should be noted that the integrated hardware can be installed at the head position of the drone, so as to facilitate obtaining forward-looking images in a timely manner and improve the accuracy of obstacle avoidance.

[0029] Referring to the content in the above-mentioned drone system embodiment, the obstacle avoidance method of the drone in the embodiment of the present application will be introduced in detail below.

[0030] Figure 2 It is one of the flowcharts of the obstacle avoidance method in the embodiment of the present application. This obstacle avoidance method can be executed by the mission computer in the foregoing embodiment, as Figure 2 shown, this obstacle avoidance method may include the following steps: Step 210: Obtain the forward-looking image and flight attitude data of the UAV in real time, and use the pre-trained target recognition model to perform target detection on each forward-looking image to obtain the target detection result.

[0031] First of all, it should be noted that the forward-looking image of the UAV is the image data obtained by the UAV along its motion axis direction (usually consistent with the nose direction) during flight.

[0032] In this embodiment, the mission computer can obtain the forward-looking image from the FPV camera of the UAV system in real time. Specifically, the FPV camera can capture the forward-looking image of the UAV in real time, and at the same time, the mission computer can obtain each forward-looking image through the interface of the FPV camera. After the mission computer obtains each forward-looking image, it can convert each forward-looking image into an image topic recognizable by the ROS1 node and publish it to each module of the mission computer. Considering the frame rate of the FPV camera itself, the frequency of the ROS1 node publishing the image topic should be consistent with the frame rate of the FPV camera.

[0033] In addition, the mission computer can obtain the flight attitude data of the UAV from the flight control chip. It should be noted that the flight attitude data can include GPS longitude and latitude information (longitude, latitude), barometric altitude (relative height from the ground), UAV attitude information (heading angle, pitch angle, roll angle), and UAV motion information (speed, acceleration). The flight control chip in the foregoing embodiment can obtain the above flight attitude data, and then use the ROS1 node, in combination with the mavros protocol and the mavlink protocol, to transmit the flight attitude data to the mission computer. The flight attitude data can be used to analyze the current flight state of the UAV, which is convenient for subsequent determination of the obstacle avoidance strategy.

[0034] After the mission computer obtains the forward-looking image, it will first preprocess the forward-looking image. The preprocessing can include scaling processing, normalization, etc., so as to scale and process the forward-looking image to be able to be input into the pre-trained target recognition model. The size of the processed forward-looking image is 608*608. Input the processed forward-looking image into the trained target recognition model, and the target recognition model can identify the object category in the forward-looking image and obtain the object position (i.e., the target detection result).

[0035] In some embodiments, the target recognition model may adopt the existing YOLOv5 model. The training steps of the target recognition model may include: obtaining a training data set, where the training data set includes multiple sample images with annotation boxes and annotation information, and processing each sample image using Mosaic data augmentation and adaptive image scaling; obtaining target anchor boxes based on the annotation boxes in each sample image and using a preset adaptive anchor box algorithm; inputting the processed sample images into a pre-constructed target recognition model. The target recognition model obtains a target recognition result based on the target anchor boxes, and calculates a loss value using the CIOU loss function based on the target recognition result and the annotation information, and updates the parameters of the target recognition model based on the loss value until the target recognition model converges, obtaining a trained target recognition model.

[0036] It should be noted that the training data set of the present application may include several sample images. The sample images may be any images with objects, and the sample images have annotation information and corresponding annotation boxes for each object.

[0037] The training method of the target recognition model provided in this embodiment is different from the conventional training method of the target recognition model in that: this embodiment of the present application uses the Mosaic data augmentation method to process the training samples, and at the same time applies the adaptive anchor box calculation method and the adaptive image scaling method in the model training process, thereby improving the training speed of the model and the accuracy of the network.

[0038] Specifically, after obtaining the training data set, the Mosaic data augmentation method can be used to perform data augmentation on the training data set. Using the Mosaic data augmentation method to process the training samples specifically includes: splicing 4 sample images in a randomly scaled, randomly cropped, and randomly arranged manner to obtain a new sample image. This augmentation method can combine several images into one, which can not only enrich the training data set, but also greatly improve the training speed of the network, and can reduce the memory requirements of the model.

[0039] Furthermore, the adaptive image scaling method is an improvement on the original image scaling. In the original image scaling algorithm, the sample image is usually directly scaled. If the aspect ratio before and after scaling is inconsistent, the imaging result will be distorted. The adaptive image scaling is to add black edges to the original image before scaling to ensure that the aspect ratio of the sample image before and after scaling is the same, so as to ensure the consistency of the imaging result.

[0040] In this embodiment, the above-mentioned Mosaic data augmentation method and adaptive image scaling method can be used to preprocess the obtained training sample set to obtain a new training sample set, and then use the new training sample set to train the target recognition model.

[0041] During the training process of the target recognition model, target anchor boxes can also be obtained based on the annotation boxes in each sample image and by using a preset adaptive anchor box algorithm, so that the target recognition model can use the adaptive target anchor boxes for recognition and training.

[0042] It should be noted that the anchor boxes described in this embodiment are the same as anchor point boxes. An anchor point box refers to a set of predefined bounding boxes that are generally consistent in size and aspect ratio with the width and height of the target objects in each sample image. These boxes have different aspect ratios and scales to cover as much as possible the width and height of the target objects that may appear in the training dataset. The adaptive anchor boxes are, during the training process, adaptively calculated according to the differences in the input training datasets to obtain the anchor point box results of the most suitable training dataset.

[0043] In some embodiments, obtaining target anchor boxes based on the annotation boxes in each sample image and by using a preset adaptive anchor box algorithm includes: when the recall rate of the initial anchor point boxes of the target recognition model is lower than a preset threshold, randomly initializing N anchor point boxes, where N is a positive integer; using a genetic algorithm, a clustering algorithm, the annotation boxes in each sample image, and each anchor point box to update each anchor point box until each anchor point box meets a preset condition, and then taking each anchor point box as the target anchor box. The preset condition is that the recall rate of each anchor point box is higher than the preset threshold.

[0044] Specifically, first, the target recognition model calculates the best recall rate of the initial anchor point boxes. When the best recall rate is lower than the preset threshold, it is necessary to recalculate the anchor points until the best recall rate is higher than the preset threshold to obtain new anchor point boxes (i.e., target anchor boxes). In this embodiment, the preset threshold can be set by the staff according to actual needs. For example, the preset threshold can be set to 0.98. The best recall rate can be calculated by the following formula:

[0045]

[0046] In the formula, w and h are the width and height of the annotation box in the training dataset respectively; and are the width and height of the current anchor point box respectively; is the number of annotation boxes in the training dataset; is the indicator function; is the threshold (default is 4.0); bpr is the best recall rate; r is an intermediate parameter.

[0047] If the target recognition model calculates that the best recall rate (i.e., the recall rate) of the initial anchor point boxes is lower than the preset threshold, the target recognition model recalculates the anchor points.

[0048] The YOLOv5 model adopted in this embodiment uses the K-means clustering algorithm to calculate the anchor points. Specifically, after randomly initializing N anchor points for the target recognition model, for each labeled bounding box in the training dataset, the distance between it and each anchor point is calculated, so as to cluster each labeled bounding box, and the position of the anchor point is updated according to the clustering result to obtain a new anchor box. The mathematical formula is as follows:

[0049]

[0050] In the formula, is the distance between the labeled bounding box i and the anchor point j; is the number of labeled bounding boxes in the cluster j; is the set of labeled bounding boxes in the cluster j; is the new anchor box; and represent the width and height of the labeled bounding box i, and represent the width and height of the new anchor box of.

[0051] In this embodiment, in order to further optimize the anchor points, the YOLOv5 model can also use the genetic algorithm to optimize the anchor points. The specific steps are as follows: Use the fitness function to evaluate the performance of each randomly generated anchor box, and select and mutate the anchor points according to the fitness. Combining the above formula for the best recall rate and the K-means clustering calculation formula, the fitness function can be deduced as follows:

[0052] Among them, is the fitness of the anchor point k; N is the number of labeled bounding boxes; is the indicator function; is the threshold (default is 4.0).

[0053] Thus, continuously repeat and execute the above genetic algorithm and clustering algorithm until the new anchor box obtained meets the requirement that the best recall rate is higher than the preset threshold, and the target anchor box can be obtained.

[0054] In this embodiment, the benchmark network of the target recognition model (i.e., the YOLOv5 model) is usually the network of some excellent classifiers with excellent performance, and this benchmark network is used to extract some general feature representations. In the YOLOv5 model, not only the CSPDarknet53 structure is used, but also the Focus structure is used as the benchmark network.

[0055] Among them, CSPDarknet53 is mainly composed of components such as convolutional layers (Conv), BN (Batch Normalization) layers, LeakReLu activation layers, cross-stage partial connections (CSP), and spatial pyramid pooling (SPPF). Connecting these components using the Figure 3 structure in

[0056] constitutes the main framework of the model.

[0057] Among them, I is the input feature map with dimensions H*W*C, K is the convolutional kernel with dimensions H*W*C*K, is the output feature map with dimensions H*W*K. c represents the data of each layer in dimension C.

[0058] The mathematical expression of the BN layer is as follows:

[0059] In the formula, is the th sample of the input feature map; is the batch mean; the calculation formula is ; is the batch variance, and the calculation formula is ; and are learnable scaling and offset parameters; is a very small constant used to ensure numerical stability.

[0060] In addition, the formula of the LeakReLu activation layer is .

[0061] The input feature map is divided into two parts: and , where goes through convolutional operations and residual connections, only requires convolutional operations, and the formula is as follows:

[0062] For spatial pyramid pooling (SPPF), the input feature map F undergoes pooling operations at different scales and then is concatenated. Its mathematical expression is as follows:

[0063] In the formula, represents taking the maximum value within the range of size k to fill the entire range; are different pooling kernel sizes, represents the feature map and are connected together.

[0064] The Focus structure is a network structure based on slicing operations. By performing slicing operations on the input image, it can effectively downsample the high-resolution input feature map into a low-resolution but increased-channel feature map. Described in mathematical terms, for the input tensor, the Focus structure slices it into small pieces and concatenates them along the channel dimension to obtain the output tensor. Its mathematical expression can be:

[0065]

[0066] where, and are the row and column indices of the input tensor; and are the row and column indices of the sliced small pieces, ranging from 0 to ; is the channel index of the input tensor; is the channel index of the output tensor, ranging from 0 to . This reorganization method enables the Focus structure to rearrange the spatial information of the input image into the channels, so that in the subsequent convolutional layers, it can process multi-scale information and reduce the amount of computation and the number of parameters.

[0067] Furthermore, the Neck network of the object recognition model (i.e., the YOLOv5 model) is usually located in the middle position between the backbone network and the head network. By using it, the diversity and robustness of the features can be further enhanced. Based on a large number of convolutional operations, the ability of network feature fusion is enhanced.

[0068] The head network is used to output the object recognition results. For different detection algorithms, the number of branches at the output end is not the same, usually including a classification branch and a regression branch. Among them, CIOU is applied as a new loss function in the model, improving the accuracy of the model. Its formula is as follows:

[0069]

[0070] where, Distance_C represents the distance of the diagonal between the predicted result and the true result; Distance_2 represents the distance of the center point between the predicted result and the true result; and represent the width and height of the true result; and represent the width and height of the prediction result; IOU represents the value obtained by dividing the intersection part of the prediction result and the ground truth by the union of the prediction result and the ground truth.

[0071] Thus, the object recognition model trained by the above method is used to recognize obstacles in the front view image, and the robustness of the object recognition model is improved, so that the accuracy of the object recognition result is improved accordingly.

[0072] Step 220: When the object detection result indicates the existence of an obstacle, use a preset object tracking algorithm to track the obstacle to obtain the tracking result of the obstacle.

[0073] It should be noted that since the obstacles appearing in the front view image are not necessarily the obstacles that need to be avoided, it is necessary to track the obstacles to further determine whether the obstacle is an obstacle that needs to be avoided.

[0074] In this embodiment, after the mission aircraft obtains the object recognition result, the object recognition result can be used to track the obstacle to obtain the motion trajectory of the obstacle in the front view images of the front and rear frames, so as to facilitate the subsequent estimation of the relative motion between the UAV and the obstacle, and then determine whether the UAV has a collision risk.

[0075] The preset object tracking algorithm in the embodiment of the present application can be the bytetrack algorithm. The bytetrack algorithm is a tracking method based on the tracking-by-detection paradigm. This paradigm constructs the association relationship between obstacles in the front and rear frames based on the detection results of the front view images of the front and rear frames to generate object IDs. Most multi-object tracking methods obtain object IDs by detecting boxes with association scores higher than a threshold. For objects with low detection scores, such as occluded objects, they will be simply discarded, which brings non-negligible problems, including a large number of missed detections and fragmented trajectories.

[0076] To solve this problem, this embodiment also adopts the BYTE association algorithm to perform tracking by associating each detection box instead of just high-score detection boxes. For low-score detection boxes, their similarity to the trajectory is used to recover real obstacles and filter out background detections.

[0077] Specifically, first, obtain the detection box and the corresponding detection score through the detector, classify the detection box. If the score is higher than T_high, classify the detection box into the high-confidence group; if the score is lower than T_high and higher than T_low, classify the detection box into the low-confidence group. Here, the detector is the object recognition model in the foregoing embodiment, and the detection box score can be directly provided by the object recognition model.

[0078] Furthermore, the matching process uses the similarity between the detection boxes and the results estimated by the Kalman filter. Here, IoU or the distance between Re-ID features can be used as the similarity metric. Then, based on the similarity, the Hungarian algorithm is used for matching, and the high-confidence detection boxes that are not matched to the trajectories and the trajectories that are not matched to the detection boxes are retained.

[0079] Among them, the calculation of the feature distance is as follows:

[0080] In the formula, is the feature distance; is the predicted box of the trajectory; is the detection box; is the intersection over union of the two boxes.

[0081] The trajectory prediction process based on the Kalman filter is as follows. First, assume that the state vector of the trajectory is , and the prediction step is divided into state prediction and covariance prediction. Among them, the state prediction is:

[0082] In the formula, is the state transition matrix, is the predicted state at time k, is the estimated state at time k-1.

[0083] The covariance prediction is:

[0084] In the formula, is the predicted error covariance; is the process noise covariance. The subsequent mathematical calculations then use the Hungarian algorithm for linear assignment to find the optimal match, and its mathematical expression is:

[0085] This match is to match the newly obtained detection box with the detection box obtained through Kalman prediction. The matched data can be used as the observation value and input into the update part of the Kalman, including state update and covariance update, to achieve the effect of correcting the prediction model. The mathematical expression of its state update is as follows:

[0086] Among them, is the detection box observation value, is the observation matrix, is the Kalman gain. And the update of the covariance is: , where I is the identity matrix.

[0087] Further, associate the remaining trajectories from the first association and the low-confidence detection bounding boxes. Then, retain the trajectories that still have not been matched to a bounding box after the second matching, and delete the bounding boxes in the low-confidence bounding boxes that have not found a corresponding trajectory after the second matching, as these bounding boxes are considered to be the background that does not contain any objects.

[0088] Finally, save the high-confidence bounding boxes that have not been matched to a corresponding trajectory as newly emerging trajectories. For the detection bounding boxes that have not been matched in both matches, initialize them as new trajectories.

[0089] The BYTE data association method is different from other methods that only perform data association on high-confidence detection bounding boxes. It almost retains all detection bounding boxes and performs identity matching, and divides them into two groups: high-confidence and low-confidence. First, associate the high-confidence detection bounding boxes into the trajectories, and then associate the low-confidence detection bounding boxes with the unmatched tracking objects to retain the low-confidence detection bounding boxes and filter the background. Thus, the movement trajectories of each obstacle can be obtained more accurately using the bytetrack object tracking algorithm.

[0090] Step 230: Based on the tracking result of the obstacle, the center point coordinates of the forward-looking image, and the area ratio of the obstacle in the forward-looking image, determine whether the obstacle is a target obstacle. And when the obstacle is a target obstacle, determine an obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data, so that the UAV performs obstacle avoidance according to the obstacle avoidance path.

[0091] It should be noted that both the center point coordinates of the forward-looking image and the area ratio of the obstacle in the forward-looking image can be determined by the mission aircraft itself.

[0092] This embodiment provides two methods for determining whether an obstacle is a target obstacle. One is to directly use the tracking result (i.e., the movement trajectory) of the obstacle to determine whether the obstacle is a target obstacle; the other is to combine the area ratio of the obstacle in the forward-looking image to determine whether the obstacle is a target obstacle. These two determination methods can be used simultaneously, so as to further improve the detection accuracy of the target obstacle.

[0093] Next, the target obstacle determination method and the obstacle avoidance strategy of this application embodiment will be introduced in detail.

[0094] In some embodiments, the tracking result of the obstacle includes the coordinate data of the obstacle in each frame of the forward-looking image and the movement trajectory of the obstacle. Figure 4 It is the second flowchart of the obstacle avoidance method of this application embodiment. As Figure 4As shown in the figure, step 230 determines whether the obstacle is a target obstacle based on the tracking result of the obstacle, the center point coordinates of the forward-looking image, and the area ratio of the obstacle in the forward-looking image. It may include the following steps: Step 410: Based on the coordinate data of the obstacle in each frame of the forward-looking image and the center point coordinates of the forward-looking image, determine whether the obstacle moves towards the center point of the forward-looking image.

[0095] Step 420: If it is determined that the obstacle moves towards the center point of the forward-looking image or the area ratio of the obstacle in the forward-looking image is greater than the preset ratio, then regard the obstacle as a target obstacle.

[0096] It should be noted that the coordinate data of the obstacle in each frame of the forward-looking image can be the coordinate data of the center point of the obstacle.

[0097] It can be understood that during flight, the relative motion between the obstacle and the aircraft needs to be considered. Considering that the FPV camera is installed at the nose of the UAV and provides the forward-looking view of the UAV, therefore, if the obstacle may collide with the UAV, its motion trend in the image must tend to the center of the image, and its size in the image must also meet certain requirements. At the same time, considering some obstacles that may suddenly appear from the outside and the situation of tracking loss, for some independent targets without a motion trajectory that reach a certain ratio in the image, they should also be determined as obstacles that need to be avoided.

[0098] Based on this, in the embodiment of the present application, the relationship between the center point of the obstacle and the center point of the forward-looking image is used to determine whether the obstacle moves towards the center point of the forward-looking image. If so, it means that the obstacle is getting closer to the UAV and there is a high probability of a collision event with the UAV; if not, it means that the obstacle is not getting closer to the UAV and may not have a collision event with the UAV.

[0099] At the same time, it can also be judged by the area ratio of the obstacle in the forward-looking image. Specifically, if the area ratio of the obstacle is greater than the preset ratio, it means that the obstacle is relatively close to the UAV, so there is a high probability of a collision event with the UAV. If the area ratio of the obstacle is less than or equal to the preset ratio, it means that the obstacle is relatively far from the UAV, so it may not have a collision event with the UAV. It should be noted that the preset ratio can be set by the staff according to actual needs. For example, the preset ratio can be 50%.

[0100] In some embodiments, based on the coordinate data of the obstacle in each frame of the front view image and the center point coordinates of the front view image, determining whether the obstacle moves towards the center point of the front view image includes: based on the coordinate data of the obstacle in each frame of the front view image, determining a plurality of displacement amounts of the obstacle, where the displacement amount is the displacement amount of the obstacle between the front view image of the current frame and the front view image of the next frame; determining the moving speed of the obstacle based on each displacement amount; using a preset formula and based on the moving speed of the obstacle and the center point coordinates of the front view image, determining whether the obstacle moves towards the center point of the front view image.

[0101] After obtaining the coordinate data of the obstacle in each frame of the front view image through the above target tracking process in this embodiment, the displacement amount of the center point of the obstacle from each frame to the next frame can be calculated first. Specifically, the displacement amount of the center point of the obstacle can be calculated through the following formula:

[0102] In the formula, is the displacement amount of the obstacle in the front view image of frame t; (x t , y t ) are the coordinate data of the obstacle in the front view image of frame t; (x t+1 , y t+1 ) are the coordinate data of the obstacle in the front view image of frame t + 1; N is the number of front view images.

[0103] Further, after obtaining the displacement amount of the obstacle in each frame of the front view image, the average displacement amount within a certain number of frames T (the number of frames T can be set according to requirements) can be calculated. Since the number of frames of the FPV camera is stable, this average displacement amount can be approximated as the moving speed of the obstacle within T frames.

[0104] In some embodiments, the moving speed of the obstacle is calculated by the following formula:

[0105] In the formula, is the moving speed of the obstacle; are the coordinates of the obstacle in the front view image of frame t; are the coordinates of the obstacle in the front view image of frame t + 1; T is the preset total number of frames of the front view image.

[0106] Further, to determine whether the obstacle moves towards the center point of the front view image through a preset formula, in some embodiments, the preset formula is:

[0107] [[ID=4"3]]In the formula, is the angle between the moving speed and the center point vector; is the moving speed of the obstacle; is the center point vector; is the center point coordinate of the front view image; a and c are the speeds in the x and y directions respectively.

[0108] If , it gets closer and closer to the center; if , it gets farther and farther away from the center. At this time, according to the initial position of the trajectory, it can also be judged whether the obstacle is moving from left to right, from right to left, from top to bottom or from bottom to top.

[0109] It is worth mentioning that considering the overall flight logic of the UAV, it is basically divided into takeoff, landing and cruising. Considering that the takeoff stage usually occurs in a relatively open scene, the main collisions occur in the cruising and landing stages. Combining with the type of obstacles, it can be inferred that in the landing stage, all obstacles need to be detected. For the cruising stage, considering that the UAV usually flies in the airspace above 50m during cruising, there will be no obstacles such as people, vehicles and utility poles, so the results of these target identifications can be blocked to reduce the amount of calculation and false alarms.

[0110] In this embodiment, since most obstacles are close to the ground during the landing process, considering that the UAV itself has a high climbing rate and more timely response, therefore, in this stage, if it is detected that the obstacle has a tendency to move towards the center of the image or the obstacle with a large area occupancy ratio, a lift signal should be directly sent to the flight control chip, and the flight control chip will execute the drive control.

[0111] During the UAV cruising stage, the mission aircraft adopts an intelligent obstacle avoidance strategy based on motion trend analysis. Since the cruising airspace is usually relatively open, the core of the obstacle avoidance logic lies in analyzing the dynamic characteristics of the obstacles in the front view image. When it is detected that the obstacle shows a motion trend of approaching the center of the image, the mission aircraft will judge that there is a potential collision risk and trigger an active avoidance mechanism. Its obstacle avoidance logic is: the reverse avoidance principle - if the obstacle moves from left to right, the UAV is controlled to yaw to the left; otherwise, it yaws to the right. For stationary obstacles, the mission aircraft implements a mirror avoidance strategy according to the spatial position of the obstacles in the front view image: when the obstacle appears on the left side of the screen, it executes a rightward avoidance, and when it appears on the right side, it executes a leftward avoidance.

[0112] This two-way hierarchical obstacle avoidance strategy not only considers the motion inertia of dynamic obstacles but also takes into account the spatial relationship of static obstacles. It maps the position change in the image coordinate system to the heading adjustment amount of the UAV through kinematic inverse solution, maximizing the coherence of the original flight path while ensuring the effectiveness of obstacle avoidance. Experimental data show that this strategy can enable the UAV to achieve a successful obstacle avoidance rate of more than 92% within a recognition distance of 30 meters, and the average heading correction amplitude only needs 8 - 12 degrees to complete safe avoidance.

[0113] Thus, by integrating the improved ByteTrack algorithm, the tracking continuity of occluded targets is enhanced using low-confidence detection boxes, and the motion trend of obstacles is predicted through Kalman filtering, improving the accuracy of target tracking. By adopting a hierarchical obstacle avoidance decision-making method, the obstacle avoidance strategy (such as reverse avoidance or emergency climb) is adaptively selected according to the flight phase (cruise / landing) and obstacle type (static / dynamic), and the response time is reduced to 86 ms. Based on the TensorRT-accelerated YOLOv5 model and the ROS-MAVLink communication architecture, end-to-end low-latency processing is achieved. In addition, through the collaborative optimization of detection-tracking-decision-making with temporal awareness in this embodiment, the obstacle avoidance robustness of the UAV in complex dynamic environments is significantly improved. Experiments show that it can still maintain an obstacle avoidance success rate of over 95% under conditions of a wind speed of 8 m / s and changing light, solving the core problems of poor real-time performance, missed detection of dynamic targets, and rigid strategies in the prior art.

[0114] Based on the above embodiments, an embodiment of the present application further provides a mission aircraft of a UAV. Figure 5 It is a schematic structural diagram of the mission aircraft of the embodiment of the present application. As Figure 5 shown, the mission aircraft 500 may include an acquisition module 510, a target recognition module 520, a target tracking module 530, and an obstacle avoidance module 540.

[0115] Among them, the acquisition module 510 is used to obtain the forward-looking image and flight attitude data of the UAV in real time. The target recognition module 520 is used to perform target detection on each forward-looking image using a pre-trained target recognition model to obtain a target detection result. The target tracking module 530 is used to track the obstacle using a preset target tracking algorithm when the target detection result indicates the presence of an obstacle, to obtain a tracking result of the obstacle. The obstacle avoidance module 540 is used to determine whether the obstacle is a target obstacle based on the tracking result of the obstacle, the center point coordinates of the forward-looking image, and the area ratio of the obstacle in the forward-looking image, and to determine an obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data when the obstacle is a target obstacle, so that the UAV avoids obstacles according to the obstacle avoidance path.

[0116] In some embodiments, the tracking result of the obstacle includes the coordinate data of the obstacle in each frame of the forward-looking image; the obstacle avoidance module 540 is specifically configured to: determine whether the obstacle moves towards the center point of the forward-looking image based on the coordinate data of the obstacle in each frame of the forward-looking image and the center point coordinates of the forward-looking image; if it is determined that the obstacle moves towards the center point of the forward-looking image or the area ratio of the obstacle in the forward-looking image is greater than a preset threshold, the obstacle is regarded as a target obstacle.

[0117] In some embodiments, the obstacle avoidance module 540 is further specifically configured to: based on the coordinate data of the obstacle in each frame of the front view image and the center point coordinates of the front view image, determine whether the obstacle is moving towards the center point of the front view image, including: based on the coordinate data of the obstacle in each frame of the front view image, determine a plurality of displacement amounts of the obstacle, where the displacement amount is the displacement amount of the obstacle between the current frame of the front view image and the next frame of the front view image; determine the moving speed of the obstacle based on each displacement amount; use a preset formula and based on the moving speed of the obstacle and the center point coordinates of the front view image, determine whether the obstacle is moving towards the center point of the front view image.

[0118] In some embodiments, the preset formula is:

[0119] In the formula, is the included angle between the moving speed vector and the center point vector; is the vector of the moving speed of the obstacle; is the center point vector; is the center point coordinates of the front view image; a and c are the speeds in the x and y directions respectively.

[0120] In some embodiments, the moving speed of the obstacle is calculated by the following formula:

[0121] In the formula, is the moving speed of the obstacle; are the coordinates of the obstacle in the t-th frame of the front view image; are the coordinates of the obstacle in the (t + 1)-th frame of the front view image; T is the preset total number of frames of the front view image.

[0122] It should be noted that for the details not disclosed in the mission machine 500 of this embodiment, please refer to the details disclosed in the embodiment of the obstacle avoidance method of the unmanned aerial vehicle in the embodiments of this specification, and will not be elaborated here.

[0123] Based on the above embodiments, the present application further provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the obstacle avoidance method for an unmanned aerial vehicle provided by the above various methods. The method includes: obtaining the forward-looking image and flight attitude data of the unmanned aerial vehicle in real time, and using a pre-trained target recognition model to perform target detection on each forward-looking image to obtain a target detection result; in the case where the target detection result indicates the presence of an obstacle, using a preset target tracking algorithm to track the obstacle to obtain a tracking result of the obstacle; based on the tracking result of the obstacle, the center point coordinates of the forward-looking image, and the area ratio of the obstacle in the forward-looking image, determining whether the obstacle is a target obstacle, and in the case where the obstacle is a target obstacle, determining an obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data, so that the unmanned aerial vehicle performs obstacle avoidance according to the obstacle avoidance path.

[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0125] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Claims

1. An obstacle avoidance method for an unmanned aerial vehicle, characterized in that, Including: Obtain the forward-looking image and flight attitude data of the drone in real time, and use the pre-trained target recognition model to perform target detection on each of the forward-looking images to obtain the target detection result; In the case that the target detection result indicates the presence of an obstacle, use a preset target tracking algorithm to track the obstacle to obtain the tracking result of the obstacle; Based on the tracking result of the obstacle, the center point coordinates of the forward-looking image, and the area ratio of the obstacle in the forward-looking image, determine whether the obstacle is a target obstacle, and in the case that the obstacle is the target obstacle, determine an obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data, so that the drone performs obstacle avoidance according to the obstacle avoidance path.

2. The obstacle avoidance method of the drone according to claim 1, wherein, The tracking result of the obstacle includes the coordinate data of the obstacle in each frame of the forward-looking image; The determining whether the obstacle is a target obstacle based on the tracking result of the obstacle, the center point coordinates of the forward-looking image, and the area ratio of the obstacle in the forward-looking image includes: Based on the coordinate data of the obstacle in each frame of the forward-looking image and the center point coordinates of the forward-looking image, determine whether the obstacle moves towards the center point of the forward-looking image; If it is determined that the obstacle moves towards the center point of the forward-looking image or the area ratio of the obstacle in the forward-looking image is greater than a preset ratio, then regard the obstacle as the target obstacle.

3. The obstacle avoidance method of the drone according to claim 2, wherein The determining whether the obstacle moves towards the center point of the forward-looking image based on the coordinate data of the obstacle in each frame of the forward-looking image and the center point coordinates of the forward-looking image includes: Based on the coordinate data of the obstacle in each frame of the forward-looking image, determine a plurality of displacement amounts of the obstacle, where the displacement amount is the displacement amount of the obstacle between the current frame of the forward-looking image and the next frame of the forward-looking image; Determine the moving speed of the obstacle based on each of the displacement amounts; Use a preset formula and based on the moving speed of the obstacle and the center point coordinates of the forward-looking image, determine whether the obstacle moves towards the center point of the forward-looking image.

4. The obstacle avoidance method for an unmanned aerial vehicle according to claim 3, characterized in that The preset formula is: Wherein, is the angle between the moving speed and the center point vector; is the moving speed of the obstacle; is the center point vector; is the center point coordinate of the forward-looking image; a and c are the speeds in the x and y directions respectively.

5. The obstacle avoidance method of the drone according to claim 3, wherein The moving speed of the obstacle is calculated by the following formula: In the formula, is the moving speed of the obstacle; is the coordinate of the obstacle in the forward-looking image at frame t; is the coordinate of the obstacle in the forward-looking image at frame t + 1; T is the preset total number of frames of the forward-looking image.

6. The obstacle avoidance method of the unmanned aerial vehicle according to any one of claims 1 to 5, characterized in that, The training steps of the target recognition model include: Obtain a training data set, the training data set includes a plurality of sample images with annotation frames and annotation information, and perform processing on each of the sample images using Mosaic data augmentation and adaptive image scaling; Based on the annotation frames in each of the sample images and using a preset adaptive anchor box algorithm, obtain target anchor boxes; Input each of the processed sample images into a pre-constructed target recognition model, the target recognition model obtains a target recognition result based on the target anchor boxes, calculates a loss value using the CIOU loss function based on the target recognition result and the annotation information, and updates the parameters of the target recognition model based on the loss value until the target recognition model converges to obtain the trained target recognition model.

7. The obstacle avoidance method of the drone according to claim 6, characterized in that, Obtaining a target anchor box based on the annotation boxes in each of the sample images and using a preset adaptive anchor box algorithm includes: When the recall rate of the initial anchor box of the target recognition model is lower than a preset threshold, randomly initialize N anchor boxes, where N is a positive integer; Use a genetic algorithm, a clustering algorithm, each of the annotation boxes in each of the sample images, and each of the anchor boxes to update each of the anchor boxes until each of the anchor boxes meets a preset condition, and then use each of the anchor boxes as the target anchor box. The preset condition is that the recall rate of each of the anchor boxes is higher than the preset threshold.

8. A mission aircraft of an unmanned aerial vehicle, characterized in that, It includes: An acquisition module, configured to acquire the front view image and flight attitude data of the drone in real time; A target recognition module, configured to perform target detection on each of the front view images using a pre-trained target recognition model to obtain a target detection result; A target tracking module, configured to, when the target detection result indicates the presence of an obstacle, use a preset target tracking algorithm to track the obstacle to obtain a tracking result of the obstacle; An obstacle avoidance module, configured to determine whether the obstacle is a target obstacle based on the tracking result of the obstacle, the center point coordinates of the front view image, and the area ratio of the obstacle in the front view image, and when the obstacle is the target obstacle, determine an obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data, so that the drone performs obstacle avoidance according to the obstacle avoidance path.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the obstacle avoidance method of the drone according to any one of claims 1-7.

10. A drone system, characterized in that, It includes the mission machine, the camera, and the flight control chip according to claim 8, and the mission machine, the camera, and the flight control chip use ros1 to implement data interaction; The camera is configured to acquire the front view image of the drone in real time and send the front view image to the mission machine; The flight control chip is configured to acquire the flight attitude data of the drone and send the flight attitude data to the mission machine; The mission machine is configured to receive the front view image and flight attitude data of the drone in real time, perform target detection on each of the front view images using a pre-trained target recognition model to obtain a target detection result, when the target detection result indicates the presence of an obstacle, use a preset target tracking algorithm to track the obstacle to obtain a tracking result of the obstacle, determine whether the obstacle is a target obstacle based on the tracking result of the obstacle, the center point coordinates of the front view image, and the area ratio of the obstacle in the front view image, and when the obstacle is the target obstacle, determine an obstacle avoidance path based on the tracking result of the target obstacle and the flight attitude data, so that the drone performs obstacle avoidance according to the obstacle avoidance path.

Images