Obstacle avoidance control method and device of unmanned aerial vehicle and storage medium

Abstract

This invention provides an obstacle avoidance control method, device, and storage medium for unmanned aerial vehicles (UAVs), relating to the field of UAV obstacle avoidance technology. The method includes: inputting the current frame image from a video stream into an obstacle detection model to obtain obstacle detection results; calculating the area expansion rate and obstacle hazard level corresponding to each obstacle based on the obstacle detection results; determining the obstacle vector and obstacle hazard level pointing from the dangerous obstacle to the two-dimensional optical imaging system, calculating and weighting the repulsion vector corresponding to each dangerous obstacle to obtain the overall repulsion vector; weighting and summing the avoidance directions corresponding to each dangerous obstacle using the obstacle hazard level to obtain the optimal avoidance direction; calculating the component of the optimal avoidance direction perpendicular to the overall repulsion vector as the true avoidance direction, and controlling the UAV to perform obstacle avoidance. This method can solve the problems of complex UAV systems and high operation and maintenance costs, reducing the complexity and maintenance costs of UAV systems.

Description

Technical Field

[0001] This invention relates to the field of obstacle avoidance technology for unmanned aerial vehicles (UAVs), and more particularly to an obstacle avoidance control method, device, and storage medium for UAVs. Background Technology

[0002] Unmanned Aerial Vehicles (UAVs) are widely used in modern society, particularly in logistics, agricultural monitoring, environmental monitoring, and infrastructure inspection. Obstacle avoidance technology is crucial for ensuring efficient and safe flight. Especially in complex three-dimensional environments, UAV flight paths can be affected by various obstacles, including static obstacles such as buildings, trees, and power lines, as well as dynamic obstacles such as birds, other aircraft, and vehicles. If an UAV fails to avoid these obstacles in time, collisions may occur, leading to equipment damage or even personal injury. Therefore, developing an efficient, real-time dynamic obstacle avoidance algorithm is essential for ensuring the safe flight of UAVs.

[0003] Currently, traditional obstacle avoidance control methods for unmanned aerial vehicles (UAVs) include: equipping the UAV with various sensors, such as LiDAR, depth cameras, and event cameras; acquiring real-time data such as environmental depth and the position of moving objects through the sensors; and dynamically calculating the safe path for the UAV by analyzing the data collected by the sensors and obstacle avoidance algorithms.

[0004] However, although obstacle avoidance technology based on lidar and depth cameras can provide real-time environmental perception, these sensors are costly, expensive, and bulky. At the same time, the deployment and maintenance requirements of high-precision sensors are also high. Under adverse conditions such as strong light, rain, snow, or fog, the accuracy of the sensors will decrease, leading to the complexity of the unmanned aerial vehicle system and high operation and maintenance costs. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an obstacle avoidance control method, apparatus, and storage medium for unmanned aerial vehicles (UAVs) to eliminate or improve one or more defects existing in the prior art. This can solve the problems of complex UAV systems and high operation and maintenance costs.

[0006] One aspect of the present invention provides an obstacle avoidance control method for an unmanned aerial vehicle (UAV); the UAV is equipped with a two-dimensional optical imaging system for continuously capturing video streams; the method includes the following steps:

[0007] The current frame image from the video stream is input into a pre-trained obstacle detection model, which outputs obstacle detection results. The obstacle detection results include the obstacle category of each obstacle in the current frame image and the corresponding obstacle detection box. The obstacle detection box is used to indicate the current pixel area and current pixel coordinates of the obstacle in the current frame image.

[0008] The obstacle detection results are compared with the corresponding detection results in the historical frame images to obtain the comparison results; the comparison results are used to indicate whether each obstacle is the same as the obstacle in the historical frame images.

[0009] If the comparison results indicate that each obstacle is the same as the obstacle in the historical frame image, the area expansion rate of each obstacle is calculated based on the current pixel area and the historical pixel area of ​​each obstacle in the historical frame image.

[0010] Multiplying the area dilation rate by the dilation rate coefficient yields the obstacle danger level for each obstacle; the dilation rate coefficient is obtained by taking the square root or logarithm of the current pixel area.

[0011] The obstacle hazard level of each obstacle is compared with the preset hazard threshold, and obstacles with an obstacle hazard level greater than the preset hazard threshold are identified as dangerous obstacles.

[0012] Based on the pixel width and pixel height of the current frame image, the actual width and actual height of the current frame image, the distance from the optical center of the two-dimensional optical imaging system to the center of the virtual imaging plane, and the current pixel coordinates, determine the obstacle vector of each dangerous obstacle pointing to the two-dimensional optical imaging system.

[0013] Based on the obstacle vector and obstacle hazard level corresponding to each obstacle, the repulsion vector corresponding to each obstacle is calculated, and the repulsion vector is obtained by weighted summation of the repulsion vectors corresponding to each obstacle hazard level.

[0014] For each obstacle avoidance direction, the vectors are weighted and summed using the corresponding obstacle danger levels. The direction pointed to by the summed vector is taken as the optimal avoidance direction. The avoidance direction is obtained based on a preset path planning algorithm.

[0015] The component of the optimal avoidance direction perpendicular to the overall repulsive force vector is calculated as the true avoidance direction, and the unmanned aerial vehicle is controlled to avoid obstacles according to the true avoidance direction.

[0016] In some embodiments of the present invention, before calculating the area dilation rate corresponding to each obstacle based on the current pixel area of ​​each obstacle in the current frame image and the historical pixel area of ​​each obstacle in the historical frame image, the method further includes:

[0017] By using a Kalman filter and the historical pixel coordinates and areas of each obstacle in historical frame images, the current pixel area and current pixel coordinates of each obstacle are smoothed.

[0018] In some embodiments of the present invention, when the comparison result indicates that the obstacle in the current frame image is different from the obstacle in the historical frame image, and there is a missing obstacle, the method further includes:

[0019] Based on the preset hazard decay rate and the historical hazard levels of lost obstacles, the predicted obstacle hazard level is obtained;

[0020] If the predicted obstacle danger level is greater than the preset danger level threshold, the missing obstacle is predicted by a Kalman filter to obtain the predicted pixel coordinates of the missing obstacle; the predicted pixel coordinates are used to calculate the dangerous obstacle vector corresponding to the missing obstacle.

[0021] The missing obstacles are identified as dangerous obstacles, and the steps of determining the dangerous obstacle vector pointing to the two-dimensional optical imaging system for each dangerous obstacle are performed.

[0022] In some embodiments of the present invention, when the video frame processing speed of the obstacle detection model is less than the frame rate of the two-dimensional optical imaging system, the video stream is converted into video frames and stored in a preset buffer.

[0023] Input the current frame image from the video stream into a pre-trained obstacle detection model, including:

[0024] The latest video frame is retrieved from the preset cache according to the preset calculation window and used as the current frame image, which is then input into the obstacle detection model.

[0025] Delete the video frames preceding the current frame image in the preset cache.

[0026] In some embodiments of the present invention, the training process of the obstacle detection model includes:

[0027] The current frame image of the sample is input into the teacher network and student network in the initial obstacle detection model, respectively, to obtain the output results of the teacher network and the student network.

[0028] By simplifying the optimal transport allocation method, the teacher network output results and the ground value labels corresponding to the current frame image of the sample are matched and divided to obtain teacher positive samples and teacher negative samples.

[0029] By simplifying the optimal transport allocation method, the teacher positive samples, teacher negative samples, and student output results are matched and divided to obtain the student positive samples.

[0030] The first distillation loss between positive student samples and positive teacher samples, the second distillation loss between positive student samples and negative teacher samples, and the ground truth loss between student output results and true labels are calculated using a preset loss function. The preset loss function includes a category loss function, a regression loss function, or a foreground-background loss function. The ratio between the ground truth loss, the first distillation loss, and the second distillation loss is set to 20:2:1.

[0031] The first distillation loss is weighted by the first weight corresponding to the positive teacher samples, and the second distillation loss is weighted by the second weight corresponding to the negative teacher samples.

[0032] The student network is iteratively trained using truth loss, first distillation loss, and second distillation loss until the student network converges to obtain the obstacle detection model.

[0033] In some embodiments of the present invention, the detection results corresponding to the historical frame images are obtained by querying the historical obstacle information table and the obstacle information queue corresponding to each historical obstacle; the historical obstacle information table includes an area expansion rate field, an obstacle hazard field, a movement direction field, and an update timestamp field; the obstacle information queue includes a historical obstacle size queue, a frame interval queue, and a historical obstacle position queue;

[0034] After multiplying the area expansion rate by the expansion rate coefficient to obtain the obstacle hazard level for each obstacle, the following steps are also included:

[0035] Add the current pixel area corresponding to each obstacle to the historical obstacle size queue for each obstacle;

[0036] Update the timestamp of the current frame image in the frame interval queue after image processing;

[0037] Add the current pixel coordinates of each obstacle to the historical obstacle position queue corresponding to each obstacle;

[0038] The historical obstacle information table is updated based on the area expansion rate, obstacle hazard level, movement direction, and update timestamp corresponding to this update for each obstacle. The movement direction is calculated based on the pixel coordinates of each obstacle in the first and last frames of the current frame image.

[0039] In some embodiments of the present invention, before inputting the current frame image from the video stream into a pre-trained obstacle detection model, the method further includes:

[0040] Based on the current time and the update timestamps of each obstacle in the historical obstacle information queue, determine the update time difference of each obstacle in the historical obstacle information queue;

[0041] Remove obstacles whose update time difference is greater than a preset time from the historical obstacle queue.

[0042] In some embodiments of the present invention, after the obstacle detection model outputs the obstacle detection result, it further includes:

[0043] Compare the category confidence score with a preset confidence threshold, and delete obstacle categories whose category confidence score is lower than the preset confidence threshold.

[0044] Another aspect of the present invention provides an obstacle avoidance control device for an unmanned aerial vehicle, including a two-dimensional optical imaging system for continuously acquiring video streams; a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the device implements the steps of the obstacle avoidance control method for the unmanned aerial vehicle described above.

[0045] In another aspect, the present invention provides a computer-readable storage medium storing a program that, when executed by a processor, is used to implement the steps of the obstacle avoidance control method for the unmanned aerial vehicle described above.

[0046] The obstacle avoidance control method and apparatus for unmanned aerial vehicles (UAVs) of the present invention can solve the problems of complex UAV systems and high operation and maintenance costs. By detecting obstacles in the video stream acquired by a two-dimensional optical imaging system, obstacle information is obtained and updated. Then, based on the relative approach speed and relative position between the obstacle and the UAV, the degree of danger posed by each obstacle to the UAV is evaluated and assigned a corresponding weight. Subsequently, using this weight and the relative position of the obstacle, the current repulsion vector of the obstacle is calculated. Finally, combined with the environmental safety information provided by the path planning system and the specific position of the obstacle, the most suitable avoidance direction and speed are determined in a plane perpendicular to the repulsion vector, thereby guiding the UAV to perform effective avoidance actions. Dynamic obstacle avoidance of the UAV can be controlled without dynamic environment modeling and real-time depth information. Therefore, it is possible to avoid equipping the UAV with sensors to obtain depth information, thereby reducing the complexity and maintenance cost of the UAV system.

[0047] In addition, to address the issue that neural networks require significant computational resources, this invention optimizes the obstacle detection model using knowledge distillation to reduce the algorithm's computational resource requirements and improve the obstacle avoidance efficiency of unmanned aerial vehicles.

[0048] Furthermore, obstacle collision time is used instead of actual depth information at a given moment to model the obstacle's 3D space as indirect depth information. Simultaneously, the obstacle's expansion rate at a certain moment is used instead of its collision time. A Kalman filter is then used to track the dynamic obstacle trajectory in the 3D space. The Kalman filter's ability to predict obstacle trajectories, combined with other methods, enables obstacle avoidance decisions. This allows for early obstacle trajectory prediction and avoidance decisions even when the obstacle detection model misses an obstacle or in the initial stages of obstacle detection, avoiding redundant obstacle avoidance actions and making the UAV's dynamic obstacle avoidance function more efficient and stable. Furthermore, combined with a reactive obstacle avoidance strategy, when the error between the predicted and observed trajectories is less than a preset threshold and the UAV has moved beyond the initial obstacle detection phase and entered a stable phase, obstacle avoidance actions are planned based on the motion trajectory predicted by the Kalman filter. When the error between the predicted and observed trajectories exceeds a preset threshold, a reactive obstacle avoidance method is employed until the error falls below the preset threshold again, improving the success rate of obstacle avoidance.

[0049] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0050] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0051] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0052] Figure 1 A flowchart illustrating an obstacle avoidance control method for an unmanned aerial vehicle provided in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram illustrating the training process of an obstacle detection model provided in an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of three-dimensional space modeling provided in an embodiment of the present invention;

[0055] Figure 4 This is a diagram showing the positional relationship of obstacles in a video frame image according to an embodiment of the present invention;

[0056] Figure 5A flowchart of an obstacle avoidance control method for an unmanned aerial vehicle provided in another embodiment of the present invention;

[0057] Figure 6 A flowchart of an obstacle avoidance control method for an unmanned aerial vehicle provided in another embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0059] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0060] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0061] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0062] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0063] The obstacle avoidance control method for the unmanned aerial vehicle provided in this application will be described in detail below.

[0064] like Figure 1 As shown, embodiments of this application provide an obstacle avoidance control method for an unmanned aerial vehicle (UAV). The UAV is equipped with a two-dimensional optical imaging system, such as a monocular camera or an infrared camera, for continuously capturing video streams. The method includes at least the following steps:

[0065] Step S101: Input the current frame image in the video stream into the pre-trained obstacle detection model, and have the obstacle detection model output the obstacle detection result.

[0066] In some embodiments of the present invention, a real-time video stream is acquired through a two-dimensional optical imaging system equipped on an unmanned aerial vehicle. A frame is extracted from the acquired video stream as the current frame image and input into an obstacle detection model to obtain the obstacle detection result. Here, the current frame image refers to the latest series of consecutive video frames in the real-time acquired video stream, and the number of frames is determined by a pre-set calculation window.

[0067] In practical implementation, due to hardware limitations, the processing speed of the obstacle detection model may be lower than the frame rate of the 2D optical imaging system. In this case, it may be impossible to perform obstacle detection on every frame of the video stream. Therefore, the real-time acquired video stream can be converted into video frames and cached in a preset buffer on the UAV. After the obstacle detection model completes obstacle detection on the previous frame, it retrieves the latest video frame from the preset buffer as the current frame image and inputs it into the obstacle detection model for obstacle detection. Undetected video frames between the current and previous frames are discarded from the preset buffer.

[0068] Specifically, when the video frame processing speed of the obstacle detection model is lower than the frame rate of the two-dimensional optical imaging system, the video stream is converted into video frames and stored in a preset cache; correspondingly, the current frame image in the video stream is input into the pre-trained obstacle detection model, including: obtaining the latest video frame from the preset cache according to a preset calculation window as the current frame image and inputting it into the obstacle detection model; and deleting the video frames before the current frame image in the preset cache.

[0069] The main factor affecting the performance of an unmanned aerial vehicle (UAV) in avoiding dynamic obstacles is the time-to-collision (TTC) time when the system detects an obstacle. A longer TTC makes it easier for the UAV to avoid obstacles. Conversely, a longer TTC results in a smaller obstacle area on the current frame. Therefore, improving the obstacle detection model's accuracy for small targets can, to some extent, improve the UAV's performance in avoiding dynamic obstacles.

[0070] Based on this, in some embodiments of the present invention, a real-time obstacle detection model is obtained by combining deep learning and knowledge distillation training. The YOLO (You Only Look Once)-Faetest network is improved using an anchor-free approach to increase inference speed; simultaneously, knowledge distillation is used to distill the improved model, which can improve detection accuracy without reducing the model's real-time performance and reduce the algorithm's computational resource requirements.

[0071] Specifically, the training process of the obstacle detection model includes at least steps S1010 to S1015:

[0072] Step S1010: Input the current frame image of the sample into the teacher network and student network in the initial obstacle detection model respectively to obtain the output results of the teacher network and the student network.

[0073] refer to Figure 2 In some embodiments of the present invention, the initial obstacle detection model includes a teacher network and a student network. The model structure of the teacher network is the same as that of the student network, including a backbone network and a detection head. The main task of the backbone network is to extract feature maps from the input sample current frame image. Typically, the backbone network uses a convolutional neural network (CNN) structure, extracting high-level features layer by layer from the original image through a series of convolutional layers, pooling layers, etc. These feature maps contain various information about the image, such as the shape, color, texture, and edges of objects. The task of the detection head is to process the feature maps extracted by the backbone network, and finally generate prediction information related to each object as the detection result output, including: obstacle class labels, obstacle box regression, and target confidence.

[0074] The obstacle category indicates the type of obstacle detected. Since unmanned aerial vehicles (UAVs) generally do not fly at very low altitudes, common obstacle categories encountered during their flight include birds, other UAVs, or helicopters.

[0075] The obstacle detection bounding box indicates the location and size of the obstacle (usually the pixel coordinates of the top-left and bottom-right corners, or the center coordinates and width and height). The target confidence score represents the level of confidence the obstacle detection model has in the presence of the obstacle.

[0076] Step S1011: The teacher network output results and the ground truth labels corresponding to the current frame image of the sample are matched and divided by a simplified optimal transport allocation method to obtain teacher positive samples and teacher negative samples.

[0077] After inputting the current frame image of the sample into the teacher network to obtain the teacher network's output, the output is first coarsely filtered based on the target confidence level, discarding samples with a target confidence level below a preset confidence threshold. By filtering out low-confidence detection results, computational load is reduced, training efficiency is improved, false positives are reduced, and label noise is decreased, making subsequent processing more efficient and accurate. This approach not only improves the overall performance of the system but also helps to enhance the learning quality and consistency between the teacher and student networks.

[0078] After coarsely filtering the teacher network output results, the remaining teacher network output results and the ground truth labels corresponding to the current frame image of the sample are divided into teacher positive samples and teacher negative samples by the Similarity-based Optimal Transport Assignment (SimOTA) method.

[0079] In this context, the actual significance of positive teacher samples is the part that is similar to the true value. The loss generated by each sample in the positive teacher samples in subsequent distillation will be weighted by the intersection over union (IoU) ratio of its corresponding true value label.

[0080] Teacher negative samples include two scenarios: the first is low-quality output caused by false positives in the teacher network; the second refers to high-quality output that is not labeled in the current frame image. Given the dataset size of the current frame image, the quality of teacher negative samples is generally positively correlated with their target confidence. Therefore, the loss generated by each sample in the subsequent distillation process will be weighted by its target confidence. The loss generated by the teacher positive samples, teacher negative samples, and the student network output is thus calculated as follows: Figure 2 Distillation loss.

[0081] Step S1012: The teacher positive samples, teacher negative samples and student output results are matched and divided by a simplified optimal transport allocation method to obtain the student positive samples.

[0082] In some embodiments of the present invention, by simplifying the optimal transport allocation method, the teacher's positive samples, teacher's negative samples and student output results are matched and divided to obtain student positive samples and student negative samples under the teacher's guidance. The student negative samples are discarded and the student positive samples are retained.

[0083] Step S1013: Calculate the first distillation loss between student positive samples and teacher positive samples, the second distillation loss between student positive samples and teacher negative samples, and the truth loss between student output results and truth labels using a preset loss function.

[0084] In some embodiments of the present invention, the preset loss function includes a category loss function, a regression loss function, or a foreground-background loss function; wherein, the category loss function includes KL divergence (KL Divergence) loss; the regression loss function includes EioU loss (Efficient-IoU) function; and the foreground-background loss function includes cross-entropy loss function.

[0085] In some embodiments of the present invention, adaptive sample matching and weight calculation are achieved through two SimOTA methods in the calculation of distillation loss.

[0086] The purpose of using the SimOTA method for the first time is to distinguish between high-quality positive samples in the teacher output that are similar to the truth labels, and negative samples that are not recorded in the truth labels but may contain effective guidance information. At the same time, it determines the first weight corresponding to the teacher's positive samples and the second weight corresponding to the teacher's negative samples.

[0087] The purpose of using the SimOTA method a second time is to match samples in the student output that are similar to the teacher's positive samples and samples that are similar to the teacher's negative samples, and to use these matched sample pairs to calculate the loss value.

[0088] Figure 2 The ground truth loss is the training loss in a typical training scenario. In some embodiments of this invention, the student's output is matched with the ground truth using the SimOTA method to calculate the class loss, regression loss, and foreground-background loss. In this case, the class loss consists of the cross-entropy loss.

[0089] Step S1014: The first distillation loss is weighted by the first weight corresponding to the positive teacher samples, and the second distillation loss is weighted by the second weight corresponding to the negative teacher samples.

[0090] The first weight refers to the intersection-union ratio (IU) of each sample in the teacher's positive sample with its corresponding true label, and the second weight refers to the target confidence level of each sample in the teacher's negative sample.

[0091] Step S1015: Iteratively train the student network using the truth loss, the first distillation loss, and the second distillation loss until the student network converges to obtain the obstacle detection model.

[0092] The ratio of truth loss, first distillation loss, and second distillation loss is set at 20:2:1, with truth loss playing a major role in model training to avoid increasing the convergence difficulty of the student network.

[0093] In addition, obstacle detection models should pay more attention to dynamic objects in the current frame image. Based on this, such as Figure 2 As shown, this invention uses Fourier transform to extract dynamic features between video frames, serving as attention gain information for the feature map. These dynamic features are independent of the knowledge distillation process and primarily serve as supplementary information during model training and inference. By extracting possible dynamic object parts in the current image from consecutive video frames, these features are used as attention information to enhance the feature map predicted by the model.

[0094] In addition, Figure 2In this process, feature loss is optional. Typically, the feature map output by the student network is input into an adaptive scaling layer consisting of convolutional or MLP layers to obtain a transformed feature at the same scale as the feature map of the teacher network. The feature loss between the transformed feature and the feature map output by the teacher is then calculated.

[0095] After training, the obstacle detection model is deployed on an unmanned aerial vehicle (UAV). During the UAV's flight, the obstacle detection model dynamically detects surrounding obstacles and outputs the detection results. The obstacle detection results output by the model include the obstacle category of each obstacle in the current frame image and the corresponding obstacle detection box for each obstacle.

[0096] In some embodiments of this invention, a series of target categories are pre-defined as potential dynamic obstacle categories. These categories are objects in the real environment that are prone to positional changes in a short period of time, causing a difference between the real environment and the environment recorded by the system, making it impossible for the path planning algorithm to incorporate them into the planning process in a timely manner. Based on this, this invention focuses on identifying target categories and performing subsequent obstacle avoidance operations. For the obstacle categories in the detection results output by the obstacle detection model, filtering is performed according to the pre-defined potential dynamic obstacle categories to filter out the objects that require priority processing.

[0097] In the obstacle detection results, the obstacle detection bounding box is used to indicate the current pixel area and current pixel coordinates of the obstacle in the current frame image. It is usually the pixel coordinates of the top left and bottom right corners of the obstacle in the current frame image, or the center coordinates and width and height.

[0098] In actual implementation, obstacle detection results also include category confidence scores for each obstacle category. These three components are stored in three lists of equal length (obstacle category list, obstacle detection box list, and category confidence score list). The corresponding positions represent the detection results for the same obstacle, and they are associated through a single obstacle identifier, which uniquely identifies the obstacle. Based on the obstacle category list in the obstacle detection model's results, a judgment is made: if it is a pre-defined dynamic obstacle category (such as birds or drones) and the category confidence score is higher than a pre-set confidence threshold, the detection result is retained; otherwise, the detection result is discarded.

[0099] Specifically, after the obstacle detection model outputs the obstacle detection results, it also includes: comparing the category confidence with a preset confidence threshold, and deleting obstacle categories whose category confidence is less than the preset confidence threshold.

[0100] Step S102: Compare the obstacle detection results with the detection results corresponding to the historical frame images to obtain comparison results. The comparison results indicate whether each obstacle is the same as the obstacle in the historical frame images.

[0101] In some embodiments of the present invention, a historical frame image refers to the previous video frame image in the video stream of the current frame image; or, a historical frame image refers to the previous video frame image in the preset buffer or in the video stream that has a frame interval with the current frame image.

[0102] The detection results corresponding to historical frame images are recorded in the current dynamic obstacle log. The current dynamic obstacle log contains a historical obstacle information table and an obstacle information queue for each historical obstacle. The historical obstacle information table includes obstacle identification field, obstacle category field, area expansion rate field, obstacle hazard field, movement direction field, and update timestamp field. The obstacle information queue includes a historical obstacle size queue, a frame interval queue, and a historical obstacle location queue.

[0103] In some embodiments of the present invention, the obstacle detection result is compared with the detection result corresponding to the historical frame image, including the comparison between obstacle category, obstacle size and obstacle position. The comparison result is the obstacle difference degree obtained based on the comparison of obstacle category, obstacle size and obstacle position.

[0104] The contribution of obstacle category to obstacle difference is a constant: 1 for obstacles of the same category and 0 for obstacles of different categories. Obstacle category difference takes into account the possibility of false positives, so it does not directly rule out the possibility that obstacles of different categories belong to the same obstacle. Obstacle position difference is calculated as the Euclidean distance between the information pixel coordinates within the obstacle multiplied by a balance coefficient. Obstacle size difference is calculated using the following formula:

[0105]

[0106] In the formula, s query This query represents the pixel area of ​​an obstacle; s target α represents the pixel area of ​​the source obstacle; α is the balance coefficient.

[0107] For an obstacle detection result output by the obstacle detection model, firstly, the difference between the result and all obstacles in the current dynamic obstacle log is calculated; then, the result is assigned to the obstacle with the smallest difference that is less than a preset difference threshold, indicating that the result and the assigned obstacle are the same obstacle.

[0108] If the difference between this detection result and all obstacles is greater than the preset difference threshold, then the obstacle corresponding to this detection result is identified as a newly added obstacle, and this detection result is used to add an obstacle object to the current dynamic obstacle log.

[0109] Step S103: If the comparison results indicate that each obstacle is the same as the obstacle in the historical frame image, calculate the area expansion rate of each obstacle based on the current pixel area and the historical pixel area of ​​each obstacle in the historical frame image.

[0110] In some embodiments of the present invention, the danger level of an obstacle is determined by its area dilation rate across n consecutive video frames. Based on this, it is necessary to first calculate the area dilation rate corresponding to each obstacle, specifically including the following two cases:

[0111] In the first case, for any obstacle, the area dilation rate calculation method considering the frame interval can be expressed by the following formula:

[0112]

[0113] Among them, v r The area dilation rate is represented by s, where n represents the nth frame of the video image. i Let s represent the pixel area of ​​obstacle r in the i-th frame of the video image. i+1 The pixel area of ​​obstacle r in the (i+1)th frame of the video image is represented by t. i This represents the frame interval between the i-th and i+1-th frames.

[0114] In the second case, for spherical obstacles, the area expansion rate based on the radius of the spherical obstacle can be calculated using the following formula:

[0115]

[0116] In the formula, Let denot be the derivative of the function R(t), where R(t) represents the area expansion rate of the obstacle from time t to t+Δt, and Δt represents the frame interval time; a represents the radius of the spherical obstacle; v represents the relative speed at which the spherical obstacle approaches the optical center of the two-dimensional optical imaging system; and x represents the distance from the spherical obstacle to the optical center of the two-dimensional optical imaging system. Initially, the distance between the spherical obstacle and the optical center is x′, then x=x′-vt.

[0117] The function R(t) can be expressed by the following equation:

[0118]

[0119] In the formula, Ω represents the solid angle of the spherical obstacle relative to the optical center, denoted as: Where x represents the distance from the spherical obstacle to the optical center of the two-dimensional optical imaging system, a represents the radius of the spherical obstacle; Δt represents the frame interval time; and Ω(t) is the solid angle occupied by the spherical obstacle relative to the optical center at time t. According to the conditions, the distance from the spherical obstacle to the optical center is x = x′ - vt. That is, Ω(t) = Ω(x(t)). The derivative of the area expansion rate with time is expressed as:

[0120]

[0121] In the formula, Ω represents the spatial solid angle occupied by the spherical obstacle relative to the optical center; Δt represents the frame interval time; a represents the radius of the spherical obstacle; v represents the relative speed at which the spherical obstacle approaches the optical center of the two-dimensional optical imaging system; and x represents the distance from the spherical obstacle to the optical center of the two-dimensional optical imaging system.

[0122] When the spherical obstacle is far away, v can be performed as in the second case described above. r The approximation shown in the expression.

[0123] Furthermore, to reduce the impact of obstacle detection model output errors on the obstacle avoidance performance of the unmanned aerial vehicle (UAV), in some embodiments of the present invention, a Kalman filter is used to smooth the current pixel area and current pixel coordinates for each obstacle. Specifically, the Kalman filter corrects the obstacle detection model output based on the historical pixel coordinates and historical pixel areas of each obstacle in historical frame images, combined with the actual observation data of the current frame.

[0124] Specifically, before calculating the area dilation rate of each obstacle based on the current pixel area of ​​each obstacle in the current frame image and the historical pixel area of ​​each obstacle in the historical frame image, the method further includes: smoothing the current pixel area and current pixel coordinates of each obstacle using a Kalman filter and the historical pixel coordinates and historical pixel areas of each obstacle in the historical frame image.

[0125] Step S104: Multiply the area expansion rate by the expansion rate coefficient to obtain the obstacle hazard level corresponding to each obstacle.

[0126] Specifically, the obstacle hazard level α is expressed as:

[0127] α=μv r

[0128] In the formula, v r represents the area expansion rate corresponding to the obstacle; μ represents the expansion rate coefficient.

[0129] In some embodiments of the present invention, the dilation coefficient is obtained by taking the square root or logarithm of the current pixel area corresponding to the obstacle. For example, taking the logarithm of the current pixel area, the dilation coefficient can be expressed as:

[0130] μ=α t (v r )=∈v r log(s)

[0131] In the formula, s represents the area of ​​the current pixel; ∈ is a constant; v r This represents the area expansion rate corresponding to the obstacle.

[0132] In addition, since the detection results corresponding to the historical frame images are obtained from the historical obstacle information table and the obstacle information queue corresponding to each historical obstacle, after calculating the obstacle danger level corresponding to each obstacle, it is necessary to update the historical obstacle information table and the obstacle information queue so that the next calculation can be carried out smoothly.

[0133] In some embodiments of the present invention, the historical obstacle information table includes an obstacle identification field, an obstacle category field, an area expansion rate field, an obstacle hazard field, a movement direction field, and an update timestamp field; the obstacle information queue includes a historical obstacle size queue, a frame interval queue, and a historical obstacle location queue.

[0134] The obstacle identification field is an integer field used to indicate the unique identifier of the obstacle; the obstacle category field is an integer field used to indicate the category of the obstacle; the area expansion rate field is a floating-point field used to record the area expansion rate of the obstacle per unit time in the current frame image; the obstacle hazard level field is a floating-point field used to indicate the degree of threat posed by the obstacle to the flight safety of the unmanned aerial vehicle, calculated from factors such as the obstacle area expansion rate; the motion direction field is a one-dimensional array field used to represent the motion direction trend of the obstacle in the image. In some embodiments of the present invention, the motion direction is calculated based on the pixel coordinates of each obstacle in the first frame image and the pixel coordinates in the last frame image in the current frame image; and the update timestamp field is a floating-point field used to indicate the timestamp of the last update of the obstacle information.

[0135] The historical obstacle size queue is a bidirectional queue used to record the pixel area of ​​the obstacle in each frame of the image; the frame interval queue is a bidirectional queue used to record the timestamp after processing each frame of the image; and the historical obstacle position queue is a bidirectional queue used to record the pixel coordinates of the obstacle in each frame of the image.

[0136] When updating obstacle information, the current obstacle size value is added to the historical obstacle size queue, the frame interval queue is updated, and the current obstacle pixel coordinate value is added to the historical obstacle position queue. The area expansion rate is calculated based on the latest frame from the above three queues and updated to the area expansion rate field. The obstacle hazard level is updated using the obstacle hazard level obtained by multiplying the area expansion rate by the expansion rate coefficient. The motion direction field is updated using the motion direction calculated from the head and tail positions of the historical obstacle position queue within the calculation window. The update timestamp corresponding to this update is updated based on the current actual time.

[0137] Specifically, after multiplying the area expansion rate by the expansion rate coefficient to obtain the obstacle danger level corresponding to each obstacle, the process also includes: adding the current pixel area corresponding to each obstacle to the historical obstacle size queue; updating the timestamp of the current frame image processing in the frame interval queue; adding the current pixel coordinates corresponding to each obstacle to the historical obstacle position queue; and updating the historical obstacle information table according to the obstacle identification field, as well as the area expansion rate, obstacle danger level, movement direction, and update timestamp corresponding to this update for each obstacle.

[0138] In some embodiments of the present invention, the time when the obstacle detection model last actually detected an obstacle is recorded by updating the timestamp. If the obstacle is not detected, it is possible that the obstacle has left the range threatening the unmanned aerial vehicle. In this case, to prevent the obstacle from remaining in the current dynamic obstacle log indefinitely and interfering with the system's decision-making, the system checks the update timestamps of all obstacles before performing obstacle detection on the current frame image. If the update time difference between the update timestamp of an obstacle and the current actual time is greater than a preset duration, meaning that the obstacle has not been detected for a certain period of time, then the obstacle is removed from the current dynamic obstacle log.

[0139] Specifically, before inputting the current frame image from the video stream into the pre-trained obstacle detection model, the method further includes: determining the update time difference of each obstacle in the historical obstacle information queue based on the current time and the update timestamp of each obstacle in the historical obstacle information queue; and removing obstacles whose update time difference is greater than a preset duration from the historical obstacle queue.

[0140] If the time difference between the update timestamp of an obstacle and the current actual time is less than or equal to a preset duration, and an obstacle recorded in the current dynamic obstacle log is not detected in a certain frame of image, the system will predict its motion state and execute the empty update method. The empty update method is used to track obstacles for a period of time during which the obstacle detection model does not detect obstacles or the obstacles exceed the image boundary. The empty update method does not update the update timestamp.

[0141] In some embodiments of the present invention, Kalman filter prediction is used to predict obstacle size, obstacle position, and obstacle direction of motion. The predicted obstacle size is added to the historical obstacle size queue corresponding to this obstacle, and the frame interval queue is updated synchronously.

[0142] In some embodiments of the present invention, due to the field of view limitation of the two-dimensional optical imaging system, the program will also perform the same prediction and empty update operations as described above after the obstacle moves out of the field of view, so as to maintain vigilance against such obstacles for a certain period of time.

[0143] Step S105: Compare the obstacle hazard level of each obstacle with the preset hazard threshold, and identify the obstacles whose obstacle hazard level is greater than the preset hazard threshold as dangerous obstacles.

[0144] Step S106: Based on the pixel width and pixel height of the current frame image, the actual width and actual height of the current frame image, the distance from the optical center of the two-dimensional optical imaging system to the center of the virtual imaging plane, and the current pixel coordinates, determine the dangerous obstacle vector pointing from each dangerous obstacle to the two-dimensional optical imaging system.

[0145] refer to Figure 3 In some embodiments of the present invention, the image pixel width is w p The image pixel height is h p The image origin is the upper left corner, and the distance from the optical center O of the two-dimensional optical imaging system to the center O' of the virtual imaging plane is d.

[0146] By specifying the value of d, the true scale corresponding to the image pixel width and height w and h can be calculated based on the field of view (FOV) of the two-dimensional optical imaging system, thereby determining the true orientation of the obstacle relative to the two-dimensional optical imaging system. Let the pixel coordinates of the obstacle in the image be (x...). o ,y o Taking this as an example, for simplicity, we set the aspect ratio of the captured image to 1:1, the horizontal field of view to f, and the true scale as follows:

[0147] So in the X of a two-dimensional optical imaging systemc Y c Z c In a coordinate system, the vector representation of a dangerous obstacle pointing from the obstacle to the two-dimensional optical imaging system is as follows:

[0148]

[0149] In the formula, w p h represents the image pixel width of a two-dimensional optical imaging system. p This indicates the image pixel height of a two-dimensional optical imaging system; (x o ,y o ) represents the pixel coordinates of the obstacle in the image; d represents the distance from the optical center O of the two-dimensional optical imaging system to the center O' of the virtual imaging plane; w represents the true scale width corresponding to the image pixel width; h represents the true scale height corresponding to the image pixel height.

[0150] In some embodiments of the present invention, the dangerous obstacles also include obstacles among the lost obstacles whose predicted obstacle hazard level is greater than a preset hazard level threshold. The predicted obstacle hazard level is calculated based on a preset hazard attenuation rate and the historical obstacle hazard level corresponding to the lost obstacle. The preset hazard attenuation rate refers to a pre-set hazard attenuation rate, including 5%, 10%, or 15%, etc. This embodiment does not limit the value of the preset hazard attenuation rate.

[0151] If the predicted obstacle risk level corresponding to the lost obstacle is greater than the preset risk level threshold, a Kalman filter is used to predict the lost obstacle, obtain the predicted pixel coordinates corresponding to the lost obstacle, and calculate the dangerous obstacle vector corresponding to the lost obstacle.

[0152] Specifically, when the comparison results indicate that the obstacles in the current frame image are different from the obstacles in the historical frame images and there are missing obstacles, the method further includes: obtaining the predicted obstacle hazard based on a preset hazard attenuation rate and the historical obstacle hazard corresponding to the missing obstacle; when the predicted obstacle hazard is greater than a preset hazard threshold, predicting the missing obstacle using a Kalman filter to obtain the predicted pixel coordinates corresponding to the missing obstacle; using the predicted pixel coordinates to calculate the dangerous obstacle vector corresponding to the missing obstacle; identifying the missing obstacle as a dangerous obstacle, and performing the step of determining the dangerous obstacle vector pointing to the two-dimensional optical imaging system for each dangerous obstacle.

[0153] Step S107: Based on the dangerous obstacle vector and obstacle hazard level corresponding to each dangerous obstacle, calculate the repulsion vector corresponding to each dangerous obstacle, and then perform a weighted summation of the repulsion vectors by the obstacle hazard level corresponding to each dangerous obstacle to obtain the overall repulsion vector.

[0154] The repulsive force vector of the obstacle on the unmanned aerial vehicle is expressed as:

[0155]

[0156] In the formula, α represents the vector of dangerous obstacles pointing towards the two-dimensional optical imaging system; t This represents the obstacle hazard level at time t.

[0157] Accordingly, the overall repulsive force vector can be expressed by the following formula:

[0158]

[0159] In the formula, α represents the vector of dangerous obstacles pointing towards the two-dimensional optical imaging system; t α represents the obstacle hazard level at time t; α represents the preset obstacle hazard threshold.

[0160] The overall hazard α is the highest hazard among the hazardous obstacles. f , is represented as:

[0161]

[0162] In the formula, s represents a dangerous obstacle; C a Let represent the set of dangerous obstacles; α(s) represents the obstacle hazard level corresponding to dangerous obstacle s.

[0163] Step S108: For the vectors corresponding to the avoidance direction of each dangerous obstacle, perform a weighted summation using the corresponding obstacle danger level, and take the direction pointed to by the summed vector as the optimal avoidance direction.

[0164] The avoidance direction is obtained based on a preset path planning algorithm.

[0165] In some embodiments of the present invention, the optimal avoidance direction is the avoidance direction of the unmanned aerial vehicle calculated on the image plane. The calculation method can be selected differently according to the requirements. For example, the direction with the sparsest obstacles on the image can be selected as the avoidance direction, or the environmental safety direction given by the path planning system can be combined with the obstacle position to calculate the avoidance direction.

[0166] In practice, when an obstacle is on one side of the image closer to the center and moves to the other side relative to the UAV at a higher tangential velocity, the opposite direction of the obstacle's movement can be selected as the avoidance direction.

[0167] For example: Reference Figure 4When an obstacle is located within the central area indicated by the dashed box and is moving at a large tangential speed, choosing the opposite direction to avoid it will allow you to escape the danger zone more quickly. When the obstacle is located outside the central area, choose the direction of the line connecting the center of the obstacle and the center of the image as the avoidance direction, and the magnitude of the line is inversely proportional to the distance between the obstacle and the center of the image.

[0168] After determining the avoidance direction for each obstacle, the avoidance directions are weighted and summed according to the obstacle danger level of each obstacle, and the vector obtained by the weighted summation is determined as the optimal avoidance direction.

[0169] Step S109: Calculate the component of the optimal avoidance direction that is perpendicular to the overall repulsive force vector as the true avoidance direction, and control the unmanned aerial vehicle to avoid obstacles according to the true avoidance direction.

[0170] In summary, the obstacle avoidance control method for unmanned aerial vehicles (UAVs) provided in this application continuously captures video streams using a two-dimensional optical imaging system. The current frame image from the video stream is input into a pre-trained obstacle detection model, which outputs obstacle detection results. Based on the current pixel area and the historical pixel areas of each obstacle in historical frame images, the area expansion rate and obstacle hazard level of each obstacle are calculated. The dangerous obstacle vector pointing from each dangerous obstacle to the two-dimensional optical imaging system is determined. Based on the dangerous obstacle vector and obstacle hazard level of each dangerous obstacle, the repulsion vector corresponding to each dangerous obstacle is calculated and weighted to obtain the overall repulsion vector. The avoidance directions corresponding to each dangerous obstacle are weighted and summed using the obstacle hazard level of each dangerous obstacle to obtain the optimal avoidance direction. The component of the optimal avoidance direction perpendicular to the overall repulsion vector is calculated as the true avoidance direction, and the UAV is controlled to follow the true avoidance direction. This system enables obstacle avoidance based on the actual avoidance direction, addressing the complexity and high maintenance costs of unmanned aerial vehicle (UAV) systems. By detecting obstacles in the video stream acquired through a 2D optical imaging system, obstacle information is obtained and updated. Then, based on the relative approach speed and position between the obstacle and the UAV, the hazard posed by each obstacle is assessed and assigned a weight. Subsequently, using this weight and the relative position of the obstacle, the current repulsion vector of the obstacle is calculated. Finally, combining environmental safety information provided by the path planning system and the specific location of the obstacle, the most suitable avoidance direction and speed are determined in a plane perpendicular to the repulsion vector, guiding the UAV to perform effective avoidance maneuvers. This system allows for dynamic obstacle avoidance without requiring dynamic environment modeling and real-time depth information, thus eliminating the need for depth sensors on the UAV and reducing system complexity and maintenance costs.

[0171] In addition, to address the issue that neural networks require significant computational resources, this invention optimizes the obstacle detection model using knowledge distillation to reduce the algorithm's computational resource requirements and improve the obstacle avoidance efficiency of unmanned aerial vehicles.

[0172] The following example illustrates another instance of the obstacle avoidance control method for unmanned aerial vehicles provided by this invention. (Refer to...) Figure 5 The method includes at least the following steps S501 to S506:

[0173] Step S501: Extract obstacle information from the required video frames to calculate the information needed to avoid obstacles, including the obstacle hazard level.

[0174] The required video frames refer to the latest series of consecutive video frame images in the real-time video stream acquired by the two-dimensional optical imaging system on the unmanned aerial vehicle. The number of frames is determined by a pre-set calculation window.

[0175] The required video frames are input into a pre-trained obstacle detection model, which outputs the obstacle detection results as the extracted obstacle information, including obstacle category, obstacle detection box, and category confidence corresponding to the obstacle category.

[0176] The obstacle detection bounding box indicates the current pixel area and coordinates of an obstacle in the current frame image. It is typically the pixel coordinates of the top-left and bottom-right corners of the obstacle in the current frame image, or the center coordinates and width / height. The training method for the obstacle detection model is described in the aforementioned embodiment of the obstacle avoidance control method for unmanned aerial vehicles, and will not be repeated here.

[0177] Based on the current pixel area and the historical pixel area of ​​each obstacle in the historical frame image, the area expansion rate of each obstacle is calculated; the area expansion rate is multiplied by the expansion rate coefficient to obtain the obstacle danger level of each obstacle.

[0178] Step S502: Determine whether the obstacle's hazard level is greater than a preset hazard threshold. If the obstacle's hazard level is greater than the preset hazard threshold, proceed to step S503; otherwise, proceed to step S501.

[0179] Step S503: Gain control of the unmanned aerial vehicle and perform evasive maneuvers.

[0180] Step S504: Determine whether the current obstacle danger level is less than the preset danger level threshold. If the current obstacle danger level is less than the preset danger level threshold, proceed to step S505; otherwise, proceed to step S501.

[0181] Step S505: Return to the original path and hand over control of the unmanned aerial vehicle.

[0182] Step S506: Determine whether the flight mission has ended. If the flight mission has ended, terminate the current flight; otherwise, proceed to step S501.

[0183] In summary, the obstacle avoidance control method for unmanned aerial vehicles (UAVs) provided in this application can solve the problems of complex UAV systems and high operation and maintenance costs. By detecting obstacles in the video stream acquired by a two-dimensional optical imaging system, obstacle information is obtained and updated. Then, based on the relative approach speed and relative position between the obstacle and the UAV, the danger posed by each obstacle to the UAV is evaluated and assigned a corresponding weight. Subsequently, using this weight and the relative position of the obstacle, the current repulsion vector of the obstacle is calculated. Finally, combining the environmental safety information provided by the path planning system and the specific position of the obstacle, the most suitable avoidance direction and speed are determined in the plane perpendicular to the repulsion vector, thereby guiding the UAV to perform effective avoidance actions. It can control the UAV to perform dynamic obstacle avoidance without dynamic environment modeling and real-time depth information. Therefore, it can avoid equipping the UAV with sensors to obtain depth information, thereby reducing the complexity and maintenance cost of the UAV system.

[0184] In existing obstacle avoidance methods for unmanned aerial vehicles (UAVs), image depth information is first acquired through an imaging system. Based on this depth information, a three-dimensional spatial model of the dynamic obstacle is created to obtain its x, y, and z coordinates in three-dimensional space. (Referencing...) Figure 3 The image is plotted with the horizontal x-axis, the vertical y-axis, and the z-axis representing depth information. A Kalman filter is then used to track the dynamic obstacle trajectories in three-dimensional space. The Kalman filter's ability to predict obstacle trajectories is combined with other methods to make obstacle avoidance decisions.

[0185] The ultimate goal of using depth information to model obstacles in 3D space is to determine whether they will collide with the UAV at some point in the future based on their future trajectory. This is similar to the concept of obstacle collision time. Obstacle collision time can be expressed as:

[0186] In the formula, x represents the distance between the obstacle and the optical center of the imaging system in the UAV; v represents the component of the obstacle's velocity along the line connecting it to the optical center.

[0187] Clearly, the obstacle collision time t is proportional to the distance x. Based on this, in some embodiments of the present invention, the obstacle collision time can be used instead of the actual depth information at this moment, serving as indirect depth information to model the three-dimensional coordinates of the obstacle. The indirect coordinates established using the indirect depth information can also be input into a Kalman filter to track and predict the obstacle trajectory. The final predicted obstacle trajectory has the same shape as the actual obstacle trajectory and is proportional to it.

[0188] For example, an obstacle approaching rapidly from a distance and an obstacle approaching slowly from nearby may collide with an unmanned aerial vehicle at the same time, and the trajectories of the two obstacles are the same under this modeling method.

[0189] Furthermore, as can be seen from the area expansion rate formula in the second case above, when the obstacle is far away, the area expansion rate... Combination Obstacle collision time can be obtained Due to the area expansion rate v r It is measurable, therefore, the obstacle expansion rate at a certain moment can be used to replace the obstacle collision time at that moment.

[0190] It is worth noting that, in the traditional sense, depth information refers to the presence of obstacles. Figure 3 The distance along the z-axis in the image is the distance to the imaging plane (xy), while the alternative method described above replaces it with the distance along the line connecting the obstacle to the optical center of the two-dimensional imaging optical system. Both methods can describe the spatial position of the obstacle, only the rules followed during Kalman filter updates are slightly different, which will not be elaborated here.

[0191] In some embodiments of the present invention, as a supplementary obstacle avoidance method to the obstacle avoidance control method of the unmanned aerial vehicle in steps S101 to S109 above, the obstacle collision time can be used instead of the actual depth information at this moment to model the three-dimensional space of the obstacle as indirect depth information. At the same time, the obstacle expansion rate at a certain moment can be used instead of the obstacle collision time at this moment. Then, a Kalman filter is used to track the dynamic obstacle trajectory in the three-dimensional space of the obstacle. The obstacle avoidance decision is made by combining the obstacle trajectory prediction capability of the Kalman filter with other methods (such as path planning algorithms or obstacle avoidance algorithms based on machine learning).

[0192] This approach allows for advance prediction of obstacle trajectories and subsequent obstacle avoidance decisions. Compared to the obstacle avoidance control methods for UAVs described in steps S101 to S109, it avoids redundant obstacle avoidance maneuvers, making the UAV's dynamic obstacle avoidance function more efficient and stable. However, a drawback is the need for sufficient observation data to predict obstacles, which requires more reaction time.

[0193] Based on this, in some embodiments of the present invention, the above-described supplementary obstacle avoidance method is applied simultaneously with the obstacle avoidance control method for unmanned aerial vehicles in steps S101 to S109 (hereinafter referred to as reactive obstacle avoidance strategy), such as... Figure 6 As shown, the method includes at least steps S601 to S612:

[0194] Step S601: Determine whether an obstacle is detected. If an obstacle is detected, proceed to step S602; otherwise, proceed to step S608.

[0195] Step S602: Update the corresponding obstacle detection information.

[0196] The update method is the same as the obstacle avoidance control method for unmanned aerial vehicles in steps S101 to S109 above, and will not be repeated here.

[0197] In step S603, the Kalman filter updates the data and predicts the obstacle's trajectory.

[0198] Step S604: Determine whether the current stage is the initial stage. If the current stage is the initial stage, proceed to step S607; otherwise, proceed to step S605.

[0199] When an obstacle is detected in its initial stage and the Kalman filter cannot accurately track and predict its trajectory, a reactive obstacle avoidance method is used; otherwise, obstacle avoidance actions can be planned based on the motion trajectory predicted by the Kalman filter.

[0200] In the initial stage, the information about a certain obstacle can be set to be updated less than a certain number of times, such as 15 times. In reality, the time it takes to leave the initial stage will vary depending on the performance of the obstacle detection model and the operating platform.

[0201] The Kalman filter used in this embodiment is different from the Kalman filter used in the above embodiments. The Kalman filter used in this embodiment can be used to replace the Kalman filter used in the above embodiments, or the Kalman filter used in the above embodiments can be retained.

[0202] Step S605: Determine whether the prediction and observation error of the Kalman filter is less than a preset threshold. If it is less than the preset threshold, proceed to step S606; otherwise, proceed to step S607.

[0203] When a certain amount of observation data is accumulated for an obstacle, the error between the predicted trajectory and the observed trajectory is less than a preset threshold, and the obstacle is no longer observed in the initial stage and enters a stable stage, obstacle avoidance actions can be planned based on the motion trajectory predicted by the Kalman filter.

[0204] During the stable phase, the obstacle's motion state may change abruptly. At this time, the error between the predicted trajectory and the observed trajectory may exceed the preset threshold. The reactive obstacle avoidance method for this obstacle will be restarted until the error between the predicted trajectory and the observed trajectory is less than the preset threshold again.

[0205] Step S606: Plan obstacle avoidance actions based on the predicted motion trajectory.

[0206] Step S607: Plan obstacle avoidance actions using a reactive obstacle avoidance strategy.

[0207] In step S608, the Kalman filter updates the data and predicts the trajectory of the obstacle.

[0208] When the obstacle detection model misses an obstacle, and there is no obstacle information update, a no-update method is performed. The no-update method refers to the obstacle avoidance control method for unmanned aerial vehicles in steps S101 to S109 above, and will not be repeated here.

[0209] If the obstacle is in its initial stage, reactive obstacle avoidance is used; if the obstacle is in its stable stage, the error judgment between the predicted trajectory and the observed trajectory is no longer performed, and the obstacle avoidance action is planned directly based on the motion trajectory predicted by the Kalman filter. The predictive ability of the Kalman filter can, to some extent, counteract missed detections.

[0210] Step S609: Determine whether the current stage is the initial stage. If the current stage is the initial stage, proceed to step S610; otherwise, proceed to step S611.

[0211] Step S610: Plan obstacle avoidance actions using a reactive obstacle avoidance strategy.

[0212] Step S611: Plan obstacle avoidance actions based on the predicted motion trajectory.

[0213] Step S612: Determine whether the obstacle avoidance task has ended. If the obstacle avoidance task has ended, end the task; otherwise, proceed to step S601.

[0214] In summary, the obstacle avoidance control method for unmanned aerial vehicles (UAVs) provided in this embodiment uses the obstacle collision time instead of the actual depth information at that moment, as indirect depth information to model the three-dimensional space of the obstacle. Simultaneously, it uses the obstacle expansion rate at a certain moment to replace the obstacle collision time at that moment, and then uses a Kalman filter to track the dynamic obstacle trajectory in the three-dimensional space. By combining the Kalman filter's ability to predict obstacle trajectories with other methods, obstacle avoidance decisions can be made. This allows for obstacle avoidance decisions to be made in advance if the obstacle detection model misses an obstacle or in the initial stage of obstacle detection, avoiding redundant obstacle avoidance actions and making the UAV's dynamic obstacle avoidance function more efficient and stable. Furthermore, combined with a reactive obstacle avoidance strategy, when the error between the predicted trajectory and the observed trajectory is less than a preset threshold, and the UAV has moved beyond the initial stage of observing the obstacle and entered a stable stage, obstacle avoidance actions are planned based on the motion trajectory predicted by the Kalman filter. When the error between the predicted trajectory and the observed trajectory exceeds the preset threshold, a reactive obstacle avoidance method is used until the error between the predicted trajectory and the observed trajectory is less than the preset threshold again, thereby improving the success rate of obstacle avoidance.

[0215] Corresponding to the above method, the present invention also provides an obstacle avoidance control method apparatus for an unmanned aerial vehicle, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the apparatus implements the steps of the aforementioned obstacle avoidance control method for an unmanned aerial vehicle.

[0216] The present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of the aforementioned obstacle avoidance control method for unmanned aerial vehicles.

[0217] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0218] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0219] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0220] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for obstacle avoidance control of an unmanned aerial vehicle, characterized in that, The unmanned aerial vehicle is equipped with a two-dimensional optical imaging system for continuously capturing video streams; the method includes the following steps: The current frame image in the video stream is input into a pre-trained obstacle detection model, and the obstacle detection model outputs obstacle detection results. The obstacle detection results include the obstacle category of each obstacle in the current frame image and the obstacle detection box corresponding to each obstacle. The obstacle detection box is used to indicate the current pixel area and current pixel coordinates of the obstacle in the current frame image. The obstacle detection results are compared with the detection results corresponding to the historical frame images to obtain a comparison result; the comparison result is used to indicate whether each obstacle is the same as the obstacle in the historical frame image; If the comparison result indicates that each obstacle is the same as the obstacle in the historical frame image, the area expansion rate corresponding to each obstacle is calculated based on the current pixel area and the historical pixel area of ​​each obstacle in the historical frame image. Multiplying the area expansion rate by the expansion rate coefficient yields the obstacle hazard level corresponding to each obstacle; the expansion rate coefficient is obtained by taking the square root or logarithm of the current pixel area. The obstacle hazard level of each obstacle is compared with a preset hazard threshold, and obstacles with an obstacle hazard level greater than the preset hazard threshold are identified as dangerous obstacles. Based on the pixel width and pixel height of the current frame image, the actual width and actual height corresponding to the current frame image, the distance from the optical center of the two-dimensional optical imaging system to the center of the virtual imaging plane, and the current pixel coordinates, determine the danger obstacle vector pointing from each of the danger obstacles to the two-dimensional optical imaging system; Based on the obstacle vector and obstacle hazard level corresponding to each of the aforementioned obstacles, the repulsion vector corresponding to each of the aforementioned obstacles is calculated, and the repulsion vector is weighted and summed by the obstacle hazard level corresponding to each of the aforementioned obstacles to obtain the overall repulsion vector; For each of the aforementioned dangerous obstacles, the vectors corresponding to the avoidance directions are weighted and summed using the corresponding obstacle danger levels. The direction pointed to by the summed vector is taken as the optimal avoidance direction. The avoidance direction is obtained based on a preset path planning algorithm. The component of the optimal avoidance direction that is perpendicular to the overall repulsive force vector is calculated as the true avoidance direction, and the unmanned aerial vehicle is controlled to avoid obstacles according to the true avoidance direction.

2. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 1, characterized in that, Before calculating the area dilation rate corresponding to each obstacle based on the current pixel area and the historical pixel areas of each obstacle in the historical frame image, the method further includes: The current pixel area and current pixel coordinates of each obstacle are smoothed by using a Kalman filter and the historical pixel coordinates and historical pixel areas of each obstacle in the historical frame image.

3. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 2, characterized in that, When the comparison result indicates that the obstacle in the current frame image is different from the obstacle in the historical frame image, and that there is a missing obstacle, the method further includes: Based on the preset danger decay rate and the historical danger of the lost obstacle, the predicted obstacle danger is obtained; If the predicted obstacle danger level is greater than the preset danger level threshold, the Kalman filter is used to predict the lost obstacle to obtain the predicted pixel coordinates corresponding to the lost obstacle; the predicted pixel coordinates are used to calculate the dangerous obstacle vector corresponding to the lost obstacle. The missing obstacles are identified as dangerous obstacles, and the step of determining the dangerous obstacle vector pointing to the two-dimensional optical imaging system for each of the dangerous obstacles is performed.

4. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 1, characterized in that, If the video frame processing speed of the obstacle detection model is lower than the frame rate of the two-dimensional optical imaging system, the video stream is converted into video frames and stored in a preset buffer. The step of inputting the current frame image from the video stream into a pre-trained obstacle detection model includes: The latest video frame is obtained from the preset cache according to the preset calculation window and used as the current frame image, and then input into the obstacle detection model; Delete the video frames preceding the current frame image in the preset cache.

5. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 1, characterized in that, The training process of the obstacle detection model includes: The current frame image of the sample is input into the teacher network and student network in the initial obstacle detection model, respectively, to obtain the output results of the teacher network and the student network. By simplifying the optimal transport allocation method, the teacher network output results and the ground value labels corresponding to the current frame image of the sample are matched and divided to obtain teacher positive samples and teacher negative samples. The simplified optimal transport allocation method is used to match and divide the teacher positive samples, the teacher negative samples, and the student network output results to obtain the student positive samples. The first distillation loss between the positive student samples and the positive teacher samples, the second distillation loss between the positive student samples and the negative teacher samples, and the ground truth loss between the student output and the ground truth label are calculated using a preset loss function. The preset loss function includes a category loss function, a regression loss function, or a foreground-background loss function. The ratio between the ground truth loss, the first distillation loss, and the second distillation loss is set to 20:2:

1. The first distillation loss is weighted by the first weight corresponding to the positive teacher samples, and the second distillation loss is weighted by the second weight corresponding to the negative teacher samples; The student network is iteratively trained using the ground truth loss, the first distillation loss, and the second distillation loss until the student network converges to obtain the obstacle detection model.

6. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 1, characterized in that, The detection results corresponding to the historical frame images are obtained by querying the historical obstacle information table and the obstacle information queue corresponding to each historical obstacle; the historical obstacle information table includes an obstacle identification field, an obstacle category field, an area expansion rate field, an obstacle hazard field, a movement direction field, and an update timestamp field; the obstacle information queue includes a historical obstacle size queue, a frame interval queue, and a historical obstacle position queue; After multiplying the area expansion rate by the expansion rate coefficient to obtain the obstacle hazard level corresponding to each obstacle, the method further includes: Add the current pixel area corresponding to each obstacle to the historical obstacle size queue corresponding to each obstacle; Update the timestamp of the current frame image after processing in the frame interval queue; Add the current pixel coordinates of each obstacle to the historical obstacle position queue corresponding to each obstacle; The historical obstacle information table is updated according to the obstacle identification field, the area expansion rate, obstacle hazard level, movement direction and update timestamp corresponding to each obstacle; the movement direction is calculated based on the pixel coordinates of each obstacle in the first frame and the last frame of the current frame image.

7. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 6, characterized in that, Before inputting the current frame image from the video stream into the pre-trained obstacle detection model, the method further includes: Based on the current time and the update timestamps of each obstacle in the historical obstacle information queue, determine the update time difference of each obstacle in the historical obstacle information queue; Obstacles whose update time difference is greater than a preset time are removed from the historical obstacle information queue.

8. The obstacle avoidance control method for an unmanned aerial vehicle according to claim 1, characterized in that, The obstacle detection results also include the category confidence scores corresponding to each obstacle category; after the obstacle detection model outputs the obstacle detection results, it also includes: The category confidence level is compared with a preset confidence threshold, and obstacle categories with a category confidence level lower than the preset confidence threshold are deleted.

9. An obstacle avoidance control device for an unmanned aerial vehicle, characterized in that, include: Two-dimensional optical imaging system for continuously acquiring video streams; A processor, a memory, and a computer program / instructions stored in the memory, the processor being configured to execute the computer program / instructions, wherein when the computer program / instructions are executed, the device implements the steps of the obstacle avoidance control method for an unmanned aerial vehicle as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement the steps of the obstacle avoidance control method for an unmanned aerial vehicle as described in any one of claims 1 to 8.

Images