A method, device, equipment and medium for recovering unmanned aerial vehicles based on visual algorithm

Abstract

The present invention belongs to the technical field of drone recovery in complex scenes, and specifically discloses a drone recovery method, device, equipment and medium based on visual algorithms. The method includes: obtaining the real-time coordinates of the drone during flight and the corresponding ground landing point coordinates; when the distance between the real-time coordinates and the ground landing point coordinates is within a preset range, determining the relative position of the drone and the ground landing point; based on the relative position, taking real-time photos of the area where the drone is located from the ground landing point upwards to obtain video image data; identifying the drone from the video image data and locating the drone; continuously tracking the drone and updating the flight trajectory of the drone; determining the descent gradient of the drone based on the flight trajectory of the drone; and using a path planning algorithm to plan the optimal landing route of the drone based on the descent gradient and the ground landing point coordinates, which is more suitable for drone recovery in complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of drone recovery in complex scenarios, and specifically relates to a drone recovery method, device, equipment and medium based on a visual algorithm. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in various fields. However, the automatic recovery of drones has always been one of the important bottlenecks restricting their further application.

[0003] In the development process of drone recovery technology, although a variety of recovery methods have been proposed and applied, there are still some obvious defects and shortcomings. For example, some drone recovery technologies still require human intervention or operation, such as manually guiding the drone to land, manually controlling the drone to enter the recovery area, etc., which increases the difficulty of operation and reduces the efficiency and safety of recovery. Summary of the invention

[0004] The purpose of the present invention is to provide a drone recovery method, device, equipment and medium based on a visual algorithm to solve the technical problems that the existing drone recovery technology still requires manual intervention or operation, is difficult to operate, and has low recovery efficiency and safety.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a method for recovering a drone based on a visual algorithm, comprising the following steps:

[0007] Acquire the real-time coordinates of the drone during flight and the corresponding coordinates of the ground landing point; when the distance between the real-time coordinates and the coordinates of the ground landing point is within a preset range, determine the relative position of the drone and the ground landing point; based on the relative position, take real-time photos of the area where the drone is located from the ground landing point upwards to obtain video image data;

[0008] Identify the drone from the video image data and locate the drone; continuously track the drone and update the flight trajectory of the drone; wherein, when the drone disappears from the video image data, use a trajectory prediction algorithm to predict the possible location of the drone, and continuously track the drone based on the possible location;

[0009] Determine the descent gradient of the drone based on its flight trajectory;

[0010] Based on the descent gradient and the coordinates of the ground landing point, a path planning algorithm is used to plan the optimal landing route of the UAV.

[0011] Further, based on the relative position, the area where the drone is located is photographed in real time from the ground landing point upward to obtain the video image data:

[0012] When the distance of the relative position is less than or equal to the preset range, the camera at the ground landing point starts to shoot the area where the drone is located; otherwise, the camera does not work.

[0013] Further, identifying the drone from the video image data and locating the drone includes:

[0014] Preprocessing the video image data to obtain continuous video frame data;

[0015] Input the video frame data into the convolutional neural network in the YOLO algorithm for feature extraction;

[0016] Based on the extracted features, the video frame data is divided into several grids and the bounding box and the corresponding confidence are predicted in each grid;

[0017] Based on the extracted features, predict the class probability vector for each bounding box;

[0018] According to the category probability vector and confidence score of each bounding box, several predicted drone position coordinates are obtained;

[0019] The non-maximum prediction algorithm is used to process the predicted UAV position coordinates to obtain the final UAV position coordinates.

[0020] Furthermore, according to the flight trajectory of the UAV, the descent gradient of the UAV is determined, including:

[0021] Determine the descent phase of the drone based on the flight trajectory of the drone;

[0022] Select the starting and ending points within the descent phase;

[0023] Use the height data of the starting point and the end point to calculate the descent height, calculate the horizontal distance between the starting point and the end point based on the position coordinates, and calculate the descent gradient based on the descent height and the horizontal distance.

[0024] Furthermore, after the step of planning the optimal landing route of the UAV using a path planning algorithm, the UAV descends according to the optimal landing route; wherein, when the UAV lands according to the optimal landing route, surrounding environment data is obtained, and the surrounding environment data is recognized using a preset recognition model; when obstacles are recognized, the route is replanned based on the path planning algorithm.

[0025] Furthermore, when the UAV lands according to the optimal landing route, the surrounding environment data is obtained, and the surrounding environment data is recognized by using a preset recognition model; when obstacles are recognized, the route is replanned based on the path planning algorithm, including:

[0026] When the drone lands according to the optimal landing route, the surrounding environment data is obtained; wherein the surrounding environment data includes: a first image taken by a camera carried by the drone in the direction of the drone's flight, and video image data taken from the ground landing point upward to the location of the drone;

[0027] Based on the first image and the video image data, respectively, a preset recognition model is used to perform obstacle recognition to obtain a first recognition result and a second recognition result; judging whether there is an obstacle in the navigation direction of the drone according to the first recognition result, and judging whether there is an obstacle in the location area of ​​the drone according to the second recognition result;

[0028] When any one of the first recognition result and the second recognition result shows an obstacle, the route is replanned based on the path planning algorithm.

[0029] Furthermore, in the step of planning the optimal landing route of the UAV using a path planning algorithm based on the descent gradient and the coordinates of the ground landing point, the path planning algorithm is an A* algorithm or an RRT algorithm.

[0030] In a second aspect, the present invention provides a UAV recovery device based on a visual algorithm, comprising:

[0031] The image acquisition module is used to obtain the real-time coordinates of the drone during flight and the corresponding coordinates of the ground landing point; when the distance between the real-time coordinates and the coordinates of the ground landing point is within a preset range, the relative position of the drone and the ground landing point is determined; based on the relative position, the area where the drone is located is photographed in real time from the ground landing point upwards to obtain video image data;

[0032] A drone identification and tracking module is used to identify a drone from the video image data and locate the drone; continuously track the drone and update the flight trajectory of the drone; wherein, when the drone disappears from the video image data, a trajectory prediction algorithm is used to predict the possible location of the drone, and the drone is continuously tracked based on the possible location;

[0033] A descent gradient calculation module is used to determine the descent gradient of the UAV based on the flight trajectory of the UAV;

[0034] The route planning module is used to plan the optimal landing route of the drone based on the descent gradient and the coordinates of the ground landing point using a path planning algorithm.

[0035] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned visual algorithm-based drone recovery method when executing the computer program.

[0036] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned drone recovery method based on visual algorithm is implemented.

[0037] Compared with the prior art, the present invention has at least the following beneficial effects:

[0038] The drone recovery method provided by the present invention performs real-time shooting of the drone location area from the ground landing point upward to obtain video image data; identifies the drone from the video image data and locates the drone; continuously tracks the drone and updates the flight trajectory of the drone; wherein, when the drone disappears from the video image data, a trajectory prediction algorithm is used to predict the possible location of the drone, and the drone is continuously tracked based on the possible location; the descent gradient of the drone is determined based on the flight trajectory of the drone; and the optimal landing route of the drone is planned based on the descent gradient and the coordinates of the ground landing point. This solution automatically identifies and tracks drones through visual algorithms, reduces manual intervention, and improves the automation of recovery. The flight coordinates and trajectory of the drone are obtained in real time, and path planning is performed accordingly, which can quickly respond to the dynamic changes of the drone. Through path planning and trajectory prediction, the collision risk of the drone during the recovery process can be reduced, and the safety of recovery can be improved. The automated process reduces human errors and delays, and improves the efficiency and accuracy of recovery.

[0039] This solution effectively solves several key problems in existing drone recovery technology by introducing visual algorithms and automation technology, including high demand for manual intervention, high difficulty in operation, low recovery efficiency and safety, etc. By acquiring drone flight data in real time and performing intelligent processing, automatic identification and tracking of drones, as well as path planning and trajectory prediction based on flight trajectories are realized, thereby improving the automation, efficiency and safety of recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0041] Figure 1 This is a flow chart of a UAV recovery method based on a visual algorithm according to an embodiment of the present invention;

[0042] Figure 2This is a structural block diagram of a UAV recovery device based on a visual algorithm according to an embodiment of the present invention;

[0043] Figure 3 The present invention is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0045] The following detailed description is an exemplary description, which is intended to provide further detailed description of the present invention. Unless otherwise specified, all technical terms used in the present invention have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used in the present invention are only for describing specific embodiments, and are not intended to limit exemplary embodiments according to the present invention.

[0046] Glossary:

[0047] ‌YOLO algorithm (You Only Look Once), an object detection system based on a single neural network.

[0048] A* algorithm (A-star Algorithm), heuristic search algorithm.

[0049] RRT algorithm (Rapidly Exploring Random Tree), rapidly expanding random tree algorithm.

[0050] The intersection over union (IoU) is a ratio that represents the ratio of the intersection area to the union area between the predicted bounding box (PredictedBounding Box) and the ground truth bounding box (Ground Truth Bounding Box).

[0051] Example 1

[0052] Example 1

[0053] Existing drone recovery technologies have the problems of high demand for manual intervention, high difficulty in operation, and low recovery efficiency and safety. Specifically, traditional methods require manual guidance of drone landing and manual control of drone entry into the recovery area, which increases the complexity and cost of operation and may lead to human errors and safety hazards. In addition, due to the limitations of manual operation, it is often difficult to respond to the dynamic changes of drones in real time, thereby reducing the efficiency and accuracy of recovery.

[0054] To solve the above problems, the present invention provides a UAV recovery method based on a visual algorithm, such as Figure 1 As shown, the following steps are included:

[0055] S100, obtaining the real-time coordinates of the drone during flight and the corresponding ground landing point coordinates; when the distance between the real-time coordinates and the ground landing point coordinates is within a preset range, determining the relative position of the drone and the ground landing point; based on the relative position, taking real-time photos of the drone location area from the ground landing point upward to obtain video image data.

[0056] S200, identifying a drone from the video image data and locating the drone; continuously tracking the drone and updating the flight trajectory of the drone; wherein, when the drone disappears from the video image data, using a trajectory prediction algorithm to predict the possible location of the drone, and continuously tracking the drone based on the possible location.

[0057] S300: Determine the descent gradient of the UAV according to the flight trajectory of the UAV.

[0058] S400, based on the descent gradient and the coordinates of the ground landing point, uses a path planning algorithm to plan the optimal landing route for the drone.

[0059] This solution uses visual algorithms to automatically identify and track drones, greatly reducing the need for manual intervention, and lowering the difficulty and cost of operation. The entire recovery process is highly automated, from drone identification, positioning, tracking to path planning, all of which are automatically completed by the algorithm, improving the degree of automation of recovery. The flight coordinates and trajectory of the drone are obtained in real time, and the path is planned accordingly, which can quickly respond to the dynamic changes of the drone, thereby improving the efficiency of recovery. Through path planning and trajectory prediction, the potential collision risk of the drone during the recovery process can be predicted and avoided in advance, improving the safety of recovery. In addition, the automated process reduces the possibility of human errors and delays, and improves the accuracy and reliability of recovery.

[0060] In order to further explain and illustrate the present solution, in some more specific embodiments, a method for recovering a drone based on a visual algorithm is also provided, comprising the following steps:

[0061] S1. Obtain the real-time coordinates of the drone during flight and the corresponding coordinates of the ground landing point; when the distance between the real-time coordinates and the coordinates of the ground landing point is within a preset range, determine the relative position of the drone and the ground landing point; based on the relative position, take real-time photos of the area where the drone is located from the ground landing point upward to obtain video image data.

[0062] Specifically, the real-time coordinates of the drone are obtained; the relative position of the drone and the ground landing point is calculated based on the real-time coordinates of the drone and the coordinates of the ground landing point; when the distance of the relative position is less than or equal to the preset range, the camera at the ground landing point starts to shoot the area where the drone is located; otherwise, the camera does not work.

[0063] Specifically, the ground landing point is set in an obstacle-free area and meets the requirements of safety distance and ground conditions. A camera is set at the ground landing point, and the camera is used to shoot the area where the drone is located in real time to obtain video image data.

[0064] Optionally, a gimbal is provided below the camera at the ground landing point, and the gimbal is used to control the rotation angle of the camera to facilitate tracking and shooting of the drone.

[0065] S2. Identify the drone from the video image data and locate the drone; continuously track the drone and update the flight trajectory of the drone; wherein, when the drone disappears from the video image data, use a trajectory prediction algorithm to predict the possible location of the drone, and continuously track the drone based on the possible location.

[0066] Specifically, identifying the drone from the video image data and locating the drone includes:

[0067] S201, preprocessing the video image data to obtain continuous video frame data;

[0068] Specifically, the input video image data is converted into a format and size that can be processed by the YOLO algorithm.

[0069] As an example, read the video image data and break it into consecutive frames. Resize each frame to match the size of the YOLO algorithm (such as 416x416 or 640x640).

[0070] In an optional embodiment, the video image data is also subjected to preprocessing steps such as normalization and denoising.

[0071] S202, inputting the video frame data into the convolutional neural network in the YOLO algorithm for feature extraction;

[0072] As an example, convolutional neural networks extract high-level features (such as shape, texture, etc.) from images through structures such as convolutional layers and pooling layers.

[0073] S203, based on the extracted features, dividing the video frame data into a number of grids and predicting a bounding box and a corresponding confidence level in each grid;

[0074] For example, the image can be divided into S×S grids, and each grid predicts B bounding boxes and their corresponding confidence scores. The bounding box is used to select the target in the image (in this case, the drone), and the confidence score indicates the probability of the target existing in the bounding box and the degree of fit between the bounding box and the true target.

[0075] S204, predicting a category probability vector of each bounding box based on the extracted features;

[0076] Specifically, for each bounding box, the YOLO algorithm outputs a class probability vector, which indicates the probability that the object in the bounding box belongs to each class. In the drone recognition task of this scheme, the class probability vector contains a higher probability of the "drone" class.

[0077] S205, obtaining a number of predicted drone position coordinates according to the category probability vector and confidence score of each bounding box;

[0078] The center coordinates (x, y), width w, and height h of the bounding box are calculated using a simplified method to represent the predicted position of the drone in the image. It should be noted that the width w and height h of the predicted bounding box for each grid cell are relative to the entire image size. The center coordinates (x, y) of the bounding box are offset from the upper left corner of the grid cell and are normalized (i.e., the value is between 0 and 1).

[0079] In an optional embodiment, the simplified method is as follows:

[0080] x=sigmoid(tx)+cx

[0081] y=sigmoid(ty)+cy

[0082] w=pw*exp(tw)

[0083] h = ph*exp(th)

[0084] Where x and y are the coordinates of the center of the bounding box; sigmoid represents the Sigmoid function; (tx,ty) is the predicted coordinate offset, (cx,cy) is the coordinate of the upper left corner of the grid cell (normalized), (pw,ph) is the width and height of the prior bounding box (anchorbox), (tw,th) is the width and height scaling factor learned by the network; w and h represent the width and height of the bounding box, respectively; exp represents the exponential function.

[0085] It should be noted that each bounding box has a confidence score, which indicates the probability of the existence of an object in the bounding box and the accuracy of the prediction. The confidence score is the product of the probability of the existence of an object in the bounding box and the IoU (intersection over union) between the bounding box and the true object.

[0086] In an optional embodiment, the simplified formula of confidence is as follows:

[0087] confidence=Pr(Object)×IoU(Predicted, Truth)

[0088] Among them, confidence is the confidence; Pr(Object) represents the probability that there is a target (drone) in the bounding box; IoU(Predicted, Truth) represents the intersection over union ratio between the bounding box and the true target.

[0089] S206: Use a non-maximum prediction algorithm to process the predicted drone position coordinates to obtain the final drone position coordinates.

[0090] Specifically, the non-maximum suppression algorithm compares the confidence scores and overlap levels of the bounding boxes and selects the optimal bounding box as the final detection result.

[0091] Specifically, the drone is continuously tracked and its flight trajectory is updated, including:

[0092] S211. In the initial frame of the video image data, the YOLO algorithm is used to identify and locate the drone, and the drone bounding box information is obtained as the tracking bounding box, and a unique ID is assigned to the drone for subsequent tracking.

[0093] S212, in the subsequent frames of the video image data, the YOLO algorithm is used again to detect all possible targets. Then, the position relationship (such as IoU value) and category consistency between these targets and the known drones in the previous frame are calculated to perform target matching. The target that matches the drone ID in the previous frame is the drone in the current frame.

[0094] S213: Update the trajectory information (such as position, speed, etc.) of the drone according to the matching result. At the same time, the flight attitude, speed and other parameters of the drone can also be calculated to provide a basis for subsequent decision-making and control.

[0095] Preferably, if the drone temporarily disappears or cannot be detected in the video frame data, a trajectory prediction algorithm can be used to predict the possible location of the drone and continue tracking it in subsequent frames.

[0096] In an optional embodiment, the trajectory prediction algorithm may adopt an extended Kalman filter method, a Kalman filter method, or an unscented Kalman filter method.

[0097] Kalman filtering is a method for estimating the state of a linear dynamic system. It uses a recursive approach to estimate the state of a system and updates these estimates through measurements, using a dynamic model of the system to predict the next state.

[0098] In this solution, Kalman filtering is used as an example to explain drone tracking:

[0099] 1. Initialization

[0100] Define the system state: First, define the state vector x_0 of the drone, including the position (such as x,y,z coordinates), speed ( v x ,v y ,v z ) etc. Optional embodiments also include other states such as acceleration, posture, etc.

[0101] Initialize State Estimate: Set the initial state estimate The initial value of .

[0102] Initialize error covariance: Set the initial error covariance matrix P 0, indicating the uncertainty in the estimate of the initial state.

[0103] Set process noise covariance Q : Set the process noise covariance matrix according to the uncertainty of the system dynamics Q .

[0104] Set the measurement noise covariance R : Set the measurement noise covariance matrix according to the uncertainty of the measurement equipment R .

[0105] Define the system model: determine the state transition matrix A and control matrix B .

[0106] Defining the measurement model: Determining the measurement matrix H , used to describe how states are mapped to measurements.

[0107] 2. Prediction Step

[0108] At each time step k , use the state estimate at the previous moment to predict the state at the current moment:

[0109] Prediction status:

[0110] Forecast error covariance:

[0111] in: Indicates that before obtaining a new measurement value, the time step k The prediction of the state vector of A represents the state transfer matrix; Represents the previous time step k −1 state estimate; B represents the control matrix; u k−1 Represents the previous time step k −1 control input; P k − Represents the prediction error covariance matrix, which represents the prediction state x k − estimates of uncertainty; P k−1 Represents the previous time step k The error covariance matrix is ​​−1; A T Represents the state transfer matrix A The transpose of Q represents the process noise covariance matrix, which represents the uncertainty in the system dynamics.

[0112] 3. Update steps

[0113] When a new measurement value z is received k , use it to update the state estimate and the error covariance:

[0114] Calculate the Kalman gain:

[0115] in: K k represents the Kalman gain, which is used to weigh the relative importance between the predicted state and the measured value. H Represents the measurement matrix, which describes how the system states are mapped to measurement values. H T Represents the measurement matrix H The transpose of . R represents the measurement noise covariance matrix, which represents the uncertainty in the measurement process. ( HP k − H T + R ) −1 Represents the inverse of the measurement prediction error covariance, which is used to calculate the Kalman gain.

[0116] Update the state estimate:

[0117] in, Represents the time step k The state estimate combines the predicted state and the new measurements. Represents the time step k The measured value. Represents the predicted measurement value, that is, according to the predicted state and the measurement matrix H Computes a prediction of the measured value. Represents the weight of the measurement residual (the difference between the actual measurement and the predicted measurement) used to correct the predicted state.

[0118] Update the error covariance:

[0119] in, P k Represents the time step k The error covariance matrix represents the state estimation uncertainty estimate. I Represents the identity matrix. Used to adjust the prediction error covariance matrix based on the Kalman gains and the measurement matrix.

[0120] 4. Repeat prediction and update

[0121] Repeat the above prediction and update steps, and continuously update the drone's state estimate as new measurements are acquired.

[0122] In a preferred embodiment, it also includes the following:

[0123] 5. Fusion of other sensor data

[0124] GPS data: Use GPS location data as one of the measurements and fuse it with visual or other sensor data through Kalman filtering to improve the accuracy of position estimation.

[0125] IMU data: Use IMU data (such as accelerometers and gyroscopes) to estimate the attitude and velocity of the drone and fuse it with the position estimate through Kalman filtering.

[0126] S3. Determine the descent gradient of the UAV based on the flight trajectory of the UAV.

[0127] Specifically, the descent phase of the drone is determined based on the flight trajectory of the drone. For example, it can be identified by monitoring the changes in altitude data. When the altitude data begins to drop continuously, it is considered that the drone has entered the descent phase. Select the starting point (which can be the position point when the descent starts) and the end point (ground landing point) during the descent phase. Use the altitude data of the starting point and the end point to calculate the descent height (Hdescent), and calculate the horizontal distance (Dhorizontal) between the starting point and the end point based on the position coordinates. Finally, calculate the descent gradient (Gradient) based on the descent height and horizontal distance. The descent gradient calculation formula is: The descent height is the difference between the current altitude of the drone and the altitude of the landing point, and the horizontal distance is the straight-line distance from the drone to the landing point.

[0128] In the optional scheme, after the descent gradient is calculated, it can be adjusted according to factors such as the speed, weight, hanging equipment, weather, etc. of the drone to ensure a safe landing.

[0129] In an alternative solution, the current position of the drone (X_current, Y_current) is determined using GPS on the drone, etc. The current altitude of the drone (H_current) is measured using a barometer, radar altimeter, or other sensor, etc. The coordinates of the landing point on the ground are determined (X_landing, Y_landing).

[0130] The height of the landing point (H_landing) can be obtained based on the terrain information of the landing area or pre-measured data.

[0131] According to the current position of the drone (X_current, Y_current) and the landing point position (X_landing, Y_landing), calculate the straight-line distance between the two, that is, the horizontal distance (D_horizontal).

[0132]

[0133] The descent height (H_descent) is the difference between the current altitude of the drone and the altitude of the landing point.

[0134] H_descent=H_current−H_landing

[0135] S4. Based on the descent gradient and the coordinates of the ground landing point, a path planning algorithm is used to plan the optimal landing route of the drone.

[0136] By calculating the descent gradient, a suitable descent path can be planned for the drone, ensuring that the drone can land at the designated location in a safe and stable manner.

[0137] For example, in this solution, a path planning algorithm (such as A* algorithm, RRT algorithm, etc.) is used to generate a route according to the descent gradient and the landing point position.

[0138] In a preferred embodiment, when the UAV lands according to the optimal landing route, the surrounding environment data is acquired and the surrounding environment data is recognized using a preset recognition model; when obstacles are recognized, the route is replanned based on a path planning algorithm.

[0139] Specifically, when the UAV lands according to the optimal landing route, surrounding environment data is obtained; wherein the surrounding environment data includes: a first image taken by a camera carried by the UAV in the direction of the UAV's navigation, and video image data taken from the ground landing point upward to the area where the UAV is located; based on the first image and video image data, obstacle identification is performed using a preset recognition model to obtain a first recognition result and a second recognition result; based on the first recognition result, it is determined whether there is an obstacle in the direction of the UAV's navigation, and based on the second recognition result, it is determined whether there is an obstacle in the area where the UAV is located; when either of the first recognition result and the second recognition result shows that there is an obstacle, the route is replanned based on a path planning algorithm.

[0140] As an example, the preset recognition model may be a deep learning model, etc.

[0141] In an optional embodiment, during the landing process of the drone, the changes in the surrounding environment are detected in real time through the camera and the video sent back by the drone. If obstacles or other factors affecting the landing are found, the landing route is adjusted in time to ensure that the drone can land safely.

[0142] The drone captures environmental information in real time through the camera it carries, and uses a deep learning-based recognition model to detect obstacles, such as trees, buildings, or other flying objects.

[0143] Drones can also be equipped with lidar, ultrasonic sensors, etc. to obtain more accurate distance and location information.

[0144] In the process of drone obstacle avoidance, the RRT algorithm can help the drone quickly replan its path after detecting an obstacle. The specific method is as follows:

[0145] Assume that the current heading of the drone is θ_current (in degrees) and the target heading is θ_goal (in degrees). Since the angle is periodic (i.e. 360° and 0° are the same), this solution needs to consider the "surrounding" situation of the angle. This is achieved through modular operation, and the heading difference Δθ is calculated as:

[0146]

[0147] Adjusting in radians is done the same way as adjusting in degrees, but the base of the modulo operation is 2π instead of 360°.

[0148] Specifically, when the drone encounters an obstacle during the return flight, the obstacle avoidance program will be triggered first to perform emergency maneuvers. At the same time, the drone's path planning system will replan the route based on the current environment and drone status. This process involves calling algorithms such as A* or RRT to generate a new flight path. The new flight path will be sent to the drone's flight control system for execution. The flight control system will adjust the drone's flight attitude, speed, heading and other parameters based on the new path to ensure that the drone can safely avoid obstacles and reach the landing point.

[0149] Example 2

[0150] like Figure 2 As shown, based on the same inventive concept as the above embodiment, the present invention also provides a UAV recovery device based on a visual algorithm, which is characterized by comprising:

[0151] The image acquisition module is used to obtain the real-time coordinates of the drone during flight and the corresponding coordinates of the ground landing point; when the distance between the real-time coordinates and the coordinates of the ground landing point is within a preset range, the relative position of the drone and the ground landing point is determined; based on the relative position, the area where the drone is located is photographed in real time from the ground landing point upwards to obtain video image data;

[0152] A drone identification and tracking module is used to identify a drone from the video image data and locate the drone; continuously track the drone and update the flight trajectory of the drone; wherein, when the drone disappears from the video image data, a trajectory prediction algorithm is used to predict the possible location of the drone, and the drone is continuously tracked based on the possible location;

[0153] A descent gradient calculation module is used to determine the descent gradient of the UAV based on the flight trajectory of the UAV;

[0154] The route planning module is used to plan the optimal landing route of the drone based on the descent gradient and the coordinates of the ground landing point using a path planning algorithm.

[0155] Example 3

[0156] like Figure 3 As shown, the present invention also provides an electronic device 100 for implementing a drone recovery method based on a visual algorithm in the above embodiment;

[0157] The electronic device 100 includes a memory 101 , at least one processor 102 , a computer program 103 stored in the memory 101 and executable on the at least one processor 102 , and at least one communication bus 104 .

[0158] The memory 101 can be used to store a computer program 103. The processor 102 implements the steps of a drone recovery method based on a visual algorithm in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0159] The processor 102 may be an AI processor.

[0160] The memory 101 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 may include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0161] At least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0162] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a method for recovering a drone based on a visual algorithm, and the processor 102 can execute the plurality of instructions to implement:

[0163] Acquire the real-time coordinates of the drone during flight and the corresponding coordinates of the ground landing point; when the distance between the real-time coordinates and the coordinates of the ground landing point is within a preset range, determine the relative position of the drone and the ground landing point; based on the relative position, take real-time photos of the area where the drone is located from the ground landing point upwards to obtain video image data;

[0164] Identify the drone from the video image data and locate the drone; continuously track the drone and update the flight trajectory of the drone; wherein, when the drone disappears from the video image data, use a trajectory prediction algorithm to predict the possible location of the drone, and continuously track the drone based on the possible location;

[0165] Determine the descent gradient of the drone based on its flight trajectory;

[0166] Based on the descent gradient and the coordinates of the ground landing point, a path planning algorithm is used to plan the optimal landing route of the UAV.

[0167] Example 4

[0168] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0169] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0173] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A drone recovery method based on visual algorithm, characterized in that: The following steps are involved: Acquire the real-time coordinates of the drone during flight and the corresponding coordinates of the ground landing point; when the distance between the real-time coordinates and the coordinates of the ground landing point is within a preset range, determine the relative position of the drone and the ground landing point; based on the relative position, take real-time photos of the area where the drone is located from the ground landing point upwards to obtain video image data; Identify the drone from the video image data and locate the drone; continuously track the drone and update the flight trajectory of the drone; wherein, when the drone disappears from the video image data, use a trajectory prediction algorithm to predict the possible location of the drone, and continuously track the drone based on the possible location; Determine the descent gradient of the drone based on its flight trajectory; Based on the descent gradient and the coordinates of the ground landing point, a path planning algorithm is used to plan the optimal landing route for the drone; The drone is identified from the video image data and the drone is positioned, including: preprocessing the video image data to obtain continuous video frame data; inputting the video frame data into a convolutional neural network in a YOLO algorithm to extract features; based on the extracted features, the video frame data is divided into a plurality of grids and a bounding box and a corresponding confidence score are predicted in each grid; based on the extracted features, a category probability vector of each bounding box is predicted; according to the category probability vector and the confidence score of each bounding box, a plurality of predicted drone position coordinates are obtained; and a non-maximum prediction algorithm is used to process the predicted drone position coordinates to obtain the final drone position coordinates; Continuously track the drone and update the flight trajectory of the drone, including: identifying and locating the drone, obtaining the drone bounding box information as the tracking bounding box, and assigning a unique ID to the drone; in the subsequent frames of the video image data, use the YOLO algorithm again to detect all possible targets, and perform target matching by calculating the positional relationship and category consistency between all possible targets and the known drones in the previous frame; find the target that matches the drone ID in the previous frame, which is the drone in the current frame; based on the matching results, update the drone's trajectory information; After the step of planning the optimal landing route of the drone using a path planning algorithm, the drone descends according to the optimal landing route; wherein, when the drone lands according to the optimal landing route, surrounding environment data is obtained, and the surrounding environment data is identified using a preset recognition model; when an obstacle is identified, the route is replanned based on the path planning algorithm; specifically, the following steps are performed: when the drone lands according to the optimal landing route, surrounding environment data is obtained; wherein, the surrounding environment data includes: a first image taken in the direction of the drone's navigation using a camera carried by the drone, and video image data taken from the ground landing point upwards to the area where the drone is located; based on the first image and video image data, obstacle identification is performed using a preset recognition model to obtain a first recognition result and a second recognition result; based on the first recognition result, it is determined whether there is an obstacle in the direction of the drone's navigation, and based on the second recognition result, it is identified whether there is an obstacle in the area where the drone is located; when either of the first recognition result and the second recognition result shows that there is an obstacle, the route is replanned based on the path planning algorithm.

2. The method for recovering a drone based on a visual algorithm according to claim 1, characterized in that: Based on the relative position, in the step of taking real-time photos of the area where the drone is located from the ground landing point upward to obtain video image data: When the distance of the relative position is less than or equal to the preset range, the camera at the ground landing point starts to shoot the area where the drone is located; otherwise, the camera does not work.

3. The UAV recovery method based on visual algorithm according to claim 1 is characterized in that: According to the flight trajectory of the UAV, determine the descent gradient of the UAV, including: Determine the descent phase of the drone based on the flight trajectory of the drone; Select the starting and ending points within the descent phase; Use the height data of the starting point and the end point to calculate the descent height, calculate the horizontal distance between the starting point and the end point based on the position coordinates, and calculate the descent gradient based on the descent height and the horizontal distance.

4. The method for recovering a drone based on a visual algorithm according to claim 1, characterized in that: In the step of planning the optimal landing route of the UAV using a path planning algorithm based on the descent gradient and the coordinates of the ground landing point, the path planning algorithm is an A* algorithm or an RRT algorithm.

5. A UAV recovery device based on visual algorithm, characterized in that: include: Image acquisition module, used to obtain the real-time coordinates of the drone during flight and the corresponding ground landing point coordinates; When the distance between the real-time coordinates and the coordinates of the ground landing point is within a preset range, the relative position of the UAV and the ground landing point is determined; based on the relative position, the area where the UAV is located is photographed in real time from the ground landing point upward to obtain video image data; A drone identification and tracking module is used to identify a drone from the video image data and locate the drone; continuously track the drone and update the flight trajectory of the drone; wherein, when the drone disappears from the video image data, a trajectory prediction algorithm is used to predict the possible location of the drone, and the drone is continuously tracked based on the possible location; A descent gradient calculation module is used to determine the descent gradient of the UAV based on the flight trajectory of the UAV; The route planning module is used to plan the optimal landing route of the drone based on the descent gradient and the coordinates of the ground landing point using a path planning algorithm; The drone is identified from the video image data and the drone is positioned, including: preprocessing the video image data to obtain continuous video frame data; inputting the video frame data into a convolutional neural network in a YOLO algorithm to extract features; based on the extracted features, the video frame data is divided into a plurality of grids and a bounding box and a corresponding confidence score are predicted in each grid; based on the extracted features, a category probability vector of each bounding box is predicted; according to the category probability vector and the confidence score of each bounding box, a plurality of predicted drone position coordinates are obtained; and a non-maximum prediction algorithm is used to process the predicted drone position coordinates to obtain the final drone position coordinates; Continuously track the drone and update the flight trajectory of the drone, including: identifying and locating the drone, obtaining the drone bounding box information as the tracking bounding box, and assigning a unique ID to the drone; in the subsequent frames of the video image data, use the YOLO algorithm again to detect all possible targets, and perform target matching by calculating the positional relationship and category consistency between all possible targets and the known drones in the previous frame; find the target that matches the drone ID in the previous frame, which is the drone in the current frame; based on the matching results, update the drone's trajectory information; After the step of planning the optimal landing route of the drone using a path planning algorithm, the drone descends according to the optimal landing route; wherein, when the drone lands according to the optimal landing route, surrounding environment data is obtained, and the surrounding environment data is identified using a preset recognition model; when an obstacle is identified, the route is replanned based on the path planning algorithm; specifically, the following steps are performed: when the drone lands according to the optimal landing route, surrounding environment data is obtained; wherein, the surrounding environment data includes: a first image taken in the direction of the drone's navigation using a camera carried by the drone, and video image data taken from the ground landing point upwards to the area where the drone is located; based on the first image and video image data, obstacle identification is performed using a preset recognition model to obtain a first recognition result and a second recognition result; based on the first recognition result, it is determined whether there is an obstacle in the direction of the drone's navigation, and based on the second recognition result, it is identified whether there is an obstacle in the area where the drone is located; when either of the first recognition result and the second recognition result shows that there is an obstacle, the route is replanned based on the path planning algorithm.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the drone recovery method based on the visual algorithm described in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the drone recovery method based on a visual algorithm described in any one of claims 1 to 4 is implemented.

Images