A Fault-Tolerant Navigation Method and System for Unmanned Aerial Vehicles Based on the Combination of Communication and Sensing

Abstract

The present invention discloses a fault-tolerant navigation method and system for an unmanned aerial vehicle, belonging to the technical field of unmanned aerial vehicles, and specifically relating to a fault-tolerant navigation method and system for an unmanned aerial vehicle based on the combination of communication and perception. The present invention controls the unmanned aerial vehicle to fly in a designed route to collect image data, performs key object annotation and feature extraction on these image data, and enters the information into a template feature library; uses a monocular camera to collect information in the scene and real-time detects and extracts feature points; performs feature matching between the feature points and the template feature library, and outputs a feature matching result; determines whether to perform motion adjustment based on the feature matching result; so that the unmanned aerial vehicle maintains in the correct flight route until it reaches the destination. Thus, the present invention can solve the problems of GPS loss and inertial navigation cumulative error, improve the navigation success rate of the unmanned aerial vehicle, and the present invention can run on an embedded platform and has a low cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicles, and particularly relates to a fault-tolerant navigation method and system for unmanned aerial vehicles based on the combination of communication and perception. Background Technique

[0002] Currently, unmanned aerial vehicles are widely used in military and civilian fields, and the research on unmanned aerial vehicle navigation has gradually become a hot topic. There are various ways of unmanned aerial vehicle navigation. However, since the single-sensor navigation method requires a specific usage environment and cannot provide long-term stable navigation in complex environments, in actual applications, multiple sensors are usually combined in the form of integrated navigation. Among them, the combination of absolute positioning and relative positioning is the dominant technical form, such as GPS + inertial navigation system.

[0003] Currently, the mainstream fault-tolerant navigation mainly starts from aspects such as sensor (data) fault detection, incorrect data masking, and algorithm optimization to reduce errors. The relevant inquiries are as follows:

[0004] Patent CN113514064A proposes a robust factor graph multi-source fault-tolerant navigation method. The main innovation lies in the acquisition of navigation sensor residuals, sensor fault detection, and targeted navigation optimization. Based on the fault information, the carrier navigation information is optimized and solved using the factor graph.

[0005] Patent CN110933597A proposes a Bluetooth-based multi-unmanned vehicle collaborative fault-tolerant navigation and positioning method. The main innovation lies in obtaining the positions and relative distances of other unmanned vehicles through Bluetooth, combining the position information provided by its own navigation system, and performing fault-tolerant fusion to finally correct the navigation and positioning results.

[0006] Patent CN110426032A proposes an analytical redundancy-based fault-tolerant navigation estimation method for aircraft. The main innovation lies in achieving fault location through a fault detection filter and completing navigation calculation after isolating the faulty sensor.

[0007] Patent CN110208843A proposes a fault-tolerant navigation method based on augmented pseudorange information assistance. The main innovation lies in performing Kalman filtering based on ephemeris data, pseudolite data, and inertial navigation data, and detecting satellite fault conditions and performing fault isolation through a multi-solution separation algorithm.

[0008] Patent CN109612459A proposes a fault-tolerant navigation method for inertial sensors of quadrotor aircraft based on a dynamic model. The main innovation lies in predicting the speed, position, etc. of the aircraft based on sensor data and a dynamic model, masking incorrect sensor information through a fault detection function and a fault location function, and finally correcting and fusing sub-filters.

[0009] In the above-mentioned existing technical solutions, whether it is a single navigation method or a multi-sensor integrated navigation method, the fault-tolerant navigation of drones or aircraft mainly focuses on solving the navigation deviation and error problems caused by faulty sensors or incorrect data. Usually, the fault points (sensors) are detected and judged, so as to shield or eliminate the incorrect data caused by the fault problems, and then correction methods or error reduction methods are designed to correct the errors, so as to normalize the navigation data and realize the self-fault-tolerant navigation strategy.

[0010] Due to the characteristic of cumulative error, the inertial navigation system cannot work independently for a long time; while GPS navigation is easily affected by the environment (such as forests, tunnels, urban building clusters), resulting in weakening or even loss of GPS signals; these two are typical technical problems in drone navigation. When a drone navigates in the normal ideal state, as Figure 1 and Figure 2 shown, only relying on inertial navigation and GPS can provide accurate real-time positions. Based on the pre-set route, ideal flight navigation from the starting point to the destination can be achieved. However, when the GPS signal is poor or fails, and affected by the cumulative error of inertial navigation, it is very likely that the destination cannot be reached smoothly. Summary of the Invention

[0011] Object of the Invention: To provide a method and system for fault-tolerant navigation of drones based on the combination of communication and perception, and solve the above-mentioned problems.

[0012] Technical Solution: A method and system for fault-tolerant navigation of drones based on the combination of communication and perception, the method includes:

[0013] Control the drone to fly in the designed route to collect image data, perform key object annotation and feature extraction on these image data, and enter the information into the template feature library;

[0014] Use a monocular camera to collect information in the scene and detect and extract feature points in real time;

[0015] Perform feature matching between the feature points and the template feature library, and output the feature matching result;

[0016] Judge whether to perform motion adjustment based on the feature matching result;

[0017] Perform the motion adjustment.

[0018] In a further embodiment, the key object annotation is to manually mark the invariant markers in the scene of the flight route, and the feature extraction is to detect and extract the feature points of all the markers.

[0019] In a further embodiment, the feature matching is performed based on key frames. Starting from when the UAV begins to detect image feature points, if there are similar feature points in the template feature library, that is, the number of similar feature points > 1, this frame of image is defined as the initial key frame. Subsequently, this frame is defined as a key frame again at an interval of 10 frames, and finally a set of key frame sequences is formed.

[0020] In a further embodiment, it is determined whether to perform motion adjustment based on the feature matching result according to the ratio of the number of similar feature points to the total number of feature points in the template feature library of this frame. If it is higher than the set ratio, it is determined that there is a high degree of similarity. At this time, it is judged that the UAV is still navigating and flying on the original set route. Otherwise, it is determined that the UAV has deviated from the expected route and motion adjustment is required.

[0021] In a further embodiment, the motion adjustment includes: autonomous motion adjustment and motion adjustment under remote instructions.

[0022] In a further embodiment, the autonomous motion adjustment divides the space directly in front of the UAV into four regions with the UAV as the center point and numbers them 1, 2, 3, 4. The regions have no set boundaries. The area search method for the UAV's motion adjustment is as follows:

[0023] Search the regions in the order of 1-2-3-4. At the same time, there is a judgment on the correctness of the motion adjustment strategy, that is, if the number of similar feature points increases in three consecutive key frames, it is determined that the motion adjustment strategy is correct; otherwise, it is incorrect. If it is incorrect, start searching the next region.

[0024] In a further embodiment, the motion adjustment under remote instructions can perform real-time positioning and mapping through the SLAM method. The cloud control platform can still view the flight situation of the UAV in the map in real time. As the final guarantee for motion adjustment, the motion adjustment under remote instructions means that an instruction is actively issued manually to make the UAV perform motion adjustment.

[0025] Second aspect:

[0026] A UAV fault-tolerant navigation system based on the combination of communication and perception is used to implement the above-mentioned UAV fault-tolerant navigation method based on the combination of communication and perception. The system includes:

[0027] A monocular camera for perceiving surrounding devices;

[0028] An inertial navigation module and a GPS module for providing the UAV with an accurate real-time position;

[0029] A wireless communication module for transmitting data information to the cloud control platform;

[0030] A real-time calculation module for detecting, extracting, and storing the features of invariant markers into a template feature library, detecting, extracting, and matching the features of the scene information collected by the monocular camera with the features in the template feature library, and judging the motion adjustment according to the matching result.

[0031] Beneficial effects: The present invention belongs to the technical field of unmanned aerial vehicles, and particularly relates to a fault-tolerant navigation method and system for unmanned aerial vehicles based on the combination of communication and perception. By visually perceiving whether the unmanned aerial vehicle deviates from the original established route and simultaneously considering the communication instructions to complete the motion adjustment, the unmanned aerial vehicle can be kept on the correct flight path until it reaches the destination. Therefore, the present invention can solve the problems of GPS loss and inertial navigation cumulative error, improve the navigation success rate of the unmanned aerial vehicle, and the present invention can run on an embedded platform with low cost. Description of the Drawings

[0032] Figure 1 It is a working scene diagram of the unmanned aerial vehicle.

[0033] Figure 2 It is a schematic diagram of the working method of the prior art unmanned aerial vehicle.

[0034] Figure 3 It is a schematic diagram of the navigation system of the present invention.

[0035] Figure 4 It is a schematic diagram of the working of the navigation method and system of the present invention.

[0036] Figure 5 It is a comparison diagram of the feature points detected by the unmanned aerial vehicle and the module feature library in the method of the present invention.

[0037] Figure 6 It is a schematic diagram of the autonomous motion adjustment method of the present invention.

[0038] Figure 7 It is a flowchart of the method of the present invention. Detailed Embodiments

[0039] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0040] A fault-tolerant navigation method for unmanned aerial vehicles based on the combination of communication and perception includes:

[0041] Controlling the unmanned aerial vehicle to fly in the designed route to collect image data, performing key object annotation and feature extraction on these image data, and inputting the information into the template feature library;

[0042] Collect information in the scene using a monocular camera, and detect and extract feature points in real time;

[0043] Perform feature matching between the feature points and the template feature library, and output the feature matching result;

[0044] Judge whether to perform motion adjustment based on the feature matching result;

[0045] Perform the motion adjustment.

[0046] According to Figures 5 to 7 , combining the above method, the implementation of the present invention mainly includes the following steps:

[0047] 1. Fly the drone by the operator in the designed route to collect image data, and perform key object annotation on these image data. Specifically: Manually mark the invariant markers in the scene of the flight route, mainly typical road traffic signs, etc. (Since the invariant markers are usually separated by a certain distance, this will make the marking workload less, so that even if the set route is long, it can still be completed); Detect and extract the feature points (ORB features) of all the markers, and input this kind of information into the template feature library.

[0048] 2. When the drone is flying in navigation, using the ORB-SLAM3 method, it can collect information in the scene using a monocular camera and detect and extract ORB feature points in real time.

[0049] 3. Match the feature points in step 2 with the template feature library. Since both are collected by the drone, the difficulty of the feature matching process is relatively small. The feature matching process is based on key frames, rather than performing feature matching on each frame of both sides (the key frames here are different from the key frames in ORB-SLAM3). The definition of the key frames in this method is: Starting from when the drone starts to detect image feature points, if there are similar feature points in the template feature library (that is, the number of similar feature points > 1), this frame of image is defined as the initial key frame, and then this frame is defined as a key frame again at an interval of 10 frames, and finally a set of key frame sequences is formed.

[0050] 4. Judge whether to perform motion adjustment based on the feature matching result. Specifically: If the proportion of the number of similar feature points in the total feature points of the template feature library of this frame is more than 90%, it is considered to have a high degree of similarity at this time, and it is judged that the drone is still flying in the original set route; on the contrary, if it is less than 90%, it means that the drone has deviated from the expected route and needs to perform motion adjustment.

[0051] 5. Motion adjustment. The motion adjustment is divided into two parts: autonomous motion adjustment and motion adjustment by receiving remote instructions.

[0052] (1) Autonomous motion adjustment: A drone usually has six degrees of freedom. However, in this case, complex situations are not considered. Instead, the space directly in front of the drone is divided into four regions (numbered as shown in the following figure) with the drone at the center point, and no boundaries are set for the regions.

[0053] Set the region search strategy for drone motion adjustment: Search the regions in the order of 1 - 2 - 3 - 4. Meanwhile, there is a judgment on the correctness of the motion adjustment strategy, that is, if the number of similar feature points increases in three consecutive key frames, then the motion adjustment strategy is determined to be correct; otherwise, it is incorrect. If it is incorrect, start searching the next region.

[0054] Specifically, an example is given below: For the first motion adjustment, fly towards the upper - right 1 - st region (default 45 - degree oblique forward). If the number of similar feature points increases in three consecutive key frames, then the motion adjustment strategy is determined to be correct. Immediately switch from the oblique flight to straight - line flight when the number starts to decrease. At this time, it is considered to be on the correct established flight route. Conversely, if the number of similar feature points decreases when flying towards the 1 - st region, then the motion adjustment strategy is determined to be incorrect. Then, update the positions of the four regions with the drone as the center. At this time, fly towards the 2 - nd region and judge the number of feature points again. If the motion adjustment strategy is judged to be incorrect, conduct the next region search, that is, the 3 - rd region, and so on until the 4 - th region.

[0055] (2) Motion adjustment under remote instructions: The drone is equipped with a wireless communication module. The cloud control platform can receive the information sent by the drone at any time. When the GPS information is affected or missing, the drone cannot provide accurate position information (such as longitude and latitude) to the cloud control platform. However, through the SLAM method, real - time positioning and mapping can be carried out, and the cloud control platform can still view the flight situation (real - time position and motion trajectory) of the drone on the map in real - time. As the final guarantee for motion adjustment, motion adjustment under remote instructions means that an operator actively issues instructions to make the drone perform motion adjustment.

[0056] Through the combination of the above two motion adjustment methods, the drone will return to the original route after deviating from the established route and move forward towards the destination.

[0057] As Figure 3 As shown in the figure, a drone fault - tolerant navigation system based on the combination of communication and perception is used to implement the above - mentioned drone fault - tolerant navigation method based on the combination of communication and perception. The system includes:

[0058] A monocular camera for perceiving surrounding devices;

[0059] An inertial navigation module and a GPS module for providing accurate real - time position for the drone;

[0060] A wireless communication module for data information transmission to the cloud control platform;

[0061] A real-time calculation module, which is used for feature detection, extraction and storage of invariant markers into a template feature library, feature detection, extraction of the scene information collected by the monocular camera, and feature matching with the template feature library, and makes a motion adjustment judgment according to the matching result.

[0062] Obviously, the above embodiments are merely examples for clear illustration and not limitations on the implementation manners. For those of ordinary skill in the art, other different forms of changes or alterations can be made based on the above description. It is not necessary and impossible to list all the implementation manners here. And the obvious changes or alterations derived therefrom still fall within the protection scope of the present invention.

Claims

1. A fault-tolerant navigation method for unmanned aerial vehicles based on the combination of communication and perception, characterized in that, The method includes: Controlling a drone to fly in a designed route to collect image data, performing key object annotation and feature extraction on these image data, and entering the information into a template feature library; Using a monocular camera to collect information in the scene and real-time detecting and extracting feature points; Performing feature matching between the feature points and the template feature library and outputting a feature matching result; Judging whether to perform motion adjustment based on the feature matching result; Performing the motion adjustment; The motion adjustment includes: autonomous motion adjustment and motion adjustment under remote instructions; The autonomous motion adjustment divides the space directly in front of the drone centered on the drone into four regions and numbers them 1, 2, 3, 4. The regions have no set boundaries. The area search method for the drone's motion adjustment is as follows: Search the regions in the order of 1-2-3-4. At the same time, there is a judgment on the correctness of the motion adjustment strategy, that is, if the number of similar feature points increases in three consecutive key frames, it is determined that the motion adjustment strategy is correct, otherwise it is incorrect; if it is incorrect, start searching the next region; The motion adjustment under remote instructions can perform real-time positioning and mapping through the SLAM method. The cloud control platform can still view the flight situation of the drone on the map in real time. As the final guarantee for motion adjustment, the motion adjustment under remote instructions means that an instruction is actively issued manually to make the drone perform motion adjustment.

2. The method for fault-tolerant navigation of an unmanned aerial vehicle based on the combination of communication and perception according to claim 1, wherein The key object annotation is to manually mark the invariant markers in the scene of the flight route, and the feature extraction is to detect and extract the feature points of all the markers.

3. A fault-tolerant navigation method for an unmanned aerial vehicle based on the combination of communication and perception according to claim 1, characterized in that, The feature matching is performed according to the key frames. Starting from when the drone starts to detect image feature points, if there are similar feature points in the template feature library, that is, the number of similar feature points > 1, this frame of image is defined as the initial key frame. Subsequently, this frame is defined as a key frame again at an interval of 10 frames, and finally a set of key frame sequences is formed.

4. A fault-tolerant navigation method for an unmanned aerial vehicle based on the combination of communication and perception according to claim 1, characterized in that, Judging whether to perform motion adjustment based on the feature matching result is determined according to the ratio of the number of similar feature points to the total number of feature points in the template feature library of this frame. If it is higher than the set ratio, it is determined that there is a high degree of similarity. At this time, it is judged that the drone is still flying along the original set route. Otherwise, it is determined that the drone has deviated from the expected route and motion adjustment is required.

5. A fault-tolerant navigation system for unmanned aerial vehicles based on the combination of communication and perception, which is used to implement the fault-tolerant navigation method for unmanned aerial vehicles based on the combination of communication and perception according to any one of claims 1 to 4, characterized in that, The system includes: A monocular camera for sensing surrounding devices; An inertial navigation module and a GPS module for providing the accurate real-time position of the drone; A wireless communication module for transmitting data information to the cloud control platform; A real-time calculation module for feature detection, extraction and storage of invariant markers into the template feature library, feature detection, extraction of the information collected by the monocular camera in the scene, and feature matching with the template feature library, and judging motion adjustment according to the matching result.

Images