Unmanned aerial vehicle real-time visual positioning method and system, electronic device and storage medium

Abstract

The embodiment of the application provides a kind of unmanned plane real-time vision positioning method, system, electronic equipment and storage medium, belong to vision positioning technical field.The scheme is by obtaining the flight detection sequence of take-off stage;According to the current position information and preset landing point, control unmanned plane to descend in target landing area;According to current image information, carry out vision reference detection, judge whether landing mark is detected;If landing mark is not detected, judge whether there is reference mark information according to current flight height;If there is reference mark information, according to current image information, carry out feature point matching based on reference mark information, determine target positioning point;According to target positioning point, control unmanned plane to land to ground.The application can reduce the positioning error caused by environmental interference, improve positioning accuracy.

Description

Technical Field

[0001] This application relates to the field of visual positioning technology, and in particular to a real-time visual positioning method, system, electronic device and storage medium for unmanned aerial vehicles. Background Technology

[0002] Precise landing algorithms for drones primarily rely on machine vision recognition systems and GPS / RTK positioning systems. However, current drone vision recognition and positioning systems face several challenges in practical applications. When drones fly at high altitudes, the quality of ground images captured by cameras is often insufficient for precise positioning. Current vision recognition systems typically rely on image feature point matching for positioning, but at high altitudes or long distances, the number of feature points in the images is small and inconspicuous, making it impossible to guarantee the accuracy of the matching algorithm.

[0003] In addition, the GPS positioning system of drones has a drift problem. GPS errors not only affect the accuracy of the flight path, but may also cause the drone to deviate from the target position when landing.

[0004] Therefore, due to factors such as poor camera imaging quality, ineffective algorithm judgment, and GPS signal drift, drones struggle to achieve high-precision positioning and landing. Summary of the Invention

[0005] The main objective of this application is to propose a real-time visual positioning method, system, electronic device, and storage medium for unmanned aerial vehicles (UAVs), aiming to reduce positioning errors caused by environmental interference and improve positioning accuracy.

[0006] To achieve the above objectives, one aspect of this application proposes a real-time visual positioning method for unmanned aerial vehicles (UAVs), the method comprising:

[0007] Acquire a flight detection sequence during the takeoff phase, wherein the flight detection sequence includes first image information at different altitudes and corresponding first flight altitude and first marker information;

[0008] The drone is controlled to descend within the target landing area based on its current location information and the preset landing point during the descent phase.

[0009] Visual baseline detection is performed based on the current image information to determine whether landing markers have been detected.

[0010] If the landing marker is not detected, the presence of reference marker information is determined based on the current flight altitude, wherein the reference marker information is the first marker information associated with the current flight altitude in the flight detection sequence;

[0011] If the reference marker information exists, then based on the reference marker information, feature point matching is performed according to the current image information to determine the target location point;

[0012] The drone is controlled to land on the ground based on the target positioning point.

[0013] In some embodiments, obtaining the flight detection sequence during the takeoff phase includes the following steps:

[0014] Initialize the flight detection sequence;

[0015] The drone's ascent is controlled based on a preset ascent altitude;

[0016] Acquire the first image information of the preset altitude point and the corresponding first flight altitude;

[0017] Based on the first image information, feature points are extracted to obtain the first feature points;

[0018] Based on the first image information, visual reference detection is performed to determine whether a landing marker is detected.

[0019] If the landing marker is detected, the center point coordinates and diagonal point coordinates of the landing marker are extracted from the first image information to obtain the first marker information;

[0020] The first image information, the first flight altitude, the first feature point, and the first marker information are recorded into the flight detection sequence, and then the drone is controlled to ascend based on a preset ascent altitude until the first flight altitude equals the preset ascent altitude.

[0021] In some embodiments, the step of performing visual reference detection based on current image information to determine whether a landing marker has been detected includes the following steps:

[0022] The current image information is processed for grayscale and edge extraction to identify potential quadrilateral regions and obtain candidate regions.

[0023] The candidate regions are subjected to perspective transformation correction to obtain standard pattern data;

[0024] Extract the encoded bit array of the standard pattern data and decode it to obtain the current tag identifier;

[0025] Calculate the Hamming distance based on the current tag identifier and the landing tag identifier corresponding to the landing marker;

[0026] Determine whether the Hamming distance is lower than a preset bit threshold;

[0027] If the Hamming distance is lower than the preset bit threshold, then the landing marker is determined to have been detected; otherwise, the landing marker is determined not to have been detected.

[0028] In some embodiments, the landing marker includes an outer marker and a nested marker, the nested marker being embedded within the outer marker. The step of performing visual reference detection based on current image information to determine whether a landing marker has been detected includes the following steps:

[0029] Visual benchmark detection is performed based on the current image information to determine whether the outer and nested markers are detected.

[0030] If the total number of the outer and nested markers detected is greater than the preset detection number, then it is determined that the landing marker has been detected; otherwise, it is determined that the landing marker has not been detected.

[0031] In some embodiments, determining whether reference marker information exists based on the current flight altitude includes the following steps:

[0032] Determine the relevant altitude range based on the current flight altitude;

[0033] Determine whether the first flight altitude exists within the associated altitude range in the flight detection sequence;

[0034] If the first flight altitude is within the associated altitude range, then the first marker information corresponding to the first flight altitude is determined as reference marker information.

[0035] In some embodiments, determining the target location point by performing feature point matching based on the reference marker information and the current image information includes the following steps:

[0036] Feature points are extracted from the current image information to obtain the current feature points;

[0037] The corresponding first feature point is queried from the flight detection sequence based on the reference marker information;

[0038] Based on the current feature point and the first feature point, feature point matching is performed to obtain a matching feature point pair;

[0039] Based on the matching feature point pairs, outer points are removed to obtain matching inner point pairs;

[0040] Based on the matched inner point pairs, perform diagonal point verification to obtain the matched diagonal points;

[0041] The geometric center is calculated based on the coordinates of the diagonal points of the matched diagonal points to obtain the target positioning point.

[0042] In some embodiments, the landing marker includes an outer marker and a nested marker, the nested marker being embedded within the outer marker. The UAV real-time visual positioning method further includes the following steps:

[0043] If the landing marker is detected, determine whether the peripheral marker exists within the landing marker;

[0044] If the surrounding markers exist, the center point coordinates of the surrounding markers are extracted from the current image information, and the target positioning point is obtained based on the center point coordinates of the surrounding markers.

[0045] If the outer marker does not exist, the center point coordinates of the nested marker are extracted from the current image information, and the target positioning point is obtained based on the center point coordinates of the nested marker.

[0046] To achieve the above objectives, another aspect of this application proposes a real-time visual positioning system for unmanned aerial vehicles (UAVs), the system comprising:

[0047] The first module is used to acquire the flight detection sequence during the takeoff phase, wherein the flight detection sequence includes first image information at different altitudes and corresponding first flight altitude and first marker information;

[0048] The second module is used to control the drone to descend in the target landing area based on the current position information and the preset landing point during the landing phase.

[0049] The third module is used to perform visual reference detection based on the current image information to determine whether a landing marker has been detected.

[0050] The fourth module is used to determine whether reference marker information exists based on the current flight altitude if the landing marker is not detected, wherein the reference marker information is the first marker information associated with the current flight altitude in the flight detection sequence;

[0051] The fifth module is used to determine the target location point by performing feature point matching based on the current image information, if the reference marker information exists.

[0052] The sixth module is used to control the UAV to land on the ground based on the target positioning point.

[0053] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0054] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0055] The embodiments of this application include at least the following beneficial effects: This application provides a real-time visual positioning method, system, electronic device, and storage medium for unmanned aerial vehicles (UAVs). This solution acquires the flight detection sequence during the takeoff phase; controls the UAV to descend in the target landing area based on the current position information and a preset landing point during the landing phase; performs visual reference detection based on the current image information to determine whether a landing marker is detected; if no landing marker is detected, it determines whether reference marker information exists based on the current flight altitude; if reference marker information exists, it determines the target positioning point based on feature point matching using the reference marker information and the current image information; and controls the UAV to land on the ground based on the target positioning point. This application can reduce positioning errors caused by environmental interference and improve positioning accuracy. Attached Figure Description

[0056] Figure 1 This is a flowchart of the real-time visual positioning method for unmanned aerial vehicles provided in the embodiments of this application;

[0057] Figure 2 This is a schematic diagram of the landing marker provided in an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of feature point matching provided in an embodiment of this application;

[0059] Figure 4 This is a flowchart of the takeoff phase provided in an embodiment of this application;

[0060] Figure 5 This is a flowchart of the landing phase provided in an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the structure of the real-time visual positioning system for unmanned aerial vehicles provided in the embodiments of this application;

[0062] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0064] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0065] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0067] Before providing a detailed description of the embodiments of this application, some related technologies involved in the embodiments of this application will be described first.

[0068] The challenges faced by UAV visual recognition and positioning in practical applications are mainly reflected in the following aspects:

[0069] 1. Poor image quality during high-altitude flight: The limited field of view and image resolution of the camera result in low accuracy of image feature point extraction, which in turn affects the accuracy of subsequent image matching algorithms.

[0070] 2. Insufficient accuracy of image matching algorithm: Especially during flight, factors such as camera stability, changes in lighting and environmental noise can affect image quality, further increasing the difficulty of matching.

[0071] 3. GPS signal drift and error: After long-term flight or in complex environments (such as densely populated urban areas, tall buildings, or indoors), GPS signals are often interfered with or blocked, leading to a decrease in positioning accuracy. Tethered drones, operating for extended periods, are more prone to GPS failure or deviation than conventional drones.

[0072] In view of this, this application provides a real-time visual positioning method, system, electronic device, and storage medium for unmanned aerial vehicles (UAVs). This solution acquires the flight detection sequence during takeoff; controls the UAV to descend in the target landing area based on the current position information and a preset landing point during the landing phase; performs visual reference detection based on the current image information to determine whether a landing marker is detected; if no landing marker is detected, it determines whether reference marker information exists based on the current flight altitude; if reference marker information exists, it determines the target positioning point based on feature point matching using the reference marker information and the current image information; and controls the UAV to land on the ground based on the target positioning point. This application can reduce positioning errors caused by environmental interference and improve positioning accuracy.

[0073] The real-time visual positioning method for unmanned aerial vehicles (UAVs) provided in this application relates to the field of visual positioning technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle-mounted terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the real-time visual positioning method for UAVs, but is not limited to the above forms.

[0074] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0075] Figure 1 This is an optional flowchart of the real-time visual positioning method for unmanned aerial vehicles provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0076] Step S101: Obtain the flight detection sequence during the takeoff phase, wherein the flight detection sequence includes first image information at different altitudes and corresponding first flight altitude and first marker information.

[0077] Step S102: Control the drone to descend in the target landing area based on the current location information and the preset landing point during the landing phase.

[0078] Step S103: Perform visual reference detection based on the current image information to determine whether a landing marker has been detected.

[0079] Step S104: If no landing marker is detected, determine whether there is reference marker information based on the current flight altitude. The reference marker information is the first marker information associated with the current flight altitude in the flight detection sequence.

[0080] Step S105: If reference marker information exists, feature point matching is performed based on the reference marker information and the current image information to determine the target location point.

[0081] Step S106: Control the drone to land on the ground according to the target positioning point.

[0082] In this embodiment, the drone is divided into two phases: takeoff and landing. Landing markers are placed on the ground. These markers serve as visual references for the drone's precise landing. These visual references have differentiated designs such as different shapes, coded arrangements, or colors, and can be identified and calculated by sensors as spatial position references. Examples include planar patterns or coded markers like AprilTags. Taking AprilTags as an example, a specific algorithm can quickly identify the markers on the AprilTags to determine the camera's pose (relative to the markers). During the takeoff phase, the drone takes off from the ground and collects data at preset altitude intervals. At each altitude point, a ground image is captured by the camera as the first image information, and the corresponding flight altitude at the time of acquiring the first image information is recorded as the first flight altitude.

[0083] Furthermore, at each altitude point, the system will also check whether there are any landing marker detection records in the vicinity of the currently collected altitude. If there are records, the system will continue to rise from the current altitude point to the next preset altitude point to be collected according to the preset altitude interval.

[0084] If no record is found, a visual reference detection is performed. If the AprilTag is successfully detected, its image coordinates are recorded as the first marker information for subsequent fine descent. If the AprilTag is not successfully detected, the ascent continues, and the above acquisition and detection process is repeated until the predetermined altitude is reached or the required recording is completed, resulting in a flight detection sequence with different altitude data sets.

[0085] Optionally, during the data acquisition process, the first timestamp and first location information for each acquisition can also be recorded in the flight detection sequence. The first timestamp indicates the time of data acquisition during takeoff, and the first location information indicates the GPS positioning signal at the time of data acquisition during takeoff.

[0086] It should be noted that once the set maximum flight altitude is reached or the required number of records is completed, the takeoff phase ends, and the drone can perform other tasks. The real-time visual positioning method for drones provided in this embodiment will restart and enter the landing phase process when the drone needs to perform a precise landing task.

[0087] During the descent phase, the current location information is first obtained by reading the current GPS positioning signal. Based on this location information, a rough positioning can be made to pinpoint the target landing area. The target landing area is the approximate location of the target landing point (within five meters of accuracy, provided the satellite signal reception is not too poor). If the flight altitude is not too low, the landing markers can be ensured to be within the camera's field of view. After the drone reaches the designated target landing area, it begins its descent.

[0088] Specifically, during the descent, the drone performs real-time visual reference detection, using the same detection algorithm as during takeoff. If the AprilTag is successfully detected, the system will determine whether it is close to the ground (e.g., less than 1 meter away). If the result is yes, it means that the drone is close to the ground and can land directly, and the algorithm ends; otherwise, it continues to descend.

[0089] If the AprilTag is not detected, the system will search for a first marker within a certain range of the current flight altitude. If found, the first marker will be used as a reference marker for subsequent matching.

[0090] If reference marker information exists, the current image information is matched with the first image information at a similar altitude from a previous record. If the match is successful and a feature point corresponding to AprilTag is found, the target positioning point is calculated using the average coordinates between the matched feature points. This target positioning point is the precise landing location for the drone. The offset coordinates calculated based on the target positioning point are sent to the flight control main board to enable the drone to land accurately. If the match fails, the descent continues, and steps S103 to S106 are repeated until the drone lands precisely at the target positioning point on the ground.

[0091] This embodiment improves the robustness of the system in dynamic environments by combining different visual positioning algorithms, along with marker detection and feature point matching. It reduces positioning errors caused by environmental interference, enabling high-precision landing even when camera imaging quality is low, the judgment effect of a single algorithm is poor, or GPS signal drift occurs. This solves problems such as inaccurate drone positioning, excessive algorithm computation, and poor environmental adaptability.

[0092] In some embodiments, step S101 may include, but is not limited to, steps S201 to S207.

[0093] Step S201: Initialize the flight detection sequence.

[0094] Step S202: Control the drone to ascend based on the preset ascent altitude.

[0095] Step S203: Obtain the first image information of the preset altitude point and the corresponding first flight altitude.

[0096] Step S204: Extract feature points based on the first image information to obtain the first feature points.

[0097] Step S205: Perform visual reference detection based on the first image information to determine whether a landing marker has been detected.

[0098] Step S206: If a landing marker is detected, the center point coordinates and diagonal point coordinates of the landing marker are extracted from the first image information to obtain the first marker information.

[0099] Step S207: Record the first image information, first flight altitude, first feature point and first marker information into the flight detection sequence, and then return to control the UAV to ascend based on the preset ascent altitude until the first flight altitude equals the preset ascent altitude.

[0100] In this embodiment, during the takeoff phase of the UAV, the initialization operation of the flight detection sequence is first performed. This process creates a structured data storage space. Based on the preset ascent altitude and preset altitude interval, multiple altitude points can be determined. Each altitude point requires allocated storage space. For example, with a preset altitude interval of 1.5 meters and a preset ascent altitude of 30 meters, 20 sets of storage space need to be allocated to store the data collected at 1.5 meters, 3 meters...30 meters respectively. Different sets of data are distinguished by altitude so as to record images at different altitudes, their corresponding timestamps, flight altitudes, and GPS coordinates.

[0101] Next, a data acquisition cycle is performed. The drone is controlled to ascend based on a preset ascent altitude. Before the drone reaches the preset ascent altitude, it continues to ascend to collect data at different altitudes. The first image information is captured by the camera controlled by the intelligent vision chip (such as RV1126), and other information is obtained by RV1126 from the flight control main board.

[0102] Specifically, during ascent, the drone hovers for a certain period of time after reaching a preset altitude to ensure stability during image acquisition. Understandably, the hovering time can be adjusted based on flight environment data. For example, in strong winds, if the detected wind speed exceeds a preset environmental threshold, the hovering time will be extended according to the hovering strategy.

[0103] For example, the first image information is captured by the RV1126 control camera. The RV1126 control camera captures ground images and requests the current timestamp t, current flight altitude h, and current GPS coordinates from the flight control main control board, and stores them together with the images in the flight detection sequence to provide data support for subsequent accurate landing.

[0104] Feature points are extracted from the acquired first image information to obtain the first feature points with significant distinguishability, which are used for matching during the landing phase. The first image information is then dimensionality-reduced using a feature point extraction algorithm to retain key information useful for matching and localization, resulting in the first feature points, such as edges and diagonal points.

[0105] Furthermore, to improve the reliability of the first feature point, feature point trajectory tracking can be performed on multiple frames of images continuously acquired at the same height point, and feature points that appear stably in at least two consecutive frames can be retained as the first feature points corresponding to the first image information.

[0106] Optionally, before performing visual reference detection in step S205, it is first checked whether the April Tag has been detected. It is determined whether there is a successful April Tag detection record within the current first flight altitude ±0.5m range. If so, the flight continues to ascend and returns to step S203 to re-enter the data acquisition loop.

[0107] If it does not exist, then perform visual benchmark detection based on the first image information to determine whether a landing marker is detected in the first image information.

[0108] If a landing marker is detected in the first image information, the detection is considered successful. The information of the April Tag is extracted to obtain the information of the first marker, such as the center point coordinates (cx, cy) of the center point and the diagonal point coordinates (x1, y1, x2, y2) of the diagonal point.

[0109] Record information such as ground image, feature points, flight timestamp, flight altitude, GPS coordinates, and AprilTag information at the current altitude point, and save it to the "Flight Detection Sequence".

[0110] If no landing marker is detected in the first image information, the detection is considered to have failed. The drone continues to ascend to the next preset altitude point and returns to step S203, returning to the data acquisition loop until it reaches the set maximum flight altitude or completes the required number of records before exiting the loop and performing other tasks.

[0111] In some embodiments, step S103 may include, but is not limited to, steps S301 to S306.

[0112] Step S301: Perform grayscale processing and edge extraction on the current image information to identify potential quadrilateral regions and obtain candidate regions.

[0113] Step S302: Perform perspective transformation correction on the candidate region to obtain standard pattern data.

[0114] Step S303: Extract the encoded bit array of the standard pattern data and decode it to obtain the current tag identifier.

[0115] Step S304: Calculate the Hamming distance based on the current tag identifier and the landing tag identifier corresponding to the landing marker.

[0116] Step S305: Determine whether the Hamming distance is lower than the preset bit threshold.

[0117] Step S306: If the Hamming distance is lower than the preset number of bits threshold, then it is determined that a landing marker has been detected; otherwise, it is determined that no landing marker has been detected.

[0118] In this embodiment, the AprilTag detection algorithm identifies potential quadrilateral regions in the image by performing grayscale processing and edge extraction on the current image information, thus obtaining candidate regions.

[0119] Next, the candidate region is corrected into a standard rectangular pattern through perspective transformation to obtain standard pattern data. The encoded bit array in the standard pattern data is extracted and decoded to identify the unique identifier (ID) of the label corresponding to the standard pattern data.

[0120] The system matches the identified ID with the coded ID (i.e., the landing tag identifier) ​​corresponding to the AprilTag. For example, it calculates the Hamming distance, which is the number of different bits in the two binary coded IDs. The fewer the number of different bits, the higher the degree of matching.

[0121] The system determines whether the current tag identifier matches the landing tag identifier by checking if the Hamming distance is below a preset bit width threshold. It's understandable that the preset bit width threshold used for this determination can be dynamically adjusted based on the ID bit width and the determination criteria.

[0122] For example, the matching criterion is that the similarity reaches 90%, and the ID is 20 digits. A preset digit threshold can be set to allow two different digits. If the number of different digits is less than this preset digit threshold, it means that a landing marker has been detected; otherwise, it is considered that a landing marker has been detected.

[0123] In some embodiments, step S103 may include, but is not limited to, steps S401 to S402.

[0124] Step S401: Perform visual reference detection based on the current image information to determine whether peripheral and nested markers are detected.

[0125] Step S402: If the total number of detected peripheral and nested markers is greater than the preset detection number, then it is determined that a landing marker has been detected; otherwise, it is determined that no landing marker has been detected.

[0126] In this embodiment, to meet the requirement of being recognizable at different altitudes and to address extreme cases such as marker occlusion, a nested, multi-scale AprilTag landing marker is used. This adapts to detection needs at different flight altitudes. Even with partial occlusion, the decoding success rate of AprilTag is improved, increasing the robustness of the system.

[0127] Specifically, a landing marker has multiple tags of different sizes and may include an outer marker and several nested markers, wherein the nested markers are embedded in the outer markers and the size of the nested markers is smaller than that of the outer markers.

[0128] For example, refer to Figure 2 In this embodiment, a landing marker is constructed by embedding three smaller nested AprilTags within a larger outer AprilTag, and the IDs of the outer AprilTag and the other three nested AprilTags are stored separately in the system. During each landing marker identification process, an attempt is made to identify all four AprilTags.

[0129] It should be noted that nesting AprilTags will not affect the recognition of the outer AprilTag, because the nesting position is already recorded in the ID.

[0130] Obstruction of markers is a common problem, especially for markers such as AprilTag, which may be partially obscured or blurred during flight, causing the system to be unable to decode and recognize them correctly, thus affecting positioning accuracy.

[0131] Taking a tethered drone as an example, the tether line may obstruct the camera on the underside of the drone during landing (in addition to other unexpected situations that may cause obstruction, such as dirt in front of the camera). In this case, the algorithm can also try to identify the nested small AprilTag, so that the entire AprilTag is partially obscured and cannot be identified at all.

[0132] Understandably, when detecting landing markers, the entire AprilTag area can be considered as the target area for precise landing. Therefore, the detection of peripheral or nested markers can be considered as the presence of landing markers. That is, as long as the total number of peripheral and nested markers detected is greater than 0, it can be determined that a landing marker has been detected; otherwise, it is considered that no landing marker has been detected.

[0133] Specifically, when both outer AprilTag and nested AprilTag are detected in the AprilTag region, the outer AprilTag is prioritized as the target area for precise landing.

[0134] If no outer AprilTag is detected in the AprilTag region, but nested AprilTags are detected, and the number of nested AprilTags is greater than 1, then the average coordinates of the multiple nested AprilTags are used as the target area for precise landing.

[0135] If no outer AprilTag is detected in the AprilTag region, and only a nested AprilTag is detected, then the nested AprilTag is taken as the target region for precise landing.

[0136] In some embodiments, step S104 may include, but is not limited to, steps S501 to S503.

[0137] Step S501: Determine the associated altitude range based on the current flight altitude.

[0138] Step S502: Determine whether there is a first flight altitude within the associated altitude range in the flight detection sequence.

[0139] Step S503: If a first flight altitude is located within an associated altitude range, then the first marker information corresponding to the first flight altitude is determined as reference marker information.

[0140] In this embodiment, the reference marker information is determined by establishing a correlation mechanism between the current flight altitude and the flight detection sequence storage data, so as to achieve positioning capability in complex environments.

[0141] Specifically, based on a preset height association threshold, such as 0.5 meters, the upper limit of the association height is determined by increasing the preset association threshold on the basis of the current flight altitude. The upper limit of the association height represents the maximum value of the queried height. The lower limit of the association height is determined by decreasing the preset association threshold. The lower limit of the association height represents the minimum value of the queried height. The association height range to be queried in the flight detection sequence is determined based on the upper limit of the association height and the lower limit of the association height.

[0142] Optionally, asymmetric threshold configuration schemes are supported for setting the height association threshold. Non-clustered association height ranges can be formed by independently setting the upper and lower limit association thresholds. For example, to adapt to rapid descent, the downward search range can be expanded by setting the lower limit association threshold to 0.8 meters, or to adapt to high-altitude environments, the upward search range can be expanded by setting the upper limit association threshold to 1 meter.

[0143] In the flight detection sequence, determine whether there is a recorded first flight altitude that is within the associated altitude range. If the first flight altitude is less than the associated upper limit altitude and greater than the associated lower limit altitude, it can be confirmed that the first flight altitude is within the associated altitude range.

[0144] If a first flight altitude is located within the associated altitude range, the marker information corresponding to the first flight altitude will be determined as the reference marker information.

[0145] Preferably, if multiple first marker information that meets the conditions are recorded within the associated altitude range, the first marker information with the smallest difference between the first flight altitude and the current flight altitude is selected as the reference marker information.

[0146] In some embodiments, step S105 may include, but is not limited to, steps S601 to S607.

[0147] Step S601: Extract feature points from the current image information to obtain the current feature points.

[0148] Step S602: Query the corresponding first feature point from the flight detection sequence based on the reference marker information.

[0149] Step S603: Perform feature point matching based on the current feature point and the first feature point to obtain a matching feature point pair.

[0150] Step S604: Remove outliers from the matching feature point pairs to obtain matching inner point pairs.

[0151] Step S605: Perform diagonal point verification based on the matched inner point pairs to obtain the matched diagonal points.

[0152] Step S607: Calculate the geometric center based on the coordinates of the diagonal points of the matched diagonal points to obtain the target positioning point.

[0153] In this embodiment, when landing markers cannot be directly detected, a precise landing is performed based on a feature point matching algorithm. This process utilizes the first image information pre-stored during takeoff to perform high-precision matching with the currently acquired image information, thereby calculating the precise location of the target positioning point.

[0154] Reference Figure 3 Feature point matching algorithms are a class of techniques used to find points at the same or similar locations in different images. They are widely used in computer vision tasks such as image stitching, 3D reconstruction, object recognition, and motion estimation. The algorithm first detects stable and discriminative key points in the image. These points should remain stable under conditions such as scaling, rotation, and changes in illumination. For each detected feature point, local information from its surrounding image region is extracted and encoded into a discriminative descriptive vector.

[0155] Specifically, feature points are extracted from the current image information acquired during the landing phase to obtain current feature points with significant distinguishability in the current image information. These current feature points are then used to match the first feature points recorded during the takeoff phase.

[0156] After extracting the current feature point, the corresponding first feature point needs to be queried from the flight detection sequence as a matching benchmark. During the query process, the first flight altitude corresponding to the reference marker information is used as the primary index to query the feature points extracted at the first flight altitude. The feature descriptions in the current feature point and the first feature point are compared to find matching feature point pairs. Matching feature point pairs are represented as a tuple, which consists of the current feature point and the first feature point corresponding to the same physical space point.

[0157] Next, the RANSAC algorithm is used to remove outliers from the matching feature point pairs, and the matching points are then validated. A basic assumption of the RANSAC algorithm is that a set of data consists of "inliers" and "outliers," where inliers are the data that make up the model parameters, and outliers are the data that do not fit the model. The goal of the RANSAC algorithm is to estimate a model that fits the inliers from a set of data containing a small number of inliers and a large number of outliers.

[0158] Specifically, a portion of the matched feature point pairs is randomly selected as samples. The parameters of the fitted model are obtained based on the selected matched feature point pairs. Then, the remaining matched feature point pairs are used to verify whether the fitted model is optimal. Through multiple iterations, the model with the smallest selection error is obtained, effectively eliminating noise points and outliers in the matched feature point pairs that cause mismatches. After eliminating outliers, the matched inner point pairs are obtained.

[0159] Based on the matching intra-point pairs, diagonal point verification is performed. If a diagonal point in the matching intra-point pair matches a point near the diagonal coordinates (x1, y1, x2, y2) of the April Tag in the flight detection sequence record, then the geometric center is calculated based on the successfully matched diagonal point, yielding the corresponding center point coordinates (cx, cy). The geometric center is obtained by calculating the average position of all points. These center point coordinates are used as the target positioning point for precise landing operations.

[0160] In some embodiments, the real-time visual positioning method for drones may also include, but is not limited to, steps S701 to S703.

[0161] Step S701: If a landing marker is detected, determine whether there are any peripheral markers among the landing markers.

[0162] Step S702: If there are peripheral markers, extract the center point coordinates of the peripheral markers from the current image information, and obtain the target positioning point based on the center point coordinates of the peripheral markers.

[0163] Step S703: If there are no external markers, extract the center point coordinates of the nested markers from the current image information, and obtain the target positioning point based on the center point coordinates of the nested markers.

[0164] In this embodiment, a priority order can be set for each landing marker according to the marker type. For example, the priority of peripheral markers is greater than that of nested markers. Alternatively, the priority order can be set according to the relative position of the landing markers. For example, the priority of the landing marker located in the center of the April Tag area is greater than that of the landing markers identified in the remaining areas.

[0165] For example, refer to Figure 2 There are a total of 4 AprilTags, with two AprilTags centered and two others centered at the edges. If the two centered AprilTags (i.e., the outer tag AprilTag1 and the nested tag AprilTag2) can be identified, the larger outer tag AprilTag1 is returned first, and the coordinates of the center point of the outer tag AprilTag1 are used as the target location point.

[0166] If the outer marker April Tag1 is not recognized due to occlusion or other reasons, the smaller nested marker April Tag2 is returned, and the center coordinates of the nested marker April Tag2 are used as the target positioning point.

[0167] If two centered April Tags are not recognized due to occlusion or other reasons, then a query is performed to see if two edge April Tags (i.e., nested AprilTag3 and nested AprilTag4) can be recognized. If both edge April Tags can be recognized simultaneously, then the average coordinates of the center points of the two April Tags are used as the target center point.

[0168] When a query finds an AprilTag (nested AprilTag3 or nested AprilTag4) whose edge can only be identified, the coordinates of the center point of that AprilTag are used as the target center point.

[0169] The following is a detailed introduction and explanation of the solutions in the embodiments of the present invention, with reference to specific application examples.

[0170] The embodiments of this application illustrate the detailed steps of a method for precise landing of an unmanned aerial vehicle (UAV) using the takeoff and landing phases as examples.

[0171] Reference Figure 4 , Figure 4 The flowchart for the takeoff phase includes, but is not limited to, steps 4.1 to 4.4.

[0172] Step 4.1: Initialize Records: Initialize the flight detection sequence record, which is used to record images at different altitudes and their corresponding timestamps, flight altitudes, and GPS coordinates.

[0173] Step 4.2: Data Acquisition Loop: The Rv1126 controls the camera to capture ground images and requests the current timestamp t, current flight altitude h, and current GPS coordinates from the flight controller main control board, and stores them together with the images.

[0174] Check if AprilTag has been detected: Determine if there is a successful AprilTag detection record within ±0.5m of the current flight altitude: If it exists, continue ascending and re-enter the data acquisition loop in step 4.2; if it does not exist, perform AprilTag detection.

[0175] Step 4.3: AprilTag detection:

[0176] If detection is successful: Extract the AprilTag information (e.g., center point cx, cy and diagonal coordinates x1, y1, x2, y2). Record the current timestamp, flight altitude, GPS coordinates, and AprilTag information, and perform feature point detection on the current image for use in the landing phase matching. Save to "Flight Detection Sequence Record".

[0177] If the detection fails, continue to ascend and return to step 4.2, the data acquisition loop.

[0178] Step 4.4: End of takeoff phase: Repeat the above process until the set maximum flight altitude is reached or the required number of records is completed, then exit the loop and execute other tasks. The algorithm will restart and enter the landing phase process when the drone needs to perform a precise landing task.

[0179] Reference Figure 5 , Figure 5 A flowchart for the landing (return) phase, including but not limited to steps 5.1 to 5.4.

[0180] Step 5.1: Start landing: The drone first determines the approximate location of the landing point based on GPS positioning, and the flight controller guides the drone to reach the approximate location of the landing point.

[0181] Step 5.2: Data Acquisition Loop: The Rv1126 controls the camera to capture ground images and requests the flight controller main board to obtain the current timestamp t, current flight altitude h, and current GPS coordinates. Then, it performs AprilTag detection: If the detection is successful: extract the AprilTag's cx and cy, and continue to determine whether the current position is close to the ground (e.g., the distance is less than 1m). If it is close to the ground, it lands directly. Otherwise, it continues to land and repeats the data acquisition loop in Step 5.2.

[0182] If detection fails: Check if there is an AprilTag detection record (from the takeoff phase) within ±0.5m of the current altitude. If no record is found, continue the descent and re-execute step 5.2, the data acquisition loop. If a record is found, continue to step 5.3, image feature point matching.

[0183] Step 5.3: Image feature point matching: Detect the feature points of the current image and match the current image feature points with the feature point set of the most recently recorded image.

[0184] Outlier removal is performed using the RANSAC algorithm, followed by matching point verification: If a corner point in the matching point set matches the diagonal coordinates (x1, y1, x2, y2) of the AprilTag in the "Flight Detection Sequence Record", the corresponding center point coordinates (cx, cy) of the successful match are calculated. These center point coordinates are used as the target positioning point for precise landing operations.

[0185] It should be noted that the entire AprilTag area can be regarded as the target area for precise landing. If the outer AprilTag is identified, the difference between the center point of the outer AprilTag and the center point in the camera's field of view is used as the offset; if the nested AprilTag is identified, the difference between the center point of the nested AprilTag and the center point in the camera's field of view is used as the offset.

[0186] Step 5.4: End the descent phase: If a match is not found, continue the descent and repeat the data acquisition loop in Step 5.2.

[0187] This application's embodiments improve the accuracy of UAV landing by combining feature point matching and multi-scale AprilTag detection, enabling high-precision positioning and landing even in situations with poor UAV imaging quality, GPS signal drift, or partial obstruction of the marker. Furthermore, it enhances the robustness of UAV landing by combining feature point matching and marker detection, improving system robustness in dynamic environments and reducing positioning errors caused by environmental interference. The use of nested AprilTag markers with various sizes adapts to detection needs at different flight altitudes. Even with partial obstruction, the decoding success rate of the AprilTag is improved, increasing system robustness.

[0188] Reference Figure 6 This application also provides a real-time visual positioning system for unmanned aerial vehicles (UAVs), which can implement the above-mentioned real-time visual positioning method for UAVs. The system includes:

[0189] The first module is used to acquire the flight detection sequence during the takeoff phase, wherein the flight detection sequence includes first image information at different altitudes and corresponding first flight altitude and first marker information.

[0190] The second module is used to control the drone to descend in the target landing area based on the current location information and the preset landing point during the descent phase.

[0191] The third module is used to perform visual benchmark detection based on the current image information to determine whether a landing marker has been detected.

[0192] The fourth module is used to determine whether there is reference marker information based on the current flight altitude if no landing marker is detected. The reference marker information is the first marker information associated with the current flight altitude in the flight detection sequence.

[0193] The fifth module is used to determine the target location point by matching feature points based on the current image information, if reference marker information exists.

[0194] The sixth module is used to control the drone to land on the ground based on the target positioning point.

[0195] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0196] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described real-time visual positioning method for unmanned aerial vehicles. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0197] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0198] Reference Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0199] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0200] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the UAV real-time visual positioning method of the embodiments of this application.

[0201] The input / output interface 903 is used to implement information input and output.

[0202] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0203] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904).

[0204] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0205] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described real-time visual positioning method for unmanned aerial vehicles.

[0206] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0207] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0208] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0209] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0210] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0211] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0212] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0213] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A real-time visual positioning method for unmanned aerial vehicles (UAVs), characterized in that, The method includes the following steps: Acquire a flight detection sequence during the takeoff phase, wherein the flight detection sequence includes first image information at different altitudes and corresponding first flight altitude and first marker information; The drone is controlled to descend within the target landing area based on its current location information and the preset landing point during the descent phase. Visual baseline detection is performed based on the current image information to determine whether landing markers have been detected. If the landing marker is not detected, the presence of reference marker information is determined based on the current flight altitude, wherein the reference marker information is the first marker information associated with the current flight altitude in the flight detection sequence; If the reference marker information exists, then based on the reference marker information, feature point matching is performed according to the current image information to determine the target location point; The drone is controlled to land on the ground based on the target positioning point.

2. The method according to claim 1, characterized in that, The acquisition of the flight detection sequence during the takeoff phase includes the following steps: Initialize the flight detection sequence; The drone's ascent is controlled based on a preset ascent altitude; Acquire the first image information of the preset altitude point and the corresponding first flight altitude; Based on the first image information, feature points are extracted to obtain the first feature points; Based on the first image information, visual reference detection is performed to determine whether a landing marker is detected. If the landing marker is detected, the center point coordinates and diagonal point coordinates of the landing marker are extracted from the first image information to obtain the first marker information; The first image information, the first flight altitude, the first feature point, and the first marker information are recorded into the flight detection sequence, and then the drone is controlled to ascend based on a preset ascent altitude until the first flight altitude equals the preset ascent altitude.

3. The method according to claim 1, characterized in that, The step of performing visual reference detection based on the current image information to determine whether a landing marker has been detected includes the following steps: The current image information is processed for grayscale and edge extraction to identify potential quadrilateral regions and obtain candidate regions. The candidate regions are subjected to perspective transformation correction to obtain standard pattern data; Extract the encoded bit array of the standard pattern data and decode it to obtain the current tag identifier; Calculate the Hamming distance based on the current tag identifier and the landing tag identifier corresponding to the landing marker; Determine whether the Hamming distance is lower than a preset bit threshold; If the Hamming distance is lower than the preset bit threshold, then the landing marker is determined to have been detected; otherwise, the landing marker is determined not to have been detected.

4. The method according to claim 1, characterized in that, The landing marker includes an outer marker and a nested marker, with the nested marker embedded within the outer marker. The step of performing visual reference detection based on the current image information to determine whether the landing marker has been detected includes the following steps: Visual benchmark detection is performed based on the current image information to determine whether the outer and nested markers are detected. If the total number of the outer and nested markers detected is greater than the preset detection number, then it is determined that the landing marker has been detected; otherwise, it is determined that the landing marker has not been detected.

5. The method according to claim 1, characterized in that, The process of determining whether reference marker information exists based on the current flight altitude includes the following steps: Determine the relevant altitude range based on the current flight altitude; Determine whether the first flight altitude exists within the associated altitude range in the flight detection sequence; If the first flight altitude is within the associated altitude range, then the first marker information corresponding to the first flight altitude is determined as reference marker information.

6. The method according to claim 2, characterized in that, The step of determining the target location point by matching feature points based on the reference marker information and the current image information includes the following steps: Feature points are extracted from the current image information to obtain the current feature points; The corresponding first feature point is queried from the flight detection sequence based on the reference marker information; Based on the current feature point and the first feature point, feature point matching is performed to obtain a matching feature point pair; Based on the matching feature point pairs, outer points are removed to obtain matching inner point pairs; Based on the matched inner point pairs, perform diagonal point verification to obtain the matched diagonal points; The geometric center is calculated based on the coordinates of the diagonal points of the matched diagonal points to obtain the target positioning point.

7. The method according to claim 1, characterized in that, The landing marker includes an outer marker and a nested marker, wherein the nested marker is embedded in the outer marker. The UAV real-time visual positioning method further includes the following steps: If the landing marker is detected, determine whether the peripheral marker exists within the landing marker; If the surrounding markers exist, the center point coordinates of the surrounding markers are extracted from the current image information, and the target positioning point is obtained based on the center point coordinates of the surrounding markers. If the outer marker does not exist, the center point coordinates of the nested marker are extracted from the current image information, and the target positioning point is obtained based on the center point coordinates of the nested marker.

8. A real-time visual positioning system for unmanned aerial vehicles (UAVs), characterized in that, The system includes: The first module is used to acquire the flight detection sequence during the takeoff phase, wherein the flight detection sequence includes first image information at different altitudes and corresponding first flight altitude and first marker information; The second module is used to control the drone to descend in the target landing area based on the current position information and the preset landing point during the landing phase. The third module is used to perform visual reference detection based on the current image information to determine whether a landing marker has been detected. The fourth module is used to determine whether reference marker information exists based on the current flight altitude if the landing marker is not detected, wherein the reference marker information is the first marker information associated with the current flight altitude in the flight detection sequence; The fifth module is used to determine the target location point by performing feature point matching based on the current image information, if the reference marker information exists. The sixth module is used to control the UAV to land on the ground based on the target positioning point.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images