Unmanned aerial vehicle navigation method based on visual mileage and related equipment

Abstract

The invention provides an unmanned aerial vehicle navigation method based on visual mileage and related equipment, which ensure the stability and reliability of an unmanned aerial vehicle in a complex environment by directly performing feature extraction and matching on a ground image instead of depending on an electromagnetic signal susceptible to interference. Meanwhile, the global pose of the unmanned aerial vehicle is determined by using a pose estimation algorithm, so that the positioning precision is improved, and the autonomy and flexibility of the unmanned aerial vehicle in the navigation process are enhanced. Besides, according to the method, the initial position information and the global pose of the unmanned aerial vehicle are combined, the mileage information and the navigation information of the unmanned aerial vehicle are further deduced, a solid foundation is provided for flight control and task planning of the unmanned aerial vehicle, and therefore it is ensured that the unmanned aerial vehicle can efficiently and accurately complete various flight tasks.

Description

Technical Field

[0001] The present application relates to the field of drone technology, and in particular to a drone navigation method based on visual mileage and related equipment. Background Art

[0002] When a drone is performing a flight mission, it needs to use navigation technology to update the drone's location information in real time to ensure that the drone always knows its current accurate location during the flight and where it should fly next, thereby ensuring that the drone can complete the flight mission efficiently and safely.

[0003] Based on the above situation, the UAV navigation method used in the prior art is susceptible to electromagnetic interference, and the operation of the satellite receiver is affected by the maneuvering of the aircraft, which seriously affects the execution of the UAV flight mission. Summary of the invention

[0004] In view of this, the purpose of this application is to propose a UAV navigation method and related equipment based on visual mileage to solve the above-mentioned technical problems.

[0005] Based on the above purpose, the first aspect of the present application provides a drone navigation method based on visual mileage, comprising:

[0006] Acquire multiple frames of ground images of the drone for the same landmark point during flight, as well as initial position information of the drone;

[0007] Extracting features of landmark points in multiple frames of ground images by a feature point detection algorithm to obtain feature points, and determining descriptors of the feature points, wherein the descriptors are used to describe image information around the feature points;

[0008] For each group of two adjacent ground images in the multiple ground images, the descriptors of the feature points are matched using a matching algorithm to obtain matching pairs corresponding to each group of two adjacent ground images;

[0009] The pose estimation algorithm is used to estimate the pose of the matching pairs corresponding to each group of two adjacent ground image frames, determine the relative pose between each group of two adjacent ground image frames, and determine the global pose of the UAV based on the relative pose between each group of two adjacent ground image frames;

[0010] The mileage information of the drone is determined based on the global position of the drone and the initial position information of the drone, and the navigation information of the drone is determined according to the global position of the drone and the mileage information of the drone.

[0011] Optionally, extracting features of landmark points in multiple frames of ground images by a feature point detection algorithm to obtain feature points includes:

[0012] Using the pre-constructed Hessian matrix, the landmark points in multiple frames of ground images are processed for interest point recognition to obtain multiple interest points;

[0013] The plurality of interest points are filtered using a non-maximum suppression algorithm to obtain feature points.

[0014] Optionally, determining the descriptor of the feature point includes:

[0015] The feature points are encoded using a hash function to obtain descriptors of the feature points.

[0016] Optionally, for each group of two adjacent ground images in the multiple ground images, respectively matching the descriptors of the feature points using a matching algorithm to obtain matching pairs corresponding to each group of two adjacent ground images includes:

[0017] For each set of feature points of two adjacent ground images in multiple ground images, the following operations are performed:

[0018] Determine the relative distances between the descriptors of each feature point of the first ground image in the two adjacent ground image frames and the descriptors of other feature points of the second ground image in the two adjacent ground image frames;

[0019] Other feature points with the smallest relative distance are selected from the relative distances, and the other feature points with the smallest relative distance and the corresponding feature points in the first frame of the ground image are used as matching pairs.

[0020] Optionally, performing pose estimation on matching pairs corresponding to each group of two adjacent ground image frames using a pose estimation algorithm to determine a relative pose between each group of two adjacent ground image frames includes:

[0021] According to the matching pairs corresponding to each group of two adjacent ground image frames, the coordinates of the feature points corresponding to the matching pairs in the world coordinate system are determined by using a triangulation algorithm;

[0022] Based on the coordinates of the feature points corresponding to the matching pairs in the world coordinate system, a camera pose estimation algorithm is used to perform camera pose estimation processing to obtain the relative pose between each group of two adjacent frames of ground images.

[0023] Optionally, determining the global pose of the UAV according to the relative pose between each group of two adjacent ground image frames includes:

[0024] The relative poses between each group of two adjacent ground image frames are summed to obtain the global pose of the UAV.

[0025] Optionally, the global pose of the drone includes the current position of the drone;

[0026] The determining of the mileage information of the drone based on the global posture of the drone and the initial position information of the drone includes:

[0027] Relative distance processing is performed based on the current position of the drone and the initial position of the drone to obtain the mileage information of the drone.

[0028] Optionally, the global pose of the drone includes the flight direction of the drone and the current position of the drone;

[0029] The determining the navigation information of the drone according to the global position and mileage information of the drone includes:

[0030] The flight direction of the drone, the current position of the drone and the mileage information of the drone are combined to obtain navigation information of the drone.

[0031] Based on the same inventive concept, the second aspect of the present application provides a drone navigation device based on visual mileage, comprising:

[0032] An acquisition module is configured to acquire multiple frames of ground images of the drone for the same landmark point during flight, as well as initial position information of the drone;

[0033] A feature extraction module is configured to extract features of landmark points in multiple frames of ground images by using a feature point detection algorithm to obtain feature points and determine descriptors of the feature points, wherein the descriptors are used to describe image information around the feature points;

[0034] The matching module is configured to use a matching algorithm to match the descriptors of the feature points of each group of two adjacent ground images in the multiple ground images, and obtain matching pairs corresponding to each group of two adjacent ground images;

[0035] A pose estimation module is configured to perform pose estimation on matching pairs corresponding to each group of two adjacent ground image frames using a pose estimation algorithm, determine a relative pose between each group of two adjacent ground image frames, and determine a global pose of the UAV based on the relative pose between each group of two adjacent ground image frames;

[0036] The navigation module is configured to determine the mileage information of the drone based on the global position of the drone and the initial position information of the drone, and to determine the navigation information of the drone according to the global position of the drone and the mileage information of the drone.

[0037] Based on the same inventive concept, the third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the method described in the first aspect above when executing the computer program.

[0038] From the above, it can be seen that the visual mileage-based drone navigation method and related equipment provided by this application ensure the stability and reliability of the drone in complex environments by directly extracting and matching features from ground images rather than relying on electromagnetic signals that are susceptible to interference. At the same time, the use of a posture estimation algorithm to determine the global posture of the drone not only improves the positioning accuracy, but also enhances the autonomy and flexibility of the drone during navigation. In addition, the method combines the initial position information and global posture of the drone, and further derives the mileage information and navigation information of the drone, providing a solid foundation for the flight control and mission planning of the drone, thereby ensuring that the drone can efficiently and accurately complete various flight missions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 This is a flow chart of a UAV navigation method based on visual mileage according to an embodiment of the present application;

[0041] Figure 2 This is a structural block diagram of the UAV navigation based on visual mileage according to an embodiment of the present application;

[0042] Figure 3 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0044] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should be the usual meanings understood by people with ordinary skills in the field to which the present application belongs. The "first", "second" and similar words used in the embodiments of the present application do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0045] It is understandable that before using the technical solutions of each embodiment of the present application, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0046] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server, storage medium or other software or hardware that performs the operation of the technical solution of the present application according to the prompt message.

[0047] As an optional but non-limiting implementation, in response to receiving the user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0048] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of this application. Other methods that meet relevant laws and regulations may also be applied to the implementation method of this application.

[0049] The navigation technologies currently used on drones mainly include satellite navigation, inertial navigation, Doppler navigation, terrain-aided navigation, and geomagnetic navigation. These navigation technologies can be used alone for drone navigation. The most widely used is satellite navigation, which has the advantages of global, all-weather, continuous precision navigation and positioning capabilities, and excellent real-time performance. The disadvantage is that it is susceptible to electromagnetic interference, and the operation of the satellite receiver is affected by the maneuvering of the aircraft. For example, the signal frequency of the satellite is generally 1-2Hz. If the aircraft needs to quickly update the navigation information, the Global Positioning System (GS) alone cannot meet the needs of the aircraft.

[0050] The embodiments of the present application provide a UAV navigation method based on visual mileage, which ensures the stability and reliability of the UAV in complex environments by directly extracting and matching features from ground images instead of relying on electromagnetic signals that are susceptible to interference. At the same time, the global posture of the UAV is determined using a posture estimation algorithm, which not only improves the positioning accuracy, but also enhances the autonomy and flexibility of the UAV during navigation. In addition, the method combines the initial position information and global posture of the UAV, and further derives the mileage information and navigation information of the UAV, providing a solid foundation for the flight control and mission planning of the UAV, thereby ensuring that the UAV can complete various flight missions efficiently and accurately.

[0051] like Figure 1 As shown, the method of this embodiment includes:

[0052] Step 101, obtaining multiple frames of ground images of the same landmark point during the flight of the drone, as well as the initial position information of the drone.

[0053] In this step, the drone can be equipped with cameras, sensors and other equipment to capture environmental information or perform other tasks.

[0054] During the flight, the drone takes multiple photos of the same specific landmark on the ground (such as a building, mountain, bridge, etc.) from different angles, positions or time points, and obtains multiple images of the landmark. These images may vary due to factors such as the flight path, altitude, angle, etc. of the drone, but their common feature is that they all contain the same landmark.

[0055] In addition to the above-mentioned multiple frames of ground images, it is also necessary to obtain the location information of the drone when it starts taking these images. This location information may include the longitude, latitude, altitude (altitude or height relative to the ground), heading, etc. of the drone, depending on the required accuracy and purpose.

[0056] The initial position information refers to the position information of the drone when it starts to perform the shooting mission, which may be necessary for subsequent analysis of the drone's flight path, image shooting position, etc.

[0057] In summary, the description involves taking multiple shots of the same landmark during a drone flight and recording the initial position information of the drone, which may be used for a variety of purposes, such as map making, environmental monitoring, terrain analysis, target tracking, drone navigation, etc. By comparing and analyzing these images and the drone's position information, more information about the landmark, the drone's flight path, environmental changes, etc. can be obtained.

[0058] Step 102, extracting features of landmark points in multiple frames of ground images by a feature point detection algorithm to obtain feature points, and determining descriptors of the feature points, wherein the descriptors are used to describe image information around the feature points.

[0059] In this step, the feature point detection algorithm is used to automatically identify and locate feature points with unique properties in the image. These feature points are usually areas in the image that have obvious changes or significant characteristics, such as corners, edges, or spots. The goal of the feature point detection algorithm is to find points that can still be stably identified and matched under different viewing angles and lighting conditions.

[0060] The descriptor of a feature point is used to describe the image information around the feature point. It is usually a vector containing a series of measurement values ​​extracted from the image around the feature point. These measurement values ​​can reflect the local structure and texture information of the image around the feature point, so that the descriptor can be used to match the same feature point even when the image is rotated, scaled or the illumination changes.

[0061] Through feature point detection, feature extraction and descriptor generation, effective recognition and matching of landmark points in multiple frames of ground images can be achieved.

[0062] Step 103 , for each group of two adjacent ground image frames in the multiple ground image frames, the descriptors of the feature points are matched using a matching algorithm to obtain matching pairs corresponding to each group of two adjacent ground image frames.

[0063] In this step, in the multiple frames of ground images, each pair of consecutive or adjacent images is focused on, because the changes between adjacent frames are usually minimal, making the matching of feature points more accurate and reliable.

[0064] Through the matching algorithm, one or more sets of feature point matching pairs can be found in each pair of adjacent frames. Each matching pair contains feature points that are considered to be the same physical point in both images. These matching pairs are the basis for subsequent image analysis and can be used to calculate transformations between images (such as translation, rotation, scaling, etc.).

[0065] Step 104, using a pose estimation algorithm to perform pose estimation on the matching pairs corresponding to each group of two adjacent ground image frames, determine the relative pose between each group of two adjacent ground image frames, and determine the global pose of the drone based on the relative pose between each group of two adjacent ground image frames.

[0066] In this step, pose estimation refers to determining the position and attitude of an object (in this case, a drone) in space. Position and attitude are usually represented by a 6-dimensional vector, including 3-dimensional position (x, y, z) and 3-dimensional attitude angles (such as pitch, yaw, and roll).

[0067] For each set of two adjacent ground images, the relative pose transformation between the two frames can be calculated using a pose estimation algorithm (such as the PnP algorithm, ICP algorithm, direct method, or indirect method, etc.) and the matching pairs found previously. This transformation includes translation (the movement of the drone in space) and rotation (the change in the drone's attitude).

[0068] With the relative poses between each set of two adjacent frames, the global pose of the drone can be estimated by accumulating these relative poses.

[0069] This is a recursive or cumulative process, starting from the starting frame, gradually applying the relative pose transformation between each pair of adjacent frames, and finally obtaining the position and attitude of the drone in the global coordinate system.

[0070] It should be noted that cumulative error is a potential problem with this approach. Over time, the error may gradually accumulate, resulting in a decrease in the accuracy of the global pose estimate. Therefore, in practical applications, some optimization strategies (such as loop closure detection, pose graph optimization, etc.) may need to be adopted to reduce the impact of cumulative error.

[0071] In summary, this process uses the pose estimation algorithm to match and estimate the pose of two adjacent ground images, and finally determines the global pose of the drone. This is crucial for the drone's autonomous navigation, path planning, obstacle avoidance and other tasks.

[0072] Step 105: determining the mileage information of the drone based on the global pose of the drone and the initial position information of the drone, and determining the navigation information of the drone according to the global pose of the drone and the mileage information of the drone.

[0073] In this step, the initial position information of the drone is the starting position of the drone when it starts to perform a mission or fly. This information is crucial for the subsequent calculation of the moving distance and direction of the drone.

[0074] The initial position information can be obtained through Global Positioning System (GPS) readings before takeoff, manual input, or other positioning technologies.

[0075] Mileage information refers to the distance and direction changes that the drone has traveled from its initial position to its current position.

[0076] This can be calculated by comparing the drone’s global pose to the initial position information, for example by integrating the drone’s velocity vector (perhaps from visual odometry) to estimate how far and in which direction it has moved.

[0077] Navigation information refers to information used to guide a drone to fly along a predetermined path or perform a specific mission.

[0078] This typically includes the drone’s target position, speed, direction, and any necessary obstacle avoidance instructions.

[0079] Navigation information is determined based on the drone’s global position and mileage information. For example, if the drone needs to fly to a specific location, the navigation system will calculate the best flight path and speed based on the current global position and the known target location.

[0080] In summary, the process involves collecting data from the drone’s sensors, processing that data to determine its location and movement, and then generating navigation instructions based on that information to guide the drone to complete its mission safely and efficiently.

[0081] Through the above scheme, the stability and reliability of the UAV in complex environments are ensured by directly extracting and matching features from ground images instead of relying on electromagnetic signals that are susceptible to interference. At the same time, the global pose of the UAV is determined using the pose estimation algorithm, which not only improves the positioning accuracy, but also enhances the autonomy and flexibility of the UAV during navigation. In addition, this method combines the initial position information of the UAV with the global pose, and further derives the mileage information and navigation information of the UAV, providing a solid foundation for the flight control and mission planning of the UAV, thereby ensuring that the UAV can complete various flight missions efficiently and accurately.

[0082] In some embodiments, in step 102, extracting features of landmark points in multiple frames of ground images by a feature point detection algorithm to obtain feature points includes:

[0083] Step A1, using a pre-constructed Hessian matrix to perform interest point recognition processing on landmark points in multiple frames of ground images to obtain multiple interest points.

[0084] Step A2: Filter the plurality of interest points using a non-maximum suppression algorithm to obtain feature points.

[0085] In the above scheme, the Hessian Matrix is ​​a square matrix that describes the local curvature of a multivariate function. In computer vision, especially in feature point detection, the Hessian Matrix is ​​used to detect the local rate of change in an image. These points with high rates of change are considered to be potential "interest points" or "key points". These points are usually parts of the image that are rich in information and remain stable under different viewing angles, such as corner points, edge points, etc.

[0086] By calculating the Hessian matrix of each pixel in the image and analyzing the properties of these matrices such as the determinant or trace, the points of interest in the image can be identified. This process is usually performed on multiple frames of ground images, which means that a series of consecutive or related images are processed to identify the points of interest in each frame. These points of interest are then used as the basis for subsequent processing (such as matching, tracking, etc.).

[0087] Non-Maximum Suppression (NMS) is an algorithm used in edge detection and feature point extraction. Its purpose is to filter out the most significant and representative points from a set of candidate points to reduce redundancy and improve computational efficiency. In the context of feature point extraction, non-maximum suppression compares the response values ​​of each candidate point with other points in its neighborhood (such as the values ​​calculated based on the Hessian matrix) and retains only the local maximum as the final feature point.

[0088] Through the non-maximum suppression algorithm, those points that are not significant or redundant can be filtered out from the multiple interest points initially identified, thereby obtaining a smaller set of feature points with higher quality. These feature points usually have higher stability and recognition, and are more suitable for subsequent image processing tasks.

[0089] In summary, these two steps together constitute a process of extracting high-quality feature points from raw image data, which is very important for many computer vision applications.

[0090] Among them, feature point detection algorithms include SIFT, SURF, ORB, etc.

[0091] In some embodiments, in step 102, determining the descriptor of the feature point includes:

[0092] The feature points are encoded using a hash function to obtain descriptors of the feature points.

[0093] In the above scheme, a hash function is a function that maps an input (in this context, a feature point) to a fixed-size output (i.e., a hash value or encoding). The design of a hash function is usually aimed at minimizing the probability of collision (i.e., different inputs produce the same output) while maintaining computational efficiency. In image processing, hash functions are used to convert complex feature point information into concise binary or numerical codes that can be used for fast comparison and matching.

[0094] A hash function is used to convert the data of the feature point (which may be information such as coordinates, direction, intensity, etc.) into a compact format that is easy to store and compare. This encoding is often called the "descriptor" or "feature vector" of the feature point.

[0095] A descriptor is a representation of a feature point that contains enough information to uniquely (or largely) identify the feature point. By comparing the descriptors of feature points in different images, we can achieve tasks such as image matching, recognition, or splicing. Descriptors are usually designed to be robust to various transformations of the image (such as rotation, scaling, and lighting changes).

[0096] In summary, using hash functions to encode feature points and obtain descriptors of feature points is a key step in computer vision and image processing. It enables efficient comparison and matching of feature points in images, thereby performing various complex image analysis tasks.

[0097] In some embodiments, step 103 includes:

[0098] Step B1, for each set of feature points of two adjacent ground image frames in the multiple ground image frames, the following operations are performed:

[0099] Step B11, determining the relative distances between the descriptors of each feature point of the first ground image frame in two adjacent ground image frames and the descriptors of other feature points of the second ground image frame in two adjacent ground image frames.

[0100] Step B12, selecting other feature points with the smallest relative distance from each relative distance, and taking the other feature points with the smallest relative distance and the corresponding feature points in the first frame of the ground image as matching pairs.

[0101] In the above scheme, for each feature point in the first frame of the ground image, a certain distance (such as Euclidean distance, Manhattan distance, cosine similarity, etc.) is calculated between its descriptor and the descriptors of all other feature points in the second frame of the ground image. The relative distance here refers to the distance between descriptors, which is used to measure the similarity between two feature points.

[0102] The smaller the distance, the more similar the two feature points are, and the more likely they are the projections of the same physical point in the two frames of images.

[0103] For each feature point in the first frame, find the feature point with the smallest distance to its descriptor in the second frame. This process is equivalent to finding a more suitable match for each feature point in the first frame in the second frame.

[0104] The feature point in the first frame and the feature point with the smallest distance to its descriptor in the second frame form a matching pair.

[0105] These matching pairs can be used for subsequent analysis, such as image registration, image transformation estimation, etc.

[0106] This process is one of the basic steps of image feature matching. By comparing the descriptors of feature points in adjacent frame images, it is possible to identify which feature points correspond in the two frame images.

[0107] In some embodiments, in step 104, performing pose estimation on matching pairs corresponding to each group of two adjacent ground image frames using a pose estimation algorithm to determine the relative pose between each group of two adjacent ground image frames includes:

[0108] Step C1, according to the matching pairs corresponding to each group of two adjacent ground image frames, the coordinates of the feature points corresponding to the matching pairs in the world coordinate system are determined by using a triangulation algorithm.

[0109] Step C2, based on the coordinates of the feature points corresponding to the matching pairs in the world coordinate system, a camera pose estimation algorithm is used to perform camera pose estimation processing to obtain the relative pose between each group of two adjacent ground image frames.

[0110] In the above scheme, a triangulation algorithm is used to determine the position of an object in three-dimensional space when the same object is observed from two or more perspectives.

[0111] Here, using the matching pairs corresponding to each set of two adjacent ground images, the three-dimensional coordinates of these matching pairs (i.e., feature points) in the world coordinate system can be calculated through a triangulation algorithm. The world coordinate system is a fixed reference coordinate system used to describe the positions of all objects in the environment.

[0112] Camera pose refers to the position and attitude (i.e. direction and tilt) of the camera in three-dimensional space.

[0113] Based on the coordinates of the feature points in the world coordinate system that have been determined by triangulation, the camera pose estimation algorithm can be used to estimate the relative pose of the camera when each set of two adjacent ground images was taken. This usually involves the process of minimizing the feature point projection error, that is, finding an optimal camera pose so that the feature points projected from the pose to the image plane best match the feature points in the actual image.

[0114] Ultimately, this process outputs the relative pose between each set of two adjacent ground images, which includes the translation and rotation of the camera between these two positions.

[0115] This relative pose information is crucial for tasks such as building environment maps, tracking camera motion, and making navigation decisions.

[0116] In summary, this section describes a process of determining the trajectory of a camera in space through image matching, triangulation, and camera pose estimation.

[0117] Among them, the pose estimation algorithm is, for example, the PnP (Perspective-n-Point) algorithm or the ICP (Iterative Closest Point) algorithm.

[0118] In some embodiments, in step 104, determining the global pose of the drone according to the relative pose between each group of two adjacent ground image frames includes:

[0119] The relative poses between each group of two adjacent ground image frames are summed to obtain the global pose of the UAV.

[0120] In the above scheme, by accumulating the relative poses between all two adjacent frames of images (i.e., the process of summing or integrating), the global movement trajectory of the drone during the entire flight process can be gradually constructed. This global pose describes the exact position and orientation of the drone in three-dimensional space from the take-off point to the current moment. This is crucial for the navigation, path planning, and obstacle avoidance of the drone.

[0121] In summary, this process is to analyze the ground image sequence taken by the UAV, use the visual algorithm to calculate the relative pose changes between each two frames of images, and estimate the global pose of the UAV by accumulating these changes.

[0122] In some embodiments, the global pose of the drone includes the current position of the drone.

[0123] In step 105, determining the mileage information of the drone based on the global position of the drone and the initial position information of the drone includes:

[0124] Relative distance processing is performed based on the current position of the drone and the initial position of the drone to obtain the mileage information of the drone.

[0125] In the above scheme, the current position of the drone refers to the spatial coordinates of the drone at a certain moment, which can usually be obtained through the global positioning system (GPS) system, visual positioning system or other positioning technology on the drone. This position information may include longitude, latitude, altitude, etc.

[0126] The initial position of a drone refers to the spatial coordinates when the drone takes off or starts recording mileage information. Like the current position, the initial position is also obtained through corresponding positioning technology.

[0127] Calculate the distance the drone has moved from its initial position to its current position. This usually involves calculations of spatial geometry, such as using the Euclidean distance formula (on a two-dimensional plane) or the distance formula in three-dimensional space to calculate the distance between two points. If the earth is approximately ellipsoidal in shape, more complex geographic coordinate calculation methods may be required to obtain accurate distances.

[0128] Through the above relative distance processing, the total distance flown by the drone from takeoff (or start of recording) to the current moment can be obtained. This distance is the mileage information of the drone. Mileage information is very important for understanding the flight status of the drone, planning the flight path, and monitoring the flight efficiency.

[0129] In summary, this process uses the drone's location information (initial location and current location) to calculate the relative distance between two points to obtain the drone's flight mileage information. This method is simple and intuitive, and is one of the commonly used technical means in drone navigation and monitoring systems.

[0130] In some embodiments, the global pose of the drone includes the flight direction of the drone and the current position of the drone.

[0131] In step 105, determining navigation information of the drone according to the global position of the drone and the mileage information of the drone includes:

[0132] The flight direction of the drone, the current position of the drone and the mileage information of the drone are combined to obtain navigation information of the drone.

[0133] In the above scheme, the flight direction of the drone refers to the direction the drone is currently heading or plans to head towards. The flight direction is usually determined by the drone's navigation system or flight control system based on a preset route, user input, or an automatic obstacle avoidance algorithm.

[0134] The current position of a drone is the precise position of the drone at a certain moment relative to a reference point (such as a take-off point, a target point, or a coordinate on the earth's surface). The position information of a drone is usually obtained through GPS (Global Positioning System), Inertial Navigation System (INS) or other positioning technologies.

[0135] The mileage information of a drone indicates the total distance the drone has flown since takeoff or the distance of a specific flight segment. Mileage information is very important for evaluating the flight efficiency and remaining endurance of the drone and planning subsequent flight missions.

[0136] Combining these three elements, the navigation information of the drone is a data set that integrates the current state of the drone and the direction of future action. This navigation information is crucial for the autonomous flight, path planning, obstacle avoidance, and mission execution of the drone. By analyzing and utilizing this navigation information, the drone's control system can make more accurate and efficient flight decisions, ensuring that the drone can complete the scheduled flight mission safely and accurately.

[0137] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0138] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0139] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a drone navigation device based on visual mileage.

[0140] refer to Figure 2 , the UAV navigation device based on visual mileage comprises:

[0141] The acquisition module 201 is configured to acquire multiple frames of ground images of the drone for the same landmark point during flight, as well as initial position information of the drone;

[0142] The feature extraction module 202 is configured to extract features of landmark points in multiple frames of ground images by using a feature point detection algorithm to obtain feature points and determine descriptors of the feature points, wherein the descriptors are used to describe image information around the feature points;

[0143] The matching module 203 is configured to use a matching algorithm to match the descriptors of the feature points of each group of two adjacent ground images in the multiple ground images, and obtain matching pairs corresponding to each group of two adjacent ground images;

[0144] The pose estimation module 204 is configured to use a pose estimation algorithm to perform pose estimation on the matching pairs corresponding to each group of two adjacent ground image frames, determine the relative pose between each group of two adjacent ground image frames, and determine the global pose of the UAV according to the relative pose between each group of two adjacent ground image frames;

[0145] The navigation module 205 is configured to determine the mileage information of the drone based on the global position of the drone and the initial position information of the drone, and determine the navigation information of the drone according to the global position of the drone and the mileage information of the drone.

[0146] In some embodiments, the feature extraction module 202 is specifically configured to:

[0147] Using the pre-constructed Hessian matrix, the landmark points in multiple frames of ground images are processed for interest point recognition to obtain multiple interest points;

[0148] The plurality of interest points are filtered using a non-maximum suppression algorithm to obtain feature points.

[0149] In some embodiments, the feature extraction module 202 is specifically configured to:

[0150] The feature points are encoded using a hash function to obtain descriptors of the feature points.

[0151] In some embodiments, the matching module 203 is specifically configured to:

[0152] For each set of feature points of two adjacent ground images in multiple ground images, the following operations are performed:

[0153] Determine the relative distances between the descriptors of each feature point of the first ground image in the two adjacent ground image frames and the descriptors of other feature points of the second ground image in the two adjacent ground image frames;

[0154] Other feature points with the smallest relative distance are selected from the relative distances, and the other feature points with the smallest relative distance and the corresponding feature points in the first frame of the ground image are used as matching pairs.

[0155] In some embodiments, the posture estimation module 204 is specifically configured to:

[0156] According to the matching pairs corresponding to each group of two adjacent ground image frames, the coordinates of the feature points corresponding to the matching pairs in the world coordinate system are determined by using a triangulation algorithm;

[0157] Based on the coordinates of the feature points corresponding to the matching pairs in the world coordinate system, a camera pose estimation algorithm is used to perform camera pose estimation processing to obtain the relative pose between each group of two adjacent frames of ground images.

[0158] In some embodiments, the posture estimation module 204 is specifically configured to:

[0159] The relative poses between each group of two adjacent ground image frames are summed to obtain the global pose of the UAV.

[0160] In some embodiments, the global pose of the drone includes the current position of the drone;

[0161] The navigation module 205 is specifically configured as follows:

[0162] Relative distance processing is performed based on the current position of the drone and the initial position of the drone to obtain the mileage information of the drone.

[0163] In some embodiments, the global pose of the drone includes the flight direction of the drone and the current position of the drone;

[0164] The navigation module 205 is specifically configured as follows:

[0165] The flight direction of the drone, the current position of the drone and the mileage information of the drone are combined to obtain navigation information of the drone.

[0166] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0167] The device of the above embodiment is used to implement the corresponding visual mileage-based drone navigation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0168] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the visual mileage-based drone navigation method described in any of the above embodiments is implemented.

[0169] Figure 3 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 301, a memory 302, an input / output interface 303, a communication interface 304, and a bus 305. The processor 301, the memory 302, the input / output interface 303, and the communication interface 304 are connected to each other in communication within the device through the bus 305.

[0170] The processor 301 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0171] The memory 302 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 302 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 302 and called and executed by the processor 301.

[0172] The input / output interface 303 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0173] The communication interface 304 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0174] The bus 305 comprises a pathway for transmitting information between the various components of the device (eg, the processor 301 , the memory 302 , the input / output interface 303 , and the communication interface 304 ).

[0175] It should be noted that, although the above device only shows the processor 301, the memory 302, the input / output interface 303, the communication interface 304 and the bus 305, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0176] The electronic device of the above-mentioned embodiment is used to implement the corresponding visual mileage-based drone navigation method in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0177] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the visual mileage-based drone navigation method as described in any of the above embodiments.

[0178] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0179] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the visual mileage-based drone navigation method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0180] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0181] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0182] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0183] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A UAV navigation method based on visual mileage, characterized in that: include: Acquire multiple frames of ground images of the drone for the same landmark point during flight, as well as initial position information of the drone; Extracting features of landmark points in multiple frames of ground images by a feature point detection algorithm to obtain feature points, and determining descriptors of the feature points, wherein the descriptors are used to describe image information around the feature points; For each group of two adjacent ground images in the multiple ground images, the descriptors of the feature points are matched using a matching algorithm to obtain matching pairs corresponding to each group of two adjacent ground images; The pose estimation algorithm is used to estimate the pose of the matching pairs corresponding to each group of two adjacent ground image frames, determine the relative pose between each group of two adjacent ground image frames, and determine the global pose of the UAV based on the relative pose between each group of two adjacent ground image frames; The mileage information of the drone is determined based on the global position of the drone and the initial position information of the drone, and the navigation information of the drone is determined according to the global position of the drone and the mileage information of the drone.

2. The method according to claim 1, characterized in that The feature point detection algorithm is used to extract features of landmark points in multiple frames of ground images to obtain feature points, including: Using the pre-constructed Hessian matrix, the landmark points in multiple frames of ground images are processed for interest point recognition to obtain multiple interest points; The plurality of interest points are filtered using a non-maximum suppression algorithm to obtain feature points.

3. The method according to claim 1, characterized in that The determining of the descriptor of the feature point comprises: The feature points are encoded using a hash function to obtain descriptors of the feature points.

4. The method according to claim 1, characterized in that: The method of matching the descriptors of the feature points of each group of two adjacent ground images in the plurality of ground images by using a matching algorithm to obtain matching pairs corresponding to each group of two adjacent ground images includes: For each set of feature points of two adjacent ground images in multiple ground images, the following operations are performed: Determine the relative distances between the descriptors of each feature point of the first ground image in the two adjacent ground image frames and the descriptors of other feature points of the second ground image in the two adjacent ground image frames; Other feature points with the smallest relative distance are selected from the relative distances, and the other feature points with the smallest relative distance and the corresponding feature points in the first frame of the ground image are used as matching pairs.

5. The method according to claim 1, characterized in that The method of using a pose estimation algorithm to perform pose estimation on matching pairs corresponding to each group of two adjacent ground image frames to determine the relative pose between each group of two adjacent ground image frames includes: According to the matching pairs corresponding to each group of two adjacent ground image frames, the coordinates of the feature points corresponding to the matching pairs in the world coordinate system are determined by using a triangulation algorithm; Based on the coordinates of the feature points corresponding to the matching pairs in the world coordinate system, a camera pose estimation algorithm is used to perform camera pose estimation processing to obtain the relative pose between each group of two adjacent frames of ground images.

6. The method according to claim 1, characterized in that Determining the global posture of the UAV according to the relative posture between each group of two adjacent ground image frames includes: The relative poses between each group of two adjacent ground image frames are summed to obtain the global pose of the UAV.

7. The method according to claim 1, characterized in that The global pose of the drone includes the current position of the drone; The determining of the mileage information of the drone based on the global posture of the drone and the initial position information of the drone includes: Relative distance processing is performed based on the current position of the drone and the initial position of the drone to obtain the mileage information of the drone.

8. The method according to claim 1, characterized in that The global pose of the UAV includes the flight direction of the UAV and the current position of the UAV; The determining the navigation information of the drone according to the global position and mileage information of the drone includes: The flight direction of the drone, the current position of the drone and the mileage information of the drone are combined to obtain navigation information of the drone.

9. A drone navigation device based on visual mileage, characterized in that: include: An acquisition module is configured to acquire multiple frames of ground images of the drone for the same landmark point during flight, as well as initial position information of the drone; A feature extraction module is configured to extract features of landmark points in multiple frames of ground images by using a feature point detection algorithm to obtain feature points and determine descriptors of the feature points, wherein the descriptors are used to describe image information around the feature points; The matching module is configured to use a matching algorithm to match the descriptors of the feature points of each group of two adjacent ground images in the multiple ground images, and obtain matching pairs corresponding to each group of two adjacent ground images; A pose estimation module is configured to perform pose estimation on matching pairs corresponding to each group of two adjacent ground image frames using a pose estimation algorithm, determine a relative pose between each group of two adjacent ground image frames, and determine a global pose of the UAV based on the relative pose between each group of two adjacent ground image frames; The navigation module is configured to determine the mileage information of the drone based on the global position of the drone and the initial position information of the drone, and to determine the navigation information of the drone according to the global position of the drone and the mileage information of the drone.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images