Pose information determination method and device of unmanned aerial vehicle, electronic equipment and storage medium

Abstract

The invention discloses an unmanned aerial vehicle pose information determination method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a panoramic image of an unmanned aerial vehicle, and determining first information which is used for indicating an error between a first feature point and a second feature point; determining second information through an inertial measurement unit arranged on the unmanned aerial vehicle, wherein the second information is used for indicating an error generated in a pre-integration process of inertial measurement unit data of the unmanned aerial vehicle from a third moment to a fourth moment; determining second pose information of the unmanned aerial vehicle from a third moment to a fourth moment according to the first information, the second information and the vibration condition of the unmanned aerial vehicle; wherein the vibration condition of the unmanned aerial vehicle is used for controlling and adjusting the credibility of the second information. By adopting the scheme of the invention, accurate determination of the pose information of the unmanned aerial vehicle can be realized, and the accuracy and robustness of unmanned aerial vehicle positioning can be significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicles (UAVs), and particularly to a method, apparatus, electronic device, and storage medium for determining the pose information of a UAV. Background Art

[0002] UAVs have broad application prospects in various inspection tasks. The traditional inspection method relies on the Global Navigation Satellite System (GNSS) positioning information determined by the UAV itself first, and then the position information of the UAV can be determined after manual processing. However, the stability of the GNSS signal in this inspection method is easily affected by the environment. Especially in environments with strong electromagnetic interference, occlusion, and low light problems such as substations, it will directly cause the GNSS signal to be unstable, unable to determine accurate GNSS positioning information, and thus unable to determine the precise pose information of the UAV, and unable to accurately position the flight of the UAV, affecting the inspection efficiency. Therefore, there is an urgent need for a UAV inspection solution that can cope with such complex environments. Summary of the Invention

[0003] The present invention provides a method, apparatus, electronic device, and storage medium for determining the pose information of a UAV, which can accurately determine the pose information of the UAV and significantly improve the accuracy and robustness of UAV positioning.

[0004] In a first aspect, the present invention provides a method for determining the pose information of a UAV, including:

[0005] Obtain a panoramic image of the UAV and determine first information, where the first information is used to indicate the error between a first feature point and a second feature point. The first feature point is a feature point predicted for a first panoramic image based on the first pose information of the UAV, and the first pose information is determined based on the data of the inertial measurement unit of the UAV. The second feature point is a feature point in a second panoramic image, and the first panoramic image and the second panoramic image are respectively panoramic images collected by the UAV at adjacent first and second moments;

[0006] Determine second information through an inertial measurement unit disposed on the UAV, where the second information is used to indicate the error generated during the pre-integration process of the inertial measurement unit data of the UAV from a third moment to a fourth moment. The time interval between the third moment and the first moment is not greater than a preset time interval, and the time interval between the fourth moment and the second moment is not greater than a preset time interval. The inertial measurement unit data of the UAV is used to determine the pose information of the UAV;

[0007] Determine the second pose information of the drone from the third moment to the fourth moment according to the first information, the second information, and the vibration condition of the drone; wherein, the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information.

[0008] In a second aspect, the present invention further provides a device for determining the pose information of a drone, including:

[0009] A first information determination module, configured to obtain a panoramic image of the drone and determine first information, where the first information is used to indicate the error between a first feature point and a second feature point, the first feature point is a feature point predicted from the feature points in the first panoramic image based on the first pose information of the drone, the first pose information is determined based on the data of the inertial measurement unit of the drone, the second feature point is a feature point in the second panoramic image, and the first panoramic image and the second panoramic image are respectively panoramic images collected by the drone at adjacent first and second moments;

[0010] A second information determination module, configured to determine second information through an inertial measurement unit disposed on the drone, where the second information is used to indicate the error generated during the pre-integration process of the inertial measurement unit data of the drone from the third moment to the fourth moment, the time interval between the third moment and the first moment is not greater than a preset time interval, the time interval between the fourth moment and the second moment is not greater than a preset time interval, and the inertial measurement unit data of the drone is used to determine the pose information of the drone;

[0011] A second pose information determination module, configured to determine the second pose information of the drone from the third moment to the fourth moment according to the first information, the second information, and the vibration condition of the drone; wherein, the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0013] One or more processors;

[0014] A storage device for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the pose information of a drone provided in any embodiment of the present invention.

[0016] Fourthly, an embodiment of the present invention further provides a storage medium containing computer-executable instructions, which are used to execute the method for determining the pose information of a drone provided in any embodiment of the present invention when executed by a computer processor.

[0017] With this technical solution, by obtaining the panoramic image of the drone and determining the first information, the first information is used to indicate the error between the first feature point and the second feature point. Since the feature points in two frames of panoramic images corresponding to adjacent moments collected by the drone have a kinematic correlation, specifically, the first information is determined based on the first pose information of the drone, and the first pose information is essentially determined based on the data of the inertial measurement unit of the drone. In this way, a correlation relationship can be constructed between the panoramic image and the data of the inertial measurement unit, so that the first information acts as a constraint in terms of visual images, thereby helping to determine more accurate pose information. At the same time, the second information is also determined by the inertial measurement unit placed on the drone, and the second information is used to indicate the error generated during the pre-integration process of the drone inertial measurement unit data, realizing the quantification of the error generated during the pre-integration process. Furthermore, this error can be adjusted and improved, thereby helping to determine more accurate pose information. Under these conditions, according to the first information, the second information, and the vibration situation of the drone, the errors indicated by the two are optimized and adjusted. The vibration situation of the drone is negatively correlated with the credibility of the second information. This solution can control and adjust the credibility of the second information through the vibration situation of the drone, effectively reducing the negative impact brought by high vibration, and being able to determine more accurate inertial measurement unit data, thereby determining the second pose information of the drone. This solution can achieve the accurate determination of the pose information of the drone, significantly improving the accuracy and robustness of drone positioning; especially enabling the drone to still achieve sub-meter or even higher positioning accuracy in harsh environments such as complex electromagnetic interference, low light, and strong vibration.

[0018] The above-mentioned invention content is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description. And in order to make the above and other purposes, features, and advantages of the present invention more obvious and understandable, the following specifically illustrates the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Combined with the drawings and referring to the following specific embodiments, the above and other features, advantages, and aspects of each embodiment of the present invention will become more obvious. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic, and the original components and elements are not necessarily drawn to scale.

[0020] Figure 1It is a schematic flowchart of a method for determining the pose information of a drone provided by an embodiment of the present invention;

[0021] Figure 2 It is a schematic flowchart of another method for determining the pose information of a drone provided by an embodiment of the present invention;

[0022] Figure 3 It is a schematic structural diagram of a device for determining the pose information of a drone provided by an embodiment of the present invention;

[0023] Figure 4 It is a schematic structural diagram of an electronic device for implementing the method for determining the pose information of a drone provided by an embodiment of the present invention. Detailed Embodiments

[0024] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Instead, these embodiments are provided to more thoroughly and completely understand the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0025] It should be understood that the various steps recorded in the method embodiments of the present invention can be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this regard.

[0026] The term "including" and its variants used herein are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0027] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions executed by these devices, modules or units or their interdependent relationships.

[0028] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly stated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] Figure 1 FIG. 4 is a schematic flow chart of a method for determining the pose information of a drone provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the situation of determining the pose information of a drone. This method can be executed by a device for determining the pose information of a drone. The device for determining the pose information of a drone can be implemented in the form of software and / or hardware and is generally integrated on any electronic device with network communication functions. The electronic device can be a mobile terminal, a PC terminal, a server, etc. As Figure 1 shown, the method for determining the pose information of a drone in the embodiment of the present invention may include the following processes:

[0031] S110. Obtain a panoramic image of the drone and determine first information, where the first information is used to indicate the error between a first feature point and a second feature point. The first feature point is a feature point predicted from the feature points in the first panoramic image based on the first pose information of the drone. The first pose information is determined based on the data of the inertial measurement unit of the drone. The second feature point is a feature point in the second panoramic image. The first panoramic image and the second panoramic image are respectively panoramic images collected by the drone at adjacent first and second moments.

[0032] Wherein, a camera device is configured on the drone. After the flight environment of the drone is photographed by the camera device, the corresponding panoramic image can be determined. The panoramic image can cover a 360-degree viewing range, can provide richer environmental textures, and makes there be no blind area problem in the viewing angle. The first panoramic image and the second panoramic image are respectively panoramic images collected by the drone at adjacent first and second moments.

[0033] Since the determined panoramic images of each frame are temporally correlated, the feature points in the panoramic image will be continuously tracked as the camera device continuously photographs. This makes there be a certain transformation relationship between the feature points of each panoramic image kinematically. Specifically, for the first panoramic image and the second panoramic image, since the shooting times of the two are adjacent, if the accurate pose information of the drone corresponding to the two frames of images can be determined, then after the feature points in the first panoramic image are transformed by means of the accurate position information, theoretically, they can coincide with the feature points in the second panoramic image.

[0034] In this embodiment, the panoramic image of the drone can be obtained first. Then, based on the first pose information of the drone, the feature points in the first panoramic image are predicted to obtain the first feature points, and the first information is used to indicate the error between the first feature points and the second feature points. Here, the second feature points are the feature points in the second panoramic image, and the first pose information is determined based on the data of the inertial measurement unit of the drone. The pose information includes position information and attitude information. It should be noted that if the first pose information determined based on the data of the inertial measurement unit of the drone is accurate enough, then the error between the first feature points and the second feature points will be small enough, indicating that the determined pose information is accurate at this time. However, the determination process of the data of the inertial measurement unit of the drone is often affected by measurement noise and zero-bias error, etc., making the determined first pose information inaccurate, and then resulting in an error between the first feature points determined based on the first pose information and the second feature points. And this solution precisely determines the first information when the drone acquires the panoramic image by means of the correlation relationship between these data, so that the first information acts as a constraint in terms of visual images, and aims to continue to adjust the error between the first feature points and the second feature points to the minimum, thereby helping to determine more accurate pose information.

[0035] S120. Determine the second information through the inertial measurement unit placed on the drone. The second information is used to indicate the error generated during the pre-integration process of the inertial measurement unit data of the drone from the third moment to the fourth moment. The time interval between the third moment and the first moment is not greater than the preset time interval, and the time interval between the fourth moment and the second moment is not greater than the preset time interval. The inertial measurement unit data of the drone is used to determine the pose information of the drone.

[0036] Among them, the inertial measurement unit (Inertial Measurement Unit, IMU) is a sensor device mainly composed of a gyroscope, an accelerometer, a magnetometer, etc. The inertial measurement unit can be used to measure the inertial measurement unit data such as the angular velocity and acceleration of the drone on the X-axis, Y-axis, and Z-axis during flight. The pose information of the drone can be determined through the inertial measurement unit data. Usually, the sampling frequency of the inertial measurement unit is relatively high, and a large amount of inertial measurement unit data can be measured in a short time. To save the computational amount of these data, generally, part of the inertial measurement unit data is selected from the continuous inertial measurement unit data within a period of time, and the selected part of the inertial measurement unit data is pre-integrated, so as to integrate the inertial measurement unit data generated during this period into one result.

[0037] Specifically, in this embodiment, the inertial measurement unit data of the drone from the third moment to the fourth moment will be pre-integrated by the inertial measurement unit placed on the drone. Among them, the time interval between the third moment and the first moment is not greater than the preset time interval, and the time interval between the fourth moment and the second moment is not greater than the preset time interval. That is, it is necessary to determine the inertial measurement unit data corresponding to the two times closest to the acquisition times of the first panoramic image and the second panoramic image respectively, and pre-integrate the determined two frames of inertial measurement unit data. At the same time, due to the inherent measurement noise and zero bias error in the inertial measurement unit, these errors will also accumulate continuously during the pre-integration process. In this embodiment, the second information when the drone performs inertial measurement unit measurement is determined, and the second information is used to indicate the error generated during the pre-integration process of the inertial measurement unit data of the drone from the third moment to the fourth moment, realizing the quantification of the error generated during the pre-integration process. Then, the error can be continuously adjusted and improved subsequently. Such second information can thus act as a constraint condition. If the error generated during this pre-integration process can be minimized, the determined inertial measurement unit data will be more accurate, and thus the corresponding accurate pose information can be determined.

[0038] S130. Determine the second pose information of the drone from the third moment to the fourth moment according to the first information, the second information, and the vibration condition of the drone; wherein, the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information.

[0039] Specifically, both the first information and the second information determined above can serve as constraints on the pose relationship, and both the first information and the second information can reflect the influence relationship between the inertial measurement unit data and the respective errors corresponding to them. However, the constraint effects of the two are not exactly the same. In particular, the constraint effect of the second information is affected by the real-time vibration situation of the UAV. For example, the more severe the vibration of the UAV, the greater the error generated during the pre-integration of the inertial measurement unit data. Then, such second information is quite untrustworthy. The vibration situation of the UAV is negatively correlated with the credibility of the second information. And the solution of this embodiment exactly takes these factors into account, and can construct an association relationship based on the first information, the second information, and the vibration situation of the UAV, so as to continuously adjust and reduce the errors indicated by the first information and the second information respectively, so as to continuously optimize and adjust the inertial measurement unit data. At the same time, the vibration situation of the UAV is also used to control and adjust the credibility of the second information, which can effectively reduce the negative impact brought by the high-vibration situation, and allocate relatively reasonable credibility to different degrees of constraint conditions, so as to be able to determine the second pose information of the UAV from the third moment to the fourth moment. In other words, compared with the first pose information, the obtained second pose information is essentially a precise pose information obtained after optimization and adjustment.

[0040] Exemplarily, the first information and the second information can be fused to jointly serve as a constraint. When the errors indicated by the first information and the second information both reach the minimum, the second pose information of the corresponding UAV from the third moment to the fourth moment is determined. For another example, it can also be that when the errors indicated by the first information and the second information can reach the minimum as a whole, the second pose information of the corresponding UAV from the third moment to the fourth moment is determined. The optimization methods and optimization objectives can be set differently based on actual needs.

[0041] It should be noted that for the solution of this embodiment, on the one hand, starting from the feature points of the actually acquired panoramic image, an association relationship between relevant feature points is constructed based on the first pose information, and the first information that can reflect the feature point error is determined. On the other hand, starting from the process of pre-integrating the inertial measurement unit data, the second information that can quantify the error generated during the pre-integration process is determined. Then, with the help of such first information and second information, continuous optimization and adjustment can be carried out. At the same time, during the optimization and adjustment process, the negative impact brought by the high-vibration situation is also considered, and the vibration situation of the UAV is used to allocate relatively reasonable credibility to these two pieces of information, and then a more precise second pose information of the UAV is determined after the optimization and adjustment.

[0042] With this technical solution, by obtaining the panoramic image of the drone and determining the first information, where the first information is used to indicate the error between the first feature point and the second feature point. Since the feature points in two adjacent frames of panoramic images collected by the drone have a kinematic correlation, specifically, the first information is determined based on the first pose information, and the first pose information is essentially determined based on the data of the drone's inertial measurement unit. In this way, a correlation relationship can be constructed between the panoramic image and the data of the inertial measurement unit, so that the first information serves as a constraint in terms of visual images, thereby helping to determine more accurate pose information. At the same time, the second information is also determined by the inertial measurement unit placed on the drone, where the second information is used to indicate the error generated during the pre-integration process of the drone's inertial measurement unit data, realizing the quantization of the error generated during the pre-integration process. Furthermore, this error can be adjusted and improved, thus helping to determine more accurate pose information. Under these conditions, according to the first information, the second information, and the vibration condition of the drone, the errors indicated by the two are optimized and adjusted, where the vibration condition of the drone is negatively correlated with the credibility of the second information. This solution can control and adjust the credibility of the second information through the vibration condition of the drone, effectively reducing the negative impact in the case of high vibration, and being able to determine more accurate inertial measurement unit data, thereby determining the second pose information of the drone. This solution can achieve the accurate determination of the drone's pose information, significantly improving the accuracy and robustness of the drone's positioning; especially enabling the drone to still achieve sub-meter or even higher positioning accuracy in harsh environments such as complex electromagnetic interference, low light, and strong vibration.

[0043] Figure 2 FIG. 4 is a schematic flowchart of another method for determining the pose information of a drone provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the process of determining the second pose information of the drone from the third moment to the fourth moment according to the first information, the second information, and the vibration condition of the drone on the basis of the technical solution of the foregoing embodiment. This embodiment can be combined with each optional solution in one or more of the foregoing embodiments. As Figure 2 shown, the method for determining the pose information of the drone according to the embodiment of the present invention may include the following process:

[0044] S210. Obtain the panoramic image of the drone and determine the first information, where the first information is used to indicate the error between the first feature point and the second feature point. The first feature point is the feature point predicted for the feature points in the first panoramic image based on the first pose information of the drone, and the first pose information is determined based on the data of the inertial measurement unit of the drone. The second feature point is the feature point in the second panoramic image. The first panoramic image and the second panoramic image are the panoramic images respectively captured by the drone at adjacent first and second moments.

[0045] As an optional but non-limiting implementation manner, determining the first information may include the following steps A1 - A4:

[0046] Step A1: Determine the corresponding first panoramic feature point cloud based on the first panoramic image.

[0047] Among them, the panoramic feature point cloud is composed of a series of three-dimensional feature points, and each of the three-dimensional feature points can be the feature points recognized corresponding to the panoramic image. The panoramic feature point cloud can provide a panoramic view and rich three-dimensional information.

[0048] Optionally, in this embodiment, four sets of monocular fisheye cameras are configured on the drone. Correspondingly, determining the corresponding first panoramic feature point cloud based on the first panoramic image includes: obtaining four monocular images captured by the monocular fisheye cameras; constructing binocular images based on the four monocular images to determine the first panoramic image; extracting feature points from the first panoramic image based on a preset feature extraction algorithm to determine the first panoramic feature point cloud.

[0049] Among them, the four sets of monocular fisheye cameras can be configured in the front, rear, left, and right directions of the drone. By configuring the four sets of monocular fisheye cameras on the drone, the environmental information in a 360-degree view can be collected, achieving an omnidirectional view coverage, and effectively compensating for the blind spots and occlusion problems existing in a single view.

[0050] Specifically, four monocular images captured by a monocular fisheye camera can be obtained. For each monocular image, the monocular image can be first divided into two parts, so that eight monocular images can be obtained. Then, for each of these eight monocular images, it can be paired with the corresponding monocular image of the adjacent fisheye camera to form a pair of binocular images. And so on, four pairs of binocular images can be constructed from these eight divided monocular images, and then the first panoramic image can be determined. Next, a preset feature extraction algorithm can be used to extract feature points from the first panoramic image. For example, the preset feature extraction algorithm can be the SIFT algorithm (Scale-Invariant Feature Transform), or the ORB algorithm (Oriented FAST and Rotated BRIEF); or, a feature extraction network based on deep learning can also be used to extract feature points from the first panoramic image. Since the overlapping fields of view are included in these four pairs of binocular images, they can be used to extract three-dimensional space information, and then the first panoramic feature point cloud can be determined.

[0051] In addition, during the process of image acquisition and processing, operations such as image correction and / or enhancement processing can also be performed first. For example, due to the inherent distortion effect of the fisheye camera, the originally acquired images usually have obvious radial distortion. Then, a fisheye image correction algorithm is used to convert the fisheye image into a standard perspective image. For example, the pinhole camera model and the radial distortion model can be used to correct the image. Also, for example, the acquired images may still have problems of insufficient contrast and noise in low-light environments. Then, the Adaptive Histogram Equalization (AHE) or Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm can be used to enhance the image to improve the brightness and contrast of the image, so as to improve the reliability of subsequent feature extraction. The specific processing method can be set based on actual needs and will not be elaborated here.

[0052] Step A2: Predict the positions of the third feature points in the first panoramic feature point cloud based on the first pose information and project them in the second panoramic image to obtain the first feature points.

[0053] Among them, the third feature points are the feature points in the first panoramic feature point cloud. Specifically, for each three-dimensional feature point in the first panoramic feature point cloud corresponding to the first moment, based on the first pose information, translation transformation and rotation transformation can be performed on each third feature point, so as to predict the position where the third feature point should be at the second moment, and project the predicted three-dimensional feature points in the second panoramic image to obtain the first feature points.

[0054] Step A3: Determine the reprojection residual based on the distance difference between the first feature point and the second feature point; the second feature point is the feature point in the second panoramic image that matches the third feature point.

[0055] Among them, the second feature point is the feature point in the second panoramic image that matches the third feature point, and can be specifically determined by means of feature matching. In the second panoramic image, the distance difference between the first feature point and the second feature point can be determined. For each other third feature point, the same principle operation is performed, and the corresponding distance differences can be determined, and then the reprojection residuals can be determined. The reprojection residual can reflect the error situation between the corresponding feature points in the first panoramic image and the second panoramic image when the feature points are transformed by means of the first pose information.

[0056] Step A4: Determine the first information based on the reprojection residual.

[0057] Specifically, the corresponding first information can be determined based on the determined reprojection residual. The first information can be described in the form of a function to describe the error situation between the corresponding feature points.

[0058] Adopting this optional solution, starting from the feature points in the actually obtained panoramic image, based on the first pose information, the distance difference between the first feature point and the second feature point is determined, and then the reprojection residual is determined, realizing the establishment of an association relationship between the corresponding feature points in adjacent frame images through the pose information. This association relationship is specifically the error situation between the corresponding feature points. Through such a relationship, a visual constraint condition can be provided for the determination of the pose information through the visual image to help determine the accurate pose information of the drone.

[0059] S220. Determine the second information through the inertial measurement unit placed on the drone. The second information is used to indicate the error generated during the pre-integration process of the inertial measurement unit data of the drone from the third moment to the fourth moment. The time interval between the third moment and the first moment is not greater than the preset time interval, and the time interval between the fourth moment and the second moment is not greater than the preset time interval. The inertial measurement unit data of the drone is used to determine the pose information of the drone.

[0060] As an optional but non-limiting implementation manner, determining the second information through the inertial measurement unit placed on the drone may include the following steps B1 - B4:

[0061] Step B1: Obtain multiple frames of inertial measurement unit data obtained when the drone performs inertial measurement unit measurement and the measurement time corresponding to each frame of inertial measurement unit data.

[0062] Among them, the measurement time refers to the specific time information when the inertial measurement unit data is measured.

[0063] Specifically, the inertial measurement unit measures the data of the inertial measurement unit of the UAV according to a preset sampling frequency, and transmits the measured inertial measurement unit data and the measurement time through a standardized interface. Thus, multiple frames of inertial measurement unit data obtained during the inertial measurement unit measurement of the UAV and the measurement time corresponding to each frame of inertial measurement unit data can be acquired.

[0064] Step B2: Determine the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment.

[0065] Optionally, determining the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment includes: determining the third moment and the fourth moment based on the first moment, the second moment, the measurement time corresponding to each frame of inertial measurement unit data, and a preset time interval; and determining the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment based on the third moment and the fourth moment.

[0066] Specifically, for the measurement time corresponding to each frame of inertial measurement unit data, the third moment and the fourth moment can be determined by means of the preset time interval, so that the time interval between the third moment and the first moment is not greater than the preset time interval, and the time interval between the fourth moment and the second moment is not greater than the preset time interval. It should be noted that due to the difference in the set value of the preset time interval, there may be situations where at least two measurement times that meet the requirements of the third moment are determined, or at least two measurement times that meet the requirements of the fourth moment are determined, etc. For these special situations, the measurement time with the smallest time interval between the measurement time corresponding to each frame of inertial measurement unit data and the first moment is determined as the third moment, and the measurement time with the smallest time interval between the measurement time corresponding to each frame of inertial measurement unit data and the second moment is determined as the fourth moment. That is to say, this embodiment aims to respectively determine the two moments closest to the first moment and the second moment from each measurement time, and the two determined moments are the third moment and the fourth moment. Based on the third moment and the fourth moment, the corresponding inertial measurement unit data can be found, and thus the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment can be determined.

[0067] Step B3: Perform pre-integration on the first inertial measurement unit data and the second inertial measurement unit data to determine the pre-integration residual.

[0068] Among them, the pre-integration residual can be used to measure the difference between the pre-integration increment calculated from the actually measured inertial measurement unit data and the state estimation increment currently obtained through optimization. Specifically, the pre-integration model can perform pre-integration on the first inertial measurement unit data and the second inertial measurement unit data, and determine the pre-integration residual generated during this integration process.

[0069] Step B4: Determine the second information based on the pre-integration residual.

[0070] Specifically, the corresponding second information can be determined based on the determined pre-integration residual. The second information can be described in the form of a function to represent the error situation generated during the pre-integration process of the first inertial measurement unit data and the second inertial measurement unit data.

[0071] By adopting this optional solution, it is possible to quantify the error generated during the pre-integration process of the inertial measurement unit data of the drone from the third moment to the fourth moment, thereby facilitating subsequent solutions to continuously adjust and optimize the error generated during this pre-integration process. After adjusting based on the error situation, more accurate inertial measurement unit data can be determined, and the corresponding accurate pose information can also be determined.

[0072] S230: Determine the first weight corresponding to the first information and the second weight corresponding to the second information; the second weight is determined in real time based on the vibration condition of the drone; the vibration condition of the drone is negatively correlated with the second weight.

[0073] Specifically, after determining the first information and the second information, based on the BA idea (Bundle Adjustment), the first information and the second information can be jointly non-linearly optimized. Based on the theoretical basis of maximum likelihood estimation, the optimal parameters can be solved by weighting and summing the corresponding errors of the two. Specifically, when performing the joint, corresponding weights can be configured for the first information and the second information. The weight information of the two can reflect the influence degree of visual constraints and inertial measurement data constraints on the overall positioning accuracy, thereby balancing the importance of the two when performing the joint.

[0074] Among them, the second weight is determined in real time based on the vibration condition of the drone. The second weight can be used to characterize the credibility of the second information. The vibration condition of the drone is negatively correlated with the second weight. For example, if the vibration condition of the drone is stronger, the error generated during the pre-integration process will be larger. Correspondingly, it is necessary to reduce the credibility of this part of the second information, that is, the second weight determined in real time will be lower. This can effectively reduce the negative impact of the pre-integration error in a high-vibration environment on the overall positioning accuracy. In this embodiment, by using this method of determining the second weight in real time based on the vibration condition of the drone, the negative impact caused by high vibration can be effectively reduced, which helps to improve the overall positioning accuracy and robustness, and thus determine more accurate pose information.

[0075] As an optional but non-limiting implementation manner, determining the first weight corresponding to the first information and the second weight corresponding to the second information may include the following steps C1-C3:

[0076] Step C1: Update the first pre-integration covariance matrix based on the noise influence factor to determine the second pre-integration covariance matrix; the noise influence factor is used to reflect the influence degree of the vibration condition of the drone on the pre-integration covariance matrix; the first pre-integration covariance matrix is the pre-integration covariance matrix obtained by pre-integrating the inertial measurement unit data within the integration time steps corresponding to the third moment and the fourth moment.

[0077] Among them, the first pre-integration covariance matrix is the pre-integration covariance matrix obtained by pre-integrating the inertial measurement unit data within the integration time steps corresponding to the third moment and the fourth moment. By means of the pre-integration covariance matrix, the error situation accumulated during the integration process caused by noise can be reflected. Exemplarily, in this embodiment, the first pre-integration covariance matrix can be determined by the following formula:

[0078] ;

[0079] where i is the third moment; j is the fourth moment; is the first pre-integration covariance matrix obtained during the pre-integration process from the third moment to the fourth moment; is the state transition Jacobian matrix during the pre-integration process from the third moment to the fourth moment; is the pre-integration covariance matrix determined at the third moment; is the transpose matrix of the state transition Jacobian matrix; is the noise influence matrix during the pre-integration process from the third moment to the fourth moment; Q is the noise covariance matrix of the inertial measurement unit; is the transpose matrix of the noise influence matrix.

[0080] Specifically, considering the impact brought by the vibration of the drone, the first pre-integration covariance matrix can be updated based on the noise impact factor to determine the second pre-integration covariance matrix. Among them, the noise impact factor is used to reflect the impact degree of the vibration of the drone on the pre-integration covariance matrix; the larger the value of the noise impact factor, the greater the impact degree of the vibration of the drone on the pre-integration covariance matrix. The noise factor can be specifically determined based on the real-time vibration condition of the drone. It should be noted that the noise factor is not a fixed value. It is specifically determined based on the real-time vibration condition of the drone, and the real-time vibration condition of the drone is not static. Then the corresponding noise factor is determined based on the actual situation. In other words, the determination of the noise factor can be dynamically adjusted based on the actual situation.

[0081] As an optional but non-limiting implementation manner, updating the first pre-integration covariance matrix based on the noise impact factor includes: updating the noise covariance matrix of the inertial measurement unit based on the noise impact factor to obtain the updated noise covariance matrix of the inertial measurement unit; updating the first pre-integration covariance matrix based on the updated noise covariance matrix of the inertial measurement unit.

[0082] Among them, the noise covariance matrix of the inertial measurement unit can be used to reflect the noise level during the measurement process. There is a positive correlation between the noise impact factor and the noise covariance matrix of the inertial measurement unit. The larger the value of the noise impact factor, the larger the corresponding noise covariance matrix of the inertial measurement unit, and the noise covariance matrix of the inertial measurement unit will affect the first pre-integration covariance matrix.

[0083] Specifically, after determining the vibration condition of the drone, the corresponding noise impact factor can be determined, and then the noise covariance matrix of the inertial measurement unit is updated based on the noise impact factor to obtain the updated noise covariance matrix of the inertial measurement unit; and the first pre-integration covariance matrix is updated based on the updated noise covariance matrix of the inertial measurement unit.

[0084] As an optional but non-limiting implementation manner, updating the noise covariance matrix of the inertial measurement unit based on the noise impact factor includes: obtaining the current real-time vibration index of the drone; determining the noise impact factor based on the real-time vibration index and the preset adjustment parameter; the value of the noise impact factor is positively correlated with the vibration intensity during the actual flight of the drone; updating the noise covariance matrix of the inertial measurement unit based on the noise impact factor.

[0085] Among them, the real-time vibration index is used to reflect the current real-time vibration intensity of the UAV. The non-control module on the UAV can obtain the parameters of each sensor configured on the UAV in real time, and process these sensor parameters through the adaptive Kalman filtering method, so as to obtain the current real-time vibration index of the UAV. The preset adjustment parameter is a preset adjustment parameter that can be adapted to the inertial measurement unit of the UAV. The preset adjustment parameter can be initialized before the UAV takes off.

[0086] Specifically, after obtaining the current real-time vibration index of the UAV, the noise influence factor can be determined based on the real-time vibration index and the preset adjustment parameter; if the vibration intensity during the current flight of the UAV increases, the value of the corresponding determined noise influence factor will also increase; if the vibration intensity during the current flight of the UAV decreases, the value of the corresponding determined noise influence factor will also decrease. After determining the noise influence factor, the noise covariance matrix of the inertial measurement unit can be updated, and the current noise level can be reflected by the updated noise covariance matrix of the inertial measurement unit.

[0087] Exemplarily, in this embodiment, the noise influence factor can be determined based on the following formula:

[0088] ;

[0089] Where, is the noise influence factor; is the preset adjustment parameter; is the real-time vibration index. It can be seen from the above formula that when the actual vibration intensity of the UAV is large, the real-time vibration index will increase, which will cause the noise influence factor to be greater than 1; relatively, when the actual vibration intensity of the UAV is small, the real-time vibration index will decrease, which can make the noise influence factor close to 1.

[0090] Exemplarily, after determining the noise influence factor, this embodiment can update the noise covariance matrix of the inertial measurement unit by means of the following formula:

[0091] ;

[0092] Where, is the updated noise covariance matrix of the inertial measurement unit; is the noise influence factor; is the noise covariance matrix of the inertial measurement unit before update. It can be seen from this formula that when the value of the noise influence factor is large, the updated noise covariance matrix of the inertial measurement unit will also become larger.

[0093] Further, after updating the noise covariance matrix of the inertial measurement unit, in this embodiment, the following formula can be used to update the first pre-integration covariance matrix to obtain the second pre-integration covariance matrix:

[0094] ;

[0095] wherein, is the second pre-integration covariance matrix, that is, the updated first pre-integration covariance matrix; is the noise covariance matrix of the inertial measurement unit updated based on the noise influence factor; is the state transition Jacobian matrix during the pre-integration process from the third moment to the fourth moment; is the pre-integration covariance matrix determined at the third moment; is the transpose matrix of the state transition Jacobian matrix; is the noise influence matrix during the pre-integration process from the third moment to the fourth moment; is the transpose matrix of the noise influence matrix.

[0096] Step C2: Determine the second weight corresponding to the second information based on the inverse matrix corresponding to the second pre-integration covariance matrix.

[0097] After determining the second pre-integration covariance matrix, the inverse matrix corresponding to the second pre-integration covariance matrix can be determined, and the determined inverse matrix corresponding to the second pre-integration covariance matrix is used as the second weight corresponding to the second information.

[0098] Step C3: Determine the first weight corresponding to the first information based on the second weight.

[0099] Specifically, after determining the second weight, the first weight corresponding to the first information can be determined based on an identity matrix and the second weight. Specifically, mathematical calculations are performed using the identity matrix and the inverse matrix corresponding to the second pre-integration covariance matrix, and the second pre-integration covariance matrix can be solved and used as the first weight corresponding to the first information.

[0100] By adopting this alternative solution, the corresponding noise influence factor can be determined based on the actual vibration condition of the drone, and this way of adaptively adjusting the noise influence factor can update the pre-integration covariance matrix. Furthermore, the second weight corresponding to the second information is also adaptively and dynamically adjusted accordingly. In particular, it can reduce the negative impact of the pre-integration error in a high-vibration environment on the overall positioning accuracy, thereby helping to accurately determine the pose information of the drone.

[0101] S240. Determine the second pose information of the drone from the third moment to the fourth moment according to the first information, the second information, the first weight, and the second weight.

[0102] Specifically, a target function can be constructed based on the first information, the second information, the first weight, and the second weight, and the target function can be continuously optimized based on the BA (Bundle Adjustment) idea until the target function reaches the optimal solution. At this time, the inertial measurement unit data corresponding to the optimal solution is obtained to determine the second pose information of the drone from the third moment to the fourth moment.

[0103] Among them, the condition for the target function to reach the optimal solution can be that the number of optimizations of the target function reaches the preset number of optimizations, or the value of the target function tends to be stable. Specifically, it can be set differently based on actual needs and will not be elaborated here.

[0104] Exemplarily, the target function constructed in this embodiment can be in the following form:

[0105] ;

[0106] Among them, is the set composed of feature points in the panoramic image; i is the i-th frame of panoramic image obtained; j is the j-th frame of panoramic image obtained; the i-th frame of panoramic image and the j-th frame of panoramic image are panoramic images collected corresponding to two adjacent moments; N is the total number of frames of panoramic images collected; is the first information determined based on the i-th frame of panoramic image and the j-th frame of panoramic image; is the first weight corresponding to the first information; is the determined second information, and the determination of this second information is related to the i-th frame of panoramic image and the j-th frame of panoramic image; is the second weight corresponding to the second information.

[0107] This target function fully reflects the statistical characteristics and noise distribution of the residual term. The nonlinear least squares solution method (for example, the Levenberg-Marquardt algorithm) can be used for global optimization until the target function reaches the optimal solution. At this time, the parameter value corresponding to the optimal solution of the above target function is obtained, so as to determine the second pose information of the drone from the third moment to the fourth moment.

[0108] Adopting this technical solution, by obtaining the panoramic image of the unmanned aerial vehicle (UAV) and determining the first information, the first information is used to indicate the error between the first feature point and the second feature point. Since the feature points in two frames of panoramic images corresponding to adjacent moments collected by the UAV have a kinematic correlation relationship, specifically, the first information is determined based on the first pose information, and the first pose information is substantially determined based on the data of the UAV's inertial measurement unit. In this way, a correlation relationship can be constructed between the panoramic image and the data of the inertial measurement unit, so that the first information serves as a constraint in terms of visual images, thereby helping to determine more accurate pose information. At the same time, the second information is also determined by the inertial measurement unit placed on the UAV, and the second information is used to indicate the error generated during the pre-integration process of the UAV inertial measurement unit data, realizing the quantization of the error generated during the pre-integration process. Furthermore, this error can be adjusted and improved, so as to help determine more accurate pose information. Under these conditions, the first weight corresponding to the first information and the second weight corresponding to the second information are determined; among them, the second weight is determined in real time based on the vibration condition of the UAV. Just by means of this real-time determination method, the negative impact brought by high vibration conditions can be effectively reduced, which helps to improve the overall positioning accuracy and robustness, and more accurate inertial measurement unit data can be determined, so as to determine more accurate second pose information of the UAV. This solution can realize the accurate determination of the UAV pose information, and can significantly improve the accuracy and robustness of the UAV positioning; especially, it can enable the UAV to still achieve sub-meter or even higher positioning accuracy in harsh environments such as complex electromagnetic interference, low light, and strong vibration.

[0109] Figure 3 FIG. is a schematic structural diagram of a device for determining the pose information of a UAV provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the situation of determining the pose information of a UAV. The device for determining the pose information of the UAV can be implemented in the form of software and / or hardware, and is generally integrated on any electronic device with network communication functions. The electronic device can be a mobile terminal, a PC terminal, a server, etc. As Figure 3 shown, the device for determining the pose information of the UAV according to the embodiment of the present invention may include a first information determination module 310, a second information determination module 320, and a second pose information determination module 330. Among them:

[0110] The first information determination module 310 is configured to obtain a panoramic image of the drone and determine first information, where the first information is used to indicate the error between a first feature point and a second feature point. The first feature point is a feature point predicted from the feature points in the first panoramic image based on the first pose information of the drone, and the first pose information is determined based on the data of the inertial measurement unit of the drone. The second feature point is a feature point in the second panoramic image. The first panoramic image and the second panoramic image are respectively panoramic images collected by the drone at adjacent first and second moments.

[0111] The second information determination module 320 is configured to determine second information through the inertial measurement unit disposed on the drone. The second information is used to indicate the error generated during the pre-integration process of the inertial measurement unit data of the drone from the third moment to the fourth moment. The time interval between the third moment and the first moment is not greater than a preset time interval, and the time interval between the fourth moment and the second moment is not greater than a preset time interval. The inertial measurement unit data of the drone is used to determine the pose information of the drone.

[0112] The second pose information determination module 330 is configured to determine the second pose information of the drone from the third moment to the fourth moment according to the first information, the second information, and the vibration condition of the drone. Wherein, the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information.

[0113] Adopting this technical solution, the first information determination module obtains the panoramic image of the unmanned aerial vehicle (UAV) and determines the first information, where the first information is used to indicate the error between the first feature point and the second feature point. Since the feature points in two consecutive panoramic images collected by the UAV have a kinematic correlation, specifically, the first information is determined based on the first pose information, and the first pose information is essentially determined based on the data of the UAV's inertial measurement unit. In this way, a correlation relationship can be constructed between the panoramic image and the inertial measurement unit data, so that the first information serves as a constraint in terms of visual images, thereby helping to determine more accurate pose information. At the same time, the second information determination module is used to determine the second information through the inertial measurement unit placed on the UAV, where the second information is used to indicate the error generated during the pre-integration process of the UAV's inertial measurement unit data, realizing the quantification of the error generated during the pre-integration process. Furthermore, this error can be adjusted and improved, thus helping to determine more accurate pose information. Under these conditions, the second pose information determination module optimally adjusts the errors indicated by the first information and the second information according to the first information, the second information, and the vibration condition of the UAV. Among them, the vibration condition of the UAV is negatively correlated with the credibility of the second information. This solution can control and adjust the credibility of the second information according to the vibration condition of the UAV, effectively reducing the negative impact under high vibration conditions, and being able to determine more accurate inertial measurement unit data, thereby determining the second pose information of the UAV. This solution can achieve the accurate determination of the UAV's pose information, significantly improving the accuracy and robustness of UAV positioning; especially enabling the UAV to still achieve sub-meter or even higher positioning accuracy in harsh environments such as complex electromagnetic interference, low light, and strong vibration.

[0114] As an optional but non-limiting implementation manner, the first information determination module 310 includes a panoramic feature point cloud determination sub-module, a first feature point determination sub-module, a reprojection residual determination sub-module, and a first information determination sub-module. Among them: the panoramic feature point cloud determination sub-module is used to determine the corresponding first panoramic feature point cloud based on the first panoramic image; the first feature point determination sub-module is used to predict the position of the third feature point in the first panoramic feature point cloud based on the first pose information and project it in the second panoramic image to obtain the first feature point; the reprojection residual determination sub-module is used to determine the reprojection residual based on the distance difference between the first feature point and the second feature point; the second feature point is the feature point in the second panoramic image that matches the third feature point; the first information determination sub-module is used to determine the first information based on the reprojection residual.

[0115] As an optional but non-limiting implementation manner, the second information determination module 320 includes a data acquisition sub-module, an inertial measurement unit data determination sub-module, a pre-integration residual determination sub-module, and a second information determination sub-module. Among them: The data acquisition sub-module is configured to acquire multiple frames of inertial measurement unit data obtained when the unmanned aerial vehicle performs inertial measurement unit measurement and the measurement time corresponding to each frame of inertial measurement unit data; The inertial measurement unit data determination sub-module is configured to determine the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment; The pre-integration residual determination sub-module is configured to perform pre-integration on the first inertial measurement unit data and the second inertial measurement unit data to determine a pre-integration residual; The second information determination sub-module is configured to determine the second information based on the pre-integration residual.

[0116] As an optional but non-limiting implementation manner, the second pose information determination module 330 includes a weight determination sub-module and a second pose information determination sub-module. Among them: The weight determination sub-module is configured to determine a first weight corresponding to the first information and a second weight corresponding to the second information; The second weight is determined in real time based on the vibration condition of the unmanned aerial vehicle; The vibration condition of the unmanned aerial vehicle is negatively correlated with the second weight; The second pose information determination sub-module is configured to determine the second pose information of the unmanned aerial vehicle from the third moment to the fourth moment according to the first information, the second information, the first weight, and the second weight.

[0117] As an optional but non-limiting implementation manner, the weight determination sub-module includes a pre-integration covariance matrix update unit, a second weight determination unit, and a first weight determination unit. Among them: The pre-integration covariance matrix update unit is configured to update the first pre-integration covariance matrix based on a noise influence factor to determine a second pre-integration covariance matrix; The noise influence factor is used to reflect the influence degree of the vibration condition of the unmanned aerial vehicle on the pre-integration covariance matrix; The first pre-integration covariance matrix is a pre-integration covariance matrix obtained by performing pre-integration on inertial measurement unit data within the integration time steps corresponding to the third moment and the fourth moment; The second weight determination unit is configured to determine the second weight corresponding to the second information based on the inverse matrix corresponding to the second pre-integration covariance matrix; The first weight determination unit is configured to determine the first weight corresponding to the first information based on the second weight.

[0118] As an optional but non-limiting implementation, the pre-integrated covariance matrix update unit includes a noise covariance matrix update subunit and a pre-integrated covariance matrix update subunit. Specifically: The noise covariance matrix update subunit is configured to update the noise covariance matrix of the inertial measurement unit based on the noise influence factor to obtain the updated noise covariance matrix of the inertial measurement unit; The pre-integrated covariance matrix update subunit is configured to update the first pre-integrated covariance matrix based on the updated noise covariance matrix of the inertial measurement unit.

[0119] As an optional but non-limiting implementation, the noise covariance matrix update subunit is specifically configured to: obtain the current real-time vibration index of the drone; determine the noise influence factor based on the real-time vibration index and a preset adjustment parameter; the value of the noise influence factor is positively correlated with the vibration intensity during the actual flight of the drone; update the noise covariance matrix of the inertial measurement unit based on the noise influence factor.

[0120] The pose information determination device of the drone provided by the embodiments of the present invention can be used to execute the pose information determination method of the drone, and has corresponding functional modules and beneficial effects for executing the pose information determination method of the drone. It should be noted that the various units and modules included in the above device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the embodiments of the present invention.

[0121] Figure 4 It is a schematic structural diagram of an electronic device for implementing a method for determining the pose information of a drone provided by an embodiment of the present invention. The following refers to Figure 4 which shows a schematic structural diagram of an electronic device 400 suitable for implementing an embodiment of the present invention. The terminal device in the embodiments of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present invention.

[0122] Such as Figure 4As shown, the electronic device 400 includes at least one processor 401 and a memory communicatively connected to the at least one processor 401, such as a read-only memory (ROM) 402, a random access memory (RAM) 403, etc. The memory stores a computer program executable by the at least one processor. The processor 401 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 402 or the computer program loaded from the storage unit 408 into the random access memory (RAM) 403. In the (RAM) 403, various programs and data required for the operation of the electronic device 400 can also be stored. The processor 401, the (ROM) 402, and the (RAM) 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404. Multiple components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disc, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The processor 401 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 401 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 401 executes the method for determining the pose information of the drone provided in any embodiment of the present invention.

[0124] Specifically, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium. The computer program contains program codes for executing the method for determining the pose information of the drone shown in the flowchart. When the computer program is executed by a processing device, the above functions defined in the method for determining the pose information of the drone in the embodiment of the present invention are executed.

[0125] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0126] The electronic device provided in the embodiment of the present invention and the method for determining the pose information of the unmanned aerial vehicle provided in the above embodiment belong to the same inventive concept. Technical details not described in detail in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0127] The embodiment of the present invention provides a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the method for determining the pose information of the unmanned aerial vehicle provided in the above embodiment.

[0128] It should be noted that the computer-readable medium in the present invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present invention, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0129] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks. The computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, C++, and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0131] The units involved in the embodiments of the present invention can be implemented in software or in hardware. Among them, the name of the unit does not constitute a limitation to the unit itself in some cases. The functions described above in this article can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0132] In the context of the present invention, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM or Flash Memory), an optical fiber, a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] The above description is only a preferred embodiment of the present invention and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present invention. In addition, although the operations are depicted in a specific order, this should not be construed as requiring that the operations be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. On the contrary, the specific features and acts described above are merely example forms for implementing the claims.

Claims

1. A method for determining the pose information of an unmanned aerial vehicle, characterized in that, The method includes: Obtaining a panoramic image of the drone and determining first information, where the first information is used to indicate the error between a first feature point and a second feature point. The first feature point is a feature point predicted for a feature point in the first panoramic image based on the first pose information of the drone, and the first pose information is determined based on the data of the inertial measurement unit of the drone. The second feature point is a feature point in the second panoramic image, and the first panoramic image and the second panoramic image are respectively panoramic images collected by the drone at adjacent first and second moments; Determining second information through an inertial measurement unit placed on the drone, where the second information is used to indicate the error generated during the pre-integration process of the inertial measurement unit data of the drone from the third moment to the fourth moment. The time interval between the third moment and the first moment is not greater than a preset time interval, and the time interval between the fourth moment and the second moment is not greater than a preset time interval. The inertial measurement unit data of the drone is used to determine the pose information of the drone; Determining the second pose information of the drone from the third moment to the fourth moment according to the first information, the second information, and the vibration condition of the drone; where the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information.

2. The method according to claim 1, characterized in that, The determining of the first information includes: Determining a corresponding first panoramic feature point cloud based on the first panoramic image; Predicting the position of a third feature point in the first panoramic feature point cloud based on the first pose information and projecting it in the second panoramic image to obtain the first feature point; Determining a reprojection residual based on the distance difference between the first feature point and the second feature point; the second feature point is a feature point in the second panoramic image that matches the third feature point; Determining the first information based on the reprojection residual.

3. The method according to claim 1, characterized in that, The determining of the second information through an inertial measurement unit placed on the drone includes: Obtaining multiple frames of inertial measurement unit data obtained when the drone performs inertial measurement unit measurement and the measurement time corresponding to each frame of inertial measurement unit data; Determining the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment; Performing pre-integration on the first inertial measurement unit data and the second inertial measurement unit data to determine a pre-integration residual; Determining the second information based on the pre-integration residual.

4. The method according to claim 1, characterized in that, The determining of the second pose information of the drone from the third moment to the fourth moment according to the first information, the second information, and the vibration condition of the drone includes: Determining a first weight corresponding to the first information and a second weight corresponding to the second information; the second weight is determined in real time based on the vibration condition of the drone; the vibration condition of the drone is negatively correlated with the second weight; Determining the second pose information of the drone from the third moment to the fourth moment according to the first information, the second information, the first weight, and the second weight.

5. The method according to claim 4, characterized in that Determining the first weight corresponding to the first information and the second weight corresponding to the second information includes: Updating a first pre-integrated covariance matrix based on a noise influence factor to determine a second pre-integrated covariance matrix; the noise influence factor is used to reflect the influence degree of the vibration condition of the drone on the pre-integrated covariance matrix; the first pre-integrated covariance matrix is the pre-integrated covariance matrix obtained by pre-integrating the inertial measurement unit data within the integration time steps corresponding to the third moment and the fourth moment; Determining the second weight corresponding to the second information based on the inverse matrix corresponding to the second pre-integrated covariance matrix; Determining the first weight corresponding to the first information based on the second weight.

6. The method according to claim 5, wherein The updating the first pre-integrated covariance matrix based on the noise influence factor includes: Updating the noise covariance matrix of the inertial measurement unit based on the noise influence factor to obtain an updated noise covariance matrix of the inertial measurement unit; Updating the first pre-integrated covariance matrix based on the updated noise covariance matrix of the inertial measurement unit.

7. The method according to claim 6, characterized in that The updating the noise covariance matrix of the inertial measurement unit based on the noise influence factor includes: Obtaining the current real-time vibration index of the drone; Determining the noise influence factor based on the real-time vibration index and a preset adjustment parameter; the value of the noise influence factor is positively correlated with the vibration intensity during the actual flight of the drone; Updating the noise covariance matrix of the inertial measurement unit based on the noise influence factor.

8. A pose information determination device for a drone, characterized in that The device includes: A first information determination module, configured to obtain a panoramic image of the drone and determine first information, where the first information is used to indicate the error between a first feature point and a second feature point, the first feature point is a feature point predicted from the feature points in the first panoramic image based on the first pose information of the drone, the first pose information is determined based on the inertial measurement unit data of the drone, the second feature point is a feature point in the second panoramic image, and the first panoramic image and the second panoramic image are respectively panoramic images collected by the drone at adjacent first and second moments; A second information determination module, configured to determine second information through an inertial measurement unit disposed on the drone, where the second information is used to indicate the error generated during the pre-integration of the inertial measurement unit data of the drone from the third moment to the fourth moment, the time interval between the third moment and the first moment is not greater than a preset time interval, the time interval between the fourth moment and the second moment is not greater than a preset time interval, and the inertial measurement unit data of the drone is used to determine the pose information of the drone; A second pose information determination module, configured to determine the second pose information of the drone from the third moment to the fourth moment according to the first information, the second information, and the vibration condition of the drone; wherein, the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the pose information of the drone according to any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions are used to execute the method for determining the pose information of the drone according to any one of claims 1-7 when executed by a computer processor.

Images