Method, device, electronic device and storage medium for determining posture information of unmanned aerial vehicle

Abstract

The present invention discloses a method, device, electronic device, and storage medium for determining the position and posture information of a drone. The method comprises: acquiring a panoramic image of the drone and determining first information, the first information being used to indicate the error between a first feature point and a second feature point; determining second information using an inertial measurement unit (IMU) mounted on the drone, the second information being used to indicate the error generated during pre-integration of IMU data from a third moment to a fourth moment; and determining second position and posture information of the drone from the third moment to the fourth moment based on the first information, the second information, and the vibration of the drone; wherein the vibration of the drone is used to control and adjust the credibility of the second information. The solution of the present invention can accurately determine the position and posture information of the drone, significantly improving the accuracy and robustness of drone positioning.

Description

Technical Field

[0001] The present invention relates to the field of unmanned aerial vehicles (UAVs), and in particular to a method, device, electronic device, and storage medium for determining the posture information of a UAV. Background Art

[0002] Drones have broad application prospects in various inspection tasks. Traditional inspection methods rely on GNSS positioning information determined by the drone's onboard Global Navigation Satellite System (GNSS), which is then manually processed to determine the drone's location. However, the stability of the GNSS signal in this inspection method is easily affected by environmental interference, especially in environments with strong electromagnetic interference, obstructions, and low light levels, such as substations. These can directly cause GNSS signal instability, making it impossible to determine accurate GNSS positioning information. Consequently, it is impossible to accurately determine the drone's position and orientation, making it impossible to accurately position the drone during flight, thus affecting inspection efficiency. Therefore, there is an urgent need for a drone inspection solution that can cope with these complex environments. Summary of the Invention

[0003] The present invention provides a method, device, electronic device, and storage medium for determining the position and posture information of a drone. These methods can accurately determine the position and posture information of a drone, significantly improving the accuracy and robustness of drone positioning.

[0004] In a first aspect, the present invention provides a method for determining the posture information of a drone, comprising:

[0005] Acquire a panoramic image of a drone and determine first information, where the first information is used to indicate an error between a first feature point and a second feature point, the first feature point being a feature point predicted from a feature point in the first panoramic image based on first pose information of the drone, the first pose information being determined based on inertial measurement unit data of the drone, the second feature point being a feature point in a second panoramic image, the first panoramic image and the second panoramic image being panoramic images correspondingly captured by the drone at adjacent first and second moments, respectively;

[0006] Determining second information by an inertial measurement unit placed on the drone, the second information being used to indicate an error generated during a pre-integration process of inertial measurement unit data of the drone from a third moment to a fourth moment, the time interval between the third moment and the first moment being no greater than a preset time interval, and the time interval between the fourth moment and the second moment being no greater than a preset time interval, the inertial measurement unit data of the drone being used to determine the drone's position and posture information;

[0007] Based on the first information, the second information and the vibration condition of the drone, the second posture information of the drone from the third moment to the fourth moment is determined; wherein the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information.

[0008] In a second aspect, the present invention further provides a device for determining the posture information of a drone, comprising:

[0009] a first information determination module, configured to acquire a panoramic image of the drone and determine first information, the first information being used to indicate an error between a first feature point and a second feature point, the first feature point being a feature point predicted from a feature point in the first panoramic image based on first pose information of the drone, the first pose information being determined based on inertial measurement unit data of the drone, the second feature point being a feature point in a second panoramic image, the first panoramic image and the second panoramic image being panoramic images correspondingly captured by the drone at adjacent first and second moments, respectively;

[0010] a second information determination module, configured to determine second information using an inertial measurement unit disposed on the drone, the second information being used to indicate an error generated during a pre-integration process of inertial measurement unit data of the drone from a third moment to a fourth moment, the time interval between the third moment and the first moment being no greater than a preset time interval, and the time interval between the fourth moment and the second moment being no greater than a preset time interval, the inertial measurement unit data of the drone being used to determine the drone's position and posture information;

[0011] The second posture information determination module is used to determine the second posture information of the drone from the third moment to the fourth moment based on the first information, the second information and the vibration condition of the drone; wherein the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0013] one or more processors;

[0014] a storage device for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the posture information of a drone as provided in any embodiment of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention further provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to execute a method for determining the posture information of a drone as provided in any embodiment of the present invention.

[0017] This technical solution acquires a panoramic image from a drone and determines first information, which is used to indicate the error between a first feature point and a second feature point. Since the feature points in two frames of panoramic images corresponding to adjacent moments captured by the drone are kinematically correlated, the first information is specifically determined based on the drone's first pose information, which is essentially determined based on the drone's inertial measurement unit data. This establishes a correlation between the panoramic image and the inertial measurement unit data, allowing the first information to act as a visual image constraint, thereby helping to determine more accurate pose information. At the same time, second information is determined using the inertial measurement unit placed on the drone. The second information is used to indicate the error generated during the pre-integration process of the drone's inertial measurement unit data, quantifying the error generated during the pre-integration process. This allows the error to be adjusted and improved, thereby helping to determine more accurate pose information. Under these conditions, the errors indicated by the first and second information, as well as the drone's vibration, are optimized and adjusted. The drone's vibration is negatively correlated with the credibility of the second information. This solution can control and adjust the credibility of the second information based on the drone's vibration, effectively reducing the negative impact of high vibration conditions and determining more accurate inertial measurement unit data, thereby determining the drone's second position information. This solution can accurately determine the drone's position information and significantly improve the accuracy and robustness of the drone's positioning. In particular, it can enable the drone to achieve sub-meter or even higher positioning accuracy in harsh environments such as complex electromagnetic interference, low light, and strong vibration.

[0018] The above content of the invention is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0020] Figure 11 is a flow chart of a method for determining the posture information of a drone provided by an embodiment of the present invention;

[0021] Figure 2 A schematic flow chart of another method for determining the posture information of a drone provided by an embodiment of the present invention;

[0022] Figure 3 A schematic structural diagram of a device for determining the posture information of a drone provided by an embodiment of the present invention;

[0023] Figure 4 A schematic structural diagram of an electronic device for implementing a method for determining the posture information of a drone provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0025] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0026] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0027] It should be noted that the concepts of "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0030] Figure 1 This is a flow chart of a method for determining the posture information of a drone provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the situation where the posture information of a drone is determined. The method can be executed by a posture information determination device of a drone. The posture information determination device of a drone can be implemented in the form of software and / or hardware and is generally integrated into any electronic device with network communication function. The electronic device can be a mobile terminal, a PC or a server, etc. Figure 1 As shown, the method for determining the posture information of a drone according to an embodiment of the present invention may include the following process:

[0031] S110. Acquire a panoramic image of the drone and determine first information, where the first information is used to indicate an error between a first feature point and a second feature point. The first feature point is a feature point predicted from a feature point in the first panoramic image based on the first position information of the drone, and the first position information is determined based on the inertial measurement unit data of the drone. The second feature point is a feature point in the second panoramic image. The first panoramic image and the second panoramic image are panoramic images collected by the drone at adjacent first and second moments, respectively.

[0032] The drone is equipped with a camera that captures the drone's flight environment and generates a corresponding panoramic image. The panoramic image covers a 360-degree viewing angle, providing richer environmental textures and eliminating blind spots. The first and second panoramic images are panoramic images captured by the drone at adjacent first and second moments, respectively.

[0033] Because the determined panoramic image frames are temporally correlated, the feature points in the panoramic images are continuously tracked as the camera captures the images continuously, resulting in a kinematic conversion relationship between the feature points of each panoramic image. Specifically, for the first and second panoramic images, since they were captured adjacently, if the precise position information of the drone corresponding to the two frames can be determined, then the feature points in the first panoramic image can theoretically be converted and aligned with the feature points in the second panoramic image using this precise position information.

[0034] In this embodiment, a panoramic image of the drone can be first obtained, and then the feature points in the first panoramic image are predicted based on the first pose information of the drone to obtain the first feature point, and the error between the first feature point and the second feature point is indicated by the first information. The second feature point is the feature point in the second panoramic image, and the first pose information is determined based on the inertial measurement unit data of the drone, and the pose information includes position information and pose information. It should be noted that if the first pose information determined based on the inertial measurement unit data of the drone is sufficiently accurate, then the error between the first feature point and the second feature point will be small enough, which can indicate that the pose information determined at this time is accurate. However, the process of determining the inertial measurement unit data of the drone is often affected by measurement noise and zero bias error, which makes the determined first pose information not accurate, and further leads to errors between the first feature point and the second feature point determined based on the first pose information. This solution uses the correlation between these data to determine the first information when the drone acquires panoramic images, so that the first information acts as a constraint on the visual image, and aims to continue to adjust the error between the first feature point and the second feature point to a minimum, thereby helping to determine more accurate pose information.

[0035] S120. Determine second information through an inertial measurement unit placed on the drone, where the second information is used to indicate an error generated by the inertial measurement unit data of the drone from a third moment to a fourth moment during a pre-integration process, the time interval between the third moment and the first moment is no greater than a preset time interval, and the time interval between the fourth moment and the second moment is no greater than a preset time interval, and the inertial measurement unit data of the drone is used to determine the drone's position information.

[0036] Among them, the Inertial Measurement Unit (IMU) is a sensor device mainly composed of gyroscopes, accelerometers, and magnetometers. The IMU can be used to measure the angular velocity, acceleration, and other IMU data of the drone on the X, Y, and Z axes during flight. The position and posture information of the drone can be determined through the IMU data. Generally, the sampling frequency of the IMU is high, and a large amount of IMU data can be measured in a short period of time. In order to save the amount of calculation for these data, some IMU data is generally selected from the continuous IMU data over a period of time, and the selected IMU data is pre-integrated, so that the IMU data generated during this period of time is integrated into one result.

[0037] Specifically, in this embodiment, an inertial measurement unit (IMU) mounted on the drone performs pre-integration on the IMU data from a third moment to a fourth moment. The time interval between the third moment and the first moment is no greater than a preset time interval, and the time interval between the fourth moment and the second moment is no greater than a preset time interval. Specifically, the IMU data corresponding to the two times closest to the acquisition times of the first and second panoramic images is determined, and pre-integration is performed on these two frames of IMU data. Furthermore, due to inherent measurement noise and bias errors in IMUs, these errors accumulate during the pre-integration process. This embodiment, by determining second information from the drone's IMU measurements, indicates the error generated during the pre-integration of the IMU data from the third moment to the fourth moment, thereby quantifying the error generated during the pre-integration process and subsequently enabling continuous adjustment and improvement of this error. This second information acts as a constraint. Minimizing the error generated during the pre-integration process allows for more accurate IMU data and, consequently, precise position and pose information to be determined.

[0038] S130. Determine second position information of the drone from the third moment to the fourth moment based on the first information, the second information, and the vibration condition of the drone; wherein the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information.

[0039] Specifically, the first and second information determined above can both act as constraints on the pose relationship, and both the first and second information can reflect the impact of the inertial measurement unit data on their respective errors. However, the constraints of the two are not exactly the same. In particular, the constraint of the second information can be affected by the real-time vibration of the drone. For example, the more severe the drone's vibration, the greater the error generated during the pre-integration process of the inertial measurement unit data, and the less trustworthy the second information. The drone's vibration is negatively correlated with the credibility of the second information. This embodiment takes these factors into account and establishes a correlation based on the first and second information and the drone's vibration. This allows the errors indicated by the first and second information to be continuously adjusted and reduced, thereby achieving continuous optimization of the inertial measurement unit data. At the same time, the drone's vibration is used to control and adjust the credibility of the second information, effectively reducing the negative impact of high vibration conditions and assigning more reasonable credibility to constraints of varying degrees, thereby determining the second pose information of the drone from the third moment to the fourth moment. In other words, compared with the first pose information, the second pose information obtained is actually an accurate pose information obtained after optimization and adjustment.

[0040] For example, the first and second information can be combined to serve as constraints. When the error indicated by the first and second information are both minimized, the second pose information of the corresponding UAV from the third moment to the fourth moment is determined. For another example, when the error indicated by the first and second information are both minimized, the second pose information of the corresponding UAV from the third moment to the fourth moment is determined. The optimization methods and optimization objectives can be differentiated based on actual needs.

[0041] It should be noted that the solution of this embodiment, on the one hand, starts from the feature points of the actually acquired panoramic image, constructs the correlation between the relevant feature points based on the first pose information, and determines the first information that can reflect the error of the feature points. On the other hand, starting from the process of pre-integration of the inertial measurement unit data, it determines the second information that can quantify the error generated in the pre-integration process. Then, with the help of such first and second information, continuous optimization and adjustment can be carried out. At the same time, in the process of optimization and adjustment, the negative impact brought about by high vibration conditions is also taken into account. The vibration conditions of the drone are used to assign a more reasonable credibility to the two pieces of information, and then after optimization and adjustment, more accurate second pose information of the drone is determined.

[0042] This technical solution is used to obtain a panoramic image from a drone and determine first information. The first information is used to indicate the error between the first feature point and the second feature point. Since the feature points in the two frames of panoramic images corresponding to adjacent moments collected by the drone are kinematically correlated, the first information is specifically determined based on the first pose information, which is essentially determined based on the drone's inertial measurement unit data. In this way, a correlation can be constructed between the panoramic image and the inertial measurement unit data, so that the first information acts as a visual image constraint, thereby helping to determine more accurate pose information. At the same time, the second information is also determined by the inertial measurement unit placed on the drone. The second information is used to indicate the error generated by the drone's inertial measurement unit data during the pre-integration process, achieving the quantification of the error generated during the pre-integration process, and then the error can be adjusted and improved, thereby helping to determine more accurate pose information. Under these conditions, the errors indicated by the first and second information, as well as the drone's vibration, are optimized and adjusted. The drone's vibration is negatively correlated with the credibility of the second information. This solution can control and adjust the credibility of the second information based on the drone's vibration, effectively reducing the negative impact of high vibration conditions and determining more accurate inertial measurement unit data, thereby determining the drone's second position information. This solution can accurately determine the drone's position information and significantly improve the accuracy and robustness of the drone's positioning. In particular, it can enable the drone to achieve sub-meter or even higher positioning accuracy in harsh environments such as complex electromagnetic interference, low light, and strong vibration.

[0043] Figure 2 This is a flow chart of another method for determining the posture information of a drone provided by an embodiment of the present invention. The technical solution of this embodiment further optimizes the process of determining the second posture information of the drone from the third moment to the fourth moment based on the first information, the second information and the vibration of the drone in the above embodiment. This embodiment can be combined with various optional solutions in one or more of the above embodiments. Figure 2 As shown, the method for determining the posture information of a drone according to an embodiment of the present invention may include the following process:

[0044] S210. Acquire a panoramic image of the drone and determine first information, where the first information is used to indicate an error between a first feature point and a second feature point. The first feature point is a feature point predicted from a feature point in the first panoramic image based on the first position information of the drone, and the first position information is determined based on the inertial measurement unit data of the drone. The second feature point is a feature point in the second panoramic image. The first panoramic image and the second panoramic image are panoramic images collected by the drone at adjacent first and second moments, respectively.

[0045] As an optional but non-limiting implementation, determining the first information may include the following steps A1-A4:

[0046] Step A1: determining a corresponding first panoramic feature point cloud based on the first panoramic image.

[0047] The panoramic feature point cloud is composed of a series of 3D feature points, each of which can be a corresponding identified feature point in the panoramic image. The panoramic feature point cloud can provide a panoramic perspective and rich 3D information.

[0048] Optionally, in this embodiment, the drone is configured with four groups of monocular fisheye cameras. Accordingly, the corresponding first panoramic feature point cloud is determined based on the first panoramic image, including: acquiring four monocular images taken by the monocular fisheye camera; constructing a binocular image based on the four monocular images to determine the first panoramic image; and extracting feature points of the first panoramic image based on a preset feature extraction algorithm to determine the first panoramic feature point cloud.

[0049] Among them, four sets of monocular fisheye cameras can be configured in the front, back, left and right directions of the drone. By configuring four sets of monocular fisheye cameras on the drone, environmental information can be collected under a 360-degree perspective, achieving omnidirectional perspective coverage, which can effectively make up for the blind spots and occlusion problems existing in a single perspective.

[0050] Specifically, four monocular images captured by a monocular fisheye camera can be obtained. Each monocular image can be split into two, resulting in eight monocular images. Each of these eight monocular images can then be paired with the corresponding monocular image from the adjacent fisheye camera to form a set of binocular images. Similarly, four sets of binocular images can be constructed from these eight divided monocular images to determine the first panoramic image. Feature points can then be extracted from the first panoramic image using a preset feature extraction algorithm, such as the SIFT (Scale-Invariant Feature Transform) or the ORB (Oriented FAST and Rotated BRIEF) algorithm. Alternatively, a deep learning-based feature extraction network can be used to extract feature points from the first panoramic image. Since these four sets of binocular images contain overlapping fields of view, they can be used to extract three-dimensional spatial information, thereby determining the first panoramic feature point cloud.

[0051] In addition, during the process of acquiring and processing the image, the image may also be corrected and / or enhanced. For example, due to the inherent distortion effect of the fisheye camera, the original captured image usually has obvious radial distortion. The fisheye image correction algorithm is used to convert the fisheye image into a standard perspective image. For example, the pinhole camera model and the radial distortion model can be used to correct the image. For another example, the captured image may still have insufficient contrast and noise problems in a low-light environment. Then the adaptive histogram equalization (AHE) or contrast-limited adaptive histogram equalization (CLAHE) algorithm can be used to enhance the image to improve the brightness and contrast of the image, thereby improving the reliability of subsequent feature extraction. The specific processing method can be set based on actual needs and is not limited in detail here.

[0052] Step A2: Predict the position of the third feature point in the first panoramic feature point cloud based on the first pose information, and project it into the second panoramic image to obtain the first feature point.

[0053] The third feature point is a feature point in the first panoramic feature point cloud. Specifically, for each 3D feature point in the first panoramic feature point cloud corresponding to the first moment, a translation transformation and a rotation transformation can be performed on each third feature point based on the first pose information, thereby predicting the position of the third feature point at the second moment. The predicted 3D feature point is then projected onto the second panoramic image to obtain the first feature point.

[0054] Step A3: determining a reprojection residual based on a distance difference between the first feature point and the second feature point; the second feature point is a feature point in the second panoramic image that matches the third feature point.

[0055] The second feature point is a feature point in the second panoramic image that matches the third feature point, and can be determined specifically through feature matching. In the second panoramic image, the distance difference between the first feature point and the second feature point can be determined. The same operation principle is performed for each of the other third feature points to determine the corresponding distance differences, and then determine the reprojection residual. The reprojection residual can reflect the error between the corresponding feature points in the first panoramic image and the second panoramic image when the feature points are transformed using the first pose information.

[0056] Step A4: Determine first information based on the reprojection residual.

[0057] Specifically, the corresponding first information may be determined based on the determined reprojection residual. The first information may be a function that describes the error between the corresponding feature points.

[0058] By adopting this optional solution, starting from the feature points in the panoramic image actually obtained, the distance difference between the first feature point and the second feature point is determined based on the first pose information, and then the reprojection residual is determined, so that the feature points corresponding to adjacent frame images are associated with each other through the pose information. The association relationship is specifically the error between the corresponding feature points. Through such a relationship, a visual constraint condition can be provided for the determination of the pose information through visual images, so as to help determine the precise pose information of the drone.

[0059] S220. Determine second information through an inertial measurement unit placed on the drone, where the second information is used to indicate an error generated by the inertial measurement unit data of the drone from a third moment to a fourth moment during a pre-integration process, the time interval between the third moment and the first moment is no greater than a preset time interval, and the time interval between the fourth moment and the second moment is no greater than a preset time interval, and the inertial measurement unit data of the drone is used to determine the drone's position information.

[0060] As an optional but non-limiting implementation, determining the second information by an inertial measurement unit placed on the drone may include the following steps B1-B4:

[0061] Step B1: Acquire multiple frames of inertial measurement unit data obtained when the drone performs inertial measurement unit measurement and the measurement time corresponding to each frame of inertial measurement unit data.

[0062] The measurement time refers to the specific time information when the inertial measurement unit data is measured.

[0063] Specifically, the IMU measures the drone's IMU data at a pre-set sampling frequency and transmits the measured IMU data and measurement time via a standardized interface. This allows for the acquisition of multiple frames of IMU data and the measurement time corresponding to each frame of IMU data.

[0064] Step B2: Determine the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment.

[0065] Optionally, determining the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment includes: determining the third moment and the fourth moment based on the first moment, the second moment, the measurement time corresponding to each frame of the inertial measurement unit data, and a preset time interval; and determining the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment based on the third moment and the fourth moment.

[0066] Specifically, for the measurement times corresponding to each frame of inertial measurement unit data, a third moment and a fourth moment can be determined using a preset time interval, such that the time interval between the third moment and the first moment is no greater than the preset time interval, and the time interval between the fourth moment and the second moment is no greater than the preset time interval. It should be noted that due to differences in the values ​​set for the preset time intervals, this process may result in at least two measurement times meeting the third moment requirement, or at least two measurement times meeting the fourth moment requirement. To address these special cases, the measurement time corresponding to each frame of inertial measurement unit data with the smallest time interval from the first moment is determined as the third moment, and the measurement time corresponding to each frame of inertial measurement unit data with the smallest time interval from the second moment is determined as the fourth moment. In other words, this embodiment aims to determine the two moments closest to the first moment and the second moment from each measurement time, respectively. These two moments are referred to as the third moment and the fourth moment. Based on the third moment and the fourth moment, the corresponding inertial measurement unit data can be found, and the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment can be determined.

[0067] Step B3: Pre-integrate the first inertial measurement unit data and the second inertial measurement unit data to determine a pre-integration residual.

[0068] The pre-integration residual can be used to measure the difference between the pre-integration increment calculated using the actual measured inertial measurement unit data and the state estimate increment currently obtained through optimization. Specifically, the pre-integration model can be used to pre-integrate the first and second inertial measurement unit data, and the pre-integration residual generated during the integration process can be determined.

[0069] Step B4: Determine second information based on the pre-integration residual.

[0070] Specifically, corresponding second information can be determined based on the determined pre-integration residual. The second information can be a function that describes the error generated during the pre-integration process of the first inertial measurement unit data and the second inertial measurement unit data.

[0071] By adopting this optional solution, it is possible to quantify the error generated during the pre-integration process of the UAV's inertial measurement unit data from the third moment to the fourth moment, thereby facilitating subsequent solutions to continuously adjust and optimize the errors generated during this pre-integration process. After adjustment based on the error situation, more accurate inertial measurement unit data can be determined, and the corresponding accurate posture information can also be determined.

[0072] S230: Determine a first weight corresponding to the first information and a second weight corresponding to the second information; the second weight is determined in real time based on the vibration condition of the drone; and the vibration condition of the drone is negatively correlated with the second weight.

[0073] Specifically, after determining the first and second information, they can be jointly nonlinearly optimized based on the BA (Bundle Adjustment) concept. Based on the theoretical basis of maximum likelihood estimation, the optimal parameters are solved by taking a weighted sum of the corresponding errors of the two. Specifically, when combining the first and second information, corresponding weights can be assigned to the first and second information. The weight information of the two can reflect the degree of influence of the visual constraints and the inertial measurement data constraints on the overall positioning accuracy, thereby balancing the importance of the two when combining them.

[0074] Among them, the second weight is determined in real time based on the vibration condition of the drone, and the second weight can be used to characterize the credibility of the second information. The vibration condition of the drone is negatively correlated with the second weight. For example, if the vibration condition of the drone is stronger, the error generated in the pre-integration process will be greater, then correspondingly, it is necessary to reduce the credibility of this part of the second information, that is, the second weight determined in real time will be lower, which can effectively reduce the negative impact of the pre-integration error on the overall positioning accuracy in a high vibration environment. This embodiment uses this method of real-time determination of the second weight based on the vibration condition of the drone to effectively reduce the negative impact brought about by high vibration conditions, help improve the overall positioning accuracy and robustness, and thus determine more accurate posture information.

[0075] As an optional but non-limiting implementation, determining the first weight corresponding to the first information and the second weight corresponding to the second information may include the following steps C1-C3:

[0076] Step C1: Update the first pre-integration covariance matrix based on the noise impact factor to determine the second pre-integration covariance matrix; the noise impact factor is used to reflect the degree of influence of the vibration of the UAV on the pre-integration covariance matrix; the first pre-integration covariance matrix is ​​the pre-integration covariance matrix obtained by pre-integrating the inertial measurement unit data within the integration time steps corresponding to the third moment and the fourth moment.

[0077] Among them, the first pre-integration covariance matrix is ​​the pre-integration covariance matrix obtained by pre-integrating the inertial measurement unit data within the integration time steps corresponding to the third moment and the fourth moment. The pre-integration covariance matrix can be used to reflect the accumulated errors caused by noise during the integration process. Exemplarily, in this embodiment, the following formula can be used to determine the first pre-integration covariance matrix:

[0078] ;

[0079] Where i is the third moment; j is the fourth moment; is the first pre-integration covariance matrix obtained during the pre-integration process from the third moment to the fourth moment; is the state transition Jacobian matrix during the pre-integration process from the third moment to the fourth moment; is the pre-integrated covariance matrix determined at the third moment; is the transposed matrix of the state transfer Jacobian matrix; is the noise influence matrix during the pre-integration process from the third moment to the fourth moment; Q is the noise covariance matrix of the inertial measurement unit; The transposed matrix of the noise contribution matrix.

[0080] Specifically, considering the impact of the vibration of the drone, the first pre-integrated covariance matrix can be updated based on the noise impact factor to determine the second pre-integrated covariance matrix. The noise impact factor is used to reflect the degree of influence of the drone's vibration on the pre-integrated covariance matrix; the larger the value of the noise impact factor, the greater the influence of the drone's vibration on the pre-integrated covariance matrix. The noise factor can be specifically determined based on the real-time vibration of the drone. It should be noted that the noise factor is not a fixed value. It is specifically determined based on the real-time vibration of the drone. The real-time vibration of the drone is not static, so the corresponding noise factor is also determined based on the actual situation. In other words, the determination of the noise factor can be dynamically adjusted based on the actual situation.

[0081] As an optional but non-limiting implementation method, the first pre-integrated covariance matrix is ​​updated based on the noise impact factor, including: updating the noise covariance matrix of the inertial measurement unit based on the noise impact factor to obtain an updated noise covariance matrix of the inertial measurement unit; and updating the first pre-integrated covariance matrix based on the updated noise covariance matrix of the inertial measurement unit.

[0082] The noise covariance matrix of the inertial measurement unit (IMU) can be used to reflect the noise level during the measurement process. The noise impact factor is positively correlated with the noise covariance matrix of the IMU. A larger noise impact factor corresponds to a larger noise covariance matrix of the IMU, which in turn affects the first pre-integrated covariance matrix.

[0083] Specifically, after determining the vibration condition of the drone, the corresponding noise impact factor can be determined, and then the noise covariance matrix of the inertial measurement unit is updated based on the noise impact factor to obtain the updated noise covariance matrix of the inertial measurement unit; and the first pre-integrated covariance matrix is ​​updated based on the updated noise covariance matrix of the inertial measurement unit.

[0084] As an optional but non-limiting implementation method, the noise covariance matrix of the inertial measurement unit is updated based on the noise impact factor, including: obtaining the current real-time vibration index of the drone; determining the noise impact factor based on the real-time vibration index and preset adjustment parameters; the value of the noise impact factor is positively correlated with the vibration intensity during the actual flight of the drone; and updating the noise covariance matrix of the inertial measurement unit based on the noise impact factor.

[0085] The real-time vibration index reflects the drone's current vibration intensity. The drone's non-control module acquires the parameters of each sensor in real time and processes these parameters using an adaptive Kalman filter to obtain the drone's current real-time vibration index. Preset adjustment parameters are pre-set parameters that are compatible with the drone's inertial measurement unit. These parameters can be initialized before the drone takes flight.

[0086] Specifically, after obtaining the drone's current real-time vibration index, the noise impact factor can be determined based on the real-time vibration index and preset adjustment parameters. If the vibration intensity during the drone's current flight increases, the corresponding noise impact factor value will also increase; if the vibration intensity during the drone's current flight decreases, the corresponding noise impact factor value will also decrease. After determining the noise impact factor, the noise covariance matrix of the inertial measurement unit can be updated. The updated noise covariance matrix of the inertial measurement unit can then reflect the current noise level.

[0087] For example, in this embodiment, the noise impact factor may be determined based on the following formula:

[0088] ;

[0089] in, is the noise impact factor; is the preset adjustment parameter; is the real-time vibration index. It can be seen from the above formula that when the actual vibration intensity of the drone is large, the real-time vibration index will increase, which will lead to the noise impact factor Will be greater than 1; Conversely, when the actual vibration intensity of the drone is small, the real-time vibration index will be reduced, which can make the noise impact factor Close to 1.

[0090] For example, after determining the noise impact factor, this embodiment can update the noise covariance matrix of the inertial measurement unit using the following formula:

[0091] ;

[0092] in, is the noise covariance matrix of the updated inertial measurement unit; is the noise impact factor; is the noise covariance matrix of the inertial measurement unit before the update. It can be seen from this formula that when the value of the noise impact factor is large, the noise covariance matrix of the inertial measurement unit after the update will also become larger.

[0093] Furthermore, after the noise covariance matrix of the inertial measurement unit is updated, the first pre-integrated covariance matrix can be updated using the following formula in this embodiment to obtain a second pre-integrated covariance matrix:

[0094] ;

[0095] in, is the second pre-integrated covariance matrix, that is, the updated first pre-integrated covariance matrix; is the noise covariance matrix of the inertial measurement unit after updating based on the noise impact factor; is the state transition Jacobian matrix during the pre-integration process from the third moment to the fourth moment; is the pre-integrated covariance matrix determined at the third moment; is the transposed matrix of the state transfer Jacobian matrix; is the noise impact matrix during the pre-integration process from the third moment to the fourth moment; The transposed matrix of the noise contribution matrix.

[0096] Step C2: Determine a second weight corresponding to the second information based on the inverse matrix corresponding to the second pre-integrated covariance matrix.

[0097] After the second pre-integrated covariance matrix is ​​determined, the inverse matrix corresponding to the second pre-integrated covariance matrix may be determined, and the determined inverse matrix corresponding to the second pre-integrated covariance matrix is ​​used as the second weight corresponding to the second information.

[0098] Step C3: Determine a first weight corresponding to the first information based on the second weight.

[0099] Specifically, after determining the second weight, a first weight corresponding to the first information can be determined based on an identity matrix and the second weight. Specifically, a mathematical calculation is performed using the identity matrix and the inverse matrix corresponding to the second pre-integrated covariance matrix to obtain the second pre-integrated covariance matrix, which is used as the first weight corresponding to the first information.

[0100] By adopting this optional solution, the corresponding noise impact factor can be determined based on the actual vibration conditions of the UAV, and this method of adaptively adjusting the noise impact factor can update the pre-integration covariance matrix, and then the second weight corresponding to the second information is also adaptively adjusted dynamically. In particular, it can reduce the negative impact of the pre-integration error on the overall positioning accuracy in a high vibration environment, thereby helping to accurately determine the UAV's posture information.

[0101] S240: Determine second position information of the UAV from the third moment to the fourth moment based on the first information, the second information, the first weight, and the second weight.

[0102] Specifically, an objective function can be constructed according to the first information, the second information, the first weight and the second weight, and the objective function can be continuously optimized based on the BA idea (Bundle Adjustment) until the objective function reaches an optimal solution. The inertial measurement unit data corresponding to the optimal solution is obtained to determine the second posture information of the drone from the third moment to the fourth moment.

[0103] The condition for the objective function to reach the optimal solution may be that the number of optimizations of the objective function reaches a preset number of optimizations, or that the value of the objective function tends to be stable. The specific setting can be differentiated based on actual needs and is not further specified here.

[0104] For example, the objective function constructed in this embodiment may be in the following form:

[0105] ;

[0106] in, is a set of feature points in the panoramic image; i is the i-th panoramic image acquired; j is the j-th panoramic image acquired; the i-th panoramic image and the j-th panoramic image are panoramic images acquired at two adjacent moments; N is the total number of panoramic image frames acquired; is first information determined based on the i-th panoramic image frame and the j-th panoramic image frame; is a first weight corresponding to the first information; The second information is determined, and the determination of the second information is associated with the i-th panoramic image frame and the j-th panoramic image frame; is the second weight corresponding to the second information.

[0107] This objective function fully reflects the statistical characteristics and noise distribution of the residual term. Nonlinear least squares solution methods (e.g., Levenberg-Marquardt algorithm) can be used for global optimization until the objective function reaches the optimal solution. Parameter values ​​are used to determine the second posture information of the UAV from the third moment to the fourth moment.

[0108] This technical solution is used to obtain a panoramic image from a drone and determine first information. The first information is used to indicate the error between the first feature point and the second feature point. Since the feature points in the two frames of panoramic images corresponding to adjacent moments collected by the drone are kinematically correlated, the first information is specifically determined based on the first pose information, which is essentially determined based on the drone's inertial measurement unit data. In this way, a correlation can be constructed between the panoramic image and the inertial measurement unit data, so that the first information acts as a visual image constraint, thereby helping to determine more accurate pose information. At the same time, the second information is also determined by the inertial measurement unit placed on the drone. The second information is used to indicate the error generated by the drone's inertial measurement unit data during the pre-integration process, achieving the quantification of the error generated during the pre-integration process, and then the error can be adjusted and improved, thereby helping to determine more accurate pose information. Under these conditions, a first weight corresponding to the first information and a second weight corresponding to the second information are determined; the second weight is determined in real time based on the vibration of the drone. This real-time determination effectively reduces the negative impact of high vibration conditions, helping to improve overall positioning accuracy and robustness. This allows for more accurate inertial measurement unit data, thereby determining more accurate second position information for the drone. This solution can accurately determine the drone's position information and significantly improve the accuracy and robustness of drone positioning. In particular, it enables the drone to achieve sub-meter or even higher positioning accuracy in harsh environments such as complex electromagnetic interference, low light, and strong vibration.

[0109] Figure 3 This is a schematic diagram of the structure of a device for determining the posture information of a drone provided by an embodiment of the present invention. The embodiment of the present invention is applicable to the situation where the posture information of a drone is determined. The device for determining the posture information of a drone can be implemented in the form of software and / or hardware and is generally integrated into any electronic device with network communication function, which can be a mobile terminal, PC or server, etc. Figure 3 As shown, the apparatus for determining the posture information of a drone according to an embodiment of the present invention may include a first information determination module 310, a second information determination module 320, and a second posture information determination module 330. In particular:

[0110] A first information determination module 310 is configured to obtain a panoramic image of the drone and determine first information, where the first information is used to indicate an error between a first feature point and a second feature point, where the first feature point is a feature point predicted from a feature point in the first panoramic image based on the first pose information of the drone, the first pose information is determined based on inertial measurement unit data of the drone, the second feature point is a feature point in a second panoramic image, and the first panoramic image and the second panoramic image are panoramic images captured by the drone at adjacent first and second moments, respectively.

[0111] a second information determining module 320 configured to determine second information using an inertial measurement unit (IMU) mounted on the drone, the second information being used to indicate an error generated during a pre-integration process of IMU data of the drone from a third moment to a fourth moment, the time interval between the third moment and the first moment being no greater than a preset time interval, and the time interval between the fourth moment and the second moment being no greater than a preset time interval, the IMU data of the drone being used to determine the drone's position and posture information;

[0112] The second posture information determination module 330 is used to determine the second posture information of the drone from the third moment to the fourth moment based on the first information, the second information and the vibration condition of the drone; wherein the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information.

[0113] With this technical solution, a panoramic image of a drone is acquired and first information is determined through a first information determination module. The first information is used to indicate the error between the first feature point and the second feature point. Since the feature points in the two frames of panoramic images corresponding to adjacent moments collected by the drone are kinematically correlated, the first information is specifically determined based on the first pose information, which is essentially determined based on the drone's inertial measurement unit data. In this way, a correlation can be constructed between the panoramic image and the inertial measurement unit data, so that the first information acts as a visual image constraint, thereby helping to determine more accurate pose information. At the same time, the second information determination module is used to determine the second information through the inertial measurement unit placed on the drone. The second information is used to indicate the error generated by the drone's inertial measurement unit data during the pre-integration process, realizing the quantification of the error generated during the pre-integration process, and then the error can be adjusted and improved, thereby helping to determine more accurate pose information. Under these conditions, the second position information determination module optimizes and adjusts the errors indicated by the first and second information, as well as the drone's vibration. The drone's vibration is negatively correlated with the credibility of the second information. This solution can control and adjust the credibility of the second information based on the drone's vibration, effectively reducing the negative impact of high vibration conditions and determining more accurate inertial measurement unit data, thereby determining the drone's second position information. This solution can accurately determine the drone's position information and significantly improve the accuracy and robustness of the drone's positioning. In particular, it can enable the drone to achieve sub-meter or even higher positioning accuracy in harsh environments such as complex electromagnetic interference, low light, and strong vibration.

[0114] As an optional but non-limiting implementation, the first information determination module 310 includes a panoramic feature point cloud determination submodule, a first feature point determination submodule, a reprojection residual determination submodule, and a first information determination submodule. Among them, the panoramic feature point cloud determination submodule is used to determine the corresponding first panoramic feature point cloud based on the first panoramic image; the first feature point determination submodule is used to predict the position of the third feature point in the first panoramic feature point cloud based on the first pose information, and project it into the second panoramic image to obtain the first feature point; the reprojection residual determination submodule is used to determine the reprojection residual based on the distance difference between the first feature point and the second feature point; the second feature point is a feature point in the second panoramic image that matches the third feature point; and the first information determination submodule is used to determine the first information based on the reprojection residual.

[0115] As an optional but non-limiting implementation, the second information determination module 320 includes a data acquisition submodule, an inertial measurement unit data determination submodule, a pre-integration residual determination submodule, and a second information determination submodule. The data acquisition submodule is configured to acquire multiple frames of inertial measurement unit data and the measurement time corresponding to each frame of inertial measurement unit data obtained when the drone performs inertial measurement unit measurement; the inertial measurement unit data determination submodule is configured to determine the first inertial measurement unit data corresponding to the third moment and the second inertial measurement unit data corresponding to the fourth moment; the pre-integration residual determination submodule is configured to pre-integrate the first inertial measurement unit data and the second inertial measurement unit data to determine a pre-integration residual; and the second information determination submodule is configured to determine the second information based on the pre-integration residual.

[0116] As an optional but non-limiting implementation, the second posture information determination module 330 includes a weight determination submodule and a second posture information determination submodule. The weight determination submodule is configured to determine a first weight corresponding to the first information and a second weight corresponding to the second information; the second weight is determined in real time based on the vibration of the drone; and the vibration of the drone is negatively correlated with the second weight; and the second posture information determination submodule is configured to determine the second posture information of the drone from the third moment to the fourth moment based on the first information, the second information, the first weight, and the second weight.

[0117] As an optional but non-limiting implementation, the weight determination submodule includes a pre-integrated covariance matrix update unit, a second weight determination unit, and a first weight determination unit. The pre-integrated covariance matrix update unit is configured to update the first pre-integrated covariance matrix based on a noise impact factor to determine a second pre-integrated covariance matrix; the noise impact factor is configured to reflect the degree of influence of the vibration of the drone on the pre-integrated covariance matrix; the first pre-integrated covariance matrix is ​​a pre-integrated covariance matrix obtained by pre-integrating the inertial measurement unit data within the integration time steps corresponding to the third moment and the fourth moment; the second weight determination unit is configured to determine the second weight corresponding to the second information based on the inverse matrix corresponding to the second pre-integrated covariance matrix; and the first weight determination unit is configured to determine the first weight corresponding to the first information based on the second weight.

[0118] As an optional but non-limiting implementation, the pre-integrated covariance matrix update unit includes a noise covariance matrix update subunit and a pre-integrated covariance matrix update subunit. Wherein: the noise covariance matrix update subunit is used to update the noise covariance matrix of the inertial measurement unit based on the noise impact factor to obtain an updated noise covariance matrix of the inertial measurement unit; and the pre-integrated covariance matrix update subunit is used to update the first pre-integrated covariance matrix based on the noise covariance matrix of the updated inertial measurement unit.

[0119] As an optional but non-limiting implementation method, the noise covariance matrix updating subunit is specifically used to: obtain the current real-time vibration index of the drone; determine the noise impact factor based on the real-time vibration index and the preset adjustment parameter; the value of the noise impact factor is positively correlated with the vibration intensity during the actual flight of the drone; and update the noise covariance matrix of the inertial measurement unit based on the noise impact factor.

[0120] The apparatus for determining the posture information of a drone provided in an embodiment of the present invention can be used to execute a method for determining the posture information of a drone, and has functional modules and beneficial effects corresponding to executing the method for determining the posture information of a drone. It is worth noting that the various units and modules included in the above-mentioned apparatus are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the various functional units are only for the purpose of distinguishing them from each other and are not used to limit the scope of protection of the embodiments of the present invention.

[0121] Figure 4 This is a schematic diagram of the structure of an electronic device for implementing a method for determining the posture information of a drone provided by an embodiment of the present invention. Figure 4 , which shows a schematic structural diagram of an electronic device 400 suitable for implementing an embodiment of the present invention. The terminal devices in the embodiments of the present invention may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 4 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0122] like Figure 4As shown, electronic device 400 includes at least one processor 401 and memory, such as read-only memory (ROM) 402 and random access memory (RAM) 403, communicatively connected to at least one processor 401. The memory stores computer programs executable by the at least one processor. Processor 401 can perform various appropriate actions and processes based on the computer programs stored in ROM 402 or loaded from storage unit 408 into RAM 403. RAM 403 can also store various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. An input / output (I / O) interface 405 is also connected to bus 404. Multiple components in the electronic device 400 are connected to the I / O interface 405, including an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0123] Processor 401 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of processor 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any other suitable processor, controller, microcontroller, etc. Processor 401 executes the method for determining the pose information of a drone provided in any embodiment of the present invention.

[0124] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method for determining the pose information of a drone shown in the flowchart. When the computer program is executed by a processing device, the functions defined in the method for determining the pose information of a drone according to embodiments of the present invention are performed.

[0125] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0126] The electronic device provided in the embodiment of the present invention and the method for determining the posture information of the drone provided in the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0127] An embodiment of the present invention provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining the posture information of a drone provided in the above embodiment.

[0128] It should be noted that the computer-readable medium described above in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0129] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can interconnect with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network. The computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. Where a remote computer is involved, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0130] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The units involved in the embodiments of the present invention may be implemented in software or hardware. The name of a unit does not, in some cases, limit the unit itself. The functions described above may be at least partially performed by one or more hardware logic components. For example, exemplary types of hardware logic components that may be used include, but are not limited to, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0132] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] The above description is merely an illustration of preferred embodiments of the present invention and the underlying technical principles. Those skilled in the art should understand that the scope of the present invention is not limited to solutions formed by the specific combinations of the aforementioned technical features, but also encompasses other solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned concepts. For example, solutions formed by interchangeably using the aforementioned features with (but not limited to) technical features having similar functions disclosed in this invention. Furthermore, while operations are depicted in a specific order, this should not be construed as requiring that these operations be performed in the specific order shown or in a sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, while the above discussion includes several specific implementation details, these should not be construed as limiting the scope of the present invention. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Although the subject matter has been described using language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for determining the posture information of an unmanned aerial vehicle, characterized in that: The method comprises: Acquire a panoramic image of a drone and determine first information, where the first information is used to indicate an error between a first feature point and a second feature point, the first feature point being a feature point predicted from a feature point in the first panoramic image based on first pose information of the drone, the first pose information being determined based on inertial measurement unit data of the drone, the second feature point being a feature point in a second panoramic image, the first panoramic image and the second panoramic image being panoramic images correspondingly captured by the drone at adjacent first and second moments, respectively; Determining second information by an inertial measurement unit placed on the drone, the second information being used to indicate an error generated during a pre-integration process of inertial measurement unit data of the drone from a third moment to a fourth moment, the time interval between the third moment and the first moment being no greater than a preset time interval, and the time interval between the fourth moment and the second moment being no greater than a preset time interval, the inertial measurement unit data of the drone being used to determine the drone's position and posture information; Determining second position information of the drone from a third moment to a fourth moment based on the first information, the second information, and a vibration condition of the drone; wherein the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information; Determining second position information of the drone from a third moment to a fourth moment based on the first information, the second information, and the vibration of the drone includes: Determining a first weight corresponding to the first information and a second weight corresponding to the second information; the second weight is determined in real time based on a vibration condition of the drone; and the vibration condition of the drone is negatively correlated with the second weight; Determining second position information of the drone from a third moment to a fourth moment based on the first information, the second information, the first weight, and the second weight; The determining a first weight corresponding to the first information and a second weight corresponding to the second information includes: The first pre-integration covariance matrix is ​​updated based on the noise impact factor to determine the second pre-integration covariance matrix; the noise impact factor is used to reflect the degree of influence of the vibration of the drone on the pre-integration covariance matrix; the first pre-integration covariance matrix is ​​a pre-integration covariance matrix obtained by pre-integrating the inertial measurement unit data within the integration time steps corresponding to the third moment and the fourth moment; determining a second weight corresponding to the second information based on an inverse matrix corresponding to the second pre-integrated covariance matrix; A first weight corresponding to the first information is determined based on the second weight.

2. The method according to claim 1, characterized in that The determining of the first information includes: determining a corresponding first panoramic feature point cloud based on the first panoramic image; Predicting a position of a third feature point in the first panoramic feature point cloud based on the first pose information, and projecting the third feature point in the second panoramic image to obtain the first feature point; Determining a reprojection residual based on a distance difference between the first feature point and the second feature point, wherein the second feature point is a feature point in the second panoramic image that matches the third feature point; The first information is determined based on the reprojection residual.

3. The method according to claim 1, characterized in that The determining of the second information by an inertial measurement unit placed on the drone includes: Obtain multiple frames of inertial measurement unit data obtained when the drone performs inertial measurement unit measurement and the measurement time corresponding to each frame of inertial measurement unit data; determining first inertial measurement unit data corresponding to the third moment and second inertial measurement unit data corresponding to the fourth moment; pre-integrating the first inertial measurement unit data and the second inertial measurement unit data to determine a pre-integration residual; The second information is determined based on the pre-integration residual.

4. The method according to claim 1, wherein The updating of the first pre-integrated covariance matrix based on the noise impact factor includes: Updating a noise covariance matrix of the inertial measurement unit based on the noise impact factor to obtain an updated noise covariance matrix of the inertial measurement unit; The first pre-integrated covariance matrix is ​​updated based on the updated noise covariance matrix of the inertial measurement unit.

5. The method according to claim 4, characterized in that The updating of the noise covariance matrix of the inertial measurement unit based on the noise impact factor includes: Get the drone's current real-time vibration index; Determining the noise impact factor based on the real-time vibration index and the preset adjustment parameter; the value of the noise impact factor is positively correlated with the vibration intensity during the actual flight of the UAV; The noise covariance matrix of the inertial measurement unit is updated based on the noise impact factor.

6. A device for determining the posture information of a drone, characterized in that: The device comprises: a first information determination module, configured to acquire a panoramic image of the drone and determine first information, the first information being used to indicate an error between a first feature point and a second feature point, the first feature point being a feature point predicted from a feature point in the first panoramic image based on first pose information of the drone, the first pose information being determined based on inertial measurement unit data of the drone, the second feature point being a feature point in a second panoramic image, the first panoramic image and the second panoramic image being panoramic images correspondingly captured by the drone at adjacent first and second moments, respectively; a second information determination module, configured to determine second information using an inertial measurement unit disposed on the drone, the second information being used to indicate an error generated during a pre-integration process of inertial measurement unit data of the drone from a third moment to a fourth moment, the time interval between the third moment and the first moment being no greater than a preset time interval, and the time interval between the fourth moment and the second moment being no greater than a preset time interval, the inertial measurement unit data of the drone being used to determine the drone's position and posture information; a second posture information determination module, configured to determine second posture information of the drone from a third moment to a fourth moment based on the first information, the second information, and a vibration condition of the drone; wherein the vibration condition of the drone is used to control and adjust the credibility of the second information, and the vibration condition of the drone is negatively correlated with the credibility of the second information; The second posture information determination module includes: a weight determination submodule, configured to determine a first weight corresponding to the first information and a second weight corresponding to the second information; the second weight being determined in real time based on a vibration condition of the drone; the vibration condition of the drone being negatively correlated with the second weight; and a second posture information determination submodule, configured to determine second posture information of the drone from a third moment to a fourth moment based on the first information, the second information, the first weight, and the second weight. The weight determination submodule includes: A pre-integration covariance matrix update unit is used to update the first pre-integration covariance matrix based on a noise impact factor to determine a second pre-integration covariance matrix; the noise impact factor is used to reflect the degree of influence of the vibration of the drone on the pre-integration covariance matrix; the first pre-integration covariance matrix is ​​a pre-integration covariance matrix obtained by pre-integrating the inertial measurement unit data within the integration time steps corresponding to the third moment and the fourth moment; a second weight determining unit, configured to determine a second weight corresponding to the second information based on an inverse matrix corresponding to the second pre-integrated covariance matrix; A first weight determining unit is configured to determine a first weight corresponding to the first information based on the second weight.

7. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method for determining the posture information of a drone as described in any one of claims 1 to 5.

8. A storage medium containing computer-executable instructions, characterized in that: When executed by a computer processor, the computer executable instructions are used to execute the method for determining the posture information of a drone as described in any one of claims 1 to 5.

Images