Methods, devices, electronic equipment and storage media for determining the attitude of aircraft

Abstract

This application provides a method, apparatus, electronic device, and storage medium for determining the attitude of an aircraft, including: determining multiple two-dimensional feature points in image data; determining the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points, and determining the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points based on the two-dimensional coordinates; determining the reprojection error corresponding to each of the multiple two-dimensional feature points based on the three-dimensional coordinates; determining the state data, noise data, and covariance matrix of the aircraft based on inertial measurement data; determining the Kalman gain coefficient based on the noise data, covariance matrix, and state data; correcting the state data based on the Kalman gain coefficient to obtain target state data, and determining the attitude of the aircraft based on the target state data. This application can determine the attitude of an aircraft in real time and accurately.

Description

Technical Field

[0001] This application relates to the field of aircraft technology, and more specifically, to a method, apparatus, electronic device, and storage medium for determining the attitude of an aircraft. Background Technology

[0002] With the rapid development of technology, aircraft are increasingly used in daily life. Currently, traditional aircraft navigation technology generally relies on external sensors (such as Global Navigation Satellite Systems) and onboard sensors (such as visual and inertial sensors) to estimate the aircraft's attitude. However, positioning using external sensors is easily affected by the external environment, while onboard sensors can achieve positioning through filtering and graph optimization methods. However, graph optimization methods are computationally intensive, requiring multiple iterations and failing to meet real-time requirements, while filtering methods are prone to error accumulation. Therefore, how to accurately determine the aircraft's attitude in real time has become a pressing problem. Summary of the Invention

[0003] In view of this, embodiments of this application propose a method, apparatus, electronic device, and storage medium for determining the attitude of an aircraft, in order to improve the above-mentioned problems.

[0004] According to one aspect of the embodiments of this application, a method for determining the pose of an aircraft is provided. The method includes: acquiring image data collected by the aircraft and determining a plurality of two-dimensional feature points from the image data; determining the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and performing a three-dimensional transformation on the plurality of two-dimensional feature points according to the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points to determine the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points; performing a two-dimensional projection on the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points to obtain the reprojected two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and determining the reprojection error corresponding to each of the plurality of two-dimensional feature points according to the reprojected two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points and the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points; acquiring inertial measurement data of the aircraft, and determining the state data, noise data, and covariance matrix of the aircraft according to the inertial measurement data; determining a Kalman gain coefficient according to the reprojection error, the noise data, the covariance matrix, and the state data; correcting the state data according to the Kalman gain coefficient to obtain target state data, and determining the pose of the aircraft according to the target state data.

[0005] According to one aspect of the embodiments of this application, an apparatus for determining the pose of an aircraft is provided. The apparatus includes: a two-dimensional feature point determination module, configured to acquire image data collected by the aircraft and determine a plurality of two-dimensional feature points from the image data; a three-dimensional transformation module, configured to determine the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and perform a three-dimensional transformation on the plurality of two-dimensional feature points according to the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points to determine the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points; and a reprojection error determination module, configured to perform a two-dimensional projection based on the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points to obtain the reprojected two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and perform a three-dimensional transformation on the plurality of two-dimensional feature points according to the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points to determine the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points. The reprojection two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points and the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points determine the reprojection error corresponding to each of the multiple two-dimensional feature points; the data determination module is used to acquire the inertial measurement data of the aircraft, and determine the state data, noise data, and covariance matrix of the aircraft based on the inertial measurement data; the gain coefficient determination module is used to determine the Kalman gain coefficient based on the reprojection error, the noise data, the covariance matrix, and the state data; the pose determination module is used to correct the state data based on the Kalman gain coefficient to obtain target state data, and determine the pose of the aircraft based on the target state data.

[0006] According to one aspect of the embodiments of this application, an electronic device is provided, including: a processor; a memory, wherein computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the attitude determination method of the aircraft as described above is implemented.

[0007] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, implement the attitude determination method of the aircraft as described above.

[0008] In this application, the scheme first performs a 3D transformation based on the 2D coordinates of multiple 2D feature points in the image data to obtain the 3D coordinates of each feature point. Then, reprojection is performed based on these 3D coordinates to obtain the reprojected 2D coordinates of each feature point. The reprojection error of each feature point is then determined based on these reprojected 2D coordinates and the 2D coordinates of each feature point. Next, the aircraft's state data, noise data, and covariance matrix are determined based on inertial measurement data. The Kalman gain coefficient is then determined based on the reprojection error, state data, covariance matrix, and noise data. Finally, Kalman filtering is performed based on the Kalman gain coefficient to correct the state data, obtaining the target state data. The aircraft's pose is then determined based on this target state data. This application estimates the aircraft's pose by filtering and fusing visual and IMU data for each image frame in the image data, enabling real-time pose determination while improving accuracy.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0011] Figure 1 This is a flowchart illustrating an attitude determination method for an aircraft according to an embodiment of this application.

[0012] Figure 2 This is a flowchart illustrating an attitude determination method for an aircraft according to another embodiment of this application.

[0013] Figure 3 This is a flowchart illustrating the specific steps of step 207 according to an embodiment of this application.

[0014] Figure 4 This is a flowchart illustrating the specific steps of step 209 according to an embodiment of this application.

[0015] Figure 5 This is a flowchart illustrating a method for determining the attitude of an aircraft according to another embodiment of this application.

[0016] Figure 6 This is a flowchart illustrating an attitude determination method for an aircraft according to another embodiment of this application.

[0017] Figure 7 This is a flowchart illustrating an attitude determination method for an aircraft according to another embodiment of this application.

[0018] Figure 8 This is a flowchart illustrating an attitude determination method for an aircraft according to an embodiment of this application.

[0019] Figure 9 This is a block diagram of an attitude determination device for an aircraft according to an embodiment of this application.

[0020] Figure 10 This is a hardware structure diagram of an electronic device according to an embodiment of this application.

[0021] The accompanying drawings have illustrated specific embodiments of the present invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the inventive concept in any way, but rather to illustrate the concept of the invention to those skilled in the art through specific embodiments. Detailed Implementation

[0022] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Please see Figure 1 , Figure 1 This application illustrates an embodiment of a method for determining the attitude of an aircraft. In a specific embodiment, this method can be applied to, for example... Figure 9The aircraft attitude determination device 800 and the electronic equipment 900 equipped with the aircraft attitude determination device 800 are shown. Figure 10 The specific process of this embodiment will be described below. It is understood that this method can be executed by a cloud server with computing power or by an in-vehicle server. The following will focus on... Figure 1 The process shown will be described in detail. The method for determining the attitude of the aircraft may specifically include the following steps:

[0025] Step 110: Acquire image data collected by the aircraft and determine multiple two-dimensional feature points from the image data.

[0026] One approach is to acquire image data using the aircraft's image acquisition device. Optionally, this image data can be a stream of image frames acquired during the aircraft's flight, or multiple consecutive images acquired within a time period.

[0027] One approach is to perform corner detection on the acquired image data to identify multiple two-dimensional feature points. Optionally, the FAST (Features from Accelerated Segment Test) algorithm can be used for corner detection. Alternatively, the image data can be first processed into grayscale to obtain a grayscale image. Then, a pixel is randomly selected in the grayscale image, and based on this pixel and a preset radius, all pixels within a circle containing the pixel's location as the origin are segmented for testing. This initial screening of corner points yields multiple candidate corner points. Then, corner feature detection based on classification is used to determine whether the features corresponding to the multiple candidate corner points are indeed corner features, thus identifying reference corner points. Finally, non-maximum suppression is used to verify the corner feature, eliminating reference corner points with unstable features, and obtaining the target corner point. The target corner point is then identified as a two-dimensional feature point in the image data.

[0028] Alternatively, the image data can also be a continuous sequence of video frames. In this case, optical flow tracing can be used to detect and determine two-dimensional feature points. Optionally, before determining the two-dimensional feature points in the image data using optical flow tracing, the image data can be processed into grayscale to obtain a grayscale image. Then, pixels whose pixel values ​​change less than or equal to a change threshold in consecutive grayscale frames can be identified, and these pixels are thus identified as two-dimensional feature points in the image data. Alternatively, other algorithms can be used to determine the two-dimensional feature points in the image data; no specific limitations are imposed here.

[0029] Step 120: Determine the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and perform a three-dimensional transformation on the plurality of two-dimensional feature points according to their respective two-dimensional coordinates to determine the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points.

[0030] One approach is to determine the image coordinates of multiple two-dimensional feature points in the image data after identifying them, thereby determining the two-dimensional coordinates of each feature point.

[0031] Optionally, before performing the 3D transformation, the calibration coefficients of the aircraft's image acquisition device can be obtained. These calibration coefficients include the intrinsic parameters corresponding to the image acquisition device, which can then be used to perform the 3D transformation. Optionally, the 3D coordinates of multiple 2D feature points can be determined based on the transformation relationship between the image coordinate system, camera coordinate system, and world coordinate system, and the 2D coordinates of each 2D feature point. Optionally, the 3D transformation can be performed using the following formula:

[0032]

[0033] Where d is the distance between a two-dimensional feature point and the image acquisition device, i.e., the object distance of the image acquisition device, where (f x f y (c) is the focal length of the image acquisition device. x c y () represents the optical center coordinates of the image acquisition device, which are its intrinsic parameters. K is the intrinsic parameter matrix of the image acquisition device. The two-dimensional coordinates of the two-dimensional feature point are (u, v), and the three-dimensional coordinates of the two-dimensional feature point are (X, v). C ,Y C Z C ), where d can be the depth value of a feature point in the depth image, or it can be calculated based on the two-dimensional coordinates of the feature points matched in different image frames, as determined by the triangulation algorithm.

[0034] Step 130: Perform two-dimensional projection based on the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points to obtain the reprojected two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points, and determine the reprojection error corresponding to each of the multiple two-dimensional feature points based on the reprojected two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points and the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points.

[0035] As one approach, after determining the three-dimensional coordinates of each of the multiple two-dimensional feature points, in order to accurately determine the attitude information of the aircraft, the calculated three-dimensional coordinates of the two-dimensional feature points can be projected back onto the two-dimensional coordinate system to obtain the reprojected two-dimensional coordinates in the two-dimensional coordinate system. This allows the reprojection error of each two-dimensional feature point to be determined, thus making it easier to incorporate the reprojection error when determining the attitude of the aircraft, making the determined attitude of the aircraft more accurate.

[0036] Optionally, the Euclidean distance between the two-dimensional coordinates and the corresponding reprojected two-dimensional coordinates can be calculated based on the above formula, according to the two-dimensional coordinates and reprojected two-dimensional coordinates of each of the determined two-dimensional feature points. This Euclidean distance can then be used as the reprojection error corresponding to each of the multiple two-dimensional feature points.

[0037] Step 140: Obtain the inertial measurement data of the aircraft, and determine the state data, noise data, and covariance matrix of the aircraft based on the inertial measurement data.

[0038] As one approach, an inertial measurement unit (IMU) is installed on the aircraft. The IMU may include an accelerometer and a gyroscope sensor, which can be used to detect the aircraft's acceleration, angular velocity, yaw angle, pitch angle, roll angle, etc. Based on the collected inertial measurement data of the aircraft, the state data of the aircraft can be predicted, that is, the current attitude of the aircraft can be predicted.

[0039] Optionally, due to the limitations of sensor detection accuracy, the inertial measurement data acquired by the IMU contains a certain amount of noise. To reduce the impact of IMU noise on determining the aircraft's attitude, the noise data of the aircraft can be determined based on the inertial measurement data. Alternatively, the noise data can also be a fixed noise value determined beforehand by calibrating the aircraft's IMU, which can be obtained directly.

[0040] As one approach, after determining the aircraft's state and noise data, the state transition matrix can be determined based on the state data, and the noise matrix can be determined based on the noise data. Then, based on the state transition and noise matrices, the covariance matrix of the previous time step can be obtained. The initial covariance matrix can be set according to actual needs. The covariance matrix is ​​then updated using the state transition and noise matrices to determine the covariance matrix of the next time step. Optionally, the state transition matrix can be determined using the following formula:

[0041]

[0042] Where F is the state transition matrix, I is the identity matrix, and the rows of the matrix from top to bottom represent the rotation angles. First noise bg ,speed Second noise b a displacement The columns of the matrix represent, from left to right, the following: b g , b a , Among them, rotation angle The speed can be determined based on the angular velocity of the aircraft in the inertial measurement data. The displacement can be determined based on the linear acceleration of the aircraft in the inertial measurement data. The speed of the aircraft can be determined based on the inertial measurement data.

[0043] Step 150: Determine the covariance matrix and Kalman gain coefficient based on the reprojection error, the noise data, the covariance matrix, and the state data.

[0044] As one approach, the noise matrix can be calculated using the formula Q = ΦGQ. c G T Φ T Δt is used to determine, where Q c The noise matrix of the IMU can be set to a constant value, and G can be calculated using the following formula:

[0045]

[0046] In this matrix, the rows from top to bottom represent the rotation angles. First noise b g ,speed Second noise b a displacement The matrix columns from left to right are (n g ,n wg ,n a ,n wa ), representing angular velocity noise, the first noise of the IMU, and acceleration noise, the second noise of the IMU, respectively. Optionally, the first noise of the IMU can be the deviation of the acceleration sensor in the IMU, and the second noise of the IMU can be the deviation of the gyroscope in the IMU.

[0047] Optionally, after determining the noise matrix, it can be calculated according to formula P. t =ΦP t-1 Φ T +Q is used to determine the covariance matrix, where Q is the noise matrix and P is the noise matrix. t-1 Let Φ be the covariance matrix of the previous time step, and Φ be the integral of the state transition matrix over discrete time.

[0048] As one method, it can be based on the formula To determine the Kalman gain coefficient, where, Let H be the covariance matrix, H be the Jacobian matrix corresponding to the two-dimensional feature points, and R be the noise matrix. The Jacobian matrix is ​​determined based on the reprojection error of the two-dimensional feature points.

[0049] Step 160: Correct the state data according to the Kalman gain coefficient to obtain target state data, and determine the attitude of the aircraft according to the target state data.

[0050] As one approach, after determining the Kalman gain coefficient, it can be calculated using the formula... To correct the state data, among which, Here, K represents the state coefficients before correction, K is the Kalman gain coefficient, and r is the reprojection error corresponding to each two-dimensional feature point. The target state data may include the aircraft's current three-dimensional coordinates, yaw angle, pitch angle, roll angle, and velocity, and the aircraft's attitude can then be determined based on the target state data.

[0051] Optionally, the covariance matrix can be corrected based on the Kalman gain coefficients, thus facilitating the determination of the aircraft's attitude at the next time step based on the updated covariance matrix. Alternatively, it can be achieved through the formula... To correct the covariance matrix, where, Let H be the current covariance matrix, H be the Jacobian matrix of the two-dimensional feature points, I be the identity matrix, and K be the Kalman gain coefficient. This is the corrected covariance matrix.

[0052] In the embodiments of this application, a 3D transformation is first performed based on the 2D coordinates of multiple 2D feature points in the image data to obtain the 3D coordinates of each 2D feature point. Then, reprojection is performed based on these 3D coordinates to obtain the reprojected 2D coordinates of each 2D feature point. The reprojection error of each 2D feature point is determined based on these reprojected 2D coordinates and the 2D coordinates of each 2D feature point. Next, the aircraft's state data, noise data, and covariance matrix are determined based on inertial measurement data. The Kalman gain coefficient is then determined based on the reprojection error, state data, covariance matrix, and noise data of the 2D feature points. Finally, Kalman filtering is performed based on the Kalman gain coefficient to correct the state data, obtaining the target state data. The aircraft's pose is then determined based on the target state data. This solution estimates the aircraft's pose by filtering and fusing visual and IMU data for each image frame in the image data, enabling real-time pose determination while improving the accuracy of the pose determination.

[0053] Please see Figure 2 , Figure 2 This application illustrates an embodiment of a method for determining the attitude of an aircraft. The following will focus on... Figure 2 The process shown is described in detail. The image data includes multiple frames of images, and the method for determining the pose of the aircraft may specifically include the following steps:

[0054] Step 201: Acquire image data collected by the aircraft and determine multiple two-dimensional feature points from the image data.

[0055] Step 202: Determine the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and perform a three-dimensional transformation on the plurality of two-dimensional feature points according to their respective two-dimensional coordinates to determine the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points.

[0056] Step 203: Perform two-dimensional projection based on the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points to obtain the reprojected two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points, and determine the reprojection error corresponding to each of the multiple two-dimensional feature points based on the reprojected two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points and the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points.

[0057] Step 204: Obtain the inertial measurement data of the aircraft, and determine the state data, noise data, and covariance matrix of the aircraft based on the inertial measurement data.

[0058] Step 205: Determine the Kalman gain coefficient based on the reprojection error, the noise data, the covariance matrix, and the state data.

[0059] Step 206: Correct the state data according to the Kalman gain coefficient to obtain target state data, and determine the attitude of the aircraft according to the target state data.

[0060] The specific steps of steps 201-206 can be found in steps 110-160, and will not be repeated here.

[0061] Step 207: Determine keyframe images from the multi-frame images and determine the number of keyframe images.

[0062] One approach involves using image data that includes multiple frames. Images meeting the event conditions can be identified as keyframes. These keyframes can then be used to optimize the 3D coordinates of 2D feature points, increasing the accuracy of the 3D coordinates and consequently improving the accuracy of the aircraft's pose and state estimation. Optionally, the preset conditions can be image frames with more feature points than a threshold, or other conditions, which can be set according to actual needs.

[0063] In some embodiments, as Figure 3 As shown, step 207 includes:

[0064] Step 310: Determine the number of feature points corresponding to each frame in the multi-frame images.

[0065] One approach is to use corner detection algorithms to detect feature points in each frame of an image, thereby determining the number of feature points in each frame.

[0066] Step 320: Based on the number of feature points corresponding to each frame image, determine in the image data whether the number of matching feature points between the current frame image and the previous frame image is less than the number of points threshold.

[0067] One approach is to use optical flow tracing to match feature points in the current frame with those in the previous frame, thereby determining the number of matching feature points between the two frames. Optionally, the threshold number can be determined based on the number of feature points in the current frame and the number of matching feature points with the previous frame, or it can be set according to actual needs; no specific limitation is made here.

[0068] As another approach, keyframe images that meet preset conditions can be identified by comparing the number of feature points in consecutive frames. Alternatively, optical flow tracing can be used to determine the relationship between feature points in the current frame and the previous frame, thereby determining whether the number of feature points in the current frame is less than the number in the previous frame. Specifically, optical flow tracing involves identifying feature points in the previous frame and then matching those identified in the previous frame in the current frame to determine if the number of feature points in the current frame is less than the number in the previous frame.

[0069] Optionally, the difference between the number of feature points corresponding to the current frame image and the number of feature points in the previous frame image can be determined. When the difference is greater than the difference threshold, it can be determined that the number of feature points corresponding to the current frame image is less than the number of feature points corresponding to the Shanghai Stock Exchange image.

[0070] Step 330: If the number of points is less than the threshold, then the current frame image is determined as the keyframe image.

[0071] As one approach, if the number of feature points matching the feature points of the current frame image with the previous frame image is less than a threshold, the current frame image can be identified as a key frame image, and optimization can then be performed based on the key frame image.

[0072] As another approach, when there is an image frame in the image data where the number of feature points corresponding to the current frame image is less than the number of feature points corresponding to the previous frame image, the current frame image is determined as the key frame image, and optimization can then be performed based on the key frame image.

[0073] In some embodiments, step 207 includes: acquiring historical keyframes and determining pose information corresponding to the historical keyframes; determining pose information corresponding to each frame in the multi-frame images; determining the relative distance between the historical keyframes and each image based on the pose information corresponding to the historical keyframes and the pose information corresponding to each frame in the multi-frame images; if there is an image whose relative distance is greater than a distance threshold, then the image whose relative distance is greater than the distance threshold is determined as the keyframe image.

[0074] One approach is to determine the pose of a historical keyframe after acquiring it, based on the two-dimensional coordinates of feature points in the keyframe and the pose of the image acquisition device. Similarly, the pose of each frame can be determined based on the two-dimensional coordinates of feature points in each frame and the pose of the image acquisition device. Optionally, the Euclidean distance between the historical keyframe and each frame can be determined based on the pose of the historical keyframe and the pose of each frame, thus defining the Euclidean distance between the historical keyframe and each frame as the relative distance between them.

[0075] In some embodiments, step 207 further includes: acquiring historical keyframes and determining reference two-dimensional coordinates corresponding to feature points in the historical keyframes; determining the two-dimensional coordinates of feature points corresponding to each frame in the multi-frame images; determining the coordinate difference between the reference two-dimensional coordinates and the two-dimensional coordinates of the feature points corresponding to each image; if there is an image with a coordinate difference greater than a difference threshold, then the image with a coordinate difference greater than the difference threshold is determined as the keyframe image. As one approach, when historical keyframe images exist, the reference two-dimensional coordinates of the feature points corresponding to the historical keyframe images can be acquired, thereby allowing determination of whether the current image frame is a keyframe image based on the reference two-dimensional coordinates.

[0076] Optionally, for each frame of image, after determining the reference two-dimensional coordinates corresponding to the historical keyframe, the two-dimensional coordinates of the two-dimensional feature points in the current frame image that have the same characteristics as the historical keyframe can be determined. Then, the Euclidean distance between the reference two-dimensional coordinates and the two-dimensional feature points in the current frame image that have the same characteristics as the historical keyframe can be calculated. This Euclidean distance is determined as the coordinate difference between the reference two-dimensional coordinates and the two-dimensional feature points in the current frame image that have the same characteristics as the historical keyframe. This is used to determine the coordinate difference between the reference two-dimensional coordinates and the two-dimensional coordinates of the feature points corresponding to each image.

[0077] As another approach, after determining the historical keyframe images, the historical key timestamps corresponding to the historical keyframe images can be determined first. Then, the corresponding historical key inertial measurement data can be determined from the historical inertial measurement data, and the timestamps corresponding to each frame of the image data can be determined. Then, the current inertial measurement data corresponding to the timestamps corresponding to each frame of the image can be determined. Then, the target relative position information can be determined based on the historical inertial measurement data and the current inertial measurement data corresponding to each frame of the image. When the target relative position information is determined to be greater than a certain set threshold, the corresponding image frame can be determined as a keyframe image.

[0078] Please continue reading. Figure 2Step 208: If the number of keyframe images is greater than the number threshold, then determine the key inertial measurement data with the same timestamp as the keyframe images from the inertial measurement data.

[0079] As one approach, when the number of keyframe images exceeds a certain threshold, it can be determined that there are sufficient keyframe images. With sufficient keyframe images, head-down construction or local map optimization can be performed. Furthermore, the key timestamps corresponding to the keyframe images can be determined first, and then the key inertial measurement data corresponding to the key timestamps can be determined from the inertial measurement data. Subsequently, the more accurate pose information of the aircraft corresponding to the keyframe images can be determined based on the key inertial measurement data, thereby performing map construction or local map optimization.

[0080] Step 209: Determine the objective function based on the key inertial measurement data and the feature points corresponding to the key frame images.

[0081] As one approach, after determining the keyframe image and its corresponding key inertial measurement data, a constraint function can be constructed based on the keyframe image and keyframe inertial measurement data. Then, the optimal solution corresponding to this constraint function is obtained, and this optimal solution is determined as the aircraft's pose information corresponding to the keyframe image. This pose information is more accurate than the aircraft pose obtained in step 160. Optionally, the objective function may include three parts: specifically, the residual between the pre-integration of IMU data corresponding to two consecutive keyframe images and the relative pose of the two keyframe images; the residual between the two-dimensional coordinates obtained by reprojecting the three-dimensional feature points corresponding to the keyframe image back into the two-dimensional image and the two-dimensional coordinates of the key feature points in the keyframe image; and the constraint formed by all feature points in the marginalized keyframe image.

[0082] In some embodiments, such as Figure 4 As shown, step 209 includes:

[0083] Step 410: Determine the inertial data pre-integration corresponding to the key frame image based on the key inertial measurement data, and determine the first residual of the key frame based on the inertial data pre-integration and the key frame pose.

[0084] As a method, pre-integration refers to the accumulated observation data from continuous IMU measurements. The inertial data corresponding to the keyframe image can be determined using the following formula:

[0085]

[0086]

[0087]

[0088] Wherein, the key inertial measurement data corresponding to the i-th keyframe image, the key inertial measurement data corresponding to the j-th keyframe image, R is the rotation parameter, p is the translation parameter, ω is the angular velocity measurement value measured by the IMU, v is the linear velocity measurement value measured by the IMU, and a is the acceleration measurement value measured by the IMU. That is, the first noise at time k. The second noise at time k, g is the gravitational acceleration, η is the random walk noise, and Δt ij The cumulative duration from the i-th keyframe image to the j-th keyframe image.

[0089] Optionally, after determining the pre-integration corresponding to the key inertial measurement data, the determined pre-integration is subtracted from the relative translation, relative rotation, and relative velocity between consecutive frames to obtain the first residual.

[0090] Step 420: Determine the key two-dimensional coordinates and key three-dimensional coordinates of the key feature points in the key frame image, and determine the second residual of the key feature points based on the key three-dimensional coordinates and the key two-dimensional coordinates, wherein the second residual is the reprojection error of the key feature points.

[0091] One approach is to determine the key frame image after determining the key feature points in the key frame image from the image data, and then perform a reprojection operation based on the key two-dimensional coordinates to project them into the two-dimensional coordinate system (i.e., the image coordinate system where the key frame image is located), thereby obtaining the reprojection coordinates of the key feature points. Then, the difference between the reprojection coordinates and the key two-dimensional coordinates is calculated to determine the reprojection error corresponding to the key feature points, thereby obtaining the second residual of the key feature points.

[0092] Optionally, when reprojecting based on the key 3D coordinates corresponding to the key feature points, the key 3D coordinates can be optimized first, and then reprojected based on the optimized key 3D coordinates, thus ensuring the accuracy of the second residual.

[0093] Step 430: Determine the edge frame image based on the number of sliding windows, and determine the two-dimensional edge feature points in the edge frame image, wherein the edge frame image is the image at the very front of the sliding window.

[0094] One approach is to construct and maintain a lightweight factor graph using a sliding window, and then optimize it using graph optimization methods. Therefore, the current keyframe image can be determined by the sliding window. Keyframe images can be selected into the sliding window in order of their timestamps. When another keyframe image is about to be selected in the sliding window, the image at the very front of the sliding window can be identified as the edge frame image, that is, the image that is about to be discarded by the sliding window is the edge frame image.

[0095] Optionally, after determining the edge frame image, the corresponding two-dimensional feature points in the edge frame image can be determined from among the multiple two-dimensional feature points in the image data determined in step 120, thereby determining the edge two-dimensional feature points. Optionally, a corresponding flag can be set for each frame image in the image data. After determining the edge frame image, the corresponding image can be determined in the image data according to the flag corresponding to the edge frame image, thereby obtaining the edge two-dimensional feature points.

[0096] Step 440: Determine the objective function based on the two-dimensional feature points of the edge, the first residual, and the second residual.

[0097] As one approach, after determining the two-dimensional feature points of the edge and the first and second residuals corresponding to the keyframe images, the two-dimensional coordinates of the two-dimensional feature points of the edge, the first residual, and the second residual can be added together to determine the objective function. This objective function is then used as a constraint condition, and optimization can be performed based on this objective function.

[0098] Please continue reading. Figure 2 Step 210: Determine the optimal solution of the objective function based on the least squares method, and optimize the pose according to the optimal solution to obtain the target three-dimensional coordinates, so as to construct a map based on the target three-dimensional coordinates.

[0099] One approach is to find the optimal function for the data by minimizing the sum of squared errors (i.e., least squares method) after determining the objective function. Optionally, the optimal solution can be pose information, including spatial three-dimensional coordinates, yaw angle, pitch angle, or roll angle. Alternatively, after obtaining the optimal solution, a local map can be constructed based on the pose information of the image frame and the three-dimensional coordinates of feature points within the image frame. This allows for map optimization or global map construction based on the constructed local map.

[0100] In this embodiment, keyframe images are identified from the image data. When the number of keyframe images exceeds a threshold, key inertial measurement data with the same timestamp are determined based on the timestamps corresponding to the key image frames. This facilitates the determination of the objective function based on the key inertial measurement data and the feature points corresponding to the keyframe images. The optimal solution of the objective function is then determined using the least squares method. The aircraft's pose is then optimized based on the optimal solution to obtain the target's three-dimensional coordinates. This facilitates map construction based on the target's three-dimensional coordinates, reduces the cumulative error caused by long-term accumulation, further improves the accuracy of state estimation, and ensures the accuracy of the aircraft's pose.

[0101] Please see Figure 5 , Figure 5 This application illustrates an embodiment of a method for determining the attitude of an aircraft. The following will focus on... Figure 5 The process shown is described in detail. The image data includes multiple frames of images, and the method for determining the pose of the aircraft may specifically include the following steps:

[0102] Step 510: Acquire the image data collected by the aircraft, and determine whether there is a first frame image in the image data, wherein the first frame image is the first frame image collected by the aircraft.

[0103] As one approach, the first image frame is the earliest image frame acquired in the image data, that is, the first image captured after the image acquisition device is turned on. The first image reflects the initial state of the aircraft, and subsequent image frames are obtained based on changes in the first image. To avoid errors or insufficient numbers of two-dimensional feature points when determining multiple two-dimensional feature points, it is advisable to first determine whether the first image exists in the image data.

[0104] Optionally, the initial timestamp of the aircraft's image acquisition device being turned on can be determined first, then the timestamp corresponding to each frame of the image data can be determined separately, and then it can be determined whether there is an image frame corresponding to a timestamp with the same initial timestamp or an image frame corresponding to a timestamp whose time difference with the initial timestamp is less than the difference threshold, so as to determine whether there is a first frame of the image data.

[0105] Step 520: If it is determined that there is no first frame image, then optical flow tracing is performed on the image data to determine multiple two-dimensional matching point pairs.

[0106] As one approach, when it is determined that the first frame image does not exist, in order to avoid large deviations in the determination of feature points in consecutive frame images, feature points can be determined first in a certain image frame using optical flow tracing, and then feature points with the same characteristics can be determined in other image frames based on optical flow tracing, thereby determining multiple two-dimensional matching point pairs.

[0107] As another approach, if the existence of a first frame image is confirmed, the FAST algorithm can be used to detect corner points in the first frame image to determine multiple two-dimensional feature points of the first frame image. Then, optical flow tracking can be performed on other images based on the multiple two-dimensional feature points determined in the first frame image to determine the two-dimensional feature points corresponding to each image in the image data.

[0108] Step 530: Perform corner detection on the image data to obtain multiple two-dimensional corner points.

[0109] As one approach, since the feature points in different consecutive frames of an aircraft are in motion, there are certain differences. For example, it is difficult for all N feature points in the previous image A1 to appear in the current image A2. In order to ensure the accuracy of the aircraft's pose, the matching point pairs determined after optical flow tracking can be marked, and then corner detection can be performed on the image data using the FAST algorithm to obtain multiple two-dimensional corner points. This makes it easier to use the two-dimensional corner points as two-dimensional feature points of the image data, increasing the richness of two-dimensional feature points and thus increasing the accuracy of the aircraft's pose.

[0110] Step 540: Determine the plurality of two-dimensional feature points based on the plurality of two-dimensional matching point pairs and the plurality of two-dimensional corner points.

[0111] As one approach, after determining multiple pairs of two-dimensional matching points and multiple two-dimensional corner points, the corresponding image frame markers for each pair of two-dimensional matching points and multiple two-dimensional corner points are determined, thereby determining the corresponding two-dimensional matching points and two-dimensional corner points for each frame of the image, and further determining the corresponding multiple two-dimensional feature points for each frame of the image.

[0112] Step 550: Determine the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and perform a three-dimensional transformation on the plurality of two-dimensional feature points according to their respective two-dimensional coordinates to determine the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points.

[0113] Step 560: Perform two-dimensional projection based on the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points to obtain the reprojected two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points, and determine the reprojection error corresponding to each of the multiple two-dimensional feature points based on the reprojected two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points and the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points.

[0114] Step 570: Obtain the inertial measurement data of the aircraft, and determine the state data, noise data, and covariance matrix of the aircraft based on the inertial measurement data.

[0115] Step 580: Determine the Kalman gain coefficient based on the reprojection error, the noise data, the covariance matrix, and the state data.

[0116] Step 590: Correct the state data according to the Kalman gain coefficient to obtain target state data, and determine the attitude of the aircraft according to the target state data.

[0117] The specific steps of steps 550-590 can be found in steps 120-160, and will not be repeated here.

[0118] In this embodiment, after acquiring the image data collected by the aircraft, it is first determined whether a first frame image exists in the image data. If it is determined that a first frame image does not exist, multiple two-dimensional matching point pairs in the image data are first determined by optical flow tracing, and then corner detection is performed on the image data to determine multiple two-dimensional corner points. In this way, multiple two-dimensional feature points are determined based on multiple two-dimensional matching point pairs and multiple two-dimensional corner points, ensuring the richness of two-dimensional feature points and thus increasing the accuracy of the aircraft's pose.

[0119] Please see Figure 6 , Figure 6 This application illustrates an embodiment of a method for determining the attitude of an aircraft. The following will focus on... Figure 6 The process shown is described in detail. The image data package includes multiple frames of images, and the method for determining the pose of the aircraft may specifically include the following steps:

[0120] Step 610: Acquire image data collected by the aircraft and determine multiple two-dimensional feature points from the image data.

[0121] Step 620: Determine whether the image data comes from different image acquisition devices.

[0122] One approach is to determine the source of each frame of an image by identifying the image information of each frame in the image data, thereby determining whether each frame of the image data was acquired by the same image acquisition device.

[0123] Step 630: If it is determined that the image data comes from different image acquisition devices, then a three-dimensional transformation is performed based on the binocular triangulation algorithm and the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points to determine the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points.

[0124] As one approach, once it is determined that the image data comes from different image acquisition devices, the same image frames and feature point pairs corresponding to the same scene acquired by different image acquisition devices are first obtained. Then, a three-dimensional transformation is performed based on the binocular triangulation algorithm and the two-dimensional coordinates corresponding to the feature point pairs to determine the three-dimensional coordinates of the feature point pairs in world coordinates. In this way, the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points are obtained.

[0125] Step 640: If it is determined that the image data comes from the same image acquisition device, then a three-dimensional transformation is performed according to the multi-frame triangulation algorithm and the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points to determine the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points.

[0126] In one approach, when it is determined that each frame of the image data comes from the same image acquisition device, the three-dimensional coordinates of the two-dimensional feature points with the same characteristics in the preceding and following frames of the image data can be determined based on the two-dimensional coordinates of the two-dimensional feature points with the same characteristics in the preceding and following frames of the image data and the multi-frame triangulation algorithm. In this way, the three-dimensional coordinates of each of the multiple two-dimensional feature points in each frame of the image data can be determined.

[0127] Step 650: Perform two-dimensional projection based on the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points to obtain the reprojected two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and determine the reprojection error corresponding to each of the plurality of two-dimensional feature points based on the reprojected two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points and the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points.

[0128] Step 660: Obtain the inertial measurement data of the aircraft, and determine the state data, noise data, and covariance matrix of the aircraft based on the inertial measurement data.

[0129] Step 670: Determine the Kalman gain coefficient based on the reprojection error, the noise data, the covariance matrix, and the state data.

[0130] Step 680: Correct the state data according to the Kalman gain coefficient to obtain target state data, and determine the attitude of the aircraft according to the target state data.

[0131] The specific steps of steps 610 and 650-680 can be found in steps 110 and 130-160, and will not be repeated here.

[0132] In this embodiment, the algorithm for 3D transformation of multiple 2D feature points is determined by first determining whether the image data was acquired by different acquisition devices. Specifically, when it is determined that the image data was acquired by the same image acquisition device, multi-frame triangulation is used to perform 3D transformation of multiple 2D feature points; when it is determined that the image data was acquired by different image acquisition devices, binocular triangulation is used to perform 3D transformation of multiple 2D feature points. This improves the accuracy of multiple 2D feature points for their respective 3D coordinates, thereby improving the accuracy of the aircraft's pose.

[0133] Please see Figure 7 , Figure 7 This application illustrates an embodiment of a method for determining the attitude of an aircraft. The following will focus on... Figure 7 The process shown is described in detail. The image data includes multiple frames of images, and the method for determining the pose of the aircraft may specifically include the following steps:

[0134] Step 710: Acquire image data collected by the aircraft and determine multiple two-dimensional feature points from the image data.

[0135] Step 720: Determine the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and perform a three-dimensional transformation on the plurality of two-dimensional feature points according to the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points to determine the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points.

[0136] Step 730: Among the plurality of two-dimensional feature points, determine the target feature point whose repetition count is greater than the count threshold, and determine the reference three-dimensional coordinates of the target feature point based on the target two-dimensional coordinates of the target feature point.

[0137] As one approach, after determining the three-dimensional coordinates of multiple two-dimensional feature points, in order to ensure the accuracy of the aircraft's attitude determination, the three-dimensional coordinates of the feature points that meet the preset conditions among the multiple two-dimensional feature points can be optimized, making the three-dimensional coordinates of the feature points that meet the preset conditions among the multiple two-dimensional feature points more accurate, thereby improving the accuracy of the reprojection error, and thus improving the accuracy of the flight attitude.

[0138] Optionally, when the same feature point is repeated multiple times in the image data, it can be determined that the feature point has been observed multiple times by the image acquisition device. This allows for the optimization of the feature point's three-dimensional coordinates using a non-linear optimization method, resulting in more accurate three-dimensional coordinates. Optionally, the threshold for the number of observations can be 2 or other values, which can be set according to actual needs and are not specifically limited here. Optionally, after determining the target feature point, the reference three-dimensional coordinates corresponding to the target feature point can be determined in step 720.

[0139] Step 740: Reproject the target feature point according to the reference three-dimensional coordinates to obtain the reference two-dimensional coordinates of the target feature point, and determine the reference reprojection error according to the target two-dimensional coordinates and the reference two-dimensional coordinates of the target feature point.

[0140] As one method, after determining the reference 3D coordinates of the target feature point, these coordinates can be projected onto the image coordinate system where the image data resides, thus obtaining the reference 2D coordinates of the target feature point. The reference reprojection error corresponding to the target feature point can then be determined by calculating the difference between the 2D coordinates of the target feature point and its reference 2D coordinates. Optionally, this reference reprojection error can be the Euclidean distance or the cosine distance between the 2D coordinates of the target feature point and its reference 2D coordinates; the form of the reprojection error can be set according to actual needs and is not specifically limited here.

[0141] Step 750: Correct the reference three-dimensional coordinates of the target feature points according to the reference reprojection error, and determine the reprojection error corresponding to each of the plurality of two-dimensional feature points according to the corrected reference three-dimensional coordinates.

[0142] One approach is to use the reference reprojection error of the target feature points as an optimization function, and then correct the reference 3D coordinates of the target feature points by minimizing the optimization function. Optionally, the optimization function can be... ∑||u i ―T i P|| 2 Where P is the reference 3D coordinate of the target feature point, i is the i-th frame image, and u i T represents the target two-dimensional coordinates of the target feature point. i The calibration parameters for the image acquisition device may include the intrinsic and extrinsic parameters of the image acquisition device (i.e., the pose of the image acquisition device relative to the aircraft), P * The corrected reference 3D coordinates for the target feature points.

[0143] Optionally, when determining the reprojection error corresponding to each of the multiple two-dimensional coordinate points, reprojection can be performed based on the corrected reference three-dimensional coordinates corresponding to each of the multiple two-dimensional coordinate points, thereby determining the reprojection error corresponding to each of the multiple two-dimensional feature points.

[0144] Step 760: Obtain the inertial measurement data of the aircraft, and determine the state data, noise data, and covariance matrix of the aircraft based on the inertial measurement data.

[0145] Step 770: Determine the Kalman gain coefficient based on the reprojection error, the noise data, the covariance matrix, and the state data.

[0146] Step 780: Correct the state data according to the Kalman gain coefficient to obtain target state data, and determine the attitude of the aircraft according to the target state data.

[0147] The specific steps of steps 710-720 and 760-780 can be found in steps 110-120 and 140-160, and will not be repeated here.

[0148] In this embodiment, by identifying target feature points among multiple two-dimensional feature points with a repetition count greater than a threshold, and determining the reprojection error corresponding to the target feature point based on its target two-dimensional coordinates, the reference three-dimensional coordinates of the target feature can be corrected based on the reprojection error of the target feature point. The reprojection error corresponding to the target feature point is then determined based on the corrected reference three-dimensional coordinates, and the aircraft's pose is determined based on the reprojection error corresponding to the target feature point, thus ensuring the accuracy of the aircraft's pose.

[0149] Figure 8 This is a flowchart illustrating an aircraft attitude determination method according to an embodiment of this application, as shown below. Figure 8 As shown, firstly, image data acquired by the aircraft's image acquisition device and inertial measurement data acquired by the inertial measurement unit are obtained. Then, it is determined whether the image data is the first frame image. If it is determined to be the first frame image, corner detection is performed on the first frame image to determine multiple two-dimensional feature points of the first frame image. If it is not the first frame image, feature points are extracted from the current frame image and the previous frame image using optical flow tracing to obtain multiple two-dimensional feature points. Then, a three-dimensional transformation is performed based on the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points to parameterize the multiple two-dimensional feature points and obtain the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points. Reprojection is then performed based on the three-dimensional coordinates of the multiple two-dimensional feature points, and multiple two-dimensional feature points are determined based on the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points and the reprojected two-dimensional coordinates. The reprojection error corresponding to each two-dimensional feature point is calculated. Then, the current state data of the aircraft and the covariance matrix corresponding to the IMU are predicted based on the inertial measurement data. The Kalman gain coefficient is determined based on the current state data, covariance matrix, and reprojection error of the aircraft. The current state data and covariance matrix of the aircraft are updated by the Kalman rate filtering method to obtain the pose of the aircraft. Furthermore, it is determined whether there is a keyframe image in the image data. When a keyframe image is found, it is added to the corresponding local map. The local map is optimized based on the keyframe image to determine the pose and coordinates of the map points corresponding to the keyframe image. Finally, the three-dimensional coordinates corresponding to multiple two-dimensional feature points are corrected based on the pose and coordinates of the map points corresponding to the keyframe image.

[0150] Figure 9 This is a block diagram of an attitude determination device for an aircraft according to an embodiment of this application, as shown below. Figure 9 As shown, the attitude determination device 800 of the aircraft includes: a two-dimensional feature point determination module 810, a three-dimensional transformation module 820, a reprojection error determination module 830, an inertial data processing module 840, a filtering module 850, and an attitude determination module 860.

[0151] A two-dimensional feature point determination module 810 is used to acquire image data collected by the aircraft and determine multiple two-dimensional feature points from the image data; a three-dimensional transformation module 820 is used to determine the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points, and perform three-dimensional transformation on the multiple two-dimensional feature points according to the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points to determine the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points; a reprojection error determination module 830 is used to perform two-dimensional projection based on the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points to obtain the reprojected two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points, and determine the reprojected two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points. The reprojection error corresponding to each of the multiple two-dimensional feature points is determined by the coordinates of the coordinates of each feature point. An inertial data processing module 840 is used to acquire the inertial measurement data of the aircraft and determine the state data, noise data, and covariance matrix of the aircraft based on the inertial measurement data. A filtering module 850 is used to determine the Kalman gain coefficient based on the reprojection error, the noise data, the covariance matrix, and the state data. A pose determination module 860 is used to correct the state data based on the Kalman gain coefficient to obtain target state data and determine the pose of the aircraft based on the target state data.

[0152] In some embodiments, the image data includes multiple frames of images, and the pose determination device 800 of the aircraft further includes: a keyframe image determination module, used to determine keyframe images in the multiple frames of images and determine the number of keyframe images; a key inertial measurement data determination module, used to determine key inertial measurement data with the same timestamp as the keyframe images from the inertial measurement data if the number of keyframe images is greater than a number threshold; an objective function determination module, used to determine an objective function based on the key inertial measurement data and the feature points corresponding to the keyframe images; and an optimization module, used to determine the optimal solution of the objective function based on the least squares method, and optimize the pose and the three-dimensional coordinates corresponding to each of the multiple feature points based on the optimal solution to obtain target three-dimensional coordinates, so as to construct a map based on the target three-dimensional coordinates.

[0153] In some embodiments, the keyframe image determination module includes: a feature point number determination unit, configured to determine the number of feature points corresponding to each frame image in the multi-frame images; a judgment unit, configured to determine whether there exists a current frame image and a previous frame image with a matching feature point number less than a point number threshold in the image data based on the feature point number corresponding to each frame image; and a first keyframe image determination unit, configured to determine the current frame image as the keyframe image if the number of matching feature points is less than the point number threshold. In other embodiments, the keyframe image determination module includes: a historical keyframe acquisition unit, configured to acquire historical keyframes and determine the pose information corresponding to the historical keyframes; a pose information determination unit, configured to determine the pose information corresponding to each frame image in the multi-frame images; a relative distance determination unit, configured to determine the relative distance between the historical keyframes and each image based on the pose information corresponding to the historical keyframes and the pose information corresponding to each frame image in the multi-frame images; and a second keyframe image determination unit, configured to determine the image with a relative distance greater than a distance threshold as the keyframe image if there exists an image with a relative distance greater than a distance threshold. In some embodiments, the objective function determination module includes: a first residual determination unit, configured to determine the inertial data pre-integration corresponding to the key frame image based on the key inertial measurement data, and determine the first residual of the key frame based on the inertial data pre-integration and the key inertial measurement data; a second residual determination unit, configured to determine the key two-dimensional coordinates and key three-dimensional coordinates of key feature points in the key frame image, and determine the second residual of the key feature points based on the key three-dimensional coordinates and the key two-dimensional coordinates, wherein the second residual is the reprojection error of the key feature points; an edge two-dimensional feature point determination unit, configured to determine edge frame images based on the number of sliding windows, and determine the edge two-dimensional feature points in the edge frame images, wherein the edge frame images are the images at the foremost edge of the sliding windows; and an objective function determination unit, configured to determine the objective function based on the edge two-dimensional feature points, the first residual, and the second residual.

[0154] In some embodiments, the two-dimensional feature point determination module 810 includes: a second determination unit, configured to acquire image data collected by the aircraft and determine whether a first frame image exists in the image data, wherein the first frame image is the first frame image collected by the aircraft; a matching unit, configured to perform optical flow tracing on the image data to determine multiple two-dimensional matching point pairs if it is determined that no first frame image exists; a corner detection unit, configured to perform corner detection on the image data to obtain multiple two-dimensional corner points; and a two-dimensional feature point determination unit, configured to determine the multiple two-dimensional feature points based on the multiple two-dimensional matching point pairs and the multiple two-dimensional corner points.

[0155] In some embodiments, the 3D conversion module 820 includes: a third determining unit, configured to determine whether the image data comes from different image acquisition devices; a first conversion unit, configured to, if it is determined that the image data comes from different image acquisition devices, perform 3D conversion based on a binocular triangulation algorithm and the 2D coordinates corresponding to each of the plurality of 2D feature points, and determine the 3D coordinates corresponding to each of the plurality of 2D feature points; or a second conversion unit, configured to, if it is determined that the image data comes from the same image acquisition device, perform 3D conversion based on a multi-frame triangulation algorithm and the 2D coordinates corresponding to each of the plurality of 2D feature points, and determine the 3D coordinates corresponding to each of the plurality of 2D feature points.

[0156] In some embodiments, the attitude determination device 800 of the aircraft further includes: a reference three-dimensional coordinate determination module, configured to determine a target feature point among the plurality of two-dimensional feature points whose repetition count is greater than a threshold, and determine the reference three-dimensional coordinates of the target feature point based on the target two-dimensional coordinates of the target feature point; a reference reprojection error determination module, configured to perform reprojection based on the reference three-dimensional coordinates of the target feature point to obtain the reference two-dimensional coordinates of the target feature point, and determine the reference reprojection error based on the target two-dimensional coordinates and the reference two-dimensional coordinates of the target feature point; and a coordinate correction module, configured to correct the reference three-dimensional coordinates of the target feature point based on the reference reprojection error, and determine the reprojection error corresponding to each of the plurality of two-dimensional feature points based on the corrected reference three-dimensional coordinates.

[0157] According to one aspect of the embodiments of this application, an electronic device is also provided, such as... Figure 10 As shown, the electronic device 900 includes a processor 910 and one or more memories 920. The one or more memories 920 are used to store program instructions executed by the processor 910. When the processor 910 executes the program instructions, it implements the above-described attitude determination method for the aircraft.

[0158] Furthermore, the processor 910 may include one or more processing cores. The processor 910 runs or executes instructions, programs, code sets, or instruction sets stored in the memory 920, and retrieves data stored in the memory 920. Optionally, the processor 910 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 910 may integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor and may be implemented using a separate communication chip.

[0159] According to one aspect of this application, a computer-readable storage medium is also provided, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0160] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0161] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

Claims

1. A method for determining the attitude of an aircraft, characterized in that, The method includes: The system acquires image data collected by the aircraft and determines multiple two-dimensional feature points from the image data. Determine the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and perform a three-dimensional transformation on the plurality of two-dimensional feature points according to the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points to determine the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points. Two-dimensional projection is performed based on the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points to obtain the reprojected two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points, and the reprojection error corresponding to each of the multiple two-dimensional feature points is determined based on the reprojected two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points and the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points. Acquire the inertial measurement data of the aircraft, and determine the state data, noise data, and covariance matrix of the aircraft based on the inertial measurement data; The Kalman gain coefficient is determined based on the reprojection error, the noise data, the covariance matrix, and the state data. The state data is corrected according to the Kalman gain coefficient to obtain target state data, and the attitude of the aircraft is determined according to the target state data.

2. The method according to claim 1, characterized in that, The image data includes multiple frames of images. After correcting the state data according to the Kalman gain coefficient to obtain target state data, and determining the pose of the aircraft based on the target state data, the method further includes: Keyframe images are identified from the multiple frames of images, and the number of keyframe images is determined. If the number of keyframe images is greater than a threshold, then key inertial measurement data with the same timestamp as the keyframe images are determined from the inertial measurement data. The objective function is determined based on the key inertial measurement data and the feature points corresponding to the key frame images; The optimal solution of the objective function is determined based on the least squares method, and the pose and the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points are optimized according to the optimal solution to obtain the target three-dimensional coordinates, so as to construct a map based on the target three-dimensional coordinates.

3. The method according to claim 2, characterized in that, Determining keyframe images from the multiple frames includes: Determine the number of feature points corresponding to each frame in the multi-frame images; Based on the number of feature points corresponding to each frame image, determine in the image data whether the number of matching feature points between the current frame image and the previous frame image is less than the number of points threshold. If the number of points is less than the threshold, then the current frame image is determined as the keyframe image.

4. The method according to claim 2, characterized in that, Determining keyframe images from the multiple frames includes: Acquire historical keyframes and determine the pose information corresponding to the historical keyframes; Determine the pose information corresponding to each frame in the multi-frame image; Based on the pose information corresponding to the historical keyframe and the pose information corresponding to each frame in the multi-frame image, the relative distance between the historical keyframe and each frame image is determined. If there is an image whose relative distance is greater than a distance threshold, then the image whose relative distance is greater than a distance threshold is determined as the keyframe image.

5. The method according to claim 2, characterized in that, The step of determining the objective function based on the key inertial measurement data and the feature points corresponding to the key frame images includes: The inertial data pre-integration corresponding to the key frame image is determined based on the key inertial measurement data, and the first residual of the key frame is determined based on the inertial data pre-integration and the key inertial measurement data; Determine the key two-dimensional coordinates and key three-dimensional coordinates of key feature points in the key frame image, and determine the second residual of the key feature points based on the key three-dimensional coordinates and the key two-dimensional coordinates, wherein the second residual is the reprojection error of the key feature points; The edge frame image is determined based on the number of sliding windows, and the two-dimensional feature points of the edge in the edge frame image are determined, wherein the edge frame image is the image at the frontmost edge of the sliding window; The objective function is determined based on the two-dimensional feature points of the edge, the first residual, and the second residual.

6. The method according to any one of claims 1-5, characterized in that, The process of acquiring image data collected by the aircraft and determining multiple two-dimensional feature points based on the image data includes: The image data collected by the aircraft is acquired, and it is determined whether a first frame image exists in the image data, wherein the first frame image is the first frame image collected by the aircraft; If it is determined that there is no first frame image, then optical flow tracing is performed on the image data to determine multiple two-dimensional matching point pairs; Corner detection is performed on the image data to obtain multiple two-dimensional corner points; The plurality of two-dimensional feature points are determined based on the plurality of two-dimensional matching point pairs and the plurality of two-dimensional corner points.

7. The method according to any one of claims 1-5, characterized in that, The step of performing a three-dimensional transformation on the plurality of two-dimensional feature points based on their respective two-dimensional coordinates to determine the three-dimensional coordinates of each of the plurality of two-dimensional feature points includes: Determine whether the image data comes from different image acquisition devices; If it is determined that the image data comes from different image acquisition devices, then a 3D transformation is performed based on the binocular triangulation algorithm and the 2D coordinates corresponding to each of the multiple 2D feature points to determine the 3D coordinates corresponding to each of the multiple 2D feature points; or If it is determined that the image data comes from the same image acquisition device, then a three-dimensional transformation is performed based on the multi-frame triangulation algorithm and the two-dimensional coordinates corresponding to each of the multiple two-dimensional feature points to determine the three-dimensional coordinates corresponding to each of the multiple two-dimensional feature points.

8. The method according to claim 1, characterized in that, After performing a three-dimensional transformation on the plurality of two-dimensional feature points based on their respective two-dimensional coordinates to determine their respective three-dimensional coordinates, the method further includes: Among the plurality of two-dimensional feature points, a target feature point is determined whose repetition count is greater than a threshold, and a reference three-dimensional coordinate of the target feature point is determined based on the target two-dimensional coordinate of the target feature point; Reprojection is performed based on the reference three-dimensional coordinates of the target feature point to obtain the reference two-dimensional coordinates of the target feature point, and the reference reprojection error is determined based on the target two-dimensional coordinates and the reference two-dimensional coordinates of the target feature point. The reference three-dimensional coordinates of the target feature points are corrected based on the reference reprojection error, and the reprojection error corresponding to each of the plurality of two-dimensional feature points is determined based on the corrected reference three-dimensional coordinates.

9. An attitude determination device for an aircraft, characterized in that, The device includes: A two-dimensional feature point determination module is used to acquire image data collected by the aircraft and determine multiple two-dimensional feature points from the image data; A 3D conversion module is used to determine the 2D coordinates corresponding to each of the plurality of 2D feature points, and to perform 3D conversion on the plurality of 2D feature points according to the 2D coordinates corresponding to each of the plurality of 2D feature points, thereby determining the 3D coordinates corresponding to each of the plurality of 2D feature points. The reprojection error determination module is used to perform two-dimensional projection based on the three-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points to obtain the reprojected two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points, and to determine the reprojection error corresponding to each of the plurality of two-dimensional feature points based on the reprojected two-dimensional coordinates and the two-dimensional coordinates corresponding to each of the plurality of two-dimensional feature points. An inertial data processing module is used to acquire the inertial measurement data of the aircraft, and determine the state data, noise data, and covariance matrix of the aircraft based on the inertial measurement data. The filtering module is used to determine the Kalman gain coefficient based on the reprojection error, the noise data, the covariance matrix, and the state data; The pose determination module is used to correct the state data according to the Kalman gain coefficient to obtain target state data, and to determine the pose of the aircraft according to the target state data.

10. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 8.

Images