Airplane pose estimation method based on pose decoupling and dynamic three-dimensional model

Abstract

This invention discloses an aircraft pose estimation method based on pose decoupling and a dynamic 3D model, belonging to the field of aircraft pose estimation. This method employs a pose decoupling strategy, dividing aircraft pose estimation into two relatively independent subtasks to reduce interference between them. It also leverages the advantages of convolutional neural networks, Transformer networks, and multilayer perceptrons to construct an attitude angle estimation network, RotNet, accurately estimating the aircraft's attitude angles in a single image. A dynamic 3D model strategy is used, utilizing the attitude angles estimated by RotNet to rotate the aircraft's 3D model in a reference coordinate system and combining it with image features. This model is then used as input to the position estimation network, TransNet, to estimate the aircraft's position. This invention can accurately estimate aircraft pose information by inputting only a single image and the aircraft's 3D model, controlling the attitude angle error within 1 degree and the position error within 1 meter.

Description

Technical Field

[0001] This invention relates to the field of aircraft pose estimation, and particularly to an aircraft pose estimation method based on pose decoupling and dynamic three-dimensional models. Background Technology

[0002] As one of the most important modes of transportation globally, the safe operation of aircraft is of paramount importance. Statistics show that the vast majority of aviation accidents occur during takeoff and landing; therefore, flight safety during these two critical phases is especially crucial. To ensure safe takeoff and landing, accurately estimating the aircraft's spatial position and attitude angles (i.e., "attitude") within a reference coordinate system during these processes is a critical step.

[0003] Traditional aircraft pose estimation primarily relies on inertial navigation systems (INS) and global positioning systems (GPS). However, INS suffers from accumulated errors, and GPS is susceptible to signal interference, especially in complex environments. Furthermore, vision-based aircraft pose estimation methods have attracted considerable attention, with related methods mainly categorized into single-stage and two-stage methods. Single-stage methods directly regress the aircraft's pose from the input data, offering high computational efficiency but relatively low accuracy. Two-stage methods first detect the 2D-3D correspondence of the aircraft and then utilize Random Sample Consensus (RANSAC) and Perspective-n-Point (PnP) algorithms to estimate the aircraft's pose. While this method offers higher accuracy, it is highly dependent on the accuracy of the correspondence and has a slower computation speed. These methods, however, are characterized by low cost, inexpensive installation, and simple operation and maintenance.

[0004] Vision-based pose estimation typically uses RGB images, point clouds, 3D models, or combinations thereof. Multimodal inputs can achieve information complementarity, improving robustness and accuracy. However, most methods are designed for small-scale, close-range scenes, where extracting reliable features is crucial for accurate pose estimation. Aircraft, as large, distant objects, often have their features reduced and blurred due to distance, making it difficult to capture effective representations using traditional methods. Therefore, designing a high-precision, real-time aircraft pose estimation method applicable to large-scale targets is essential for aircraft takeoff and landing. Summary of the Invention

[0005] Purpose of the invention: To address the above problems, the purpose of this invention is to provide an aircraft pose estimation method based on pose decoupling and dynamic 3D model.

[0006] Technical solution: The aircraft pose estimation method based on pose decoupling and dynamic 3D model of the present invention includes the following steps:

[0007] Acquire aircraft images and corresponding pose information from different perspectives and at different times during flight, and construct a dataset.

[0008] Features at different levels are extracted from aircraft images using a feature extraction network;

[0009] Features from different levels are input into the attitude angle estimation network to estimate the aircraft's rotation matrix;

[0010] Calculate the rotation loss value, iteratively optimize the attitude angle estimation network based on the rotation loss value, and use the optimized attitude angle estimation network to obtain the target rotation matrix of the aircraft.

[0011] The target rotation matrix is ​​used to rotate the 3D model of the aircraft, and the position estimation network is used to predict the position of the rotated 3D model of the aircraft.

[0012] Calculate the position loss value, iteratively optimize the position estimation network based on the loss value, and use the optimized position estimation network to predict the aircraft's position.

[0013] Furthermore, the steps for extracting features at different levels from aircraft images using a feature extraction network include:

[0014] The aircraft image is input into the ResNet network to obtain features f2, f3, f4 and f5 output from different layers of the ResNet network.

[0015] Furthermore, the steps for estimating the aircraft's rotation matrix by inputting features from different levels into the attitude angle estimation network include:

[0016] Features f2, f3, f4, and f5 are input into the Feature Pyramid Network (FPN) for fusion processing to obtain the edge feature Fside.

[0017] The features f3, f4, and f5 are converted into two-dimensional feature representations by using a flattening operation, and then stacked along the first dimension of the channel to obtain the backbone network features.

[0018] The features of the backbone network are input into the feature optimization Transformer module, which contains the encoder and decoder, to obtain global features;

[0019] The global features are input into a multilayer perceptron to obtain the first feature;

[0020] The first feature is subjected to tensor dimension transformation to make the first feature and the edge feature Fside consistent in dimension. Then, the edge feature and the first feature are stacked to obtain the pose fusion feature.

[0021] The attitude fusion features are input into the RotHead attitude angle regression network to predict the aircraft's rotation matrix. R .

[0022] Furthermore, the steps for predicting the position of the rotated 3D aircraft model using a position estimation network include:

[0023] The 3D model of the aircraft is made of point clouds. To represent, where Let represent the number of points in the point cloud. Then, the rotated 3D model of the aircraft is... The multilayer perceptron (MLP) is used to process the features of the rotated 3D model of the aircraft. Then, the Einstein summation convention is used to perform tensor dimension transformation on the processing results to obtain the second feature, so that the second feature is consistent with the edge feature Fside in dimensionality.

[0024] The second feature is stacked with the edge feature Fside to obtain the geometric-image feature;

[0025] Feature extraction of feature f5 is performed using the PoseNet image pose network, and its output is stacked with geometric-image features to obtain position fusion features;

[0026] The position stack features are input into the position regression network TransHead for processing to obtain the translation matrix representing the aircraft's position. Where K is the camera's intrinsic parameter, It is an estimated depth value. It is the estimated projection position of the center point of the aircraft's 3D bounding box in the pixel coordinate system.

[0027] Furthermore, the formula for calculating the rotational loss value is:

[0028] ,

[0029] In the formula, Let R be the rotation matrix transformed from the aircraft's actual attitude angles, and let R be the rotation matrix of the aircraft predicted by the attitude angle regression network RotHead. Let N represent the coordinates of key points on the aircraft fuselage in the fuselage coordinate system, where N represents the number of points. This represents the L1 loss function.

[0030] Furthermore, the formula for calculating the position loss value is:

[0031] ,

[0032] In the formula, It is the estimated projection position of the center point of the aircraft's 3D bounding box in the pixel coordinate system. It is the true projection position of the center point of the aircraft's 3D bounding box in the pixel coordinate system. It is an estimated depth value. For the actual depth value, This represents the L1 loss function.

[0033] Beneficial effects: Compared with the prior art, the significant advantages of this invention are:

[0034] 1. This invention can accurately estimate the aircraft's attitude information by inputting a single image, and control the attitude angle error within 1° and the position error within 1m.

[0035] 2. The network used in this invention is simple, and while maintaining high precision, the runtime on a single RTX 4090D can be controlled within 10ms;

[0036] 3. This invention adopts a dynamic three-dimensional model input strategy, which increases the amount of information in the input information;

[0037] 4. This invention adopts a pose decoupling strategy to separate the aircraft's attitude angle estimation from its position estimation, thereby reducing interference between the two. Attached Figure Description

[0038] Figure 1 This is a flowchart of the present invention;

[0039] Figure 2 This is a schematic diagram of the present invention;

[0040] Figure 3 This is a schematic diagram of a dynamic 3D model conversion method. Detailed Implementation

[0041] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the embodiments of the present invention, and not all structures.

[0042] In the following description, specific details such as target system architecture and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0043] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0044] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0045] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0046] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include the target features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0047] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to understand the invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various modifications are readily apparent as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0048] The aircraft pose estimation method based on pose decoupling and a dynamic 3D model described in this embodiment uses a pose decoupling strategy to divide aircraft pose estimation into two relatively independent subtasks, reducing interference between the two subtasks. It also fully leverages the advantages of convolutional neural networks, Transformer networks, and multilayer perceptrons to construct an attitude angle estimation network, RotNet, to accurately estimate the aircraft's attitude angles. Then, a dynamic 3D model strategy is used to rotate the aircraft's 3D model in the reference coordinate system using the attitude angles estimated by RotNet, making it the input to the position estimation network, TransNet, for estimating the aircraft's position. Combined with... Figures 1 to 2 As shown, specifically, the implementation process of the aircraft pose estimation method includes the following steps:

[0049] Step 1: Obtain aircraft images and corresponding pose information from different perspectives and at different times during the flight, and construct a dataset.

[0050] To obtain more comprehensive images of the aircraft during flight, this example takes pictures of the aircraft at different times and from different perspectives to obtain images containing the aircraft. The images are then cropped to retain the area where the aircraft is located.

[0051] Step 2: Extract features at different levels from the aircraft image using the feature extraction network.

[0052] Furthermore, the steps for extracting features at different levels from aircraft images using a feature extraction network include:

[0053] The aircraft image is input into the ResNet network to obtain features f2, f3, f4 and f5 output from different layers of the ResNet network.

[0054] Feature f2 is the side output feature of the Conv2_x layer in the ResNet network, feature f3 is the side output feature of the Conv3_x layer, feature f4 is the side output feature of the Conv4_x layer, and feature f5 is the side output feature of the Conv5_x layer.

[0055] Step 3: Input the features from different levels into the attitude angle estimation network to estimate the aircraft's rotation matrix.

[0056] Furthermore, the steps for estimating the aircraft's rotation matrix by inputting features from different levels into the attitude angle estimation network include:

[0057] Features f2, f3, f4, and f5 are input into the Feature Pyramid Network (FPN) for fusion processing to obtain the edge feature Fside, which has a tensor dimension of 256×64×64.

[0058] The flattening operation is used to convert features f3, f4, and f5 from three-dimensional features into two-dimensional feature representations, and then stack them along the first dimension of the channel to fuse feature information from different levels and obtain the backbone network features.

[0059] The backbone network features are input into the feature optimization Transformer module, which contains encoders and decoders, to mine the long-range dependencies of aircraft features from the fused features, thereby enhancing the global modeling capability of feature representation and obtaining global features.

[0060] To further extract and process features, the global features are input into a multilayer perceptron (MLP) to obtain the first feature;

[0061] The tensor dimension of the first feature is transformed using the Einstein Summation Convention to make the first feature and the edge feature Fside consistent in dimension, changing the tensor dimension from 512×256 to 512×64×64; then the edge feature and the first feature are stacked to obtain the pose fusion feature.

[0062] The attitude fusion features are input into the RotHead attitude angle regression network to predict the aircraft's rotation matrix. R .

[0063] The pose angle regression network RotHead consists of two convolutional layers and three fully connected layers arranged sequentially.

[0064] Step 4: Calculate the rotation loss value, iteratively optimize the attitude angle estimation network based on the rotation loss value, and use the optimized attitude angle estimation network to obtain the target rotation matrix of the aircraft.

[0065] The pose angle estimation network RotNet consists of a feature optimization Transformer module, an MLP, and a pose angle regression network RotHead.

[0066] Furthermore, the formula for calculating the rotational loss value is:

[0067] ,

[0068] In the formula, Let R be the rotation matrix transformed from the aircraft's actual attitude angles, and let R be the rotation matrix of the aircraft predicted by the attitude regression network RotHead. Let N represent the coordinates of key points on the aircraft fuselage in the fuselage coordinate system, where N represents the number of points. This represents the L1 loss function.

[0069] Step 5: Rotate the 3D model of the aircraft using the target rotation matrix, and use the position estimation network to predict the position of the rotated 3D model of the aircraft.

[0070] like Figure 3 As shown, the aircraft is rotated in the reference coordinate system according to the estimated aircraft rotation matrix R. The 3D model of the aircraft in the image is designed to ensure that its projection onto the image has the same geometric shape as the aircraft in the input image, where the body coordinate system is... .

[0071] Furthermore, the steps for predicting the position of the rotated 3D aircraft model using a position estimation network include:

[0072] The 3D model of the aircraft is made of point clouds. The representation is performed, where the point cloud is generated by computer-aided design software (such as CAD). Let represent the number of points in the point cloud. Then, the rotated 3D model of the aircraft is... The multilayer perceptron (MLP) consisting of three linear layers is used to perform feature processing on the rotated 3D model of the aircraft. Then, the Einstein summation convention is used to perform tensor dimension transformation on the processing result to obtain the second feature, so that the second feature is consistent with the edge feature Fside in dimension.

[0073] The second feature is stacked with the edge feature Fside to obtain the geometric-image feature;

[0074] The PoseNet image pose network is used to extract features from feature f5, and its output is stacked with geometric-image features to obtain position fusion features. The PoseNet image pose network includes a transposed convolutional layer, a first convolutional layer, a second convolutional layer, a first upsampling layer, a third convolutional layer, a fourth convolutional layer, a second upsampling layer, a fifth convolutional layer, and a sixth convolutional layer arranged in sequence.

[0075] The position fusion features are input into the position regression network TransHead for processing to obtain the translation matrix representing the aircraft's position. Where K is the camera's intrinsic parameter, It is an estimated depth value. It is the estimated projection position of the center point of the aircraft's 3D bounding box in the pixel coordinate system.

[0076] The TransHead location regression network consists of two convolutional layers and three fully connected layers arranged sequentially.

[0077] Step 6: Calculate the position loss value, iteratively optimize the position estimation network based on the loss value, and use the optimized position estimation network to predict the aircraft's position.

[0078] The TransNet location estimation network consists of an MLP, an image pose network PoseNet, and a location regression network TransHead. The MLP consists of three linear layers.

[0079] Furthermore, the formula for calculating the position loss value is:

[0080] ,

[0081] In the formula, It is the estimated projection position of the center point of the aircraft's 3D bounding box in the pixel coordinate system. It is the true projection position of the center point of the aircraft's 3D bounding box in the pixel coordinate system. It is an estimated depth value. For the actual depth value, This represents the L1 loss function.

[0082] To further demonstrate the accuracy of the aircraft pose estimation method based on pose decoupling and dynamic 3D model described in this invention for position and attitude estimation, the following example is provided.

[0083] The comparison methods used include YOLO6D, PVNet, WDR, 3DSke, and Alas. The evaluation metrics include ADD, Re, Te, 2DProj, r (°), t (meters), and Runtime (milliseconds), where ADD represents the 3D spatial accuracy, calculated to satisfy... The sample proportion, where L is the maximum fuselage length, and [R, T] is the predicted pose. The true pose metric reflects the model's ability to control aircraft keypoint errors within 2% of the fuselage length in 3D space. 2DProj (2D Projection Accuracy) is the 2D projection matching accuracy, calculating the proportion of samples where the average reprojection error between the predicted pose projected aircraft keypoints and the true pose projected keypoints is less than 2 pixels. It evaluates the model's visual alignment effect in image space, directly impacting the practicality of target detection and tracking. Re (Rotation Accuracy) is the attitude estimation accuracy, statistically analyzing the proportion of samples with aircraft attitude angle errors less than 2°, used to evaluate the model's accuracy in estimating the aircraft's 3D orientation. Te (Translation Accuracy) is the position estimation accuracy, statistically analyzing the proportion of samples with aircraft center position errors less than 2 meters, reflecting the model's accuracy in spatial positioning of the aircraft. r (°) represents the attitude angle error, t (meters) represents the position error, and Runtime refers to the processing time per image. The comparison results are shown in Table 1. It can be concluded that the accuracy of the present invention in three-dimensional space ADD is 42.55% higher than that of AlaS, and the accuracy of two-dimensional projection, i.e. 2DProj, is also the highest. While maintaining the highest computing efficiency, i.e., the processing time of a single image is less than 10 milliseconds, the present invention can simultaneously control the attitude angle error r and the position error t to within 1° and 1 meter, respectively, thus demonstrating the dual advantages of the present invention in terms of measurement accuracy and real-time performance.

[0084] Table 1

[0085] .

Claims

1. An aircraft pose estimation method based on pose decoupling and a dynamic 3D model, characterized in that, Includes the following steps: Acquire aircraft images and corresponding pose information from different perspectives and at different times during flight, and construct a dataset. Features at different levels are extracted from aircraft images using a feature extraction network; Features from different levels are input into the attitude angle estimation network to estimate the aircraft's rotation matrix; Calculate the rotation loss value, iteratively optimize the attitude angle estimation network based on the rotation loss value, and use the optimized attitude angle estimation network to obtain the target rotation matrix of the aircraft. The target rotation matrix is ​​used to rotate the 3D model of the aircraft, and the position estimation network is used to predict the position of the rotated 3D model of the aircraft. Calculate the position loss value, iteratively optimize the position estimation network based on the loss value, and use the optimized position estimation network to predict the aircraft's position. The steps for extracting features at different levels from aircraft images using a feature extraction network include: The aircraft image is input into the ResNet network to obtain features f2, f3, f4 and f5 output from different layers of the ResNet network; The steps for estimating the aircraft's rotation matrix by inputting features from different levels into the attitude angle estimation network include: Features f2, f3, f4, and f5 are input into the Feature Pyramid Network (FPN) for fusion processing to obtain the edge feature Fside. The features f3, f4, and f5 are converted into two-dimensional feature representations by using a flattening operation, and then stacked along the first dimension of the channel to obtain the backbone network features. The features of the backbone network are input into the feature optimization Transformer module, which contains the encoder and decoder, to obtain global features; The global features are input into a multilayer perceptron to obtain the first feature; The first feature is subjected to tensor dimension transformation to make the first feature and the edge feature Fside consistent in dimension. Then, the edge feature and the first feature are stacked to obtain the pose fusion feature. The attitude fusion features are input into the RotHead attitude angle regression network to predict the aircraft's rotation matrix. R .

2. The aircraft pose estimation method based on pose decoupling and dynamic 3D model according to claim 1, characterized in that, The steps for predicting the position of a rotated 3D aircraft model using a position estimation network include: The 3D model of the aircraft is made of point clouds. To represent, where Let represent the number of points in the point cloud. Then, the rotated 3D model of the aircraft is... The multilayer perceptron (MLP) is used to process the features of the rotated 3D model of the aircraft. Then, the Einstein summation convention is used to perform tensor dimension transformation on the processing results to obtain the second feature, so that the second feature is consistent with the edge feature Fside in dimensionality. The second feature is stacked with the edge feature Fside to obtain the geometric-image feature; Feature extraction of feature f5 is performed using the PoseNet image pose network, and its output is stacked with geometric-image features to obtain position fusion features; The position fusion features are input into the position regression network TransHead for processing to obtain the translation matrix representing the aircraft's position. Where K is the camera's intrinsic parameter, It is an estimated depth value. It is the estimated projection position of the center point of the aircraft's 3D bounding box in the pixel coordinate system.

3. The aircraft pose estimation method based on pose decoupling and dynamic 3D model according to any one of claims 1 to 2, characterized in that, The formula for calculating the rotational loss value is: ， In the formula, Let R be the rotation matrix transformed from the aircraft's actual attitude angles, and let R be the rotation matrix of the aircraft predicted by the attitude angle regression network RotHead. Let N represent the coordinates of key points on the aircraft fuselage in the fuselage coordinate system, where N represents the number of points. This represents the L1 loss function.

4. The aircraft pose estimation method based on pose decoupling and dynamic 3D model according to any one of claims 1 to 2, characterized in that, The formula for calculating the position loss value is: ， In the formula, It is the estimated projection position of the center point of the aircraft's 3D bounding box in the pixel coordinate system. It is the true projection position of the center point of the aircraft's 3D bounding box in the pixel coordinate system. It is an estimated depth value. For the actual depth value, This represents the L1 loss function.

Images