Optical flow based attitude estimation for automated aerial refueling

Abstract

This application relates to optical flow based pose estimation for automated in-flight refueling. Pose estimation for in-flight refueling. The method includes receiving input comprising a first video frame of an object. The first frame includes a first set of 2D keypoints corresponding to 3D keypoints on a model of the object. Input comprising a second video frame of the object is received. The second frame is sequentially in time after the first frame. The first and second frames are fed into an optical flow algorithm that estimates a second set of 2D keypoints of the second frame based on the set of 2D keypoints in the first frame. The second set of 2D keypoints and camera parameters are fed into a perspective-n-point algorithm that estimates a six degree of freedom pose of the object in the second frame. The position of an in-flight refueling boom is adjusted in real time based on the estimated 6DOF pose of the object.

Description

Technical Field

[0001] This disclosure generally relates to tanker aircraft, and more specifically to attitude estimation for controlling aircraft refueling operations. Background Technology

[0002] Using air-to-air refueling, the position and orientation (6DoF) of the receiving aircraft following the tanker are estimated. This position and orientation are referred to as attitude. The tanker is equipped with a refueling boom having an end cap that can be manipulated to insert into the receiving port of the receiving aircraft to initiate the refueling operation.

[0003] For example, cameras on a refueling aircraft can generate images of the refueling boom and the receiving aircraft. Machine learning models can perform real-time 6DoF pose estimation as part of the processing that controls the refueling operation using the refueling boom. Summary of the Invention

[0004] An illustrative embodiment provides a computer-implemented method for attitude estimation during aerial refueling. The method includes: receiving input of a first video frame containing an object, wherein the first video frame includes a first set of two-dimensional (2D) keypoints corresponding to pre-selected three-dimensional (3D) keypoints on a 3D model of the object; receiving input of a second video frame containing the object, wherein the second video frame is temporally sequential after the first video frame; feeding the first and second video frames into an optical flow (OF) algorithm, which estimates a second set of 2D keypoints in the second video frame based on the first set of 2D keypoints in the first video frame; feeding the second set of 2D keypoints and a plurality of camera parameters into a perspective n-point (PnP) algorithm, which estimates the six degrees of freedom (6DOF) attitude of the object in the second video frame; and adjusting the position of the aerial refueling boom in real time based on the estimated 6DOF attitude of the object.

[0005] Another illustrative embodiment provides a system for attitude estimation during in-flight refueling. The system includes a storage device storing program instructions and one or more processors operatively connected to the storage device and configured to execute the program instructions to cause the system to: receive input of a first video frame containing an object, wherein the first video frame includes a first set of two-dimensional (2D) keypoints corresponding to pre-selected three-dimensional (3D) keypoints on a 3D model of the object; receive input of a second video frame containing the object, wherein the second video frame is temporally sequential after the first video frame; feed the first and second video frames into an optical flow (OF) algorithm; estimate a second set of 2D keypoints in the second video frame using the OF algorithm based on the first set of 2D keypoints in the first video frame; feed the second set of 2D keypoints and a plurality of camera parameters into a perspective n-point (PnP) algorithm; estimate the six degrees of freedom (6DOF) attitude of the object in the second video frame using the PnP algorithm; and adjust the position of the in-flight refueling boom in real time based on the estimated 6DOF attitude of the object.

[0006] Another illustrative embodiment provides a computer program product for attitude estimation in aerial refueling. The computer program product includes a computer-readable storage medium having program instructions contained thereon for performing the following operations: receiving input of a first video frame containing an object, wherein the first video frame includes a first set of two-dimensional (2D) keypoints corresponding to pre-selected three-dimensional (3D) keypoints on a 3D model of the object; receiving input of a second video frame containing the object, wherein the second video frame is temporally sequential after the first video frame; feeding the first video frame and the second video frame to an optical flow (OF) algorithm; estimating a second set of 2D keypoints in the second video frame using the OF algorithm based on the first set of 2D keypoints in the first video frame; feeding the second set of 2D keypoints and a plurality of camera parameters to a perspective n-point (PnP) algorithm; estimating the six degrees of freedom (6DOF) attitude of the object in the second video frame using the PnP algorithm; and adjusting the position of the aerial refueling boom in real time based on the estimated 6DOF attitude of the object.

[0007] These features and functions may be implemented independently in different embodiments of this disclosure, or may be combined in other embodiments, in which further details can be seen with reference to the following description and accompanying drawings. Attached Figure Description

[0008] The appended claims set forth novel features that are considered characteristic of the illustrative embodiments. However, the illustrative embodiments, their preferred modes of use, further objects and features, will be best understood by referring to the following detailed description of the illustrative embodiments of this disclosure, when read in conjunction with the accompanying drawings, wherein: Figure 1 This is an illustration of attitude estimation of the receiver aircraft according to an illustrative embodiment; Figure 2 A flowchart illustrating the conceptual algorithm for 6DOF attitude estimation is shown; Figure 3 A block diagram illustrating Kalman filter tracking of 6DOF attitude is shown; Figure 4 A block diagram illustrating optical flow (OF) 6DOF attitude estimation according to an illustrative embodiment is shown; Figure 5 A block diagram illustrating the post-fusion of optical flow and machine learning for detecting 2D keypoints according to an illustrative embodiment is depicted. Figure 6 A block diagram illustrating an early fusion of optical flow and machine learning for detecting 2D keypoints according to an illustrative embodiment is depicted; Figure 7 Depicts object detection, keypoint detection, and pose estimation within a video frame according to illustrative embodiments; Figure 8 A visualization of 2D key points on the window of the receiver aircraft, which is partially obscured by the refueling boom; Figure 9 A flowchart describing the comparison of keypoint detection using heatmaps and without heatmaps is provided. Figure 10 A block diagram illustrating heatmap-guided keypoint detection training according to an illustrative embodiment is depicted; Figure 11 A flowchart illustrating a method for attitude estimation for aerial refueling according to an illustrative embodiment is depicted; Figure 12 A flowchart illustrating a method for the post-integration of optical flow and machine learning attitude estimation for aerial refueling, according to an illustrative embodiment, is depicted. Figure 13 A flowchart illustrating an early integration of optical flow and machine learning attitude estimation for aerial refueling, according to an illustrative embodiment, is depicted. Figure 14 A flowchart illustrating a method for detecting 2D key points in a video frame according to an illustrative embodiment is depicted; Figure 15 A flowchart illustrating a method for training a neural network to detect 2D keypoints in video image frames according to an illustrative embodiment is depicted; and Figure 16 This is an illustration of a block diagram of a data processing system according to an illustrative embodiment. Detailed Implementation

[0009] The illustrative embodiments recognize and consider that previous attitude estimation methods employ machine learning to perform fast, real-time 6DOF attitude estimation for receiver aircraft used in live air-to-air refueling. These methods rely on deep neural networks to detect two-dimensional (2D) keypoints on objects in each video frame, and then use a tracker to smooth the frame-by-frame 6DOF attitude estimation. However, these previous methods may fail in 2D keypoint detection for a given frame in the video. Although heuristics exist in previous methods to prevent such spurious errors from affecting the tracked 6DOF attitude results, a wider range of systematic errors may exist because neural network-based keypoint detectors cannot perform under certain conditions due to insufficient training.

[0010] The illustrative embodiments also recognize and consider that, for the case of tracking the attitude of a receiver aircraft used for air-to-air refueling, there can be multiple flight dynamics to account for the various receiver aircraft and the dynamics associated with disturbances such as turbulence and unique pilot preferences. Without fine-tuning the tracker parameters, these dynamic characteristics cannot typically be tracked well with a Kalman filter.

[0011] An exemplary embodiment provides an optical flow (OF)-based method for 6DOF pose estimation (position and rotation in 3D space) of a rigid object captured in video. The system input includes previous video frames, 2D keypoints (i.e., landmarks or feature points) detected on the object in the previous video frames, and the current video frame. The output includes the 2D keypoints for the current video frame. This process is repeated frame-by-frame on the input video in real time. The 2D keypoint information can be integrated as input to find the 6DOF pose of the object through optimization using a PnP (Perspective N-Point) algorithm, given corresponding 3D points on the object, and camera intrinsic and extrinsic parameters. The optical flow method of the exemplary embodiment is adaptable to new situations not included in the training dataset used for deep learning models.

[0012] OF-based pose estimation can be integrated with deep learning 6DOF pose estimation as an additional method for estimating 6DOF pose, thereby increasing redundancy and robustness. The 2D keypoint output and / or 6DOF pose estimation of the OF method can be fused with the 2D keypoint detection or 6DOF pose estimation of the deep learning method. Furthermore, the OF 2D keypoint output and 6DOF pose estimation can be used as a heuristic check of the corresponding signals in the pipeline of the deep learning method.

[0013] Exemplary embodiments also provide a fast heatmap-free method for keypoint detection, which is initially trained with heatmaps to improve accuracy without reducing runtime speed.

[0014] Figure 1 This is an illustration of attitude estimation of a receiver aircraft according to an illustrative embodiment. In this illustrative example, an air-to-air refueling operation is shown between a tanker aircraft 100 and a receiver aircraft 101. As depicted in this example, the receiver aircraft 101 follows behind the tanker aircraft 100. The tanker aircraft 100 is equipped with a boom 102 that connects to the refueling port 103 of the receiver aircraft 101 for the refueling operation.

[0015] In this illustrative example, images from a camera and video data generated by camera system 104 are used to guide boom 102 to refueling port 103 and receiver 101. The video data contains metadata such as timestamps and GPS information. This information forms video 105.

[0016] During normal operation of the tanker aircraft 100, the image processing algorithms and models in computer 110 can use the video to determine the attitude estimate of the receiver aircraft 101. In these illustrative examples, this attitude estimate is a 6-DOF attitude estimate of the receiver aircraft 101. 6DOF includes x, y, z, yaw, pitch, and roll.

[0017] This attitude estimation can be used by the automatic controller in computer 110 to guide boom 102 into oil receiver 103 of receiver 101.

[0018] Figure 2 A flowchart illustrating the conceptual algorithm for 6DOF pose estimation is provided. For each input frame 202, processing 200 performs object detection 204, which estimates the local region where the object is located in the image. Figure 7 Box 702 in the image is an example of object detection within a video frame.

[0019] Based on the local region found by object detection 204, processing 200 performs keypoint detection 206, which detects 2D keypoints on the object that correspond to pre-selected keypoints on the object's 3D model. Figure 7 Point 704 in the image is an example of a 2D keypoint detected in a video frame.

[0020] After 2D keypoints are detected in the video frame, the 2D to 3D transformation is calculated using the PnP algorithm. This transformation allows the 6DOF pose estimation of PnP to be solved using the detected 2D keypoints, pre-selected 3D keypoints on the 2D model, and camera intrinsic and extrinsic parameters. Figure 7The contour lines in frame 706 show an example pose estimation from a video frame.

[0021] The estimated pose is then tracked using a Kalman filter (KF) or similar tracker 210, which employs a heuristic approach to when to start and stop tracking to temporally track the 6DOF pose estimate over video frames 212 (see [link to relevant documentation]). Figure 3 ).

[0022] Figure 3 A block diagram illustrating Kalman filter tracking of 6DOF attitude is provided. Architecture 300 is... Figure 2 A detailed example of pose tracking 210 is provided. A 6DOF estimate for the current video frame (frame N) is predicted using the 6DOF pose solution from the previous video frame (frame N-1), which is a KF prediction. The KF prediction is also updated by the current frame N 6DOF pose solution from the PnP algorithm. This KF update provides a 6DOF pose response informed by the observations of the current frame.

[0023] Figure 4 A block diagram illustrating optical flow (OF) 6DOF pose estimation according to an illustrative embodiment is provided. The OF architecture 400 acquires several pieces of information. First, a video of an object moving in a temporally consistent manner is provided, such as... Figure 1 The video shows a receiver aircraft, such as receiver 101. The next given information is 3D points in model space from a 3D model of the object (receiver aircraft), used to solve for the 6DOF pose. These 3D points in model space should correspond to 2D keypoints detected in the video frames, such as... Figure 7 The 2D keypoint 704 is given. Another set of parameters are the camera's intrinsic parameters, such as focal length, aspect ratio, principal point, image resolution, distortion parameters, etc. The last set of parameters are the camera's extrinsic parameters, which include the camera's 6DOF pose in world space.

[0024] Using the information given above, find 2D keypoints on the object in the current video frame. These 2D keypoints correspond to pre-selected 3D keypoints on the object's 3D model. Then, store the video frame and its 2D keypoints, and when OF Architecture 400 advances to the next frame, the video frame becomes frame N-1, and the next frame becomes the new frame N.

[0025] The new frame N (frame N-1) and the 2D keypoints of frame N-1 are fed into the OF algorithm. The OF algorithm outputs the 2D keypoints of frame N. These 2D keypoints of frame N are then fed into the PnP algorithm, which solves for the 6DOF pose. The PnP algorithm can also be used with the RANSAC (Random Sample Consensus) algorithm.

[0026] If the object under discussion is partially obscured, for example, the receiver is obscured by the refueling boom (see...). Figure 8 If the OF architecture 400 can use the known position of the boom to ignore key points obscured by it and remove them from use in the PnP step, then the OF architecture 400 can use the known position of the boom to ignore key points obscured by it and remove them from use in the PnP step. Figure 8 In the example shown, some key points at the middle window corner are not visible because they are obscured by the boom and therefore not used by PnP.

[0027] Can Figure 4 The OF method described in the paper is integrated into a broader deep learning-based 6DOF pose estimation pipeline (e.g., Figure 2 and Figure 3 The pipeline shown in the image is used to assist frame-by-frame machine learning (ML) keypoint detection. In cases where the ML keypoint detector is not working, spurious video frames may exist, resulting in incorrect 2D keypoint estimates that negatively impact 6DOF pose estimation. OF using additional information from previous frames serves as a backup method for estimating 6DOF to remedy this potential problem.

[0028] Figure 5 A block diagram illustrating a late-stage fusion of optical flow and machine learning for detecting 2D keypoints according to an illustrative embodiment is depicted. In this late-stage fusion architecture 500, the OF method is used to check whether the 6DOF Pnp solution from the current (N) frame of the deep learning method is within the expected bound. If it is not within the bound, OF-based 6DOF pose estimation is used as an alternative.

[0029] The OF algorithm and the ML keypoint model operate in parallel. Frame N is fed into both the OF and ML models. Frame N-1 and its detected 2D keypoints are also fed into the OF. The individual predicted 2D keypoints from each processed frame N are fed into PnP to produce OF 6DOF pose estimates and ML 6DOF pose estimates, respectively. The OF 6DOF and ML 6DOF estimates are then compared. If the difference between the two 6DOF estimates exceeds a specified threshold, the OF-generated 6DOF pose estimate is selected. If the difference does not exceed the threshold, the ML-generated 6DOF pose estimate is selected.

[0030] Regardless of which 6DOF attitude estimate is chosen, it is used to update the Kalman filter's 6DOF predictions, such as Figure 3 The architecture is 300.

[0031] Figure 6 A block diagram illustrating an early fusion of optical flow and machine learning for detecting 2D keypoints according to an illustrative embodiment is depicted. In this early fusion architecture 600, keypoint estimates generated by OF and ML are compared before applying the PnP algorithm.

[0032] If the difference between the OF and ML estimated keypoints exceeds a specified threshold, the OF-estimated keypoints are fed into PnP. If the difference does not exceed the threshold, the ML-estimated keypoints are selected for PnP. The PnP algorithm then generates a 6DOF pose estimate for frame N, which is used to update the KF prediction.

[0033] Figure 9 A block diagram comparing heatmap-based and heatmap-less keypoint detection is described. The first element for a typical heatmap-based keypoint detector 900 is the backbone 902, which can be an encoder similar to ResNet (Residual Network). The backbone 902 transforms the image into a latent space.

[0034] An upsampling process of 904 is performed to ensure that the latent space has the same shape as the original input image. These outputs, with the same x and y dimensions as the input image, constitute the heatmap. For example, if the input image is 512x512 resolution, the heatmap is Nx512x512, where N is the number of keypoints. For each keypoint, there is a heatmap. The "hot" parts of the heatmap are likely where the network perceives the keypoint to be located.

[0035] The heatmap-to-coordinate module 906 can be used to obtain the actual coordinates of each keypoint using various methods (e.g., using the coordinates with the maximum heat, using a weighted average, etc.). The loss function 908 of the heatmap-based keypoint detector 900 is then compared with the baseline ground truth to train the backbone 902.

[0036] A drawback of the heatmap-based keypoint detector 900 architecture is that upsampling 904 is typically slow. Upsampling 904 requires operating on larger inputs (e.g., 512x512), which contrasts with the backbone 902, which operates in compressed form (e.g., 128x128), requiring significantly more FLOPS (floating-point operations per second).

[0037] In contrast to heatmap-based methods, the RLE (Residual Log-Likelihood Estimation) architecture 910 uses a direct regression module 914 of keypoints after encoding by the backbone 912, without upsampling (which does not involve heatmaps). Typically, direct regression leads to inaccurate estimates. Therefore, RLE adds a normalization flow module 916, which aids in training. The loss function 918 from the normalization flow module 916 is compared to the ground truth.

[0038] However, at runtime, the only two blocks used are the backbone 912 and the direct regression module 914, which makes RLE faster but less accurate due to the absence of a normalized flow.

[0039] Figure 10 A block diagram illustrating heatmap-guided keypoint detection training according to an illustrative embodiment is depicted. Figure 10 The heatmap-guided keypoint training architecture 1000 combines heatmap-based keypoint detection and regression (RLE) keypoint detection during training, but only uses the regression path at runtime.

[0040] This architecture maintains the same runtime speed for RLE, but trains both heatmap and heatless subnets simultaneously. The advantage of this method is that heatmap training is incorporated into the loss function 1002 during training. Studies show that the heatmap method performs better than the heatless method, achieving higher accuracy. In the keypoint training architecture 1000, the backbone 902 is shared by both methods, allowing heatmap-based detections to teach it the optimal embedding and optimal latent space. The direct regression module 914 can then use this optimal embedding and optimal latent space during runtime to obtain more accurate results.

[0041] Loss function 1002 also includes a loss term for direct regression. In this case, the RLE loss (with some normalization / weighting) is used directly, and the loss term is used for the heatmap-based method, which includes heatmap loss plus keypoint loss (because the heatmap to coordinate module 906 is a DSNT (differentiable spatial to numerical transformation) layer).

[0042] Optionally, a consistency loss can also be included in loss function 1002 to ensure that the coordinated outputs of the two branches are identical. (i.e., addL_cons = ||(Coords – coords_heatmap)) vis_kps||. That is, for all visible keypoints, obtain the average keypoint error between the coordinates of two predicted keypoints.

[0043] It should be noted that this training method is not unique to RLE. Any future direct regression methods of any type will use the current training method (without any runtime speed impact) to achieve accuracy gains.

[0044] Figure 11 A flowchart illustrating a method for attitude estimation for aerial refueling according to an illustrative embodiment is provided. Process 1100 can... Figure 4 The OF architecture 400 shown is implemented in this architecture.

[0045] Processing 1100 begins by receiving input of a first video frame containing an object, wherein the first video frame includes a first set of two-dimensional (2D) key points that correspond to pre-selected three-dimensional (3D) key points on a 3D model of the object (operation 1102).

[0046] Process 1100 then receives input of a second video frame containing the object, wherein the second video frame is sequentially placed after the first video frame in time (operation 1104).

[0047] The first video frame (including the first group of 2D key points) and the second video frame are fed into the optical flow (OF) algorithm (operation 1106).

[0048] The second set of 2D key points in the second video frame is estimated using the OF algorithm based on the first set of 2D key points in the first video frame (Operation 1108).

[0049] The second set of 2D keypoints and multiple camera parameters are fed into the Perspective n-Point (PnP) algorithm (Operation 1110). The PnP algorithm estimates the six-degree-of-freedom (6DOF) pose of the object in the second video frame (Operation 1112).

[0050] Based on the estimated 6DOF attitude of the object, the position of the aerial refueling boom is adjusted in real time (operation 1114). Then, processing 1100 ends.

[0051] Figure 12 A flowchart illustrating a method for late-stage integration of optical flow and machine learning attitude estimation for aerial refueling, according to an illustrative embodiment, is depicted. Process 1200 can be implemented in the late-stage fusion architecture 500, and is... Figure 11 The extension of processing 1100 in the process.

[0052] Process 1200 begins by feeding the first and second video frames into the machine learning model (operation 1202). The machine learning model estimates the third set of 2D keypoints in the second video frame based on the first set of 2D keypoints in the first video frame (operation 1204).

[0053] The third set of 2D keypoints and camera parameters are fed into the second PnP algorithm (Operation 1206). The second PnP algorithm estimates the second 6DOF pose of the object in the second video frame (Operation 1208).

[0054] Process 1200 determines the difference between the 6DOF pose estimated from the second set of 2D keypoints and the second 6DOF pose estimated from the third set of 2D keypoints (operation 1210), and determines whether the difference exceeds a specified threshold (operation 1212).

[0055] In response to determining that the difference exceeds a specified threshold, in operation 1114, the 6DOF attitude estimated from the second set of 2D key points is selected to adjust the position of the aerial refueling boom (operation 1214).

[0056] In operation 1114, in response to determining that the difference does not exceed a specified threshold, the second 6DOF attitude estimated from the third set of 2D key points is selected to adjust the position of the aerial refueling boom (operation 1216).

[0057] The Kalman filter 6DOF predictions are then updated based on the selected 6DOF attitude estimate (operation 1218). Then, processing 1200 concludes the process.

[0058] Figure 13 A flowchart illustrating an early integration method for optical flow and machine learning attitude estimation for aerial refueling, according to an illustrative embodiment, is depicted. Process 1300 can be implemented in a later-early fusion architecture 600, and is... Figure 11 The extension of processing 1100 in the process.

[0059] Process 1300 begins by feeding the first and second video frames into the machine learning model (operation 1302). The machine learning model estimates the third set of 2D keypoints in the second video frame based on the first set of 2D keypoints in the first video frame (operation 1304).

[0060] Process 1300 determines the difference between the second group of 2D key points and the third group of 2D key points (operation 1306), and determines whether the difference exceeds a specified threshold (operation 1308).

[0061] In response to the determination that the difference exceeds the specified threshold, select the second group of 2D key points (Operation 1310).

[0062] In response to the determination that the difference does not exceed the specified threshold, the third group of 2D key points is selected (Operation 1312).

[0063] The selected set of 2D key points is fed into the PnP algorithm (operation 1314), and the PnP algorithm estimates the 6DOF pose of the object in the second video to adjust the aerial refueling boom (operation 1316).

[0064] Then, the Kalman filter 6DOF prediction is updated based on the 6DOF attitude estimation (operation 1218). Then, processing ends at 1300.

[0065] Figure 14 A flowchart illustrating a method for detecting 2D key points in a video frame according to an illustrative embodiment is provided. An example of how process 1400 detects 2D key points in a first video frame during operation 1102 on process 1100 is shown, and it is possible to... Figure 10 The heatmap-guided keypoint training architecture is implemented in 1000.

[0066] Processing 1400 begins by feeding the input image from the first video frame into the neural network (operation 1402). The neural network then converts the first video frame into the latent space (operation 1404).

[0067] Process 1400 performs direct regression on the latent space to determine the coordinates of keypoints within the input image, where the coordinates have a standard deviation and a mean deviation (operation 1406). Then, process 1400 ends.

[0068] Process 1500 is a flowchart illustrating a method for training a neural network to detect 2D keypoints in video image frames according to an illustrative embodiment. Process 1500 can be implemented in... Figure 10 The heatmap-guided keypoint training architecture is in 1000.

[0069] Process 1500 begins by feeding the training images into the neural network (operation 1502). The neural network converts the training images into a latent space (operation 1504). The latent space is then upsampled to match the dimensions of the training images (operation 1506).

[0070] A heatmap is generated from the upsampled latent space (operation 1508), and the coordinates of key points in the input image are determined from the hot spots detected in the heatmap (operation 1510).

[0071] Process 1500 also performs direct regression on the latent space to determine the coordinates of keypoints within the training images, where the coordinates have a standard deviation and a mean deviation (operation 1512). Then, process 1500 normalizes the standard deviation and mean deviation of the keypoint coordinates (operation 1514).

[0072] Process 1500 determines the loss function by comparing the baseline true value with keypoint coordinates determined from both the heatmap and direct regression after normalizing the standard deviation and average deviation (operation 1516). Determining the loss function may also include a loss term that converges to the keypoint coordinates determined from the heatmap and direct regression. The neural network is then updated based on the loss function (operation 1518). Process 1500 then concludes.

[0073] Now go to Figure 16 A block diagram of a data processing system is depicted according to an illustrative embodiment. The data processing system 1600 can be used to implement... Figure 1 Computer 110 in the middle, and can be used to implement Figures 11 to 15The processing steps are 1100 to 1500. In this illustrative example, the data processing system 1600 includes a communication framework 1602 that provides communication between the processor unit 1604, memory 1606, permanent memory 1608, communication unit 1610, input / output (I / O) unit 1612, and display 1614. In this example, the communication framework 1602 takes the form of a bus system.

[0074] Processor unit 1604 is used to execute instructions for software that can be loaded into memory 1606. Processor unit 1604 may be multiple processors, multiprocessor cores, or some other type of processor, depending on the specific implementation. In an embodiment, processor unit 1604 includes one or more conventional general-purpose central processing units (CPUs). In an alternative embodiment, processor unit 1604 includes one or more graphics processing units (GPUs).

[0075] Memory 1606 and permanent memory 1608 are examples of storage device 1616. A storage device is any hardware capable of storing information on a temporary basis, on a permanent basis, or on both a temporary and permanent basis, such as, but not limited to, at least one of data, program code in a functional form, or other suitable information. In these illustrative examples, storage device 1616 may also be referred to as a computer-readable storage device. In these examples, memory 1606 may be, for example, random access memory or any other suitable volatile or non-volatile storage device. Permanent memory 1608 may take various forms depending on the specific implementation.

[0076] For example, permanent storage 1608 may include one or more components or devices. For example, permanent storage 1608 may be a hard disk drive, flash memory, rewritable optical disk, rewritable magnetic tape, or some combination thereof. The medium used in permanent storage 1608 may also be removable. For example, a removable hard disk may be used for permanent storage 1608. In these illustrative examples, communication unit 1610 provides communication with other data processing systems or devices. In these illustrative examples, communication unit 1610 is a network interface card.

[0077] The input / output unit 1612 allows data input and output to other devices that can be connected to the data processing system 1600. For example, the input / output unit 1612 can provide a connection for user input via at least one of a keyboard, mouse, or some other suitable input device. Furthermore, the input / output unit 1612 can send output to a printer. The display 1614 provides a mechanism for displaying information to the user.

[0078] Instructions for at least one of an operating system, application, or program may reside in storage device 1616, which communicates with processor unit 1604 via communication frame 1602. Processing in different embodiments may be executed by processor unit 1604 using computer-implemented instructions, which may reside in memory (e.g., memory 1606).

[0079] These instructions are referred to as program code that can be read and executed by the processor in processor unit 1604, computer-usable program code, or computer-readable program code. The program code in different embodiments may be contained on different physical or computer-readable storage media, such as memory 1606 or permanent memory 1608.

[0080] Program code 1618 is functionally located on a computer-readable medium 1620, which is selectively removable and can be loaded onto or transferred to a data processing system 1600 for execution by a processor unit 1604. Program code 1618 and computer-readable medium 1620 form a computer program product 1622 in these illustrative examples. In one example, computer-readable medium 1620 may be a computer-readable storage medium 1624 or a computer-readable signal medium 1626.

[0081] In these illustrative examples, computer-readable storage medium 1624 is a physical or tangible storage device for storing program code 1618, and not a medium for propagating or transmitting program code 1618. Computer-readable storage medium 1624 as used herein should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0082] Alternatively, program code 1618 may be transmitted to data processing system 1600 using computer-readable signal medium 1626. Computer-readable signal medium 1626 may be, for example, a propagated data signal containing program code 1618. For example, computer-readable signal medium 1626 may be at least one of electromagnetic signals, optical signals, or any other suitable type of signal. These signals may be transmitted over at least one communication link, such as a wireless communication link, fiber optic cable, coaxial cable, wire, or any other suitable type of communication link.

[0083] The different components described for data processing system 1600 do not imply an architectural limitation on the ways in which different embodiments can be implemented. Different illustrative embodiments can be implemented in a data processing system that includes components other than those shown in data processing system 1600 or components that replace those shown in data processing system 1600. Figure 16 The other components shown may differ from the illustrative example shown. Different embodiments can be implemented using any hardware device or system capable of running program code 1618.

[0084] As used here, the phrase "at least one of" when used with a list of items means that different combinations of one or more of the listed items can be used, and it may be necessary to have only one of each item in the list. In other words, "at least one of" refers to any combination of items and the number of items that can be used from the list, but not all items in the list are required. An item can be a specific object, thing, or category.

[0085] This application involves the following provisions: 1. A computer-implemented method for attitude estimation during aerial refueling, the method comprising the following steps: Receive input containing a first video frame of an object, wherein the first video frame includes a first set of two-dimensional key points, the first set of two-dimensional key points corresponding to pre-selected three-dimensional key points on a three-dimensional model of the object; Receive input containing a second video frame of the object, wherein the second video frame is sequentially following the first video frame in time; The first video frame and the second video frame are fed into the optical flow algorithm; The optical flow algorithm is used to estimate the second set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. The second set of two-dimensional key points and multiple camera parameters are fed into the perspective n-point algorithm; The six-DOF pose of the object in the second video frame is estimated using the perspective n-point algorithm; and The position of the aerial refueling boom is adjusted in real time based on the estimated six-DOF attitude of the object.

[0086] 2. The method according to Clause 1, further comprising the following steps: The first video frame and the second video frame are fed into the machine learning model; The machine learning model estimates the third set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. The third set of two-dimensional key points and the camera parameters are fed into the second perspective n-point algorithm; The second six-degree-of-freedom pose of the object in the second video frame is estimated using the second perspective n-point algorithm; Determine the difference between the six-DOF pose estimated from the second set of two-dimensional keypoints and the second six-DOF pose estimated from the third set of two-dimensional keypoints; In response to determining that the difference exceeds a specified threshold, the six-DOF attitude estimated from the second set of two-dimensional keypoints is selected to adjust the position of the aerial refueling boom; and In response to determining that the difference does not exceed the specified threshold, the second six-degree-of-freedom attitude estimated from the third set of two-dimensional key points is selected to adjust the position of the aerial refueling boom.

[0087] 3. The method according to Clause 2, further comprising the step of updating the Kalman filter six-degree-of-freedom prediction based on the selected six-degree-of-freedom attitude estimation.

[0088] 4. The method according to Clause 1, further comprising the following steps: The first video frame and the second video frame are fed into the machine learning model; The machine learning model estimates the third set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. Determine the differences between the second set of two-dimensional key points and the third set of two-dimensional key points; In response to determining that the difference exceeds a specified threshold, the second group of two-dimensional key points is selected; In response to determining that the difference does not exceed the specified threshold, the third set of two-dimensional key points is selected; The selected set of two-dimensional key points is fed into the perspective n-point algorithm; and The six-DOF attitude of the object in the second video frame is estimated using the perspective n-point algorithm to adjust the aerial refueling boom.

[0089] 5. The method according to Clause 4, further comprising the step of updating the Kalman filter six-degree-of-freedom prediction based on the estimated six-degree-of-freedom pose.

[0090] 6. The method according to Clause 1, wherein the two-dimensional keypoints in the first video frame are detected by the following steps: The input image from the first video frame is fed into the neural network; The first video frame is converted into a latent space using the neural network; and Direct regression is performed on the latent space to determine the coordinates of key points within the input image, wherein the key point coordinates have a standard deviation and a mean deviation.

[0091] 7. The method according to Clause 6, wherein the neural network is trained by the following steps: The training images are fed into the neural network; The training images are converted into a latent space using the neural network. The latent space is upsampled to match the dimensions of the training image. Heatmaps are generated from the oversampled latent space; The coordinates of the key points in the input image are determined from the hot spots detected in the heat map; Direct regression is performed on the latent space to determine the coordinates of key points within the training image, wherein the key point coordinates have a standard deviation and a mean deviation; Normalize the standard deviation and the average deviation of the key point coordinates; The loss function is determined by comparing the true baseline value with the keypoint coordinates determined from both the heatmap and the direct regression after normalizing the standard deviation and the mean deviation; and The neural network is updated according to the loss function.

[0092] 8. The method according to Clause 7, wherein the loss function further includes a loss term that converges from the keypoint coordinates determined by the heatmap and the direct regression.

[0093] 9. A system for attitude estimation during aerial refueling, the system comprising: Storage device, the storage device storing program instructions; One or more processors, operatively connected to the storage device and configured to execute the program instructions to cause the system to: Receive input containing a first video frame of an object, wherein the first video frame includes a first set of two-dimensional key points, the first set of two-dimensional key points corresponding to pre-selected three-dimensional key points on a three-dimensional model of the object; Receive input containing a second video frame of the object, wherein the second video frame is sequentially following the first video frame in time; The first video frame and the second video frame are fed into the optical flow algorithm; The optical flow algorithm is used to estimate the second set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. The second set of two-dimensional key points and multiple camera parameters are fed into the perspective n-point algorithm; The six-DOF pose of the object in the second video frame is estimated using the perspective n-point algorithm; and The position of the aerial refueling boom is adjusted in real time based on the estimated six-DOF attitude of the object.

[0094] 10. The system according to Clause 9, wherein the processor further executes instructions to cause the system to: The first video frame and the second video frame are fed into the machine learning model; The machine learning model estimates the third set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. The third set of two-dimensional key points and the camera parameters are fed into the second perspective n-point algorithm; The second six-degree-of-freedom pose of the object in the second video frame is estimated using the second perspective n-point algorithm; Determine the difference between the six-DOF pose estimated from the second set of two-dimensional keypoints and the second six-DOF pose estimated from the third set of two-dimensional keypoints; In response to determining that the difference exceeds a specified threshold, the six-DOF attitude estimated from the second set of two-dimensional keypoints is selected to adjust the position of the aerial refueling boom; and In response to determining that the difference does not exceed the specified threshold, the second six-degree-of-freedom attitude estimated from the third set of two-dimensional key points is selected to adjust the position of the aerial refueling boom.

[0095] 11. The system according to Clause 10, wherein the processor further executes instructions to cause the system to update the Kalman filter six-degree-of-freedom predictions based on the selected six-degree-of-freedom attitude estimation.

[0096] 12. The system according to Clause 9, wherein the processor further executes instructions to cause the system to: The first video frame and the second video frame are fed into the machine learning model; The machine learning model estimates the third set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. Determine the differences between the second set of two-dimensional key points and the third set of two-dimensional key points; In response to determining that the difference exceeds a specified threshold, the second group of two-dimensional key points is selected; In response to determining that the difference does not exceed the specified threshold, the third set of two-dimensional key points is selected; The selected set of two-dimensional key points is fed into the perspective n-point algorithm; and The six-DOF attitude of the object in the second video frame is estimated using the perspective n-point algorithm to adjust the aerial refueling boom.

[0097] 13. The system according to Clause 12, wherein the processor further executes instructions to cause the system to update the Kalman filter six-degree-of-freedom predictions based on the estimated six-degree-of-freedom pose.

[0098] 14. The system according to Clause 9, wherein the two-dimensional keypoints in the first video frame are detected by: The input image from the first video frame is fed into the neural network; The first video frame is converted into a latent space using the neural network; and Direct regression is performed on the latent space to determine the coordinates of key points within the input image, wherein the key point coordinates have a standard deviation and a mean deviation.

[0099] 15. The system according to Clause 14, wherein the neural network is trained by: The training images are fed into the neural network; The training images are converted into a latent space using the neural network. The latent space is upsampled to match the dimensions of the training image. Heatmaps are generated from the oversampled latent space; The coordinates of the key points in the input image are determined from the hot spots detected in the heat map; Direct regression is performed on the latent space to determine the coordinates of key points within the training image, wherein the key point coordinates have a standard deviation and a mean deviation; Normalize the standard deviation and the average deviation of the key point coordinates; The loss function is determined by comparing the true baseline value with the keypoint coordinates determined from both the heatmap and the direct regression after normalizing the standard deviation and the mean deviation; and The neural network is updated according to the loss function.

[0100] 16. The system according to Clause 15, wherein the loss function further includes a loss term that converges from the keypoint coordinates determined by the heatmap and the direct regression.

[0101] 17. A computer program product for attitude estimation in aerial refueling, the computer program product comprising: A computer-readable storage medium having program instructions contained thereon to perform the following operations: Receive input containing a first video frame of an object, wherein the first video frame includes a first set of two-dimensional key points, the first set of two-dimensional key points corresponding to pre-selected three-dimensional key points on a three-dimensional model of the object; Receive input containing a second video frame of the object, wherein the second video frame is sequentially following the first video frame in time; The first video frame and the second video frame are fed into the optical flow algorithm; The optical flow algorithm is used to estimate the second set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. The second set of two-dimensional key points and multiple camera parameters are fed into the perspective n-point algorithm; The six-DOF pose of the object in the second video frame is estimated using the perspective n-point algorithm; and The position of the aerial refueling boom is adjusted in real time based on the estimated six-DOF attitude of the object.

[0102] 18. The computer program product according to Clause 17, wherein the computer program product further includes instructions for performing the following operations: The first video frame and the second video frame are fed into the machine learning model; The machine learning model estimates the third set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. The third set of two-dimensional key points and the camera parameters are fed into the second perspective n-point algorithm; The second six-degree-of-freedom pose of the object in the second video frame is estimated using the second perspective n-point algorithm; Determine the difference between the six-DOF pose estimated from the second set of two-dimensional keypoints and the second six-DOF pose estimated from the third set of two-dimensional keypoints; In response to determining that the difference exceeds a specified threshold, the six-degree-of-freedom attitude estimated from the second set of two-dimensional key points is selected to adjust the position of the aerial refueling boom; In response to determining that the difference does not exceed the specified threshold, the second six-degree-of-freedom attitude estimated from the third set of two-dimensional keypoints is selected to adjust the position of the aerial refueling boom; and The Kalman filter six-degree-of-freedom predictions are updated based on the selected six-degree-of-freedom attitude estimation.

[0103] 19. The computer program product according to Clause 17, wherein the computer program product further includes instructions for performing the following operations: The first video frame and the second video frame are fed into the machine learning model; The machine learning model estimates the third set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame; Determine the differences between the second set of two-dimensional key points and the third set of two-dimensional key points; In response to determining that the difference exceeds a specified threshold, the second group of two-dimensional key points is selected; In response to determining that the difference does not exceed the specified threshold, the third set of two-dimensional key points is selected; The selected set of two-dimensional key points is fed into the perspective n-point algorithm; The six-DOF pose of the object in the second video is estimated using the perspective n-point algorithm to adjust the aerial refueling boom; and The Kalman filter six-degree-of-freedom predictions are updated based on the estimated six-degree-of-freedom attitude.

[0104] 20. The computer program product according to Clause 17, wherein the two-dimensional keypoints in the first video frame are detected by: The input image from the first video frame is fed into a neural network, which is trained by the following operations: The training images are fed into the neural network; The training images are converted into a latent space using the neural network. The latent space is upsampled to match the dimensions of the training image. Heatmaps are generated from the oversampled latent space; The coordinates of the key points in the input image are determined from the hot spots detected in the heat map; Direct regression is performed on the latent space to determine the coordinates of key points within the training image, wherein the key point coordinates have a standard deviation and a mean deviation; Normalize the standard deviation and the average deviation of the key point coordinates; A loss function is determined by comparing the baseline true value with the keypoint coordinates determined from both the heatmap and the direct regression after normalizing the standard deviation and the mean deviation. The loss function includes a loss term that converges from the keypoint coordinates determined from the heatmap and the direct regression. The neural network is updated according to the loss function. The first video frame is converted into a latent space using the neural network; and Direct regression is performed on the latent space to determine the coordinates of key points within the input image, wherein the key point coordinates have a standard deviation and a mean deviation.

[0105] For example, but not limited to, "at least one of Item A, Item B, or Item C" could include Item A, Item A and Item B, or Item B. The example could also include Item A, Item B, and Item C, or Item B and Item C. Of course, any combination of these items can exist. In some illustrative examples, "at least one of" could be, for example, but not limited to, two of Item A; one of Item B; and the tenth of Item C; four of Item B and seven of Item C; or other suitable combinations.

[0106] As used herein, when referring to projects, "multiple" means one or more projects. For example, "multiple different types of networks" means one or more different types of networks. In the illustrative examples, "group" used with reference projects means one or more projects. For example, a set of measures is one or more measures.

[0107] Various illustrative embodiments have been described for purposes of illustration and description, and these descriptions are not intended to be exhaustive or limited to the embodiments of the disclosed forms. The various illustrative examples describe components that perform actions or operations. In the illustrative embodiments, a component may be configured to perform the described actions or operations. For example, the component may be configured or designed to provide the component with the ability to perform actions or operations, which are described in the illustrative examples as being performed by the component. Furthermore, with regard to the terms “comprising,” “including,” “having,” “containing,” and variations thereof as used herein, these terms are intended to be inclusive in a manner similar to the term “comprising” as an open-ended transitional term, without excluding any additional or other elements.

[0108] Many modifications and variations will be apparent to those skilled in the art. Furthermore, different illustrative embodiments may provide different features compared to other desired embodiments. One or more embodiments were selected and described in order to best explain the principles of the embodiments, their practical application, and to enable others skilled in the art to understand the disclosure of various embodiments with various modifications suitable for the intended particular purpose.

Claims

1. A computer-implemented method for attitude estimation during aerial refueling, the method comprising the following steps: Receive input containing a first video frame of an object, wherein the first video frame includes a first set of two-dimensional key points, the first set of two-dimensional key points corresponding to pre-selected three-dimensional key points on a three-dimensional model of the object; Receive input containing a second video frame of the object, wherein the second video frame is sequentially following the first video frame in time; The first video frame and the second video frame are fed into the optical flow algorithm; The optical flow algorithm is used to estimate the second set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. The second set of two-dimensional key points and multiple camera parameters are fed into the perspective n-point algorithm; The six-DOF pose of the object in the second video frame is estimated using the perspective n-point algorithm; and The position of the aerial refueling boom is adjusted in real time based on the estimated six-DOF attitude of the object.

2. The method according to claim 1, further comprising the following steps: The first video frame and the second video frame are fed into the machine learning model; The machine learning model estimates the third set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. The third set of two-dimensional key points and the camera parameters are fed into the second perspective n-point algorithm; The second six-degree-of-freedom pose of the object in the second video frame is estimated using the second perspective n-point algorithm; Determine the difference between the six-DOF pose estimated from the second set of two-dimensional keypoints and the second six-DOF pose estimated from the third set of two-dimensional keypoints; In response to determining that the difference exceeds a specified threshold, the six-degree-of-freedom attitude estimated from the second set of two-dimensional key points is selected to adjust the position of the aerial refueling boom; as well as In response to determining that the difference does not exceed the specified threshold, the second six-degree-of-freedom attitude estimated from the third set of two-dimensional key points is selected to adjust the position of the aerial refueling boom.

3. The method according to claim 2, further comprising the following steps: The Kalman filter six-degree-of-freedom predictions are updated based on the selected six-degree-of-freedom attitude estimation.

4. The method according to claim 1, further comprising the following steps: The first video frame and the second video frame are fed into the machine learning model; The machine learning model estimates the third set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. Determine the differences between the second set of two-dimensional key points and the third set of two-dimensional key points; In response to determining that the difference exceeds a specified threshold, the second group of two-dimensional key points is selected; In response to determining that the difference does not exceed the specified threshold, the third set of two-dimensional key points is selected; The selected set of two-dimensional key points is fed into the perspective n-point algorithm; as well as The six-DOF attitude of the object in the second video frame is estimated using the perspective n-point algorithm to adjust the aerial refueling boom.

5. The method according to claim 4, further comprising the following step: The Kalman filter six-degree-of-freedom predictions are updated based on the estimated six-degree-of-freedom attitude.

6. The method according to claim 1, wherein, The two-dimensional key points in the first video frame are detected through the following steps: The input image from the first video frame is fed into the neural network; The first video frame is converted into a latent space using the neural network; and Direct regression is performed on the latent space to determine the coordinates of key points within the input image, wherein the key point coordinates have a standard deviation and a mean deviation.

7. The method according to claim 6, wherein, The neural network is trained through the following steps: The training images are fed into the neural network; The training images are converted into a latent space using the neural network. The latent space is upsampled to match the dimensions of the training image. Heatmaps are generated from the oversampled latent space; The coordinates of the key points in the input image are determined from the hot spots detected in the heat map; Direct regression is performed on the latent space to determine the coordinates of key points within the training image, wherein the key point coordinates have a standard deviation and a mean deviation; Normalize the standard deviation and the average deviation of the key point coordinates; The loss function is determined by comparing the true baseline value with the keypoint coordinates determined from both the heatmap and the direct regression after normalizing the standard deviation and the mean deviation; and The neural network is updated according to the loss function.

8. The method according to claim 7, wherein, The loss function also includes a loss term that converges from the keypoint coordinates determined by the heatmap and the direct regression.

9. A system for attitude estimation during aerial refueling, the system comprising: Storage device, the storage device storing program instructions; One or more processors, operatively connected to the storage device and configured to execute the program instructions to cause the system to: Receive input containing a first video frame of an object, wherein the first video frame includes a first set of two-dimensional key points, the first set of two-dimensional key points corresponding to pre-selected three-dimensional key points on a three-dimensional model of the object; Receive input containing a second video frame of the object, wherein the second video frame is sequentially following the first video frame in time; The first video frame and the second video frame are fed into the optical flow algorithm; The optical flow algorithm is used to estimate the second set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. The second set of two-dimensional key points and multiple camera parameters are fed into the perspective n-point algorithm; The six-DOF pose of the object in the second video frame is estimated using the perspective n-point algorithm; and The position of the aerial refueling boom is adjusted in real time based on the estimated six-DOF attitude of the object.

10. A computer program product for attitude estimation in aerial refueling, the computer program product comprising: A computer-readable storage medium having program instructions contained thereon to perform the following operations: Receive input containing a first video frame of an object, wherein the first video frame includes a first set of two-dimensional key points, the first set of two-dimensional key points corresponding to pre-selected three-dimensional key points on a three-dimensional model of the object; Receive input containing a second video frame of the object, wherein the second video frame is sequentially following the first video frame in time; The first video frame and the second video frame are fed into the optical flow algorithm; The optical flow algorithm is used to estimate the second set of two-dimensional key points in the second video frame based on the first set of two-dimensional key points in the first video frame. The second set of two-dimensional key points and multiple camera parameters are fed into the perspective n-point algorithm; The six-DOF pose of the object in the second video frame is estimated using the perspective n-point algorithm; and The position of the aerial refueling boom is adjusted in real time based on the estimated six-DOF attitude of the object.

Images