Method, system and aircraft for verifying execution of automatic refueling data

Abstract

Methods, systems, and aircraft for validating execution of automatic refueling data are disclosed herein. A method includes receiving a two-dimensional (2D) image from a camera, cropping the 2D image based on a predefined feature region of interest of a target object to produce a plurality of cropped images, in response to the target object being greater than a threshold distance from the camera, reresizing one or more of the cropped images to produce one or more reresized images, determining a 2D keypoint of the target object within the one or more reresized images or the plurality of cropped images, a 6 degree of freedom (6DOF) pose is estimated based on a 2D keypoint of a target object and a three-dimensional (3D) model to produce an estimated 6DOF pose, and the 6DOF pose is output.

Description

Technical Field

[0001] The present disclosure generally relates to in-air refueling and, more particularly, to controlling in-air refueling operations. Background Art

[0002] Automation of in-air refueling provides safety benefits for tanker and receiver aircraft. However, in practicing current automation techniques, it can be difficult to perform refueling operations accurately and efficiently using cameras. Summary of the Invention

[0003] In response to the current state of the art and, in particular, in response to the disadvantages of conventional in-air refueling techniques that are not fully addressed by currently available technologies, the subject matter of the present application has been developed. Accordingly, the subject matter of the present application has been developed to provide systems and methods for providing in-air refueling techniques that overcome at least some of the above disadvantages of the prior art.

[0004] The following is a non-exhaustive list of examples, which may or may not be claimed, of the subject matter disclosed herein.

[0005] In one example, a method includes: receiving a two-dimensional (2D) image from a camera; cropping the 2D image based on a region of interest of predefined features of a target object to produce a plurality of cropped images, resizing one or more of the cropped images in response to the target object being greater than a threshold distance from the camera to produce one or more resized images, determining 2D key points of the target object within the one or more resized images or the plurality of cropped images, estimating a six-degree-of-freedom (6DOF) pose based on the 2D key points of the target object and a three-dimensional (3D) model to produce an estimated 6DOF pose, and outputting the 6DOF pose.

[0006] In another example, a tanker aircraft includes a refueling boom, a camera configured to generate a 2D image of an in-flight refueling operation between a receiver aircraft and the tanker aircraft, a processor, and a non-transitory computer-readable storage medium storing code. The code is executable by the processor to perform operations including the steps of: cropping the 2D image based on a predefined feature region of the receiver aircraft to produce a plurality of cropped images; resizing one or more of the cropped images in response to the receiver aircraft being greater than a threshold distance from the camera to produce one or more resized images, determining 2D key points of the receiver aircraft within the one or more resized images or the plurality of cropped images, estimating a 6DOF pose based on the 2D key points of the receiver aircraft and a 3D model to produce an estimated 6DOF pose, and outputting the 6DOF pose.

[0007] In yet another example, a system includes a camera, a processor, and a non-transitory computer-readable storage medium storing code. The camera is configured to generate a 2D image of an in-flight refueling operation between a receiver aircraft and a tanker aircraft. The code is executable by the processor to perform operations including the steps of: cropping the 2D image based on a predefined feature region of the receiver aircraft to produce a plurality of cropped images; resizing one or more of the cropped images in response to the receiver aircraft being greater than a threshold distance from the camera to produce one or more resized images; determining 2D key points of the receiver aircraft within the one or more resized images or the plurality of cropped images; estimating a 6DOF pose based on the 2D key points of the receiver aircraft and a 3D model to produce an estimated 6DOF pose; and outputting the 6DOF pose.

[0008] The features, structures, advantages, and / or characteristics of the subject matter described in this disclosure may be combined in any suitable manner in one or more examples and / or implementations. In the following description, numerous specific details are provided to thoroughly understand examples of the subject matter of this disclosure. Those skilled in the relevant art will recognize that the subject matter of this disclosure may be practiced without one or more of the specific features, details, components, materials, and / or methods of a particular example or implementation. In other instances, additional features and advantages that may not be present in all examples or implementations may be recognized in a particular example and / or implementation. Additionally, in some instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the subject matter of this disclosure. The features and advantages of the subject matter of this disclosure will become more apparent from the following description and the appended claims, or may be learned by practice of the subject matter as set forth below. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] To more easily understand the advantages of the subject matter, a more specific description of the subject matter briefly described above will be presented by reference to specific examples shown in the accompanying drawings. It should be understood that these drawings only depict typical examples of the subject matter and are therefore not considered to limit its scope. The subject matter will be described and explained with additional features and details by using the drawings, in which:

[0010] Figure 1 is a schematic block diagram of a tanker aircraft having an automatic guiding light system according to one or more examples of the present disclosure;

[0011] Figure 2 is a schematic side view of an aircraft refueling operation according to one or more examples of the present disclosure;

[0012] Figure 3 is a schematic perspective view of an aircraft refueling operation according to one or more examples of the present disclosure;

[0013] Figure 4 is a schematic flowchart of a method for an automatic control refueling operation according to one or more examples of the present disclosure;

[0014] Figure 5-1 is according to one or more examples of the present disclosure Figure 4 schematic flowchart of the steps of the method;

[0015] Figure 5-2 is according to one or more examples of the present disclosure Figure 5-1 enlarged view of the component;

[0016] Figure 6 is a cropped image of a refueling aircraft at a first distance;

[0017] Figure 7 is a cropped image of a refueling aircraft at a second distance; and

[0018] Figure 8 is a cropped image of a refueling aircraft at a third distance. Detailed Description

[0019] References throughout this specification to "one example", "an example", or similar language mean that a particular feature, structure, or characteristic described in connection with the example is included in at least one example of the present disclosure. The appearances of the phrases "in one example", "in an example", and similar language throughout this specification may, but do not necessarily, all refer to the same example. Similarly, the use of the term "implementation" means an implementation having a particular feature, structure, or characteristic described in connection with one or more examples of the present disclosure, however, in the absence of an express correlation to indicate otherwise, an implementation may be associated with one or more examples.

[0020] Disclosed herein is a refueling system 102 located on a tanker 100 that provides a determination of whether a two-dimensional (2D) to three-dimensional (3D) attitude estimation system is correct. In various implementations, the system and method extract smaller intrinsic (full-resolution) regions of interest to train and detect key points for six degrees of freedom (6DOF) estimation with higher accuracy. The determination can be provided to an in-air refueling system for controlling the output to a receiver pilot, a boom operator, and / or an automatic in-air refueling component during in-air refueling operations. As Figure 1 shown, the refueling system 102 includes a processor 104, a camera system 106, a guiding light system 108 (e.g., a directional light system), a boom operator interface 110, an automatic refueling system 112, and a memory 114.

[0021] In various implementations, referring to Figure 1 and Figure 2, the camera system 106 includes a camera 120, a video image processor 122, and an image generator 124. The camera 120 is mounted to a fixed platform within a distal housing attached to the lower rear fuselage of the fuel dispenser 100 (e.g., see Figure 2 ). The camera 120 includes one or more lenses with remotely operable focusing and zoom capabilities. The camera 120 is located in a rear position relative to and below the fuel dispenser 100. The video image processor 122 receives digitized video images from the camera 120 and generates real-time 2D video images. The digitized video images include objects viewed by the camera 120 within the cone of vision. The image generator 124 then generates images for presentation to the boom operator.

[0022] In various embodiments, the boom operator interface 110 includes a user interface device 130 and a monitor 132. The images presented on the monitor 132 are based on information provided by the processor 104. The guidance light system 108 includes a switch unit 140 and a light array 142 (i.e., flight guidance lights). The switch unit 140 controls the activation of the light array 142 based on information provided by the processor 104. The automatic refueling system 112 controls the operation of the refueling boom 204 and / or the fuel dispenser 100 based on information provided by the processor 104.

[0023] In various embodiments, the light array 142 is located on the lower front fuselage of the fuel dispenser 100. The light array 142 is positioned to be clearly visible to the pilot of the receiving aircraft 202. The light array 142 includes various lights for providing direction information to the pilot of the receiving aircraft 202. The light array 142 may include approach light bars, elevation light bars, front / rear position light bars, four longitudinal reflectors, two lateral reflectors, or other lights.

[0024] The camera system 106 generates a two-dimensional (2D) image 300 of at least the three-dimensional space of the receiving aircraft 202. The 2D image 300 includes the approach area into which the receiving aircraft 202 enters before commencing the refueling operation. The receiving aircraft 202 includes a boom nozzle receiver 208 that is capable of coupling to the refueling boom 204 to effect fuel transfer.

[0025] It will be appreciated that refueling or close quarter operations can occur between other vehicles, not just the depicted aircraft 100, 202. Refueling operations or close quarter operations can occur during adverse weather conditions. The vehicles can be any vehicles that move relative to each other (in water, on land, in air, or in space). The vehicles can also be manned or unmanned. By way of non-limiting examples, in various embodiments, the vehicles can be motor vehicles driven by wheels and / or tracks, such as but not limited to cars, trucks, vans, etc. By way of other non-limiting examples, in various embodiments, the vehicles can include vessels, such as but not limited to boats, ships, submarines, submersibles, autonomous underwater vehicles (AUVs), etc. By way of additional non-limiting examples, in various embodiments, the vehicles can include other manned or unmanned aircraft, such as but not limited to fixed-wing aircraft, rotary-wing aircraft, and lighter-than-air (LTA) aircraft.

[0026] It will be appreciated that the image analysis techniques described herein can be used in robotics, such as for pose estimation of objects for grasping and manipulation and for data generation for pose estimation projects.

[0027] In various embodiments, the non-transitory computer-readable instructions (i.e., code) stored in the memory 114 (i.e., storage medium) cause the processor 104 to use the raw image data from a single sensor (i.e., camera 120) and to make the raw data scalable and cost-effectively integrated into an existing system. Specifically, the processor 104 predicts the key points 310 of the receiver aircraft 202 within the 2D image 300 (e.g., see Figure 3 ). The key points 310 are referenced in 2D space. The prediction is based on a trained deep neural network configured to estimate the pixel positions of the key points of the refueling boom 204 in the 2D image 300. The processor 104 then performs 2D-to-3D correspondence using a 3D point matching algorithm by projecting the 2D key points 310 into 3D space. Each of the predicted 2D key points 310 is projected from 2D space to 3D space using perspective-n-point (PnP) pose calculation to produce a prediction of the refueling boom 204 (i.e., the boom 6 degrees of freedom (6DOF) position (i.e., pose)). More generally, PnP pose calculation produces any parameterization of an object in order to locate it in 3D space. In the specific case of the boom 204, a more constrained set of parameters in the form of boom control parameters is produced (e.g., boom pitch and roll based on the boom attachment point 230).

[0028] In various embodiments, non-transitory computer-readable instructions (i.e., code) stored in the memory 114 (i.e., storage medium) cause the processor 104 to predict key points 310 of the tanker 202 within the 2D image 300 (see, e.g., Figure 3 ). The key points 310 are referenced in 2D space.

[0029] In various embodiments, the processor 104 trains a convolutional neural network (CNN) to recognize features / key points on a 3D model (computer-aided design (CAD) model) from 2D images. The CNN is based on a residual network (ResNet) architecture. The CNN removes the final pooling and fully connected layers of the architecture and replaces them with a series of deconvolution and / or upsampling layers to return an output image that matches the height and width of the input image and multiple key points that match multiple channels. Each of the channels is considered a heat map of the position of the key point in the 2D image space. Based on this heat map, the pixel at the center of the distribution represented by the heat map is selected as the position of the key point (i.e., 2D key point prediction).

[0030] In various embodiments, during the training of the CNN, a detector (e.g., the CNN) takes a rescaled bounding box crop of a video frame as input and returns a black-and-white heat map image of each key point as output. The pixel values of the heat map indicate, for each key point, the likelihood of finding the key point of the 3D virtual object at each pixel position of the image once the object is projected onto the image. To train the weights of the CNN, a ground truth heat map is constructed from the ground truth 2D pixel positions. The pixel values of the ground truth heat map are assigned the values of a Gaussian probability distribution on the 2D coordinates, where the mean is equal to the ground truth 2D pixel position and the covariance is left as a hyperparameter for training. The loss minimized during training consists of the Jensen-Shannon divergence between the heat map output of the CNN and the ground truth heat map and the Euclidean norm between the 2D key point estimate of the CNN and the ground truth 2D key points.

[0031] Each of the predicted 2D key points 310 is compared with the corresponding 3D model key points using a PnP pose algorithm to produce a 6DOF pose estimate of the position of the tanker 202 or the refueling boom 204. Then, the processor 104 analyzes the potential error of the 6DOF pose estimate. The processor 104 generates a confidence or uncertainty value associated with the 6DOF pose estimate. First, the processor 104 determines the reprojection error. The reprojection error includes the reprojection error of the i-th key point estimate. Using the solved 6DOF pose, the reprojection error is calculated as the 2D distance between the i-th estimated 2D key point and the 2D projection of the i-th 3D model key point.

[0032] Reprojection error of the i-th key point = , R, t)

[0033] N – Number of key points

[0034] The i-th 2D key point estimate, N total points

[0035] The i-th 3D model estimate (corresponding to the i-th key point estimate), N total points

[0036] – Projection operator based on camera parameters

[0037] – Rotation and translation pose parameters (6DOF)

[0038] – Number of new key point sets to be sampled

[0039] The i-th 2D key point in the j-th newly sampled key point set, N total points in each set, M total point sets

[0040] The pose parameters calculated for the j-th newly sampled key point set, M total poses

[0041] – 1D tuning factor for sampling

[0042] 2D noise from a normal distribution with mean 0 and unit covariance matrix. N*M total samples, template symbol for noise from the normal distribution

[0043] Multiplication or element-wise multiplication of 1D vectors and matrices

[0044] The 2D covariance matrix of the i-th key point estimate

[0045] The reprojection error is used to sample the distribution of the new key point sets and calculate the pose of the sampled key point sets in order to form a distribution of 6DOF pose results. Processor 104 samples M new key point sets. To sample the i-th key point in the j-th new key point set, processor 104 samples noise from a 2D normal distribution with 0 mean and unit covariance. Next, processor 104 multiplies (i.e., scales) the sampled noise by the absolute value of the reprojection error and a scaling factor for tuning the result. Processor 104 then adds the scaled noise to the 2D key point estimate. This can be interpreted as sampling from a 2D normal distribution centered at the 2D key point estimate, where the covariance is scaled by the reprojection error

[0046] , R, t)

[0047]

[0048] Next, the processor 104 obtains M new 6DOF pose estimates from the M sets of sampled key points. The M 6DOF pose estimates form a solution distribution, from which the processor 104 calculates the 6DOF standard deviation to represent the solution uncertainty. If there is a large variance in the 6DOF estimates that seem statistically reasonable, the magnitude of the uncertainty should increase accordingly.

[0049] In various embodiments, the processor 104 uses a Kalman filter to track the 6DOF pose of an object along a video path. The processor 104 uses the most recent pose to update the Kalman filter and calculates the reprojection error using the result-averaged pose of the Kalman filter. An additional Kalman filter can be used to smooth the uncertain output.

[0050] In various embodiments, the processor 104 generates the 3D position of a specific point of interest on the 3D object after rotation and translation by the predicted 6DOF pose. The processor 104 adjusts the uncertainty estimate into the 3D point output. After running the PnP algorithm to obtain the sample pose for each set of sample key points, the processor 104 uses the sample pose to rotate and translate the 3D object model to calculate the sample 3D points. The result is a distribution on the 3D point of interest. From this distribution, the processor 104 calculates the 3D standard deviation to represent the solution uncertainty.

[0051] Reference Figure 4 , process 1000 provides visualization of the steps in the 6DOF estimation for an air-to-air refueling operation. Process 1000 includes a preprocessing machine learning step 402 (e.g., object model training 422 and key point model training block 414) and a runtime geometric optimization step (shape-pose transformation block 420). In the machine learning step 402, an image is received at the object detection step (box 410). After the receiver aircraft 202 is detected at box 410, the image of the receiver aircraft 202 is cropped and fed into the key point detection step (box 412), which estimates the 2D key points of the receiver aircraft 202. Once these 2D key points are found, given the 2D key points of the receiver aircraft 202, the camera parameters, and the 3D model, the geometric optimization step uses PnP to find the 6DOF pose of the receiver aircraft 202 (box 420 of the geometric optimization step).

[0052] In key point model training (block 414), a data matrix created from multiple CAD model variations and their random scaling can be used to find singular value decomposition (SVD) components. Each of the CAD model variations has a 2D (number of key points times 3 coordinates per key point) point array, which can be transformed into a flat one-dimensional (1D) vector before being added to the data matrix. Then, the mean of all samples in the data matrix (i.e., the base model) is performed. Then, the SVD is calculated based on the difference of the data matrix and the mean of the data matrix. The mean of the data matrix and the SVD components are later used in the shape-pose transformation 420 and stored as a model in the receiver CAD model storage 430.

[0053] Machine learning step 402 further includes object detection 410, which detects the tanker 202 from the raw image data received from the camera system 106. Detection of the tanker 202 during object detection 410 utilizes a trained deep neural network from object model training 422, which processes the input to output a bounding box around the region of interest of the tanker 202.

[0054] During key point model training 414, a neural network is trained to predict semantic key points across all variations of the tanker 202. The neural network is referred to as a key point detector (e.g., associated with key point detection 412). In this case, domain randomization 426 is used to achieve better results using the annotated training data stored in the simulation database 428. Another neural network is used as an object detector during object detection 410, which is trained on the input image and the bounding box. At runtime, the input image from the camera system 106 is cropped by the object detector (object detection 410), and the input image is then fed into the key point detector (key point detection 412). The object detector neural network associated with object detection 410 predicts a bounding box from the full-frame image from the camera system 106 to obtain a more localized image of the tanker to provide the input to the key point detector during key point detection 412.

[0055] A convolutional neural network (CNN) is used for the deep learning-based key point detector associated with key point detection 412. All variations of the previously created key point set are used to train the key point detector on the input image.

[0056] In various embodiments, referring to FIG. 5, the input image 502 is decomposed into smaller groups of features (cropped image 500), thus allowing full resolution of the desired features and avoiding resizing, which can result in a loss of processing speed. This keeps the runtime speed at the desired level. The higher-resolution cropped images near the region of interest of the input image provide as much detail as possible about the relevant features. For air-to-air receiver 6DOF estimation, a cropped image of the tanker 202 that shares the same (native) resolution as the input image is taken and resized to the input resolution of the keypoint detection neural network. Since the keypoint detection neural network must be able to process in real time, the input resolution should match the input resolution used for training the keypoint detection neural network. If the input image is resized, pixel information is lost, resulting in a degraded representation around the relevant image features. For resolution, the image of the tanker 202 is cropped into smaller parts and then resized. When resized to match the neural network input resolution, the resized cropped image degrades less. When the tanker 202 reaches a specific position relative to the camera 120, some or all of the resizing of the smaller cropped images is not performed, thereby using the native / full resolution representation of those regions. Since the smaller cropped images are focused around a local area, such as around the container 208 ( Figure 3 ), more details of that region are retained. This improves the accuracy of the tanker 6DOF estimation.

[0057] Referring Figure 5-1 and Figure 5-2 , cropped images 500 are taken from the input image 502 based on the regions of interest of the features. A cropped image 500 is provided for each region of interest of the features. In some examples, all cropped images 500 are provided for all keypoints (box 420) used in PnP for 6DOF estimation. Since resizing the cropped images can result in image degradation, native crops (i.e., a single crop that includes all features / keypoints of the tanker) with the same resolution as the keypoint detection neural network can be used to avoid resizing. The processor 104 or the user can choose to use native crops, resized cropped images, or both native crops and resized cropped images. Objects farther away in a single initial cropped image 502 appear smaller than closer objects. When using native crops, the features of interest in the zooming method (i.e., the smaller cropped images) may not occupy as much space when the tanker 202 is farther away, so resizing is used during these instances such that the features occupy most of the cropped image.

[0058] As Figure 6As shown, the receiver aircraft 202 only enters the refueling area. The receptacle features occupy less space in the cropped image 600 of the original resolution input image. However, the receptacle features in the resized image 602 are degraded.

[0059] As Figure 7 shown, when the receiver aircraft 202 is closer to the camera 120, the resized cropped image 700 and the cropped image 702 in the original resolution are very similar, and both produce clear images. The cropped image 702 is selected because the pixel values are not affected by the resizing.

[0060] As Figure 8 shown, when the receiver aircraft 202 is at or nearly at the ideal refueling position, the resized cropped image 800 and the cropped image 802 in the original resolution are almost the same. The cropped image 802 is selected because the pixel values are not affected by the resizing.

[0061] When the receiver aircraft is estimated to be within a certain distance from the tanker aircraft, there are conditions for using the native resolution image, which is where conditions such as the cropped image 802 are observed. This distance threshold generally corresponds consistently to when the receiver aircraft is within the contact envelope (which means the receiver aircraft can physically contact the tanker aircraft via the refueling boom).

[0062] The following is a non-exhaustive list of examples, which may or may not be claimed, of the subject matter disclosed herein.

[0063] The following part of this paragraph depicts Example 1 of the subject matter disclosed herein. According to Example 1, a method includes: receiving a 2D image from a camera; cropping the 2D image based on a region of interest of predefined features of a target object to generate a plurality of cropped images, resizing one or more of the cropped images in response to the target object being greater than a threshold distance from the camera to generate one or more resized images, determining 2D key points of the target object within the one or more resized images or the plurality of cropped images, estimating a 6DOF pose based on the 2D key points of the target object and a 3D model to generate an estimated 6DOF pose, and outputting the 6DOF pose.

[0064] The following part of this paragraph depicts Example 2 of the subject matter disclosed herein. According to Example 2 including Example 1 above, each of the cropped images includes a plurality of key points.

[0065] The following part of this paragraph depicts Example 3 of the subject matter disclosed herein. According to Example 3 including any one of Example 1 or 2 above, the target object is a receiver aircraft in a close air operation.

[0066] The following part of this paragraph depicts Example 4 of the subject matter disclosed herein. According to Example 4, which includes Example 3 above, the close air operation is a refueling operation.

[0067] The following part of this paragraph depicts Example 5 of the subject matter disclosed herein. According to Example 5, which includes any one of the previous examples above, resizing includes resizing to a resolution associated with a previously trained keypoint detection neural network.

[0068] The following part of this paragraph depicts Example 6 of the subject matter disclosed herein. According to Example 6, which includes Example 5 above, resizing is responsive to the resolution of the cropped image.

[0069] The following part of this paragraph depicts Example 7 of the subject matter disclosed herein. According to Example 7, which includes any one of the previous examples above, an initial single crop of the 2D image is performed before generating multiple cropped images.

[0070] The following part of this paragraph depicts Example 8 of the subject matter disclosed herein. According to Example 8, which includes any of the foregoing examples above, an automatic refueling operation is performed in response to a 6DOF pose.

[0071] The following part of this paragraph depicts Example 9 of the subject matter disclosed herein. According to Example 9, a fuel tanker includes: a refueling boom; a camera configured to generate a 2D image of an in-flight refueling operation between a receiver aircraft and the fuel tanker; a processor; and a non-transitory computer-readable storage medium storing code that can be executed by the processor to perform an operation including the following steps: cropping the 2D image based on a predefined feature region of the receiver aircraft to generate multiple cropped images; resizing one or more of the cropped images in response to the receiver aircraft being greater than a threshold distance from the camera to produce one or more resized images, determining 2D keypoints of the receiver aircraft within the one or more resized images or the multiple cropped images, estimating a 6DOF pose based on the 2D keypoints of the receiver aircraft and a 3D model to produce an estimated 6DOF pose, and outputting the 6DOF pose.

[0072] The following part of this paragraph depicts Example 10 of the subject matter disclosed herein. According to Example 10, which includes Example 9 above, each of the cropped images includes multiple keypoints.

[0073] The following part of this paragraph depicts Example 11 of the subject matter disclosed herein. According to Example 11, which includes the previous examples above, resizing includes resizing to a resolution associated with a previously trained keypoint detection neural network.

[0074] The following part of this paragraph depicts Example 12 of the subject matter disclosed herein. According to Example 12, which includes Example 11 above, the resizing is responsive to the resolution of the cropped image.

[0075] The following part of this paragraph depicts Example 13 of the subject matter disclosed herein. According to Example 13, which includes any of the previous examples above, the code is further configured to cause the processor to perform an initial single crop of the 2D image before generating multiple cropped images.

[0076] The following part of this paragraph depicts Example 14 of the subject matter disclosed herein. According to Example 14, which includes any of the previous examples above, the fuel tanker further includes an automatic refueling system and an arm operator system or a flight guidance light system. The code is further configured to cause the processor to control the operation of the automatic refueling system, the arm operator system, or the flight guidance light system based on the 6DOF attitude of the receiver aircraft.

[0077] The following part of this paragraph depicts Example 15 of the subject matter disclosed herein. According to Example 15, a system includes: a camera configured to generate a 2D image of an in-flight refueling operation between a receiver aircraft and a fuel tanker; a processor; and a non-transitory computer-readable storage medium storing code. The code is executable by the processor to perform operations including the steps of: cropping the 2D image based on a predefined feature region of the receiver aircraft to generate multiple cropped images; resizing one or more of the cropped images in response to the receiver aircraft being greater than a threshold distance from the camera to generate one or more resized images, determining 2D key points of the receiver aircraft within the one or more resized images or the multiple cropped images, estimating a 6DOF attitude based on the 2D key points of the receiver aircraft and a 3D model to generate an estimated 6DOF attitude, and outputting the 6DOF attitude.

[0078] The following part of this paragraph depicts Example 16 of the subject matter disclosed herein. According to Example 16, which includes Example 15 above, each of the cropped images includes multiple key points.

[0079] The following part of this paragraph depicts Example 17 of the subject matter disclosed herein. According to Example 17, which includes Example 16 above, the resizing includes resizing to a resolution associated with a previously trained key point detection neural network.

[0080] The following part of this paragraph depicts Example 18 of the subject matter disclosed herein. According to Example 18, which includes any of the previous examples above, the resizing is responsive to the resolution of the cropped image.

[0081] The following part of this paragraph depicts Example 19 of the subject matter disclosed herein. According to Example 19, which includes any of the previous examples, the code is further configured to cause the processor to perform an automatic refueling operation in response to a 6DOF pose.

[0082] The following part of this paragraph depicts Example 20 of the subject matter disclosed herein. According to Example 20, which includes any of the previous examples above, the code is further configured to generate an uncertainty value of the 6DOF pose.

[0083] Those skilled in the art will understand that the different illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. Some embodiments and implementations have been described above in terms of functional and / or logical block components (or modules) and various processing steps. However, it should be understood that such block components (or modules) can be implemented by any number of hardware, software, and / or firmware components configured to perform the specified functions. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above in terms of their functional aspects. Whether the functions are implemented as hardware or software depends on the particular application and the design constraints imposed on the overall system. Those skilled in the art can implement the described functionality in different ways for each particular application, but such implementation decisions should not be construed as causing a departure from the scope of the present invention. For example, an implementation of a system or component may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, look-up tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will recognize that the embodiments described herein are merely exemplary implementations.

[0084] The different illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0085] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination thereof. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In an alternative, the storage medium may be integral with the processor. The processor and the storage medium may reside in an ASIC.

[0086] Techniques and technologies may be described herein in terms of functional and / or logical block components and with reference to symbolic representations of operations, processing tasks, and functions that may be performed by different computing components or devices. Such operations, tasks, and functions are sometimes referred to as being computer-executed, computerized, software-implemented, or computer-implemented. In fact, one or more processor devices may perform the described operations, tasks, and functions by manipulating electrical signals that represent data bits at memory locations in the system memory, as well as other processing of the signals. A memory location that holds data bits is a physical location having specific electrical, magnetic, optical, or organic properties corresponding to the data bits. It should be understood that the different block components shown in the figures may be implemented by any number of hardware, software, and / or firmware components configured to perform the specified functions. For example, embodiments of a system or component may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, look-up tables, etc., which may perform various functions under the control of one or more microprocessors or other control devices.

[0087] In the above description, certain terms may be used, such as "upward", "downward", "upper", "lower", "horizontal", "vertical", "left", "right", "above", "below", etc. Where applicable, these terms are used to provide some clarity in the description when dealing with relative relationships. However, these terms are not intended to imply absolute relationships, positions, and / or orientations. For example, with respect to an object, the "upper" surface can simply be turned over to become the "lower" surface. Nevertheless, it is still the same object. Additionally, unless otherwise explicitly specified, the terms "including", "comprising", "having", and their variants mean "including but not limited to". Unless otherwise explicitly specified, an enumerated list of items does not imply that any or all of the items are mutually exclusive and / or mutually inclusive. Unless otherwise explicitly specified, the terms "a", "an", and "the" also refer to "one or more". Additionally, the term "plural" can be defined as "at least two". Further, unless otherwise indicated, as defined herein, a plurality of specific features do not necessarily mean every specific feature in the entire set or class of a particular feature.

[0088] In addition, in this specification, examples where one element is "coupled" to another element may include direct coupling and indirect coupling. Direct coupling can be defined as one element being coupled to another element and making contact with the other element to some extent. Indirect coupling can be defined as a coupling between two elements that do not directly contact each other, but rather have one or more additional elements between the coupled elements. Further, as used herein, fixing one element to another element can include direct fixing and indirect fixing. Additionally, as used herein, "adjacent" does not necessarily mean in contact. For example, one element can be adjacent to another element without contacting that element.

[0089] As used herein, when used with a list of items, the phrase "at least one of..." means that different combinations of one or more of the listed items can be used, and only one item from the list may be required. The items can be specific objects, things, or categories. In other words, "at least one of..." means that any combination or multiple items from the list can be used, but not all items from the list may be required. For example, "at least one of item A, item B, and item C" can mean item A; item A and item B; item B; item A, item B, and item C; and item B and item C. In some cases, "at least one of item A, item B, and item C" can, for example, mean but not be limited to two item As, one item B, and ten item Cs; four item Bs and seven item Cs; or some other suitable combination.

[0090] Unless otherwise indicated, the terms "first", "second", etc. are used herein only as labels and are not intended to impose ordinal, positional, or hierarchical requirements on the items to which these terms refer. Additionally, a reference to, for example, a "second" item does not require or preclude the presence of, for example, a "first" or lower-numbered item and / or a "third" or higher-numbered item.

[0091] As used herein, a system, apparatus, structure, article, element, component, or piece of hardware that is "configured to" perform a specified function is indeed capable of performing the specified function without any further modification, rather than merely having the possibility of performing the specified function after further modification. In other words, a system, apparatus, structure, article, element, component, or piece of hardware that is "configured to" perform a specified function is specifically selected, created, implemented, utilized, programmed, and / or designed for the purpose of performing the specified function. As used herein, "configured to" represents an existing characteristic of a system, apparatus, structure, article, element, component, or piece of hardware that enables the system, apparatus, structure, article, element, component, or piece of hardware to perform the specified function without further modification. For purposes of this disclosure, a system, apparatus, structure, article, element, component, or piece of hardware described as "configured to" perform a particular function may alternatively or additionally be described as "adapted to" and / or "operable to" perform that function.

[0092] The illustrative flowcharts included herein are generally set forth as logical flowcharts. As such, the depicted order and labeled steps represent an example of the presented method. Other steps and methods may be conceived that are equivalent in function, logic, or effect to one or more steps or portions thereof of the illustrated method. Additionally, the format and symbols employed are provided to explain the logical steps of the method and are understood not to limit the scope of the method. Although various arrow types and line types may be employed in the flowcharts, they are understood not to limit the scope of the corresponding method. In fact, some arrows or other connectors may be used to indicate only the logical flow of the method. For example, an arrow may indicate a waiting or monitoring period of unspecified duration between enumerated steps of the depicted method. Additionally, the order in which a particular method occurs may or may not strictly adhere to the order of the corresponding steps shown.

[0093] Those skilled in the art will recognize that at least a portion of the controllers, devices, units, and / or processes described herein can be integrated into a data processing system. Those skilled in the art will recognize that a data processing system generally includes a system unit housing, a video display device, a memory such as volatile or non-volatile memory, a processor such as a microprocessor or digital signal processor, computing entities such as an operating system, drivers, a graphical user interface, and application programs, one or more interaction devices (e.g., a touchpad, a touch screen, an antenna, etc.), and / or a control system including a feedback loop and control motors (e.g., for sensing position and / or speed feedback; for controlling motors for moving and / or adjusting components and / or quantities). The data processing system can be implemented using suitable commercially available components such as those typically found in data computing / communication and / or network computing / communication systems.

[0094] As used in the foregoing / following disclosure, the term controller / processor can refer to a collection of one or more components arranged in a particular manner, or can refer to a collection of one or more general-purpose components that can be configured to operate in a particular manner at one or more particular points in time, and / or can also be configured to operate in one or more other manners at one or more other times. For example, the same hardware or the same portion of hardware can be configured / reconfigured in sequential / parallel time as a first type of controller (e.g., at a first time), configured / reconfigured as a second type of controller (e.g., at a second time, which in some examples can coincide with, overlap, or follow the first time), and / or configured / reconfigured as a third type of controller (e.g., at a third time, which in some examples can coincide with, overlap, or follow the first time and / or the second time), etc. Reconfigurable and / or controllable components (e.g., general-purpose processors, digital signal processors, field programmable gate arrays, etc.) can be configured as a first controller for a first purpose, then a second controller for a second purpose, then a third controller for a third purpose, etc. The transition of reconfigurable and / or controllable components can occur in as little as a few nanoseconds, or can occur over a period of minutes, hours, or days.

[0095] In some such examples, when the controller is configured to perform a second purpose, the controller may no longer be able to perform the first purpose until the controller is reconfigured. The controller can switch between configurations as different components / modules in as little as a few nanoseconds. The controller can be reconfigured instantaneously. For example, the reconfiguration from a first controller to a second controller can occur just when the second controller is needed. The controller can be reconfigured in stages. For example, even before the first controller has completed its operation, parts of the first controller that are no longer needed can be reconfigured into the second controller. Such reconfigurations can occur automatically or can be prompted by an external source whether that source is another component, instruction, signal, condition, external stimulus, etc.

[0096] For example, the central processing unit / processor of the controller, etc., can function as a component / module for displaying graphics on a screen, a component / module for writing data to a storage medium, a component / module for receiving user input, and a component / module for multiplying two large prime numbers by configuring its logic gates according to its instructions at different times. Such reconfigurations can be invisible to the naked eye and, in some embodiments, can include the activation, deactivation, and / or rerouting of various parts of the component (e.g., switches, logic gates, inputs, and / or outputs). Thus, in the examples found in the foregoing / following disclosure, if an example includes or enumerates multiple components / modules, the example includes the possibility that the same hardware can implement more than one of the enumerated components / modules simultaneously or at discrete times or timings. The implementation of multiple components / modules, whether using more components / modules, fewer components / modules, or the same number of components / modules as the number of components / modules, is merely an implementation choice and generally does not affect the operation of the components / modules themselves. Thus, it should be understood that any recitation of multiple discrete components / modules in this disclosure includes the implementation of those components / modules as any number of underlying components / modules, including but not limited to a single component / module that reconfigures itself over time to perform the functions of multiple components / modules, and / or multiple components / modules that are similarly reconfigured, and / or a dedicated reconfigurable component / module.

[0097] In some cases, one or more components are herein referred to as "configured to", "configured by", "configurable to", "operable / operative for", "adapted / adaptable", "able to", "conformable to", etc. Those skilled in the art will recognize that such terms (e.g., "configured to") generally encompass active state components and / or inactive state components and / or standby state components, unless the context otherwise requires.

[0098] The foregoing detailed description has set forth various embodiments of the apparatus and / or process by use of block diagrams, flowcharts, and / or examples. As long as these block diagrams, flowcharts, and / or examples contain one or more functions and / or operations, those skilled in the art will recognize that each function and / or operation within these block diagrams, flowcharts, or examples can be implemented, individually and / or collectively, by a wide range of hardware, software (e.g., a high-level computer program that serves as a hardware specification), firmware, or virtually any combination thereof, limited to subject matter eligible for patent protection. In an embodiment, several portions of the subject matter described herein can be implemented via an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or other integrated form. However, those skilled in the art will recognize that, subject to eligible patent protection, all or a portion of some aspects of the embodiments disclosed herein can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as virtually any combination thereof, and that designing the circuitry and / or writing the code and / or firmware for the software (e.g., a high-level computer program that serves as a hardware specification) will be well within the skill of one of ordinary skill in the art in light of this disclosure. Further, those skilled in the art will recognize that the mechanisms of the subject matter described herein are capable of being distributed in a variety of forms as a program product, and that, regardless of the particular type of signal bearing medium used to actually effectuate the distribution, illustrative embodiments of the subject matter described herein apply. Examples of signal bearing media include, but are not limited to, the following: recordable media such as floppy disks, hard disk drives, compact discs (CDs), digital video discs (DVDs), digital tapes, computer memories, and the like; and transmission type media such as digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links (e.g., transmitters, receivers, transmission logic, reception logic, etc.)).

[0099] Regarding the appended claims, those skilled in the art will recognize that the operations recited therein can generally be performed in any order. Moreover, although the different operational flows are presented in sequence, it should be understood that the different operations can be performed in other orders different from those shown or can be performed simultaneously. Unless the context otherwise dictates, examples of such alternative orderings can include overlapping, interleaving, interrupting, reordering, incrementing, preparatory, supplementary, simultaneous, reverse, or other variant orderings. Additionally, unless the context otherwise dictates, terms like "in response to," "involve," or other past tense adjectives generally are not intended to exclude such variants. Without departing from the spirit or essential characteristics of the subject matter, the subject matter can be embodied in other specific forms. The described examples are to be considered in all respects only as illustrative and not restrictive. All changes that fall within the meaning and scope of the equivalents of the claims are included within their scope.

Claims

1. A method for verifying data for performing automatic refueling, comprising: receiving a two-dimensional image from a camera (120); cropping the two-dimensional image based on a predefined feature region of interest of the target object to generate a plurality of cropped images; resizing one or more of the cropped images to produce one or more resized images in response to the target object being greater than a threshold distance from the camera (120); determining two-dimensional key points of the target object within the one or more resized images or the plurality of cropped images; estimating a 6-DOF pose based on the 2D key points and the 3D model of the target object to generate an estimated 6-DOF pose; as well as The estimated 6-DOF pose is output.

2. The method according to claim 1, wherein: Each of the cropped images includes a plurality of key points.

3. The method according to claim 1, wherein: The target object is a receiving aircraft (202) in close air operations.

4. The method according to claim 3, wherein: The close air operation is a refueling operation.

5. The method according to claim 1, wherein: Resizing includes resizing to a resolution associated with a previously trained keypoint detection neural network.

6. The method according to claim 5, wherein: The resizing is responsive to a resolution of the cropped image.

7. The method according to claim 1, further comprising: Prior to generating the plurality of cropped images, an initial single cropping of the two-dimensional image is performed.

8. The method according to claim 1, further comprising: An automatic refueling operation is performed in response to the 6-DOF posture.

9. A fuel dispenser (100), comprising: refueling arm (204); A camera (120) configured to generate a two-dimensional image of an in-flight refueling operation between a receiving aircraft and the tanker (100); a processor (104); and A non-transitory computer-readable storage medium storing code, the code being executable by the processor (104) to perform operations comprising: cropping the two-dimensional image based on a predefined characteristic region of interest of the receiving aircraft to generate a plurality of cropped images; resizing one or more of the cropped images to produce one or more resized images in response to the receiving aircraft being greater than a threshold distance from the camera (120); determining two-dimensional key points of the receiving aircraft within the one or more resized images or the plurality of cropped images; estimating a 6-DOF attitude based on the 2D key points and the 3D model of the receiving aircraft to generate an estimated 6-DOF attitude; as well as The estimated 6-DOF pose is output.

10. The fuel dispenser (100) according to claim 9, wherein: Each of the cropped images includes a plurality of key points.

11. The fuel dispenser (100) according to claim 9, wherein: Resizing includes resizing to a resolution associated with a previously trained keypoint detection neural network.

12. The fuel dispenser (100) according to claim 11, wherein: The resizing is responsive to a resolution of the cropped image.

13. The fuel dispenser (100) according to claim 9, wherein: The code is further configured to cause the processor (104) to perform an initial single cropping of the two-dimensional image prior to generating the plurality of cropped images.

14. The fuel dispenser (100) according to claim 9, wherein: The fuel dispenser (100) further comprises: an automatic refueling system (112); and Boom operator system (110); or Flight guidance light system (108), The code is further configured to cause the processor (104) to control the operation of the automatic refueling system (112), the boom operator system (110), or the flight guidance light system (108) based on the 6-DOF attitude of the receiving aircraft (202).

15. A system (102) for verifying automatic refueling data, comprising: A camera (120) configured to generate a two-dimensional image of an in-flight refueling operation between a receiving aircraft and a refueling aircraft; Processor (104); as well as A non-transitory computer-readable storage medium storing code, the code being executable by the processor (104) to perform operations comprising: cropping the two-dimensional image based on a predefined characteristic region of interest of the receiving aircraft to generate a plurality of cropped images; resizing one or more of the cropped images to produce one or more resized images in response to the receiving aircraft being greater than a threshold distance from the camera (120); determining two-dimensional key points of the receiving aircraft within the one or more resized images or the plurality of cropped images; estimating a 6-DOF attitude based on the 2D key points and the 3D model of the receiving aircraft to generate an estimated 6-DOF attitude; as well as The estimated 6-DOF pose is output.

16. The system (102) of claim 15, wherein: Each of the cropped images includes a plurality of key points.

17. The system (102) of claim 16, wherein: Resizing includes resizing to a resolution associated with a previously trained keypoint detection neural network.

18. The system (102) of claim 15, wherein: The resizing is responsive to a resolution of the cropped image.

19. The system (102) of claim 15, wherein: The code is further configured to cause the processor (104) to perform an automatic refueling operation in response to the 6-DOF gesture.

20. The system (102) of claim 15, wherein: The code is further configured to generate an uncertainty value for the 6-DOF pose.

Images