Air refueling system and method

By using the camera to receive 2D images in the aerial refueling system and combining deep neural network and PnP attitude calculation, the 6-degree-of-free attitude of the refueling casing and oil receiver is estimated, and the accuracy and efficiency of the existing technology of aerial refueling operation is solved. , achieving more efficient and safe refueling control.

CN118941625APending Publication Date: 2024-11-12THE BOEING CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410292296.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-09
Filing Date
2024-03-14
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing aerial refueling technology is difficult to achieve efficient and accurate when refueling with cameras, especially with challenges in automated control.

Method used

A system and method are developed to receive 2D images using the camera, determine 2D key points based on a predetermined model of the target object, and estimate 6-degree-of-freedom poses through the 3D model, generate and output uncertainty values ​​of 6DOF poses to achieve refueling Precise control of operation. The system includes a camera, processor and computer-readable storage medium that stores code. Through deep neural networks and PnP attitude calculation and other technologies, the position and attitude estimation of the refueling sleeve and the oil receiver are realized.

Benefits of technology

Improves the accuracy and efficiency of air refueling operations, provides real-time control and directional guidance for refueling operations, and enhances safety, especially stability in adverse weather conditions or in close-range operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941625A_ABST
    Figure CN118941625A_ABST
Patent Text Reader

Abstract

The invention relates to an air refueling system and method. Methods, systems, and aircraft for validating execution of automatic refueling data are disclosed herein. A method includes: receiving a two-dimensional (2D) image from a camera; determining a 2D key point of a target object located in the 2D image based on a predetermined model of the target object; estimating a 6DOF attitude based on the 2D key point and the three-dimensional model of the target object; an uncertainty value of the 6DOF attitude is generated; and outputting the uncertainty value of the 6DOF attitude.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates generally to aerial refueling, and more particularly to controlling aerial refueling operations. Background Art

[0002] Automation of aerial refueling provides safety benefits to both the tanker and the receiving aircraft. However, accurately and efficiently performing refueling operations using cameras can be difficult when practicing current automation technology. Summary of the invention

[0003] The subject matter of the present application has been developed in response to the prior art, and in particular, in response to the shortcomings of conventional aerial refueling techniques that have not been fully addressed by the prior art. Therefore, the subject matter of the present application has been developed to provide systems and methods for providing aerial refueling techniques that overcome at least some of the above-mentioned shortcomings of the prior art.

[0004] The following is a non-exhaustive list of examples of subject matter disclosed herein that may or may not be claimed.

[0005] In one example, a method includes: receiving a 2D image from a camera; determining two-dimensional (2D) key points of a target object located within the 2D image based on a predetermined model of the target object; estimating a six-degree-of-freedom (6DOF) pose based on the 2D key points and a three-dimensional (3D) model of the target object; generating an uncertainty value for the 6DOF pose; and outputting the uncertainty value for the 6DOF pose.

[0006] In another example, a tanker includes a refueling ferrule, a camera configured to generate a 2D image of an in-flight refueling operation between a receiving aircraft and a tanker, a processor, and a non-transitory computer-readable storage medium storing code. The code is executable by the processor to perform operations including: determining 2D key points of a target object located within the 2D image based on a predetermined model of the target object; estimating a 6DOF pose based on the 2D key points and a 3D model of the target object; generating an uncertainty value for the 6DOF pose; and outputting the uncertainty value for the 6DOF pose.

[0007] In yet another example, a system includes: a camera configured to generate a 2D image of an in-flight refueling operation between a receiving aircraft and a refueling aircraft; a processor; and a non-transitory computer-readable storage medium storing code. The code is executable by the processor to perform operations including: determining 2D key points of a target object located within the 2D image based on a predetermined model of the target object; estimating a 6DOF pose based on the 2D key points and a 3D model of the target object; generating an uncertainty value for the 6DOF pose; and outputting the uncertainty value for the 6DOF pose.

[0008] The described features, structures, advantages and / or characteristics of the subject matter of the present disclosure may be combined in any suitable manner in one or more examples and / or implementations. In the following description, many specific details are provided to give a thorough understanding of the examples of the subject matter of the present disclosure. Those skilled in the relevant art will recognize that the subject matter of the present disclosure may be practiced without one or more of the specific features, details, parts, materials and / or methods of a particular example or embodiment. In other cases, additional features and advantages may be recognized in certain examples and / or embodiments that may not be present in all examples or embodiments. In addition, in some cases, known structures, materials or operations are not shown or described in detail to avoid blurring the various aspects of the subject matter of the present disclosure. According to the following description and the appended claims, the features and advantages of the subject matter of the present disclosure will become more completely apparent, or may be learned by the practice of the subject matter as described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order that the advantages of the subject matter may be more readily understood, a more particular description of the subject matter briefly described above will be presented by reference to specific examples shown in the accompanying drawings. It should be understood that these drawings depict only typical examples of the subject matter and that they are not to be considered limiting of its scope. The subject matter will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0010] Figure 1 is a schematic block diagram of a fuel dispenser with an automatic pilot light system according to one or more examples of the present disclosure;

[0011] Figure 2 is a schematic side view of an aircraft refueling operation according to one or more examples of the present disclosure;

[0012] Figure 3 is a schematic perspective view of an aircraft refueling operation according to one or more examples of the present disclosure;

[0013] Figure 4 is a schematic diagram of an image of a portion of an aircraft according to one or more examples of the present disclosure;

[0014] Figure 5 is a schematic flow chart of a method for automatically controlling a refueling operation according to one or more examples of the present disclosure; and

[0015] Figure 6 is a schematic flow chart of a method for automatically controlling a refueling operation according to one or more examples of the present disclosure. DETAILED DESCRIPTION

[0016] References throughout this specification to "one example," "example," or similar language mean that a particular feature, structure, or characteristic described in conjunction with the example is included in at least one example of the present disclosure. The phrases "in one example," "in an example," and similar language throughout this specification may, but do not necessarily, all refer to the same example. Similarly, use of the term "embodiment" means an embodiment having a particular feature, structure, or characteristic described in conjunction with one or more examples of the present disclosure, however, an embodiment may be associated with one or more examples without an explicit correlation to indicate otherwise.

[0017] A refueling system 102 located on a tanker 100 is disclosed herein that provides a determination of whether a two-dimensional (2D) to three-dimensional (3D) attitude estimation system is correct. This determination may be provided to the aerial refueling system for controlling outputs to a receiving aircraft pilot, a cannula operator, and / or an automated aerial refueling component during an aerial refueling operation. Figure 1 As shown, refueling system 102 includes a processor 104 , a camera system 106 , a guidance light system 108 (eg, guidance light system), a casing operator interface 110 , an automated refueling system 112 , and a memory 114 .

[0018] In various embodiments, reference Figure 1 and Figure 2 , camera system 106 includes camera 120, video image processor 122, and image generator 124. Camera 120 is generally mounted to a fixed platform within a cast housing that is attached to the lower rear fuselage of tanker 100. Camera 120 includes one or more lenses with remotely operated focus and zoom capabilities. Camera 120 is located at a rear position relative to tanker 100 and below tanker 100. Video image processor 122 receives a digitized video image from camera 120 and generates a real-time 2D video image. The digitized video image includes objects viewed within the viewing cone by camera 120. Image generator 124 then generates an image for presentation to a casing operator.

[0019] In various embodiments, sleeve operator interface 110 includes user interface device 130 and monitor 132. The images presented on monitor 132 are based on information provided by processor 104. Guidance light system 108 includes switch unit 140 and light array 142 (i.e., guidance lights). Switch unit 140 controls activation of light array 142 based on information provided by processor 104. Automatic refueling system 112 controls the operation of refueling sleeve 204 and / or refueling machine 100 based on information provided by processor 104.

[0020] In various embodiments, light array 142 is located on the lower forward fuselage of tanker 100. Light array 142 is positioned to be clearly visible to the pilot of receiving aircraft 202. Light array 142 includes various lights for providing directional information to the pilot of receiving aircraft 202. Light array 142 may include an approach light bar, an elevation light bar, a forward / aft position light bar, four longitudinal reflectors, two transverse reflectors, or other lights.

[0021] refer to Figure 3 , camera system 106 generates a two-dimensional (2D) image 300 of the three-dimensional space including refueling ferrule 204 and receiving aircraft 202. 2D image 300 includes an approach area that receiving aircraft 202 enters before commencing a refueling operation. Receiving aircraft 202 includes a ferrule nozzle receiver 208 that can be coupled to refueling ferrule 204 to complete a fuel transfer.

[0022] It is understood that refueling or close quarter operations may occur between other vehicles, not just the depicted aircraft (e.g., tanker 100, receiver 202). Refueling or close quarter operations may occur during adverse weather conditions. The vehicles may be any vehicles that move relative to each other (in water, on land, in air, or in space). The vehicles may also be manned or unmanned. Given by way of non-limiting example, in various embodiments, the vehicles may be motor vehicles driven by wheels and / or tracks, such as, but not limited to, cars, trucks, vans, etc. Given by way of further non-limiting example, in various embodiments, the vehicles may include ships, such as, but not limited to, boats, ships, submarines, submersibles, autonomous underwater vehicles (AUVs), etc. Given by way of further non-limiting example, in various embodiments, the vehicles may include other manned or unmanned aircraft, such as, but not limited to, fixed-wing aircraft, rotary-wing aircraft, and lighter-than-air (LTA) aircraft.

[0023] In various embodiments, non-transitory computer-readable instructions (i.e., code) stored in memory 114 (i.e., storage medium) cause processor 104 to use raw image data from a single sensor (i.e., camera 120) and make the raw data scalable and cost-effective to integrate into existing systems. Specifically, processor 104 predicts key points 310 (see, e.g., FIG. 1 ) of receiving aircraft 202 within 2D image 300. Figure 3). Key points 310 are referenced in 2D space. The prediction is based on a trained deep neural network that is configured to estimate the pixel locations of the key points of the refueling ferrule 204 in the 2D image 300. The processor 104 then uses a 3D point matching algorithm to perform 2D to 3D correspondence by projecting the 2D key points 310 into 3D space. Each of the predicted 2D key points 310 is projected from 2D space to 3D space using a perspective n-point (PnP) pose calculation to produce a prediction of the refueling ferrule 204 (i.e., the ferrule 6 degree of freedom (DOF) position (i.e., pose)). More generally, the PnP pose calculation produces any parameterization of an object to position it in 3D space. In the specific case of the refueling ferrule 204, a more constrained set of parameters is produced in the form of ferrule control parameters (e.g., ferrule pitch and roll based on the ferrule attachment point 230).

[0024] In various embodiments, non-transitory computer readable instructions (ie, code) stored in memory 114 (ie, storage medium) cause processor 104 to predict key points 310 (see, e.g., FIG. 3 ) of receiving aircraft 202 within 2D image 300. Figure 3 ) or a key point 320 of the refueling sleeve 204 (see e.g. Figure 2 ). The key points 310, 320 are referenced in 2D space.

[0025] In various embodiments, the processor 104 trains a convolutional neural network (CNN) to identify features / key points on a 3D model (computer-aided design (CAD) model) from a 2D image. The CNN is based on a residual network (Res Net) architecture. The CNN removes the final pooling and fully connected layers of the architecture and replaces them with a series of deconvolution and / or upsampling layers to return an output image that matches the height and width of the input image, where the number of key points matches the multiple channels. Each channel is considered to be a heat map of the location of the key point in the 2D image space. From the heat map, the pixel at the center of the distribution represented by the heat map is selected as the location of the key point (i.e., 2D key point prediction).

[0026] In various embodiments, reference Figure 4, during the training of the CNN, a detector (e.g., a CNN) takes as input an image 400, or in our case a rescaled bounding box crop of a video frame, and returns as output a black and white heatmap image 402 for each keypoint. The pixel values ​​of the heatmap indicate for each keypoint the likelihood of finding the keypoint of the 3D virtual object at each pixel location of the image once the object has been projected onto the image. To train the weights of the CNN, a ground truth heatmap is constructed from the ground truth 2D pixel locations. The pixel values ​​of the ground truth heatmap are assigned the values ​​of a Gaussian probability distribution on 2D coordinates with a mean equal to the ground truth 2D pixel location and the covariance remaining as a hyperparameter for training. The loss minimized during training consists of the Jensen-Shannon divergence between the heatmap output of the CNN and the ground truth heatmap and the Euclidean norm between the CNN's 2D keypoint estimate and the ground truth 2D keypoint.

[0027] Each of the predicted 2D keypoints 310, 320 is compared to the corresponding 3D model keypoint using the PnP pose algorithm to generate a 6DOF pose estimate of the position of the receiving aircraft (e.g., the tanker 202) or the refueling sleeve 204. The processor 104 then analyzes the potential errors of the 6DOF pose estimate. The processor 104 generates a confidence or uncertainty value associated with the 6DOF pose estimate. First, the processor 104 determines the reprojection error. The reprojection error includes the reprojection error of the i-th keypoint estimate. Using the solved 6DOF pose, the reprojection error is calculated as the 2D distance between the i-th estimated 2D keypoint and the 2D projection of the i-th 3D model keypoint.

[0028]

[0029] N - number of key points

[0030] - The i-th 2D keypoint estimate, a total of N points

[0031] k i ,i∈N-the i-th 3D model estimate (corresponding to the i-th keypoint estimate), a total of N points

[0032] P-Projection operator based on camera parameters

[0033] R, t-rotation and translation attitude parameters (6DOF)

[0034] M - the number of new keypoint sets to sample

[0035] - the i-th 2D keypoint in the j-th set of newly sampled keypoints, with a total of N points in each set and a total of M point sets

[0036] R j ,t j ,j∈[M]-the pose parameters of the jth set of newly sampled key points, a total of M poses

[0037] λ - 1D tuning factor used in the adoption

[0038] - 2D noise from normal distribution with mean 0 and identity covariance matrix. Total N*M samples, template symbol of noise from normal distribution.

[0039] * - 1D multiplication or point-wise multiplication for vectors and matrices

[0040] - 2D covariance matrix for the i-th keypoint estimate

[0041] The reprojection error is used to sample the distribution of the new key point set and calculate the pose of the sampled key point set to form a distribution of 6DOF pose results. The processor 104 samples M new key point sets. In order to sample the i-th key point in the j-th new key point set, the processor 104 samples the noise from a 2D normal distribution with 0 mean and constant covariance. Next, the processor 104 multiplies (i.e., scales) the sampled noise by the absolute value of the reprojection error and the scaling factor for tuning the result. The processor 104 then adds the scaled noise to the 2D key point estimate. This can be interpreted as sampling from a 2D normal distribution centered on the 2D key point estimate, where the covariance is scaled by the reprojection error.

[0042]

[0043]

[0044] Next, from the M sets of sampled key points, the processor 104 obtains M new 6DOF pose estimates. The M 6DOF pose estimates form a distribution of solutions, and the processor 104 calculates the 6DOF standard deviation based on the distribution to represent the solution uncertainty. If there is a large variance in the statistically reasonable 6DOF estimate, the magnitude of the uncertainty should be increased accordingly.

[0045] In various embodiments, the processor 104 uses a Kalman filter to track the 6DOF pose of the object during the video. The processor 104 updates the Kalman filter with the most recent pose and uses the resulting average pose of the Kalman filter to calculate the reprojection error. Additional Kalman filters can be used to smooth the uncertainty output.

[0046] In various embodiments, the processor 104 generates the 3D position of a particular point of interest on the 3D object after being rotated and translated by the predicted 6DOF pose. The processor 104 adjusts the uncertainty estimate to the 3D point output. After running the PnP algorithm to obtain a sample pose for each sample key point set, the processor 104 uses the sample pose to rotate and translate the 3D object model to calculate the sample 3D point. The result is a distribution on the 3D point of interest. Based on the distribution, the processor 104 calculates the 3D standard deviation to represent the solution uncertainty.

[0047] In various implementations, processor 104 uses λ=3 and M=128 sets of 2D keypoints. Other parameters may be used.

[0048] refer to Figure 5 , method 500 includes outputting an estimated position of a target object and a certainty value of the output estimated position of the target object. Block 505 of method 500 includes receiving a 2D image from a refueling camera. Block 510 of method 500 includes estimating key points of an aircraft image within the received 2D image. Block 515 of method 500 includes comparing the predicted 2D key points with corresponding 3D model key points via PnP to generate a 6DOF pose of the aircraft image. Block 520 of method 500 includes generating a confidence value for the 6DOF pose of the aircraft image. Block 530 of method 500 includes outputting the 6DOF pose of the aircraft image and the confidence value to an appropriate aircraft or refueling system.

[0049] In some examples, block 520 of method 500 also includes various sub-steps, such as Figure 6 As shown. Box 605 includes calculating the mean / covariance of the key point heat map. Box 610 includes calculating the reprojection error of each key point based on the PnP 6DOF solution. Box 615 includes scaling the covariance from box 605 using the reprojection error. Box 620 includes sampling the new key points based on predetermined parameters. Box 625 includes calculating the 6DOF pose of each new key point set. Box 630 includes calculating the standard deviation of all calculated 6DOF poses. Finally, box 635 includes smoothing the resulting uncertainty based on the standard deviation from box 630 in time using a temporal filter.

[0050] The following is a non-exhaustive list of examples of subject matter disclosed herein that may or may not be claimed.

[0051] The following portion of this paragraph describes Example 1 of the subject matter disclosed herein. According to Example 1, a method includes: receiving a 2D image from a camera (which may be a single camera in some examples); determining 2D key points of a target object located within the 2D image based on a predetermined model of the target object; estimating a 6DOF pose based on the 2D key points and the 3D model of the target object; generating an uncertainty value for the 6DOF pose; and outputting the uncertainty value for the 6DOF pose.

[0052] The following portion of this paragraph depicts Example 2 of the subject matter disclosed herein. According to Example 2 including Example 1 above, determining 2D keypoints is also based on a trained neural network configured to output keypoint heatmaps, wherein pixel intensity values ​​associated with each keypoint heatmap indicate a keypoint detection probability.

[0053] The following portion of this paragraph depicts Example 3 of the subject matter disclosed herein. According to Example 3 including any of Examples 1 or 2 above, the method further comprises: calculating a covariance for each of the keypoint heatmaps; calculating a reprojection error; scaling the covariance of the heatmap based on the reprojection error to produce a scaled covariance; generating a sample of a new keypoint based on the scaled covariance; and generating a new 6DOF pose based on the sample of the new keypoint. Generating the uncertainty value comprises calculating a standard deviation of the new 6DOF position.

[0054] The following portion of this paragraph depicts Example 4 of the subject matter disclosed herein. According to Example 4 including Example 3 above, calculating the reprojection error includes generating 2D keypoints from the 6DOF pose to produce reprojected 2D keypoints and comparing the 2D keypoints from the heatmap to the reprojected 2D keypoints.

[0055] The following portion of this paragraph depicts Example 5 of the subject matter disclosed herein. According to Example 5, which includes any of Examples 3 or 4 above, generating the uncertainty value further comprises applying a smoothing algorithm to the standard deviation.

[0056] The following portion of this paragraph depicts Example 6 of the subject matter disclosed herein. According to Example 6 including Example 5 above, the smoothing algorithm includes a Kalman filter.

[0057] The following portion of this paragraph depicts Example 7 of the subject matter disclosed herein. According to Example 7 including any of Examples 1 to 6 above, outputting the uncertainty value of the 6DOF attitude further comprises outputting the uncertainty value of the 6DOF attitude to an automatic refueling system, a casing operator system, or a pilot guidance light system.

[0058] The following portion of this paragraph depicts Example 8 of the subject matter disclosed herein. According to Example 8, a tanker includes: a refueling sleeve; a camera configured to generate a 2D image of an in-flight refueling operation between a receiving aircraft and a tanker; a processor; and a non-transitory computer-readable storage medium storing code. The code is executable by the processor to perform operations including: determining 2D key points of a target object located within the 2D image based on a predetermined model of the target object; estimating a 6DOF pose based on the 2D key points and a 3D model of the target object; generating an uncertainty value for the 6DOF pose; and outputting the uncertainty value for the 6DOF pose.

[0059] The following portion of this paragraph depicts Example 9 of the subject matter disclosed herein. According to Example 9 including Example 8 above, determining the 2D keypoints is also based on a trained neural network configured to output keypoint heatmaps, wherein the pixel intensity value associated with each keypoint heatmap indicates a keypoint detection probability.

[0060] The following portion of this paragraph depicts Example 10 of the subject matter disclosed herein. According to Example 10 including Example 9 above, the operations further include: calculating a covariance of each key point heat map in the key point heat map; calculating a reprojection error; scaling the covariance of the heat map based on the reprojection error to produce a scaled covariance; generating samples of new key points based on the scaled covariance; and generating a new 6DOF pose based on the samples of the new key points. Generating the uncertainty value includes calculating a standard deviation of the new 6DOF pose.

[0061] The following portion of this paragraph depicts Example 11 of the subject matter disclosed herein. According to Example 11 including Example 10 above, calculating the reprojection error includes generating 2D keypoints from the 6DOF pose to produce reprojected 2D keypoints and comparing the 2D keypoints to the reprojected 2D keypoints.

[0062] The following portion of this paragraph depicts Example 12 of the subject matter disclosed herein. According to Example 12 including Example 13 above, the smoothing algorithm includes a Kalman filter.

[0063] The following portion of this paragraph depicts Example 13 of the subject matter disclosed herein. According to Example 13, which includes any of Examples 10 to 12 above, code may be executed by a processor to determine one or more geometric relationships between key points projected into 3D space and key points on a 3D digital model associated with a recipient aircraft, and generate a confidence value based on the one or more geometric relationships.

[0064] The following portion of this paragraph depicts Example 14 of the subject matter disclosed herein. According to Example 14 including any one of Examples 8 to 13 above, the tanker further comprises an automatic refueling system, a cannula operator system, or a pilot guidance light system. Outputting the uncertainty value and the 6DOF attitude further comprises: outputting the uncertainty value and the 6DOF attitude to the automatic refueling system, the cannula operator system, or the pilot guidance light system.

[0065] The following portion of this paragraph depicts Example 15 of the subject matter disclosed herein. According to Example 15, a system includes: a camera configured to generate a 2D image of an in-flight refueling operation between a receiving aircraft and a refueling aircraft; a processor; and a non-transitory computer-readable storage medium storing code. The code is executable by the processor to perform operations including: determining 2D key points of a target object located within the 2D image based on a predetermined model of the target object; estimating a 6DOF pose based on the 2D key points and a 3D model of the target object; generating an uncertainty value for the 6DOF pose; and outputting the uncertainty value for the 6DOF pose.

[0066] The following portion of this paragraph depicts Example 16 of the subject matter disclosed herein. According to Example 16 including Example 15 above, determining the 2D keypoints is also based on a trained neural network configured to output keypoint heatmaps, wherein the pixel intensity value associated with each keypoint heatmap indicates a keypoint detection probability.

[0067] The following portion of this paragraph depicts Example 17 of the subject matter disclosed herein. According to Example 17 including Example 16 above, the operation further comprises: calculating the covariance of each key point heat map in the key point heat map; calculating a reprojection error; scaling the covariance of the heat map based on the reprojection error to produce a scaled covariance; generating samples of new key points based on the scaled covariance; and generating a new 6DOF pose based on the samples of the new key points. Generating the uncertainty value comprises calculating a standard deviation of the new 6DOF pose.

[0068] The following portion of this paragraph depicts Example 18 of the subject matter disclosed herein. According to Example 18 including Example 17 above, calculating the reprojection error includes generating 2D keypoints from the 6DOF pose to produce reprojected 2D keypoints and comparing the 2D keypoints to the reprojected 2D keypoints.

[0069] The following portion of this paragraph depicts Example 19 of the subject matter disclosed herein. According to Example 19, which includes any of Examples 17 or 18 above, generating the uncertainty value further comprises applying a Kalman filter to the standard deviation.

[0070] The following portion of this paragraph depicts Example 20 of the subject matter disclosed herein. According to Example 20 including any of Examples 15 to 19 above, outputting the uncertainty value of the 6DOF attitude further comprises outputting the uncertainty value of the 6DOF attitude to an automatic refueling system, a cannula operator system, or a pilot director light system.

[0071] It will be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software or a combination of the two. Some embodiments and implementations are described above according to functions and / or logic block components (or modules) and various processing steps. However, it should be understood that such block components (or modules) can be implemented by any number of hardware, software and / or firmware components configured to perform a specified function. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above according to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art can implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing departure from the scope of the present invention. For example, the implementation of a system or component can use various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices. In addition, it will be appreciated by those skilled in the art that the embodiments described herein are merely exemplary implementations.

[0072] The various illustrative logical blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0073] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, a software module executed by a processor, or a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an ASIC.

[0074] This article can be described technology and technique according to function and / or logic block assembly and with reference to the symbolic representation of operation, processing task and function that can be performed by various computing components or devices. Such operation, task and function are sometimes referred to as being computer-implemented, computerized, software-implemented or computer-implemented. In practice, one or more processor devices can perform the described operation, task and function by manipulating the electrical signal of the data bit at the memory location in the system memory and other processing of the signal. The memory location of the data bit is maintained to be a physical location with a specific electrical, magnetic, optical or organic property corresponding to the data bit. It should be understood that the various block components shown in the figure can be implemented by any number of hardware, software and / or firmware components configured to perform a specified function. For example, the implementation of the system or component can adopt various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices.

[0075] In the above description, certain terms may be used, such as "upper", "lower", "up", "lower", "horizontal", "vertical", "left", "right", "upper", "lower", etc. These terms are used to provide some clear descriptions when dealing with relative relationships where applicable. However, these terms are not intended to imply absolute relationships, positions and / or orientations. For example, with respect to an object, an "upper" surface can simply be turned into a "lower" surface by turning the object over. However, it is still the same object. In addition, unless otherwise expressly stated, the terms "include", "comprise", "have" and their variations mean "including but not limited to". Unless otherwise expressly stated, the list of enumerated items does not mean that any or all items are mutually exclusive and / or mutually inclusive. Unless otherwise expressly stated, the terms "one", "an" and "the" also refer to "one or more". In addition, unless otherwise stated, as defined herein, multiple specific features do not necessarily mean the entire set of specific features or each specific feature of a category.

[0076] In addition, examples of an element "coupled" to another element in this specification may include direct and indirect coupling. Direct coupling may be defined as an element that is coupled to another element and in some contact with the other element. Indirect coupling may be defined as a coupling between two elements that are not in direct contact with each other, but have one or more additional elements between the coupled elements. In addition, as used herein, fixing an element to another element may include direct fixing and indirect fixing. In addition, as used herein, "adjacent" does not necessarily mean contact. For example, an element may be adjacent to another element without being in contact with the element.

[0077] As used herein, when used with a list of items, the phrase "at least one of" means that different combinations of one or more of the listed items may be used, and only one of the items in the list may be required. An item may be a specific object, thing, or category. In other words, "at least one of" means that any combination of the items or number of items in the list may be used, but all of the items in the list may not be required. For example, "at least one of item A, item B, and item C" may mean item A; item A and item B; item B; item A, item B, and item C; or item B and item C. In some cases, "at least one of item A, item B, and item C" may mean, for example, but not limited to, two of item A, one of item B, and ten of item C; four of item B and seven of item C; or some other suitable combination.

[0078] Unless otherwise indicated, the terms "first," "second," etc. are used herein merely as labels and are not intended to impose order, position, or hierarchical requirements on the items to which these terms refer. Furthermore, reference to, for example, a "second" item does not require or preclude the presence of, for example, a "first" or lower numbered item and / or, for example, a "third" or higher numbered item.

[0079] As used herein, a system, device, structure, article, element, component, or hardware that is "configured to" perform a specified function is actually capable of performing the specified function without any changes, rather than merely having the potential to perform the specified function after further modification. In other words, for the purpose of performing the specified function, a system, device, structure, article, element, component, or hardware that is "configured to" perform a specified function is specifically selected, created, implemented, utilized, programmed, and / or designed. As used herein, "configured to" represents an existing characteristic of a system, device, structure, article, element, component, or hardware that enables the system, device, structure, article, element, component, or hardware to perform the specified function without further modification. For the purposes of this disclosure, a system, device, structure, article, element, component, or hardware that is described as "configured to" perform a particular function may additionally or alternatively be described as "suitable for" and / or "operable to" perform that function.

[0080] The schematic flow chart included herein is generally described as a logical flow chart. Therefore, the order depicted and the steps of the marking indicate an example of the method presented. It is conceivable that other steps and methods that are equivalent to one or more steps or parts thereof of the method shown in function, logic or effect. In addition, the format and symbols adopted are provided to explain the logical steps of the method, and are understood to not limit the scope of the method. Although various arrow types and line types can be adopted in the flow chart, they are understood to not limit the scope of the corresponding method. In fact, some arrows or other connectors can be used to indicate only the logical flow of the method. For example, an arrow can indicate a waiting or monitoring period of unspecified duration between the enumeration steps of the method depicted. In addition, the order in which a particular method occurs may or may not strictly follow the order of the corresponding steps shown.

[0081] Those skilled in the art will recognize that at least a portion of the controllers, devices, units and / or processes described herein can be integrated into a data processing system. Those skilled in the art will recognize that a data processing system typically includes one or more of the following: a system unit housing, a video display device, a memory such as a volatile or non-volatile memory, a processor such as a microprocessor or a digital signal processor, a computing entity such as an operating system, a driver, a graphical user interface and an application program, one or more interactive devices (e.g., a touch pad, a touch screen, an antenna, etc.) and / or a control system including a feedback loop and a control motor (e.g., feedback for sensing position and / or velocity; a control motor for moving and / or adjusting components and / or quantities). The data processing system can be implemented using suitable commercially available components, such as those typically found in data computing / communication and / or network computing / communication systems.

[0082] As used in the foregoing / below disclosure, the term controller / processor may refer to a collection of one or more components arranged in a particular manner, or a collection of one or more general components that may be configured to operate in a particular manner at one or more particular points in time and / or also configured to operate in one or more additional ways at one or more additional points in time. For example, the same hardware or the same portion of hardware may be configured / reconfigured in sequential / parallel time as a first type of controller (e.g., at a first time), a second type of controller (e.g., at a second time, which in some cases may coincide with, overlap, or follow the first time), and / or a third type of controller (e.g., at a third time, which in some cases may coincide with, overlap, or follow the first time and / or the second time), etc. The transformation of the reconfigurable and / or controllable components may occur in as little as a few nanoseconds, or may occur over a period of minutes, hours, or days.

[0083] In some such examples, when a controller is configured to perform a second purpose, the controller may no longer be able to perform the first purpose until it is reconfigured. The controller can switch between configurations as different components / modules in as little as a few nanoseconds. The controller can be reconfigured on the fly, for example, reconfiguration of a controller from a first controller to a second controller can occur when the second controller is needed. The controller can be reconfigured in stages, for example, even before the first controller has completed its operation, parts of the first controller that are no longer needed can be reconfigured into the second controller. Such reconfiguration can occur automatically, or can occur by prompting an external source, whether the source is another component, instruction, signal, condition, external stimulus, etc.

[0084] For example, the central processing unit / processor of the controller, etc., may be operated at various times as a component / module for displaying graphics on a screen, a component / module for writing data to a storage medium, a component / module for receiving user input, and a component / module for multiplying two large prime numbers by configuring its logic gates according to its instructions. This reconfiguration may be invisible to the naked eye, and in some embodiments may include activating, deactivating, and / or rewiring various parts of the component, such as switches, logic gates, inputs, and / or outputs. Therefore, in the examples found in the aforementioned / below disclosure, if the example includes or describes multiple components / modules, the example includes the possibility that the same hardware can implement more than one of the described components / modules simultaneously or at discrete times or timings. The implementation of multiple components / modules, whether using more components / modules, fewer components / modules, or the same number of components / modules as the number of components / modules, is merely an implementation choice and generally does not affect the operation of the components / modules themselves. Thus, it should be understood that any recitation of multiple discrete components / modules in the present disclosure includes implementing those components / modules as any number of underlying components / modules, including but not limited to a single component / module that reconfigures itself over time to perform the functions of multiple components / modules, and / or multiple components / modules that are similarly reconfigured, and / or dedicated reconfigurable components / modules.

[0085] In some cases, one or more components may be referred to herein as being "configured to," "configured by," "configurable to," "operable / operable to," "adapted / adaptable," "capable of," "conform / conform to," etc.

[0086] The foregoing detailed description has set forth various embodiments of the apparatus and / or process by using block diagrams, flow charts, and / or examples. Where such block diagrams, flow charts, and / or examples contain one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation within such block diagrams, flow charts, or examples may be implemented individually and / or collectively by a wide range of hardware, software (e.g., a high-level computer program used as a hardware specification), firmware, or nearly any combination thereof, subject to patentable subject matter. In embodiments, portions of the subject matter described herein may be implemented via an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or other integrated format. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein may be equivalently implemented in whole or in part in an integrated circuit, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as nearly any combination thereof, limited to patentable subject matter, and that designing circuits and / or writing code for software (e.g., a high-level computer program used as a hardware specification) and / or firmware will be within the skills of those skilled in the art in light of the present disclosure. Furthermore, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed in a variety of forms as a program product, and that the illustrative embodiments of the subject matter described herein apply regardless of the particular type of signal-bearing medium used to actually carry out the distribution. Examples of signal-bearing media include, but are not limited to, the following: recordable media such as floppy disks, hard drives, compact disks (CDs), digital video disks (DVDs), digital tapes, computer memory, etc.; and transmission-type media such as digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links (e.g., transmitters, receivers, transmission logic, reception logic, etc.), etc.).

[0087] With regard to the appended claims, it will be understood by those skilled in the art that the operations described therein can generally be performed in any order. Moreover, although various operational flows are presented in sequence, it should be understood that various operations can be performed in an order other than the order shown or can be performed simultaneously. Unless the context otherwise provides, examples of such alternative sorting may include overlapping, interlaced, interrupted, reordered, incremental, prepared, supplemented, simultaneous, reverse or other variant sorting. In addition, terms such as "responsive to", "related to" or other past adjectives are generally not intended to exclude such variants, unless the context otherwise provides. Without departing from the spirit or essential features of this subject, this subject may be embodied in other specific forms. The described examples are considered to be illustrative and non-restrictive in all respects. All changes falling within the meaning and scope of the equivalents of the claims should be included within their scope.

Claims

1. A method, the method include: receiving a two-dimensional image (200) from a camera (120); Determining two-dimensional key points (210, 220) of the target object (202, 204) located within the two-dimensional image (200) based on a predetermined model of the target object (202, 204); estimating a 6-DOF pose based on the 2D key points (210, 220) and a 3D model of the target object (202, 204); generating an uncertainty value for the 6-DOF posture; and Output the uncertainty value of the 6-DOF posture.

2. The method according to claim 1, in, Determining the two-dimensional keypoints (210, 220) is also based on a trained neural network configured to output a keypoint heat map (402), wherein a pixel intensity value associated with each of the keypoint heat maps indicates a keypoint detection probability.

3. The method according to claim 2, further comprising: include: Calculating the covariance of each of the keypoint heatmaps; Calculate the reprojection error; scaling the covariance of the keypoint heatmap based on the reprojection error to produce a scaled covariance; generating samples of new keypoints based on the scaled covariance; as well as Generate a new 6-DOF pose based on the sample of the new key points; The generating of the uncertainty value comprises: calculating the standard deviation of the new 6-DOF posture.

4. The method according to claim 3, in, Calculating the reprojection error includes: generating 2D keypoints from the 6-DOF pose to produce reprojected 2D keypoints; and The two-dimensional keypoints are compared to the reprojected two-dimensional keypoints.

5. The method according to claim 3, in, Generating the uncertainty value also includes applying a smoothing algorithm to the standard deviation.

6. The method according to claim 5, in, The smoothing algorithm includes a Kalman filter.

7. The method according to claim 1, in, Outputting the uncertainty value of the 6-DOF attitude further includes outputting the uncertainty value of the 6-DOF attitude to an automatic refueling system (112), a casing operator system (110), or a pilot guidance light system (108).

8. A fuel dispenser (100), the fuel dispenser include: Refueling sleeve (204); a camera (120) configured to generate a two-dimensional image (200) of an in-flight refueling operation between a receiving aircraft (202) and the refueling aircraft (100); a processor (104); and A non-transitory computer-readable storage medium (114) storing code executable by the processor (104) to perform operations including: Determining two-dimensional key points (210, 220) of the target object (202, 204) located within the two-dimensional image (200) based on a predetermined model of the target object (202, 204); estimating a 6-DOF posture based on the 2D key points (210, 220) and a 3D model of the target object (202, 204); generating an uncertainty value for the 6-DOF posture; and Output the uncertainty value of the 6-DOF posture.

9. The fuel dispenser (100) according to claim 8, in, Determining the two-dimensional keypoints (210, 220) is also based on a trained neural network configured to output a keypoint heat map (402), wherein a pixel intensity value associated with each of the keypoint heat maps indicates a keypoint detection probability.

10. The fuel dispenser (100) according to claim 9, in, The operations also include: Calculating the covariance of each of the keypoint heatmaps; Calculate the reprojection error; scaling a covariance of the keypoint heatmap based on the reprojection error to produce a scaled covariance; generating samples of new keypoints based on the scaled covariance; and Generate a new 6-DOF pose based on the sample of the new key points; Wherein, generating the uncertainty value comprises: calculating the standard deviation of the new 6-DOF posture.

11. The fuel dispenser (100) according to claim 10, in, Calculating the reprojection error includes: generating 2D keypoints from the 6-DOF pose to produce reprojected 2D keypoints; and The two-dimensional keypoints are compared to the reprojected two-dimensional keypoints.

12. The fuel dispenser (100) according to claim 10, in, Generating the uncertainty value by calculating a standard deviation includes applying a smoothing algorithm to the standard deviation.

13. The fuel dispenser (100) according to claim 12, in, The smoothing algorithm includes a Kalman filter.

14. The fuel dispenser (100) according to claim 8, in: The fuel dispenser (100) further comprises: Automatic refueling system (112); a casing operator system (110); or a pilot guidance light system (108); and Outputting the uncertainty value and the 6-DOF attitude further includes: outputting the uncertainty value and the 6-DOF attitude to the automatic refueling system (112), the casing operator system (110), or the pilot guidance light system (108).

15. A system (102), wherein: include: a camera (120) configured to generate a two-dimensional image (200) of an in-flight refueling operation between a receiving aircraft (202) and a refueling aircraft (100); a processor (104); and A non-transitory computer-readable storage medium (114) storing code executable by the processor to perform operations including: Determining two-dimensional key points (210, 220) of the target object (202, 204) located within the two-dimensional image (200) based on a predetermined model of the target object (202, 204); estimating a 6-DOF posture based on the 2D key points (210, 220) and a 3D model of the target object (202, 204); generating an uncertainty value for the 6-DOF posture; and Output the uncertainty value of the 6-DOF posture.

16. The system (102) according to claim 15, in, Determining the two-dimensional keypoints (210, 220) is also based on a trained neural network configured to output a keypoint heat map (402), wherein a pixel intensity value associated with each of the keypoint heat maps indicates a keypoint detection probability.

17. The system (102) according to claim 16, in, The operations also include: Calculating the covariance of each of the keypoint heatmaps; Calculate the reprojection error; scaling a covariance of the keypoint heatmap based on the reprojection error to produce a scaled covariance; generating samples of new keypoints based on the scaled covariance; and Generate a new 6-DOF pose based on the sample of the new key points; Wherein, generating the uncertainty value comprises: calculating the standard deviation of the new 6-DOF posture.

18. The system (102) according to claim 17, in, Calculating the reprojection error includes: generating 2D keypoints from the 6-DOF pose to produce reprojected 2D keypoints; and The two-dimensional keypoints are compared to the reprojected two-dimensional keypoints.

19. The system (102) of claim 17, in, Generating the uncertainty value further includes applying a Kalman filter to the standard deviation.

20. The system (102) of claim 15, in, Outputting the uncertainty value of the 6-DOF attitude further includes outputting the uncertainty value of the 6-DOF attitude to an automatic refueling system (112), a casing operator system (110), or a pilot guidance light system (108).