System and method for calculating the 3D position of a filter carrier tip

By installing a camera system and 3D digital model on the refueling machine, combined with deep learning algorithms, tracking the position and attitude of the refueling casing, the accuracy and efficiency of the existing technology of aerial refueling operations are solved, and stability in complex environments is achieved come on.

CN118941929APending Publication Date: 2024-11-12THE BOEING CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410292186.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-09
Filing Date
2024-03-14
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

Existing aerial refueling technology is difficult to achieve accurate and efficient refueling operations, especially in complex environments, such as unfavorable weather conditions.

Method used

By installing a camera system on the tanker, using 3D digital models and image technology, tracking the position of the refueling casing and the end of the casing, combining deep learning algorithms and neural networks, the 6-degree-of-freedom attitude and tip position of the casing are determined to achieve autonomous control by realizing autonomous control. and visual guidance.

Benefits of technology

It improves the accuracy and efficiency of the refueling process, can carry out refueling operations stably in complex environments, and enhances the system's autonomous control capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941929A_ABST
    Figure CN118941929A_ABST
Patent Text Reader

Abstract

The invention relates to a system and a method for calculating the 3D position of the tip of a refueling machine sleeve for air refueling. Methods, systems, and aircraft for performing image analysis to assist in refueling operations are disclosed herein. A refueling machine includes a single camera, a refueling sleeve, a camera configured to generate a 2D image of the refueling sleeve, a processor, and a non-transitory computer-readable storage medium storing code. The code is executable by the processor to perform operations including: receiving a 2D image from a single camera; determining a 2D key point of the refueling arm located within the 2D image based on a predetermined point model of the refueling arm; determining key points in the 3D space based on the 2D key points to generate 3D key points; determining a 6DOF attitude by using the 2D key points and the 3D key points; and estimating the position of the tip of the refueling arm based on the 6DOF attitude.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates generally to aerial refueling, and more particularly to controlling aerial refueling operations. Background Art

[0002] Cameras provide information for aerial refueling operations. However, using cameras to accurately and efficiently conduct refueling operations can be difficult when practicing current technology. Summary of the invention

[0003] The subject matter of the present application has been developed in response to the prior art, and in particular, in response to the shortcomings of conventional aerial refueling techniques that have not been fully addressed by the prior art. Therefore, the subject matter of the present application has been developed to provide systems and methods for providing aerial refueling techniques that overcome at least some of the above-mentioned shortcomings of the prior art.

[0004] The following is a non-exhaustive list of examples of subject matter disclosed herein that may or may not be claimed.

[0005] In one example, a method includes: receiving a two-dimensional (2D) image from a single camera of a refueling machine; determining 2D key points of a refueling sleeve located within the 2D image based on a predetermined point model of the refueling sleeve; determining a 6-degree-of-freedom (6DOF) pose using the 2D key points and corresponding three-dimensional (3D) key points; and estimating a position of a tip of the refueling sleeve based on the 6DOF pose.

[0006] In another example, a refueling machine includes a single camera, a refueling sleeve, a camera configured to generate a 2D image of the refueling sleeve, a processor, and a non-transitory computer-readable storage medium storing code. The code is executable by the processor to perform operations including: receiving the 2D image from the single camera; determining 2D key points of the refueling telescopic arm located within the 2D image based on a predetermined point model of the refueling telescopic arm; determining a 6DOF pose using the 2D key points and corresponding 3D key points; and estimating a position of a tip of the refueling telescopic arm based on the 6DOF pose.

[0007] In yet another example, a refueling system includes a processor and a non-transitory computer-readable storage medium storing code. The code is executable by the processor to perform operations including: receiving a 2D image from a single camera; determining 2D key points of the refueling arm based on a predetermined point model of the refueling arm located within the 2D image; determining a 6DOF pose using the 2D key points and corresponding 3D key points; and estimating a position of a tip of the refueling arm based on the 6DOF pose.

[0008] The described features, structures, advantages and / or characteristics of the subject matter of the present disclosure may be combined in any suitable manner in one or more examples and / or implementations. In the following description, many specific details are provided to give a thorough understanding of the examples of the subject matter of the present disclosure. Those skilled in the relevant art will recognize that the subject matter of the present disclosure may be practiced without one or more of the specific features, details, parts, materials and / or methods of a particular example or embodiment. In other cases, additional features and advantages may be recognized in certain examples and / or embodiments that may not be present in all examples or embodiments. In addition, in some cases, known structures, materials or operations are not shown or described in detail to avoid blurring the various aspects of the subject matter of the present disclosure. According to the following description and the appended claims, the features and advantages of the subject matter of the present disclosure will become more completely apparent, or may be learned by the practice of the subject matter as described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order that the advantages of the subject matter may be more readily understood, a more particular description of the subject matter briefly described above will be presented by reference to specific examples shown in the accompanying drawings. It should be understood that these drawings depict only typical examples of the subject matter and that they are not to be considered limiting of its scope. The subject matter will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0010] Figure 1 is a schematic block diagram of a fuel dispenser with an automatic pilot light system according to one or more examples of the present disclosure;

[0011] Figure 2 is a schematic side view of an aircraft refueling operation according to one or more examples of the present disclosure;

[0012] Figure 3 is a representation of an image of an aircraft refueling operation generated by a camera on a tanker according to one or more examples of the present disclosure;

[0013] Figure 4 is a schematic side view of a refueling sleeve in a first orientation according to one or more examples of the present disclosure;

[0014] Figure 5 is a schematic side view of a refueling sleeve in a second orientation according to one or more examples of the present disclosure;

[0015] Figure 6 is a schematic side view of a refueling sleeve in a third orientation according to one or more examples of the present disclosure;

[0016] Figure 7 is a schematic diagram of an intersection of a camera plane and a refueling sleeve extension according to one or more examples of the present disclosure;

[0017] Figure 8 According to one or more examples of the present disclosure Figure 7 A schematic side view of the intersection of ;

[0018] Fig. 9 According to one or more examples of the present disclosure Figure 7 a schematic perspective view of the intersection of ; and

[0019] Fig.10 is a schematic flow chart of a method for determining a cannula tip position according to one or more examples of the present disclosure. DETAILED DESCRIPTION

[0020] References throughout this specification to "one example," "example," or similar language mean that a particular feature, structure, or characteristic described in conjunction with the example is included in at least one example of the present disclosure. The phrases "in one example," "in an example," and similar language throughout this specification may, but do not necessarily, all refer to the same example. Similarly, use of the term "embodiment" means an embodiment having a particular feature, structure, or characteristic described in conjunction with one or more examples of the present disclosure, however, an embodiment may be associated with one or more examples without an explicit correlation to indicate otherwise.

[0021] A refueling system 102 located on a refueling machine 100 is disclosed herein, which tracks a refueling sleeve and sleeve tip of the refueling machine by utilizing control sensors, a three-dimensional (3D) digital model, and images to estimate the position of the sleeve and sleeve tip in space. The tracked information can be used for autonomous control purposes or visual guidance during the refueling process.

[0022] like Figure 1 and Figure 2 As shown, the refueling system 102 includes a processor 104 , a camera system 106 , a guide light system 108 (eg, a guide light system), a cannula operator interface 110 , an automatic refueling system 112 , a cannula tip sensor 116 , and a memory 114 .

[0023] In various embodiments, the camera system 106 includes a camera 120, a video image processor 122, and an image generator 124. The camera 120 is generally mounted to a fixed platform that is located within a cast housing attached to the lower rear fuselage of the tanker 100. The camera 120 includes one or more lenses with remotely operated focus and zoom capabilities. The camera 120 is located in a rear position relative to and below the tanker 100. The video image processor 122 receives a digitized video image from the camera 120 and generates a real-time 2D video image. The digitized video image includes objects viewed within the viewing cone by the camera 120, specifically, the casing (e.g., the refueling casing 204) and the receiving aircraft 202. The image generator 124 then generates an image for presentation on the casing operator's monitor 132.

[0024] refer to Figure 3 , camera system 106 generates a two-dimensional (2D) image 300 of a 3D space that includes at least deployed refueling sleeve 204 ( Figure 2 ) and may also include a receiving machine 202 ( Figure 2 2D image 300 includes an approach area that receiving aircraft 202 enters prior to commencing a refueling operation. Receiving aircraft 202 includes a ferrule nozzle receiver 208 that can be coupled to refueling ferrule 204 to complete a fuel transfer.

[0025] In various embodiments, sleeve operator interface 110 includes user interface device 130 and monitor 132. The images presented on monitor 132 are based on information provided by processor 104. Guide light system 108 includes switch unit 140 and light array 142 (i.e., guide lights). Switch unit 140 controls activation of light array 142 based on information provided by processor 104. Automatic refueling system 112 controls the operation of refueling sleeve 204 and / or refueling machine 100 based on information provided by processor 104.

[0026] It is understood that refueling or closing quarter operations may occur between other vehicles, not just the aircraft depicted (e.g., tanker 100, receiver 202). Refueling or closing quarter operations may occur during adverse weather conditions. Vehicles may be any vehicles that move relative to each other (in water, on land, in the air, or in space). Vehicles may also be manned or unmanned. Given by way of non-limiting example, in various embodiments, the vehicle may be a motorized vehicle driven by wheels and / or tracks, such as, but not limited to, a car, truck, van, etc. Given by way of additional non-limiting example, in various embodiments, the vehicle may include a vessel, such as, but not limited to, a ship, a boat, a submarine, a submersible, an autonomous underwater vehicle (AUV), etc. Given by way of additional non-limiting example, in various embodiments, the vehicle may include other manned or unmanned aircraft, such as, but not limited to, fixed-wing aircraft, rotary-wing aircraft, and lighter-than-air (LTA) aircraft.

[0027] In various embodiments, reference Figure 2 , light array 142 is located on the lower forward fuselage of tanker 100. Light array 142 is positioned to be clearly visible to the pilot of receiving aircraft 202. In addition, light array 142 includes various lights for providing directional information to the pilot of receiving aircraft 202. Light array 142 may include an approach light bar, an elevation light bar, a forward / rear position light bar, four longitudinal reflectors, two transverse reflectors, or other lights.

[0028] In various embodiments, non-transitory computer-readable instructions (i.e., code) stored in memory 114 (i.e., storage medium) enable processor 104 to use raw image data from a single sensor (i.e., camera 120) and make the raw data scalable and cost-effective to integrate into existing systems. Specifically, processor 104 predicts key points 302, 304 of refueling sleeve 204 within 2D image 300 (see, e.g., Figure 3). The key points 302 are referenced in 2D space. The prediction is based on a trained deep neural network that is configured to estimate the pixel locations of the key points of the refueling sleeve 204 in the 2D image 300. The processor 104 then performs a 2D to 3D correspondence by projecting the 2D key points 302 into 3D space using a 3D point matching algorithm. Each of the predicted 2D key points 302 is projected from 2D space to 3D space using a perspective n-point (PnP) pose calculation to produce a prediction of the refueling sleeve 204. The PnP pose calculation solves for the 6 degrees of freedom (DoF) parameters (3D translation and rotation) by optimizing: (1) the 2D predicted key points 302 from the neural network, (2) the 3D corresponding key points from the 3D model, and (3) the camera intrinsic K matrix and distortion (these determine how the 3D world is projected onto the 2D image). In another embodiment, the optimization process iteratively tries different 6DoF solutions until the reprojection error is minimized (which is the difference between (1) above and projecting (2) onto the image using the camera parameters of (3)). The cannula tip 6DOF position (i.e., pose) is then determined using the cannula tip extension values ​​received from the cannula tip sensor 116 and the predicted cannula 6DOF pose in addition to the predicted cannula 6DOF pose or using a deep learning algorithm coupled with ray tracing from the camera 120.

[0029] In various embodiments, four exemplary methods / solutions are provided for estimating the relative position of the cannula tip. The methods / solutions include: (1) ray tracing with 6DOF; (2) cannula tip extension value with 6DOF; (3) performing method / solution (1) with the actual pose that the cannula can be in; and (4) performing method / solution (2) with the actual pose that the cannula can be in.

[0030] In various implementations, processor 104 performs object bounding box detection, keypoint detection, 6DOF pose estimation, and cannula tip 6DOF pose estimation.

[0031] Bounding Box Detection

[0032] In various embodiments, processor 104 generates a bounding box around a region of interest within 2D image 300 generated by camera 120. The region of interest is the area where refueling cannula 204 is most likely to be located based on knowledge of the cannula's dimensions and where camera 120 is pointing. Knowledge of the cannula's dimensions may include information from cannula sensors including cannula tip sensors 115 that may identify cannula locations, such as Figures 4 to 6As shown. The bounding box minimizes the amount of unnecessary features within the 2D image 300 that will be analyzed by the processor 104. The bounding box is first determined using information from sensors associated with the refueling sleeve 204. Using these sensors and the object file of the refueling sleeve 204, the processor 104 estimates the position and orientation of the 3D virtual model of the refueling sleeve 204 in 3D space. The estimated 3D model is then projected onto the image of the virtual camera in 2D space to produce a projected virtual sleeve. A bounding box is then formed around the pixels that contain the projected virtual sleeve. The bounding box includes a buffer to include more of the image in the event that the projected virtual sleeve does not match the real world.

[0033] Additionally, after a period of time has passed and the refueling cannula 204 is being actively tracked, only the estimated position and orientation (i.e., the cannula's 6DoF parameters) may be used to form the bounding box. The cannula sensor of the refueling machine is initially used because there is no initial estimate of the cannula's position in 3D space or 2D space. Once estimation and tracking begins, the cannula's tracked 6DoF parameters may be used to feed back into the bounding box process.

[0034] Keypoint Detection

[0035] In various embodiments, the processor 104 identifies key points 302 on the casing imaged by the camera. Before the processor 104 identifies the key points 302, a convolutional neural network (CNN) is trained to identify features / key points on the 3D virtual / digital model from the 2D image. The CNN is based on a residual network (Res Net) architecture. The CNN removes the final pooling and fully connected layers of the architecture and replaces them with a series of deconvolution or upsampling layers to return an output image that matches the height and width of the input image, where the number of key points matches the number of channels. Each channel is considered to be a heat map of the location of the key point in the 2D image space. From the heat map, the pixel at the center of the distribution represented by the heat map is selected as the location of the key point (i.e., 2D key point prediction).

[0036] In various embodiments, during training of the CNN, the CNN takes as input a rescaled bounding box crop of an image, or in our case a video frame, and returns as output a black and white heatmap image for each keypoint. The pixel values ​​of the heatmap indicate for each keypoint the likelihood of finding the keypoint of the 3D virtual object at each pixel location of the image once the object has been projected onto the image. To train the weights of the CNN, a ground truth heatmap is constructed from the ground truth 2D pixel locations. The pixel values ​​of the ground truth heatmap are assigned values ​​of a Gaussian probability distribution on 2D coordinates with a mean equal to the ground truth 2D pixel location and the covariance remaining as a hyperparameter for training. The loss minimized during training consists of the Jensen-Shannon divergence between the heatmap output of the CNN and the ground truth heatmap and the Euclidean norm between the CNN's 2D keypoint estimate and the ground truth 2D keypoint.

[0037] In various embodiments, the training of the CNN uses a domain randomization (DR) process, in which several 3D enhancements are performed as well as 2D enhancements. The 3D enhancements done are randomly selected cannula positions and orientations, as well as the coating of the object and random materials and the intensity and random positioning of the lighting. The 2D enhancements perform random discarding, blurring, lighting and contrast changes of the image. By adjusting all of these 3D and 2D parameters, a wide range of views of what the cannula may look like in different positions, materials and lighting is obtained.

[0038] refer to Figure 3 , in one example, thirty-seven cannula key points (KPs) + one cannula tip KP (for a total of thirty-eight KPs) are determined. The first key point has a first color indicating that the key point is fixed / static and is used to solve PnP. The second key point has a second color indicating that the key point is located on a portion of the cannula that can move and may not be used in PnP. The third key point has a third color indicating that the key point is located on a portion of the cannula that can move some (but not a significant amount). The cannula tip is included as a key point for 3D cannula tip estimation, but it is not used in the PnP solution for cannula 6DOF estimation.

[0039] 6DOF Pose Estimation

[0040] Processor 104 converts the 2D keypoint predictions into 6DOF estimates. The first method uses only the PnP algorithm, and the second method solves PnP and then solves the control fitting algorithm. Because both methods use PnP, they use the correspondence between the 2D pixel positions of the detected keypoints and the 3D keypoint positions on the object model to rotate and position the object in space so that the camera's view of the 3D keypoints matches the 2D pixel positions. In various embodiments, a random sample consensus (RANSAC) PnP algorithm (e.g., OpenCV) is used.

[0041] The control fitting algorithm receives the PnP 6DOF solution and performs further optimization to find a physically feasible 6DOF solution (i.e., true pose) that takes into account the fixed points between the cannula and the tanker. An inverse calculation is used to convert the 6DOF solution to estimated input cannula control parameters, using the robot model, camera to cannula external features, and triangulation. Once the control parameters are estimated, a least squares fit is used to optimize the most likely set of input control parameters. From there, a Kalman filter is used to help smooth the results.

[0042] Typically, the inverse calculation depends on the kinematics of the robotic arm. In this case, the robotic arm is a cannula. In order to derive the kinematic equations to describe the movement of the cannula, there are 4 cannula control parameters (2 for pitch, 1 for roll, and 1 for telescopic extension). The first three are revolute joints, and the last three are prismatic joints. The kinematics of these joints are guided by the 3D transformation (hence the "camera to cannula" non-intrinsic and "triangulation") to describe how 3D points along the cannula are transformed in 3D space when the cannula control parameters change. This is the so-called forward calculation of the kinematics. Therefore, the inverse calculation tries to solve for the cannula control parameters that give a specific 3D keypoint projection in the 2D image.

[0043] There is a way to project 3D points on the casing onto a 2D image based on the casing control parameters (forward calculation), and a way to recover the casing control parameters based on the projected 3D points (reverse calculation). These calculations are used for the control fitting algorithm. The control fitting algorithm attempts to update the 6DOF casing pose solution from PnP so that it is physically achievable. Because the casing is fixed to the tanker, it can only move in certain ways based on its kinematics. This limits the set of 6DOF poses that the casing can be in. PnP does not take these restrictions into account, so the point of the control algorithm is to take these restrictions into account in order to produce a more accurate 6DOF pose answer. It iteratively solves for a set of casing control parameters via least squares optimization that minimizes the error between the 2D predicted key points from the neural network and the projected key points from a physically possible set of casing control parameters. The initial condition for the optimization is the 6DOF pose solution from PnP.

[0044] Cannula tip position estimation

[0045] In various embodiments, after the processor 104 uses the 6DOF of the cannula as described above, the processor 104 determines the 3D position of the cannula tip. Because the cannula tip is only one point, it is not possible to perform PnP on it alone. PnP requires multiple points to be valid. The following are two methods to find the 3D position of the cannula tip.

[0046] Sensor extension method

[0047] The default cannula tip 3D position in the model space is adjusted by the cannula extension parameters generated by the cannula tip sensor 116. The extended cannula tip 3D position is then rotated and translated based on the predicted cannula 6DOF coordinate transformation.

[0048] Ray tracing method

[0049] In various embodiments, the processor 104 receives the 6DOF estimate and additional calculations using the 2D keypoint prediction. The cannula tip 708 is one of the 2D keypoints. The 2D cannula tip 708 is used as a basis for projecting the plane 704 outward from the origin of the camera 120 into space in order to find the 3D cannula tip estimate (point) 706, which is the intersection of the plane 704 and the cannula telescope ray 702. In various embodiments, the orientation of the plane 704 is determined using several keypoints 302 found from the CNN. Figures 7 to 9 Different views (plan view 700, side view 800, perspective view 900) of a plane 704 intersecting a 3D point 706 are shown. In the plan view 700, the slope of the 2D key point 302 is found and used to determine the slope of the plane 704. The cannula tip 708 is found in 3D at some arbitrary distance using inverse camera projection along with another arbitrary point in 2D on the plane 704. These 3D points together with the origin of the camera 120 parameterize the plane 704 in 3D. Using 6DOF estimation, the cannula orientation and position in space are known. The processor 104 then finds the intersection 706 of the vector / ray 702 extended from the cannula with the plane 704 from the origin of the camera. The intersection 706 is the 3D position estimate of the cannula tip. The 3D position of the cannula tip generated by the ray tracing method or the extension method is used to automatically send instructions to the receiving aircraft or the cannula operator system or automatically control the automatic refueling system.

[0050] refer to Fig.10, method 1000 includes a 2D keypoint machine learning method 1002 and a geometric optimization method 1004. Block 1006 of the 2D keypoint machine learning method 1002 includes generating a bounding box using casing computer-aided design (CAD) model (i.e., digital model) information from a database 1008 and casing control angle values ​​sensed by a casing sensor 1010. Block 1012 of the 2D keypoint machine learning method 1002 includes detecting keypoints using a deep learning neural network based on simulated training data (RGB image data) 1014, domain randomization 1016, data preprocessing 1018, and keypoint model training 1020. Data preprocessing 1018 includes normalizing data samples by the mean and standard deviation of the entire input data set, reorganizing channels of data into a format typical for a neural network (e.g., number of images in a batch, number of channels, image height, image width), moving data from a CPU cache to a GPU, etc. In the geometry optimization method 1004, block 1022 includes computing a 2D to 3D transform using PnP and a cannula CAD (digital) model 1024. Block 1026 of the geometry optimization method 1004 includes controlling fitting of cannula parameters to cannula external features 1028 using a stored cannula robot model and a camera. Block 1026 of the control fitting (optional) is where the initial pose guess from block 1022 is further refined to obtain a true pose (i.e., the cannula is fixed to the fuel dispenser and within a set of poses physically achievable by the cannula robot model, also referred to as the kinematics of the cannula). Block 1030 of the geometry optimization method 1004 includes performing cannula tip estimation using cannula tip extension input or ray tracing to output a 3D cannula tip value and / or a 3D model projection for visualization.

[0051] The following is a non-exhaustive list of examples of subject matter disclosed herein that may or may not be claimed.

[0052] The following portion of this paragraph describes Example 1 of the subject matter disclosed herein. According to Example 1, a method includes: receiving a 2D image from a single camera of a refueling machine; determining 2D key points of a refueling sleeve located within the 2D image based on a predetermined point model of the refueling sleeve; determining a 6DOF pose using the 2D key points and corresponding 3D key points; and estimating a position of a tip of the refueling sleeve based on the 6DOF pose.

[0053] The following portion of this paragraph depicts Example 2 of the subject matter disclosed herein. According to Example 2 including Example 1 above, determining the 6DOF pose includes performing a perspective n-point algorithm using the 2D keypoints and the 3D keypoints to produce the 6DOF pose.

[0054] The following portion of this paragraph depicts Example 3 of the subject matter disclosed herein. According to Example 3 including any of Examples 1 or 2 above, estimating the position of the tip of the refueling sleeve includes receiving a sleeve tip extension value and estimating the position of the tip of the refueling sleeve based on the sleeve tip extension value.

[0055] The following portion of this paragraph depicts Example 4 of the subject matter disclosed herein. According to Example 4, which includes any one of Examples 1 to 3 above, estimating the position of the tip of the refueling sleeve includes receiving ray tracing values ​​of the tip of the refueling sleeve, and estimating the position of the tip of the refueling sleeve based on the ray tracing values.

[0056] The following portion of this paragraph describes Example 5 of the subject matter disclosed herein. According to Example 5, which includes Example 4 above, receiving ray tracing values ​​includes: identifying a cannula tip keypoint within a 2D image; generating a 3D cannula tip keypoint based on the cannula tip keypoint; identifying a horizontal plane between the camera and the 3D cannula tip keypoint; and generating ray tracing values ​​by identifying an intersection of a longitudinal cannula axis and the horizontal plane.

[0057] The following portion of this paragraph depicts Example 6 of the subject matter disclosed herein. According to Example 6, including any of Examples 1 to 5 above, the method further includes sending the position of the tip of the refueling sleeve to an output device.

[0058] The following portion of this paragraph depicts Example 7 of the subject matter disclosed herein. According to Example 7 including Example 6 above, the output device includes an automatic refueling system, a pilot guidance light system, or a casing operator system.

[0059] The following part of this paragraph depicts Example 8 of the subject matter disclosed herein. According to Example 8, a refueling machine includes a single camera, a refueling sleeve, a camera configured to generate a 2D image of the refueling sleeve, a processor, and a non-transitory computer-readable storage medium storing code. The code can be executed by the processor to perform operations including: receiving the 2D image from the single camera; determining 2D key points of the refueling telescopic arm located within the 2D image based on a predetermined point model of the refueling telescopic arm; determining a 6DOF pose using the 2D key points and corresponding 3D key points; and estimating a position of a tip of the refueling telescopic arm based on the 6DOF pose.

[0060] The following portion of this paragraph depicts Example 9 of the subject matter disclosed herein. According to Example 9 including Example 8 above, transforming the 2D keypoints to 3D space includes performing a perspective n-point algorithm using the 2D keypoints and the 3D keypoints to produce a 6DOF pose.

[0061] The following portion of this paragraph depicts Example 10 of the subject matter disclosed herein. According to Example 10, which includes any of Examples 8 or 9 above, the fuel dispenser further includes a sleeve tip sensor, and estimating the position of the tip of the fueling sleeve includes receiving a sleeve tip extension value from the sleeve tip sensor and estimating the position of the tip of the fueling sleeve based on the sleeve tip extension value.

[0062] The following portion of this paragraph depicts Example 11 of the subject matter disclosed herein. According to Example 11, which includes any of Examples 8 to 10 above, estimating the position of the tip of the refueling sleeve includes receiving ray tracing values ​​of the tip of the refueling sleeve, and estimating the position of the tip of the refueling sleeve based on the ray tracing values.

[0063] The following portion of this paragraph describes Example 12 of the subject matter disclosed herein. According to Example 12 including Example 11 above, receiving ray tracing values ​​includes: identifying a cannula tip keypoint within a 2D image to generate a 3D cannula tip keypoint; identifying a horizontal plane between the camera and the 3D cannula tip keypoint; and generating ray tracing values ​​by identifying an intersection of a longitudinal cannula axis and the horizontal plane.

[0064] The following portion of this paragraph depicts Example 13 of the subject matter disclosed herein. According to Example 13, which includes any of Examples 8 to 12 above, the fuel dispenser further includes an output device, and the processor further performs the following operations: sending the position of the tip of the fueling sleeve to the output device.

[0065] The following portion of this paragraph depicts Example 14 of the subject matter disclosed herein. According to Example 14 including Example 13 above, the output device includes an automatic refueling system, a pilot guidance light system, or a casing operator system.

[0066] The following portion of this paragraph depicts Example 15 of the subject matter disclosed herein. According to Example 15, a refueling system includes a processor and a non-transitory computer-readable storage medium storing code. The code is executable by the processor to perform operations including: receiving a 2D image from a single camera; determining 2D key points of the refueling arm based on a predetermined point model of the refueling arm located within the 2D image; determining a 6DOF pose using the 2D key points and corresponding 3D key points; and estimating a position of a tip of the refueling arm based on the 6DOF pose.

[0067] The following portion of this paragraph depicts Example 16 of the subject matter disclosed herein. According to Example 16 including Example 15 above, transforming the 2D keypoints to 3D space includes performing a perspective n-point algorithm using the 2D keypoints and the 3D keypoints to produce a 6DOF pose.

[0068] The following portion of this paragraph depicts Example 17 of the subject matter disclosed herein. According to Example 17, including any of Examples 15 or 16 above, estimating the position of the tip of the refueling sleeve includes receiving a sleeve tip extension value from a sleeve tip sensor and estimating the position of the tip of the refueling sleeve based on the sleeve tip extension value.

[0069] The following portion of this paragraph depicts Example 18 of the subject matter disclosed herein. According to Example 18, which includes any one of Examples 15 to 17 above, estimating the position of the tip of the refueling sleeve includes determining ray tracing values ​​of the tip of the refueling sleeve, and estimating the position of the tip of the refueling sleeve based on the ray tracing values.

[0070] The following portion of this paragraph describes Example 19 of the subject matter disclosed herein. According to Example 19 including Example 18 above, receiving ray tracing values ​​includes: identifying a cannula tip keypoint within a 2D image to generate a 3D cannula tip keypoint; identifying a horizontal plane between the camera and the 3D cannula tip keypoint; and generating ray tracing values ​​by identifying an intersection of a longitudinal cannula axis and the horizontal plane.

[0071] The following portion of this paragraph depicts Example 20 of the subject matter disclosed herein. According to Example 20 including any of Examples 15 to 19 above, the processor further performs the following operations: sending the position of the tip of the refueling cannula to an automatic refueling system, a pilot guidance light system, or a cannula operator system.

[0072] It will be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software or a combination of the two. Some embodiments and implementations are described above according to functions and / or logic block components (or modules) and various processing steps. However, it should be understood that such block components (or modules) can be implemented by any number of hardware, software and / or firmware components configured to perform a specified function. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above according to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art can implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing departure from the scope of the present invention. For example, the implementation of a system or component can use various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices. In addition, it will be appreciated by those skilled in the art that the embodiments described herein are merely exemplary implementations.

[0073] The various illustrative logical blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0074] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, a software module executed by a processor, or a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an ASIC.

[0075] This article can be described technology and technique according to function and / or logic block assembly and with reference to the symbolic representation of operation, processing task and function that can be performed by various computing components or devices. Such operation, task and function are sometimes referred to as being computer-implemented, computerized, software-implemented or computer-implemented. In practice, one or more processor devices can perform the described operation, task and function by manipulating the electrical signal of the data bit at the memory location in the system memory and other processing of the signal. The memory location of the data bit is maintained to be a physical location with a specific electrical, magnetic, optical or organic property corresponding to the data bit. It should be understood that the various block components shown in the figure can be implemented by any number of hardware, software and / or firmware components configured to perform a specified function. For example, the implementation of the system or component can adopt various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which can perform various functions under the control of one or more microprocessors or other control devices.

[0076] In the above description, certain terms may be used, such as "upper", "lower", "up", "lower", "horizontal", "vertical", "left", "right", "upper", "lower", etc. These terms are used to provide some clear descriptions when dealing with relative relationships where applicable. However, these terms are not intended to imply absolute relationships, positions and / or orientations. For example, with respect to an object, an "upper" surface can simply be turned into a "lower" surface by turning the object over. However, it is still the same object. In addition, unless otherwise expressly stated, the terms "include", "comprise", "have" and their variations mean "including but not limited to". Unless otherwise expressly stated, the list of enumerated items does not mean that any or all items are mutually exclusive and / or mutually inclusive. Unless otherwise expressly stated, the terms "one", "an" and "the" also refer to "one or more". In addition, unless otherwise stated, as defined herein, multiple specific features do not necessarily mean the entire set of specific features or each specific feature of a category.

[0077] In addition, examples of an element "coupled" to another element in this specification may include direct and indirect coupling. Direct coupling may be defined as an element that is coupled to another element and in some contact with the other element. Indirect coupling may be defined as a coupling between two elements that are not in direct contact with each other, but have one or more additional elements between the coupled elements. In addition, as used herein, fixing an element to another element may include direct fixing and indirect fixing. In addition, as used herein, "adjacent" does not necessarily mean contact. For example, an element may be adjacent to another element without being in contact with the element.

[0078] As used herein, when used with a list of items, the phrase "at least one of" means that different combinations of one or more of the listed items may be used, and only one of the items in the list may be required. An item may be a specific object, thing, or category. In other words, "at least one of" means that any combination of the items or number of items in the list may be used, but all of the items in the list may not be required. For example, "at least one of item A, item B, and item C" may mean item A; item A and item B; item B; item A, item B, and item C; or item B and item C. In some cases, "at least one of item A, item B, and item C" may mean, for example, but not limited to, two of item A, one of item B, and ten of item C; four of item B and seven of item C; or some other suitable combination.

[0079] Unless otherwise indicated, the terms "first," "second," etc. are used herein merely as labels and are not intended to impose order, position, or hierarchical requirements on the items to which these terms refer. Furthermore, reference to, for example, a "second" item does not require or preclude the presence of, for example, a "first" or lower numbered item and / or, for example, a "third" or higher numbered item.

[0080] As used herein, a system, device, structure, article, element, component, or hardware that is "configured to" perform a specified function is actually capable of performing the specified function without any changes, rather than merely having the potential to perform the specified function after further modification. In other words, for the purpose of performing the specified function, a system, device, structure, article, element, component, or hardware that is "configured to" perform a specified function is specifically selected, created, implemented, utilized, programmed, and / or designed. As used herein, "configured to" represents an existing characteristic of a system, device, structure, article, element, component, or hardware that enables the system, device, structure, article, element, component, or hardware to perform the specified function without further modification. For the purposes of this disclosure, a system, device, structure, article, element, component, or hardware that is described as "configured to" perform a particular function may additionally or alternatively be described as "suitable for" and / or "operable to" perform that function.

[0081] The schematic flow chart included herein is generally described as a logical flow chart. Therefore, the order depicted and the steps of the marking indicate an example of the method presented. It is conceivable that other steps and methods that are equivalent to one or more steps or parts thereof of the method shown in function, logic or effect. In addition, the format and symbols adopted are provided to explain the logical steps of the method, and are understood to not limit the scope of the method. Although various arrow types and line types can be adopted in the flow chart, they are understood to not limit the scope of the corresponding method. In fact, some arrows or other connectors can be used to indicate only the logical flow of the method. For example, an arrow can indicate a waiting or monitoring period of unspecified duration between the enumeration steps of the method depicted. In addition, the order in which a particular method occurs may or may not strictly follow the order of the corresponding steps shown.

[0082] Those skilled in the art will recognize that at least a portion of the controllers, devices, units and / or processes described herein can be integrated into a data processing system. Those skilled in the art will recognize that a data processing system typically includes one or more of the following: a system unit housing, a video display device, a memory such as a volatile or non-volatile memory, a processor such as a microprocessor or a digital signal processor, a computing entity such as an operating system, a driver, a graphical user interface and an application program, one or more interactive devices (e.g., a touch pad, a touch screen, an antenna, etc.) and / or a control system including a feedback loop and a control motor (e.g., feedback for sensing position and / or velocity; a control motor for moving and / or adjusting components and / or quantities). The data processing system can be implemented using suitable commercially available components, such as those typically found in data computing / communication and / or network computing / communication systems.

[0083] As used in the foregoing / below disclosure, the term controller / processor may refer to a collection of one or more components arranged in a particular manner, or a collection of one or more general components that may be configured to operate in a particular manner at one or more particular points in time and / or also configured to operate in one or more additional ways at one or more additional points in time. For example, the same hardware or the same portion of hardware may be configured / reconfigured in sequential / parallel time as a first type of controller (e.g., at a first time), a second type of controller (e.g., at a second time, which in some cases may coincide with, overlap, or follow the first time), and / or a third type of controller (e.g., at a third time, which in some cases may coincide with, overlap, or follow the first time and / or the second time), etc. The transformation of the reconfigurable and / or controllable components may occur in as little as a few nanoseconds, or may occur over a period of minutes, hours, or days.

[0084] In some such instances, when a controller is configured to perform a second purpose, the controller may no longer be able to perform the first purpose until it is reconfigured. The controller can switch between configurations as different components / modules in as little as a few nanoseconds. The controller can be reconfigured on the fly, for example, reconfiguration of a controller from a first controller to a second controller can occur when the second controller is needed. The controller can be reconfigured in stages, for example, even before the first controller has completed its operation, parts of the first controller that are no longer needed can be reconfigured into the second controller. Such reconfiguration can occur automatically, or can occur by prompting an external source, whether the source is another component, instruction, signal, condition, external stimulus, etc.

[0085] For example, the central processing unit / processor of the controller, etc., may be operated at various times as a component / module for displaying graphics on a screen, a component / module for writing data to a storage medium, a component / module for receiving user input, and a component / module for multiplying two large prime numbers by configuring its logic gates according to its instructions. This reconfiguration may be invisible to the naked eye, and in some embodiments may include activating, deactivating, and / or rewiring various parts of the component, such as switches, logic gates, inputs, and / or outputs. Therefore, in the examples found in the aforementioned / below disclosure, if the example includes or describes multiple components / modules, the example includes the possibility that the same hardware can implement more than one of the described components / modules simultaneously or at discrete times or timings. The implementation of multiple components / modules, whether using more components / modules, fewer components / modules, or the same number of components / modules as the number of components / modules, is merely an implementation choice and generally does not affect the operation of the components / modules themselves. Thus, it should be understood that any recitation of multiple discrete components / modules in the present disclosure includes implementing those components / modules as any number of underlying components / modules, including but not limited to a single component / module that reconfigures itself over time to perform the functions of multiple components / modules, and / or multiple components / modules that are similarly reconfigured, and / or dedicated reconfigurable components / modules.

[0086] In some cases, one or more components may be referred to herein as being "configured to," "configured by," "configurable to," "operable / operable to," "adapted / adaptable," "capable of," "conform / conform to," etc.

[0087] The foregoing detailed description has set forth various embodiments of devices and / or processes through the use of block diagrams, flow charts, and / or examples. Where such block diagrams, flow charts, and / or examples contain one or more functions and / or operations, it will be understood by those skilled in the art that each function and / or operation within such block diagrams, flow charts, or examples may be implemented individually and / or collectively by a wide range of hardware, software (e.g., a high-level computer program used as a hardware specification), firmware, or nearly any combination thereof, limited to patentable subject matter under 35 U.S.C. 101. In embodiments, portions of the subject matter described herein may be implemented via an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or other integrated format. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein may be equivalently implemented in whole or in part in an integrated circuit, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as nearly any combination thereof, limited to patentable subject matter under 35 U.S.C. 101, and that designing circuits and / or writing code for software (e.g., a high-level computer program used as a hardware specification) and / or firmware will be within the skills of those skilled in the art in light of the present disclosure. Furthermore, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed in a variety of forms as a program product, and that the illustrative embodiments of the subject matter described herein apply regardless of the particular type of signal-bearing medium used to actually carry out the distribution. Examples of signal-bearing media include, but are not limited to, the following: recordable media such as floppy disks, hard drives, compact disks (CDs), digital video disks (DVDs), digital tapes, computer memory, etc.; and transmission-type media such as digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links (e.g., transmitters, receivers, transmission logic, reception logic, etc.), etc.).

[0088] With regard to the appended claims, it will be understood by those skilled in the art that, among others, the operations described can generally be performed in any order. Moreover, although various operational flows are presented in sequence, it should be understood that various operations can be performed in an order other than the order shown or can be performed simultaneously. Unless the context otherwise provides, examples of such alternative sorting may include overlapping, interlaced, interrupted, reordered, incremental, prepared, supplemented, simultaneous, reverse or other variant sorting. In addition, terms such as "responsive to", "related to" or other past adjectives are generally not intended to exclude such variants, unless the context otherwise provides. Without departing from the spirit or essential features of this subject, this subject may be embodied in other specific forms. The described examples are considered to be illustrative and non-restrictive in all respects only. All changes falling within the meaning and scope of the equivalents of the claims should be included within their scope.

Claims

1. A method comprising the following steps: Receiving a two-dimensional image (300) from a camera (120) of a fuel dispenser (100); Determining two-dimensional key points (302) of the refueling sleeve (204) located in the two-dimensional image (300) based on a predetermined point model of the refueling sleeve (204); Determine a 6-DOF posture using the 2D key points (302) and corresponding 3D key points in 3D space; as well as The position of the tip of the refueling sleeve (204) is estimated based on the 6-DOF pose.

2. The method according to claim 1, wherein: Determining the 6-DOF pose includes performing a perspective n-point algorithm using the 2D keypoints (302) and the 3D keypoints to generate the 6-DOF pose.

3. The method according to claim 1, wherein: Estimating the position of the tip of the refueling sleeve (204) includes: receiving a cannula tip extension value; and The position of the tip of the refueling sleeve (204) is estimated based on the sleeve tip expansion value.

4. The method according to claim 1, wherein: Estimating the position of the tip of the refueling sleeve (204) includes: receiving ray tracing values ​​of the tip of the refueling sleeve (204); and The position of the tip of the refueling sleeve (204) is estimated based on the ray tracing values.

5. The method according to claim 4, wherein: Receiving the ray tracing value comprises: identifying a cannula tip keypoint within the two-dimensional image (708); Generating a three-dimensional cannula tip key point (706) based on the cannula tip key point (708); identifying a horizontal plane (704) between the camera (120) and the three-dimensional cannula tip keypoint (706); and The ray tracing values ​​are generated by identifying the intersection of the longitudinal casing axis (702) and the horizontal plane (704).

6. The method according to claim 1, further comprising the steps of: The position of the tip of the refueling sleeve (204) is transmitted to an output device.

7. The method according to claim 6, wherein: The output device includes an automatic refueling system (112), a pilot guidance light system (108), or a casing operator interface (110).

8. A fuel dispenser (100), comprising: Camera(120); A refueling sleeve (204), wherein the camera (120) is configured to generate a two-dimensional image (300) of the refueling sleeve (204); a processor (104); and A non-transitory computer-readable storage medium (114) storing code executable by the processor to perform operations including: receiving the two-dimensional image (300) from the camera (120); Determining two-dimensional key points (302) of the refueling sleeve (204) located in the two-dimensional image (300) based on a predetermined point model of the refueling sleeve (204); Determine a 6-DOF pose using the 2D keypoints (302) and corresponding 3D keypoints in 3D space; and The position of the tip of the refueling sleeve (204) is estimated based on the 6-DOF pose.

9. The fuel dispenser (100) according to claim 8, wherein: Transforming the two-dimensional keypoints (302) into three-dimensional space includes performing a perspective n-point algorithm using the two-dimensional keypoints (302) and the three-dimensional keypoints to generate the six-degree-of-freedom pose.

10. The fuel dispenser (100) according to claim 8, wherein: The fuel dispenser (100) further includes a cannula tip sensor (116); and Estimating the position of the tip of the refueling sleeve (204) includes: receiving a cannula tip extension value from the cannula tip sensor (116); and The position of the tip of the refueling sleeve (204) is estimated based on the sleeve tip expansion value.

11. The fuel dispenser (100) according to claim 8, wherein: Estimating the position of the tip of the refueling sleeve (204) includes: receiving ray tracing values ​​of the tip of the refueling sleeve (204); and The position of the tip of the refueling sleeve (204) is estimated based on the ray tracing values.

12. The fuel dispenser (100) according to claim 11, wherein: Receiving the ray tracing value comprises: identifying cannula tip keypoints (708) within the two-dimensional image to generate three-dimensional cannula tip keypoints (706); identifying a horizontal plane (704) between the camera (120) and the three-dimensional cannula tip keypoint (706); and The ray tracing values ​​are generated by identifying the intersection of the longitudinal casing axis (702) and the horizontal plane (704).

13. The fuel dispenser (100) according to claim 8, wherein: The fuel dispenser (100) further includes an output device; and The processor also transmits the position of the tip of the refueling sleeve to the output device.

14. The fuel dispenser (100) according to claim 13, wherein: The output device includes an automatic refueling system (112), a pilot guidance light system (108), or a casing operator interface (110).

15. A refueling system (102), the refueling system comprising: Processor (104); as well as A non-transitory computer-readable storage medium (114) storing code executable by the processor to perform operations including: receiving a two-dimensional image (300) from a camera (120); Determining two-dimensional key points (302) of the refueling sleeve (204) located in the two-dimensional image (300) based on a predetermined point model of the refueling sleeve (204); Determine a 6-DOF posture using the 2D key points (302) and corresponding 3D key points; as well as The position of the tip of the refueling sleeve (204) is estimated based on the 6-DOF pose.

16. The refueling system (102) of claim 15, wherein: Transforming the two-dimensional keypoints (302) into three-dimensional space includes performing a perspective n-point algorithm using the two-dimensional keypoints (302) and the three-dimensional keypoints to generate the six-degree-of-freedom pose.

17. The refueling system (102) of claim 15, wherein: Estimating the position of the tip of the refueling sleeve (204) includes: receiving a cannula tip extension value from a cannula tip sensor (116); and The position of the tip of the refueling sleeve (204) is estimated based on the sleeve tip expansion value.

18. The refueling system (102) of claim 15, wherein: Estimating the position of the tip of the refueling sleeve (204) includes: determining ray tracing values ​​for the tip of the refueling sleeve (204); and The position of the tip of the refueling sleeve (204) is estimated based on the ray tracing values.

19. The refueling system (102) of claim 18, wherein: Receiving the ray tracing value comprises: identifying cannula tip keypoints (708) within the two-dimensional image to generate three-dimensional cannula tip keypoints (706); identifying a horizontal plane (704) between the camera (120) and the three-dimensional cannula tip keypoint (706); and The ray tracing values ​​are generated by identifying the intersection of the longitudinal casing axis (702) and the horizontal plane (704).

20. The refueling system (102) of claim 15, wherein: The processor also performs the following operations: sending the position of the tip of the refueling sleeve (204) to an automatic refueling system (112), a pilot guidance light system (108), or a sleeve operator interface (110).