Correction of time-consistent estimated position for aerial refueling

Abstract

To provide fuel receptacle position estimation for aerial refueling (derived from aircraft position estimation).SOLUTION: A video stream 200, comprising a plurality of video frames 200a each showing an aircraft 110 to be refueled, is received from a single camera 108. An initial position estimate is determined for the aircraft 110 for the plurality of video frames 200a, generating an estimated flight history for the aircraft 110. The estimated flight history for the aircraft 110 is used to determine a temporally consistent refined position estimate, based on known aircraft flight path trajectories in an aerial refueling setting. A position 422 of a fuel receptacle on the aircraft 110 is determined, based on the refined position estimate for the aircraft 110, and an aerial refueling boom 104 may be controlled to engage the fuel receptacle.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to aerial refueling of aircraft. [Background Art]

[0002] Aerial refueling, which is refueling from an aircraft to another aircraft, is generally manually performed by a highly skilled human refueling boom operator. In some systems, a human operator waits inside the observation window to visually check the refueling boom and the recipient aircraft. This system requires significant additional costs to provide space for accommodating a human operator at the rear of the refueling platform.

[0003] Some refueling systems employ stereovision via dual cameras. A human operator wears dedicated goggles to view images from the dual cameras as three-dimensional (3D) images. As another refueling system, some systems perform distance measurement using light detection and ranging (LIDAR) or radar to assist the operation of a human operator. However, these refueling systems require expensive additional equipment. [Summary of Invention]

[0004] Hereinafter, examples of the present disclosure will be described in detail with reference to the accompanying drawings. The following summary is presented for the purpose of describing embodiments and implementations of the present disclosure. However, this summary is not intended to limit all examples to a specific configuration or operations in a specific order.

[0005] Embodiments of the present disclosure include a solution for aerial refueling, the solution comprising: receiving a video stream comprising a plurality of video frames, each video frame indicating an aircraft to be refueled; identifying an initial estimated position for the aircraft for each of the plurality of video frames, wherein the initial estimated positions for the plurality of video frames constitute the estimated flight history of the aircraft; using an estimation correction unit to identify a temporally consistent estimated position for the aircraft based on at least the estimated flight history of the aircraft; identifying the position of the aircraft's refueling port based on at least the modified estimated position for the aircraft; and controlling an aerial refueling boom to engage with the refueling port based on at least the position of the refueling port. [Brief explanation of the drawing]

[0006] In the detailed description of the embodiments of this disclosure, reference will be made to the accompanying drawings listed below.

[0007] [Figure 1] This figure shows a configuration 100 that employs a time-consistent estimated position correction for aerial refueling, according to one embodiment. [Figure 2A] This figure shows a typical video frame 200a in an aerial refueling operation according to one embodiment. [Figure 2B] This figure shows video frame 200a with annotations. [Figure 3] This is a block diagram showing a computer vision (CV) architecture 300 usable in a configuration 100 according to one embodiment. [Figure 4] Block diagram of an aircraft attitude estimation pipeline 400 according to one embodiment. [Figures 5A-5C] This figure shows translation and rotation for attitude estimation correction according to one embodiment. [Figure 5D-5F] This figure shows the correction of posture estimation according to one embodiment. [Figure 6]This figure shows the neural network (NN) estimation correction unit 600 for temporal refinement. [Figure 7] This figure shows the optimization estimation correction unit 700 for temporal adjustment. [Figure 8A-8B] This figure shows the concept of the feature extraction results according to one embodiment. [Figure 9] Block diagram showing a boom tip attitude estimation pipeline 900 according to one embodiment. [Figure 10] This flowchart 1000 shows a method for estimating attitude during aerial refueling, usable in the arrangement configuration 100 shown in Figure 1, according to one embodiment. [Figure 11] This flowchart 1100 shows an alternative method for estimating the attitude of the refueling port and boom tip during aerial refueling, which can be used in the arrangement configuration 100 shown in Figure 1, according to one embodiment. [Figure 12] This is a block diagram showing a computer device 1200 suitable for implementing various aspects of the present disclosure according to one embodiment. [Figure 13] This is a block diagram showing a method 1300 for manufacturing and using an apparatus employing various aspects of the present disclosure, according to one embodiment. [Figure 14] This is a block diagram showing an apparatus 1400, according to one embodiment, which can suitably employ various aspects of the present disclosure. [Figure 15] This is a schematic perspective view showing a specific flying device 1401 according to one embodiment. [Modes for carrying out the invention]

[0008] Various embodiments are described below with reference to the attached drawings. Identical or similar components are given the same reference numerals throughout the drawings whenever possible. Throughout this disclosure, any references to specific embodiments and designs are for illustrative purposes only and are not intended to limit all embodiments unless otherwise stated.

[0009] The above overview and the detailed descriptions of some embodiments described later will be better understood by referring to the accompanying drawings. In this specification, elements and steps described in the singular form following an indefinite article do not necessarily exclude multiple elements or steps. Furthermore, reference to one embodiment or example does not exclude the existence of other embodiments that include the features described in that embodiment or example. In addition, unless otherwise stated, an embodiment that "includes" or "has" one or more elements having a particular property may further include other elements that do not possess that property.

[0010] The aspects and embodiments described herein relate to the estimated position of the fuel receptacle in aerial refueling (derived from the estimated position of the aircraft). A video stream is received from a single camera, containing multiple video frames, each showing the aircraft being refueled. An initial position estimate of the aircraft is determined for each of the multiple video frames, thereby generating an estimated flight history of the aircraft. Using the estimated flight history of the aircraft, a temporally consistent refined position is determined based on the known aircraft flight trajectory in the aerial refueling setup. The position of the aircraft's fuel receptacle is determined based on the refined position of the aircraft, and the aerial refueling boom is controlled to engage with the fuel receptacle. In some embodiments, determining the refined position from the estimated flight history can be done using a deep learning neural network (NN) or optimization (e.g., bundle tuning).

[0011] Aspects of the present disclosure achieve technical effects of, for example, improved computer operation through improved efficiency of arithmetic hardware and improved resource allocation, compared to conventional systems that rely on, for example, processing of a plurality of different measurement inputs. According to aspects of the present disclosure, the position and orientation of a three-dimensional (3D) object (e.g., an aircraft fuel filler inlet) can be estimated from a video stream acquired by a single camera, and thus, for example, autonomous aerial refueling operations and human-assisted aerial refueling operations can be supported. For example, according to aspects of the present disclosure, for the purpose of automating control of a refueling boom during refueling, the relative position and orientation between the fuel filler inlet of a receiver aircraft and the refueling boom of a refueling platform can be determined. In some embodiments, the position and orientation information is represented by six degrees of freedom (6DoF), comprising a 3D position (x-coordinate, y-coordinate, and z-coordinate) and a 3D orientation (roll, pitch, and yaw).

[0012] Since position tracking occurs incrementally, for each video frame among a plurality of video frames, an initial estimate of the 3D position and 3D orientation of the aircraft (collectively, an initial estimate of the six-degree-of-freedom position) is determined. An estimated flight history of the aircraft is thereby generated, and the generated estimated flight history is passed to an estimate refiner that determines an estimated position corrected to be temporally consistent for the aircraft. Since the position of the fuel filler inlet of the aircraft can be determined from the corrected estimated position of the aircraft, it becomes possible to control the aerial refueling boom to engage with the fuel filler inlet.

[0013] Referring more specifically to the accompanying drawings, FIG. 1 shows an arrangement 100 including a refueling platform 102 and a receiver aircraft 110. Both the refueling platform 102 and the aircraft 110 are examples of a flight device 1401 that will be described in detail with reference to FIGS. 14 and 15. In the arrangement 100, the refueling platform 102 refuels the aircraft 110 using an aerial refueling boom 104. A camera 108 provides a video stream 200 (shown in FIG. 3) that is used for pose estimation.

[0014] Figure 2A shows a typical video frame 200a illustrating an aerial refueling operation. For clarity, Figure 2A shows only the clean version of video frame 200a. In contrast, Figure 2B shows an annotated version of video frame 200a. Video frame 200a shows a cropped image of a portion of the aircraft 110 and the aerial refueling boom 104, for example, within an aircraft bounding box 210. The aircraft bounding box 210 is generated in the initial stages of the aircraft attitude estimation pipeline 400, which will be described later with reference to Figure 4. The refueling port 116 of the aircraft 110 is also indicated by being enclosed by a fiducial marker 118. Video frame 200a further shows the aerial refueling boom 104, and its boom tip 106 is indicated by being enclosed by a boom tip bounding box 206. The boom tip bounding box 206 is generated in the initial stages of the boom tip attitude estimation pipeline 900, which will be described later with reference to Figure 9. During the operation, the aerial refueling boom 104 engages its boom tip 106 with the refueling port 116 to supply fuel to the aircraft 110. The location of the aircraft 110's refueling port 116 is easily identifiable by the image marker 118.

[0015] Figure 3 is a block diagram showing a computer vision (CV) architecture 300 that performs attitude estimation of the fuel inlet and boom tip in the configuration 100. Figure 3 shows the components of architecture 300, while Figures 4, 6, 7, and 9 show details of the components. The operation of architecture 300 will be described in detail with reference to Figure 10 (flowchart 1000). Architecture 300 receives a video stream 200 containing multiple video frames (e.g., video frame 200a, video frame 200b, and video frame 200c) from camera 108. The processing of video frame 200a will be described later. The processing of other video frames in the multiple video frames 200a to 200c is the same as the processing described for video frame 200a.

[0016] Architecture 300 includes an aircraft attitude estimation pipeline 400 shown and described with reference to FIGS. 4, 6 and 7, and a boom tip attitude estimation pipeline 900 shown and described with reference to FIG. 9. The aircraft attitude estimation pipeline 400 outputs a refueling port position 422 (the three-dimensional position and three-dimensional orientation of the refueling port of the aircraft 110) as, for example, a position with six degrees of freedom (DoF). In some embodiments, the aircraft attitude estimation pipeline 400 further outputs an aircraft model projection 424. The boom tip attitude estimation pipeline 900 outputs a boom tip position 922 (the three-dimensional position and three-dimensional orientation of the boom tip) as, for example, a position with six degrees of freedom. In some embodiments, the boom tip attitude estimation pipeline 900 further outputs a boom model projection 924. The refueling port position 422 and the boom tip position 922 are supplied to tracking logic 302 that identifies a distance 304 between the boom tip 106 and the refueling port 116, both shown in FIG. 2B.

[0017] The tracking logic 302 determines the boom control parameters 308. The determined boom control parameters are supplied to the boom control unit 310, which autonomously moves the aerial refueling boom 104 to a position where the boom tip 106 engages with the refueling port 116. In other words, the boom control unit 310 controls the aerial refueling boom 104 to engage with the refueling port 116. In some embodiments, the tracking logic 302 determines whether the control to engage the aerial refueling boom 104 with the refueling port 116 falls within the range of the safety parameters 306, and generates an alert 316 if it falls outside the range. The boom control parameters 308 used herein include variables that describe the movements that the aerial refueling boom 104 can perform (e.g., roll, pitch, yaw, translation, extension, retraction, rotation, etc.), and may also include limits and speeds of such movements. According to the boom control parameter 308, given constraints on the boom's pivot position and the camera's intrinsic and external parameters, the aerial refueling boom 104 can be controlled. For example, it is possible to control how the aerial refueling boom 104 rotates (roll and pitch) and how the connecting portion of the aerial refueling boom 104 extends.

[0018] In some embodiments, the aircraft model projection 424 and / or the boom model projection 924 are supplied to a video compilation unit 312. The video compilation unit overlays the supplied aircraft model projection 424 and / or the boom model projection 924 onto a video frame 200a to generate an overlay image 320. In some embodiments, the overlay image 320 and / or alert 316 are presented to a human operator 314 via a presentation component 1206 (for example, by displaying the overlay image 320 on a video monitor screen). In some embodiments, the human operator 314 inputs boom control parameters 308 using an input / output (I / O) component 1210 (e.g., a joystick, mouse, keyboard, touchscreen, keypad, and / or other input device) to control the aerial refueling boom 104 so that the boom tip 106 is in a position to engage with the refueling port 116.

[0019] Figure 4 is a block diagram showing the aircraft attitude estimation pipeline 400, which constitutes part of the architecture 300. The video frame 200a is supplied to the aircraft bounding box detection unit 402, which identifies the aircraft bounding box 210. In some embodiments, the aircraft bounding box detection unit 402 includes a neural network (NN), such as a deep CNN. In some embodiments, the aircraft bounding box detection unit 402 generates a cropped image 404 by cropping and extracting the region corresponding to the aircraft bounding box 210 from the video frame 200a. Cropping allows subsequent processing to ignore unnecessary parts of the video frame 200a and handle only the portion enclosed by the rectangle. Using only the relevant region is also effective in reducing computation time and allows the use of more computationally intensive algorithms in subsequent processing of the aircraft attitude estimation pipeline 400.

[0020] In some embodiments, filter 406 performs filtering of video data, and for example, a Kalman filter can be used. Kalman filtering is a method for estimating an unknown variable using a series of measurements observed over time. Although the observed measurements contain noise and other errors, it is possible to estimate a more reliable value than when using a single measurement by estimating the joint probability distribution of the variable estimated in each time frame. Therefore, in some embodiments, filter 406 processes multiple video frames (e.g., video frames 200a to 200c).

[0021] The output of filter 406 is supplied to aircraft feature extraction unit 408, which outputs aircraft features 802 (see Figures 8A and 8B). In some embodiments, aircraft feature extraction unit 408 includes a keypoint detection unit, which can be implemented using a neural network such as a residual network (ResNet). Keypoint detection identifies the locations of points on a 3D object in a video frame that can be used for 6-degree-of-freedom position estimation. Keypoints are selected as locations that can be consistently recognized on a 3D object, such as the wingtips or nose of an aircraft.

[0022] The aircraft features 802 are supplied to the aircraft 2D / 3D conversion unit 410. In some embodiments, the aircraft 2D / 3D conversion unit 410 uses a perspective n-point (PnP) algorithm. The PnP algorithm is a method that, given the 3D positions of N points on an object and the 2D projection of these points in a camera image, estimates the orientation of a camera calibrated for that object (e.g., a position with 6 degrees of freedom). In some embodiments, the PnP algorithm uses the correspondence between the 2D pixel positions of detected keypoints (aircraft features 802) and the 3D keypoint positions on the 3D aircraft model 416 to match the 3D keypoints on the aircraft model 416 in the camera view of the simulation with the 2D pixel positions of the detected keypoints.

[0023] The aircraft 2D / 3D conversion unit 410 obtains the world position of an object by utilizing the fact that its angle, external information (extrinsics), and geometry are known at each point in time. Camera parameter information is used, which includes parameters used in a camera model that mathematically describes the relationship between the 3D coordinates of a point in the target scene and the 2D coordinates of the projection position of the light emitted from that point on the image plane. Intrinsic parameters are variables also known as internal parameters, and are variables specific to the camera itself, such as focal length and lens distortion. External parameters are variables also known as external parameters or camera orientation, and are used to describe the transformation between the camera coordinate system and the world coordinate system outside the camera. The camera's external information, resolution, magnification, and other intrinsic information are assumed to be known.

[0024] The aircraft 2D / 3D conversion unit 410 identifies the raw aircraft position and filters it with the filter 412 to generate, for example, an initial estimated position 502 with 6 degrees of freedom (shown as a dotted line in Figures 5A to 5E). The initial estimated position 502 with 6 degrees of freedom is a combination of the initial estimated 3D position and initial estimated 3D orientation of the aircraft 110 (for example, the attitude of the aircraft 110).

[0025] The initial estimated position 502 for video frame 200a is added to a set of initial estimated positions for other video frames in the video stream (e.g., initial estimated position 502b, initial estimated position 502c, initial estimated position 502d), thereby generating the estimated flight history 418 of aircraft 110. The estimation correction unit 430 identifies a temporally consistent corrected estimated position 504 for aircraft 110, based at least on the estimated flight history 418 of aircraft 110. The estimation correction unit 430 generates a temporally consistent corrected estimated position 504 by comparing the known aircraft flight trajectory in the aerial refueling setting (trajectory as viewed from refueling platform 102) with the estimated flight history 418.

[0026] There are multiple architectures that can implement the estimation correction unit 430. Examples include the NN estimation correction unit 600 based on a deep learning NN, as shown in Figure 6, and the optimizer estimate refiner 700, as shown in Figure 7. In other words, the estimation correction unit 430 in some embodiments is implemented using the NN estimation correction unit 600, and the estimation correction unit 430 in some embodiments is implemented using the optimizer estimate refiner 700. In one embodiment, for example, if the root mean square error (RMSE) between the actual fuel filler position at the initial estimated position before correction and the estimated fuel filler position is 2.3 inches, the NN estimation correction unit 600 can improve this RMSE to approximately 1.83 inches, and the optimizer estimate refiner 700 can improve this RMSE to approximately 1.76 inches.

[0027] In some embodiments, instead of directly outputting the corrected estimated position 504, the estimation correction unit 430 outputs correction parameters 440 (shift amounts ΔT and ΔR) used in equations 1 and 2 for calculating the corrected estimated position 504. The output correction parameters 440 include a translation refinement 442 and a rotational refinement 444. The corrected estimated position 504 is determined by applying the correction parameters 440 to the initial estimated position 502, as shown in equation 1 for translational correction and as shown in equation 2 for rotational correction.

number

number

[0028] The 6-degree-of-freedom refueling port position 422 can be determined from the 6-degree-of-freedom aircraft position 420. In other words, if the aircraft position 420 is known, the refueling port position 422 located at a predetermined location on the aircraft 110 can also be determined. In some embodiments, the aircraft position 420 and / or the refueling port position 422 can be further processed using a Kalman filter (a filter that filters over multiple video frames in time).

[0029] The aircraft model projection image 424 can be generated by rendering the aircraft model 416 using the model rendering component 414 according to the corrected estimated position 504. The aircraft model projection image 424 is suitable for overlaying onto other images (e.g., images obtained from video frame 200a), so that the original image is still visible in the overlay image, except for the parts that are obscured by the rendering of the aircraft model. In some embodiments, the corrected estimated position 504 is fed back into the estimated flight history 418.

[0030] Figure 5A shows translational correction in 2D space, where the corrected estimated position 504 is shifted laterally relative to the initial estimated position 502 in the xy image space corresponding to the 2D space. In some embodiments, translational correction is performed in 3D, i.e., along the z axis (where the 2D image space corresponds to the xy axes, and z corresponds to the depth direction in the camera view). As a result, as shown in Figure 5B, the scale of the image space of the initial estimated position 502 is scaled to match the size of the corrected estimated position 504 (as the estimated position moves closer to or further away from the virtual camera in rendering). Figure 5C shows rotational correction, where the corrected estimated position 504 is rotated relative to the initial estimated position 502. Note that rotational correction is 3D rotation (roll, pitch, and yaw). Figure 5D shows aircraft 110 and the initial estimated position 502, Figure 5E shows the initial estimated position 502 and the revised estimated position 504 together, and Figure 5F shows aircraft 110 and the revised estimated position 504.

[0031] Figure 6 shows an NN estimation correction unit 600, which can be selected as one embodiment of the estimation correction unit 430, and which includes an NN 602 (e.g., a deep learning NN). The NN 602 incorporates the estimated flight history 418, including the initial estimated positions 502, 502b, 502c, and 502d, into the history of the time-length index T. The initial estimated position 502 is the initial estimated position at time = t, the initial estimated position 502b is the initial estimated position at time = (t-1), the initial estimated position 502c is the initial estimated position at time = (t-2), and the initial estimated position 502d is the initial estimated position at time = (t-(T-1)). In some embodiments, T = 6. The corrected estimated position 504 is the estimated position obtained by correcting the estimated position at time = t.

[0032] The initial 6-degree-of-freedom estimated positions 502, 502b, 502c, and 502d include 3D estimated position and 3D orientation estimates represented by x, y, z, r, p, and yw, where x, y, and z are position parameters, and r (roll), p (pitch), and yw (yaw) are orientation (attitude) parameters. NN602 outputs correction parameters 440 used to identify the corrected estimated position 504, but in some embodiments, NN602 can directly output the corrected estimated position 504 itself. The corrected estimated position 504 consists of correction parameters x', y', z', r', p', and yw'. In some embodiments, NN602 first calculates ΔR as the quaternion of rotation ΔR q The output is then converted into a rotation matrix using external coordinates.

[0033] Since NN602 needs to be trained on realistic aircraft flight paths, the training data 606 is generated from known aircraft flight paths. The flight model 608 models the flight kinematics of various refueling aircraft platforms and provides ground truth data. Random noise 609 is data for adding random positional deviations and noise to each of the six positional parameters. The training unit 604 uses the training data 606, which is obtained by perturbing the flight position (obtained from the flight model 608) with noise 609, and the ground truth parameters (the correct aircraft position, also obtained from the flight model 608).

[0034] In some embodiments, training-time augmentations of the flight profile are performed to improve training robustness. A first noise component, shown in Equation 3, is added to the aircraft position with 6 degrees of freedom (for example, the initial state is the correct position).

number

[0035] Furthermore, the second noise component shown in Equation 4 adds Gaussian random noise to each of the six position parameters.

number

[0036] There are several implementation options for NN602, including those using multilayer perceptrons (MLPs), CNNs, and long-shortening memory (LSTM) networks. These options are shown as implementations 610, 620, and 630, respectively. In implementation 610, upon receiving input 640 (e.g., flight history 418), it is passed to the flattening component 612. Here, the T×6 dimensional input is processed and propagated into the hidden layer of MLP614, which has 100 neurons, and the final set of six MLP616 forms a single modified output 618.

[0037] In implementation 620, upon receiving input 640, it is passed to CNN622. Here, a one-dimensional (1D) CNN takes the temporal correlation of each of the six position parameters and converts them into (T-1) × 60 activation shapes, which pass through MLP624 to a hidden layer containing 50 neurons, and then to a set of MLP626 to produce a single modified output 628. In implementation 630, upon receiving input 640, it is passed to the Long-Short-Term Memory (LSTM) network 632. The sets of MLP634 and MLP636 are similar to the sets of MLP624 and MLP626, respectively, although their parameters differ slightly, and produce a single modified output 638.

[0038] Figure 7 shows an optimized estimation correction unit 700, which is another option for an embodiment of the estimation correction unit 430. The optimized estimation correction unit 700 uses temporal bundle adjustment (TBA). Bundle adjustment is a method for minimizing the reprojection error between observation points and estimation points on an image, which is expressed as the sum of squares of a nonlinear real-valued function. In bundle adjustment, the initial camera parameter estimation and structural parameter estimation are combined and corrected as a single pair to find the set of parameters that can most accurately predict the position of the observation points in the available image set.

[0039] Each of the video frames 200a to 200d is supplied to the aircraft feature extraction unit 408, where aircraft features (e.g., keypoints) are extracted. For example, aircraft feature 802 is extracted from video frame 200a, aircraft feature 802b from video frame 200b, aircraft feature 802c from video frame 200c, and aircraft feature 802d from video frame 200d. In some embodiments, the aircraft feature extraction unit 408 includes a CNN that detects 2D pixel values ​​that are keypoints on the image of the aircraft 110 in the video frames constituting the video stream 200. The aircraft 2D / 3D conversion unit 410 (e.g., PnP algorithm) obtains an initial 6DoF position estimate. This estimate includes the initial estimated position 502 (already described with reference to Figure 4) for the current (most recent) video frame 200a at time t.

[0040] The initial estimated positions for each preceding video frame 200b-200d (e.g., initial estimated positions 502b, 502c, and 502d) have already been identified. Furthermore, the initial estimated positions for each preceding video frame 200b-200d have been corrected using TBA702, and the corrected estimated positions 504b, 504C, and 504d have already been identified.

[0041] For the current video frame 200a at time t, the initial estimated position 502, along with the corrected estimated positions 504b~502d, is used as the first value to be tried as a solution to the TBA702 minimization objective function. This generates either the corrected parameter 440 or the corrected estimated position 504 as output 710. In one example implementation of TBA702, the optimization objective shown in Equation 5 is used.

number

[0042] The first line of Equation 5 calculates the sum of PnP reprojection losses for the current window out of T frames, weighted by the keypoint confidence value c. The second and third lines of Equation 5 constitute the time bundle adjustment (TBA) portion of the optimization objective function. This portion contains two terms. The first term, listed in the second line of Equation 5, minimizes the PnP reprojection distance between each frame and the preceding frame in a time window. The second term, listed in the third line of Equation 5, minimizes the distance between 3D keypoints in world space, after transformation by the current 6-degree-of-freedom estimated position, between each frame and the preceding frame in a time window. These two terms optimize the set of 6-degree-of-freedom parameters so that the positional displacement of each keypoint between frames is minimized while maintaining the overall shape and geometry of the keypoints in both 2D and 3D. By regularizing the hyperparameters for optimization, the weights of each optimization objective function can be adjusted so that each objective function contributes to the search for a 6-degree-of-freedom solution.

[0043] Figure 8A shows aircraft 110 and key points extracted for aircraft 110 from video frame 200a (or other video frames constituting video stream 200), i.e., aircraft features 802 (white circles). Figure 8B omits aircraft 110 for clarity and shows only aircraft features 802.

[0044] Figure 9 is a block diagram showing the boom tip attitude estimation pipeline 900. Video frame 200a is supplied to a boom tip bounding box detection unit 902 that identifies the boom tip bounding box 206. In some embodiments, the boom tip bounding box detection unit 902 generates a cropped image 904 by cropping and cutting out the region corresponding to the boom tip bounding box 206 from video frame 200a. In some embodiments, the filter 906 filters the video data, and for example, a Kalman filter that processes over multiple video frames (e.g., video frame 200a and multiple additional video frames 200b) can be used.

[0045] In some embodiments, the boom tip bounding box detection unit 902 includes a neural network (NN), such as a deep CNN. The output of the boom tip bounding box detection unit 902 (in some embodiments, the cropped and / or filtered output) is supplied to the boom tip feature extraction unit 908. In some embodiments, the boom tip feature extraction unit 908 includes a keypoint detection unit implemented using a neural network, such as ResNet. The boom tip feature extraction unit 908 outputs a boom tip feature map, which is supplied to the boom tip 2D / 3D conversion unit 910. The boom tip 2D / 3D conversion unit 910 identifies the boom tip position 922, for example, with 6 degrees of freedom. In some embodiments, the identification result is filtered by a filter 912.

[0046] In some embodiments, the boom tip 2D / 3D conversion unit 910 also identifies a boom position 920, which is the position of the 6-degree-of-freedom aerial refueling boom 104. In some embodiments, the boom position 920 is identified using the boom tip position 922 (for the distal end of the aerial refueling boom 104), and for the base end of the aerial refueling boom 104, it is identified using the known information that the base end is in a predetermined position on the refueling platform 102. The boom model projection 924 is generated by rendering the 3D refueling boom model 916 using the model rendering component 914 according to the boom position 920.

[0047] Referring to Figure 10, flowchart 1000 illustrates a method of aerial refueling (e.g., a method for attitude estimation for aerial refueling). In some embodiments, the process shown in Figure 10 is performed, at least in part, by one or more processors 1204 in the computer device 1200 shown in Figure 12 executing instructions 1202a (stored in memory 1202). For example, the estimation correction unit 430 may be created (e.g., trained) and validated in the first computer device 1200 and then deployed to a second (different) computer device 1200 located on the refueling platform 102. Processes 1002 and 1004 are executed before deployment in process 1006.

[0048] The flight profile model is created in process 1002 based on the aircraft flight path observed during aerial refueling operations. In some embodiments, this process includes receiving simulated and / or measured flight profile information. The estimation correction unit 430 is created or trained in process 1004. In some embodiments, process 1004 includes generating training data 606 for NN602 using a simulator that simulates the flight path in an aerial refueling setting to generate pairs of aircraft position and ground truth aircraft data, labeling the aircraft position using the ground truth aircraft data, and training NN602 using the training data 606 by the training unit 604. The estimation correction unit 430 is deployed in process 1006.

[0049] Process 1008 includes selecting an aircraft model 416 based on at least the aircraft 110 to be refueled. In this process, refueling schedule information for aircraft 110, user input, or automatic object recognition can be used. In some embodiments, the aircraft model 416 is a 3D triangular mesh model. Process 1010 includes receiving a video stream 200 containing video frame 200a (and other video frames 200b and 200c from a plurality of video frames 200a to 200c) showing the aircraft 110 to be refueled. In some embodiments, the video stream 200 is supplied by a single camera (e.g., camera 108). In some embodiments, the video stream 200 is a monocular video stream.

[0050] Process 1012 includes identifying the initial estimated position 502 (or initial estimated position 502b, 502c, etc.) of the aircraft 110 for each of the multiple video frames 200a to 200c. In process 1014, the initial estimated positions 502 to 502d for the multiple video frames are collected and combined to create the estimated flight history 418 of the aircraft 110. In some embodiments, each of the initial estimated positions includes an estimated position with 6 degrees of freedom (e.g., 3D estimated position and 3D orientation estimated).

[0051] Process 1016 is a process performed using processes 1018 to 1026 (at least partially performed by the estimation correction unit 430), and identifies an estimated position 504 that has been corrected to be temporally consistent for the aircraft 110, based on at least the estimated flight history 418 of the aircraft 110. The estimation correction unit 430 corrects the estimated flight history 418 according to a known aircraft flight trajectory. In some embodiments, the corrected estimated position 504 includes an estimated position with 6 degrees of freedom. In some embodiments, the estimation correction unit includes a NN 602. In some embodiments, the estimation correction unit 430 includes an optimization unit that uses bundle adjustment. In process 1018, keypoints (aircraft features 802) are extracted from the video stream 200 (e.g., from video frame 200a) using the aircraft feature extraction unit 408 included in the estimation correction unit 430 (e.g., the optimization estimation correction unit 700). In some embodiments, the aircraft feature extraction unit 408 includes a NN such as a CNN.

[0052] In process 1020, the estimation correction unit 430 is used to determine correction parameters 440 (estimated position correction parameters) based on at least the estimated flight history 418 of the aircraft 110. In some embodiments, the correction parameters 440 include a translation correction amount 442 and a rotation correction amount 444. According to some exemplary arrangement configurations 100, the correction parameters 440 can be determined even if part of the aircraft 110 is not visible in the video frame 200a. In process 1022, the rotation correction amount 444 is converted from a rotation quaternion to a rotation matrix, and in process 1024, the correction parameters 440 are applied to the initial estimated position 502 in the estimated flight history 418, thereby generating a corrected estimated position 504 (to be consistent in time). In process 1026, the corrected estimated position 504 is filtered, and in some embodiments, a Kalman filter is used for this filtering.

[0053] In the determination process 1028, it is determined whether to repeat processes 1010 to 1026 again, or whether the required number of repetitions has already been reached. Through this process, the fuel filler port position 422 is identified by repeatedly generating a continuously corrected estimated position 504.

[0054] If flowchart 1000 proceeds to the next step, process 1030 includes identifying the refueling port position 422 of the aircraft 110 based on the modified estimated position 504 for at least the aircraft 110. In process 1032, the position of the boom tip 106 on the aerial refueling boom 104 (e.g., a 6-degree-of-freedom boom tip position 922) is identified. In process 1034, an overlay image 320 is generated. The overlay image 320 includes at least an aircraft model 416 and an aircraft model projection image 424 based on the modified estimated position 504, and a video frame 200a showing the aircraft 110 or the next video frame showing the aircraft 110. In some embodiments, the overlay image 320 further includes at least a refueling boom model 916 and a boom model projection image 924 based on the boom tip position 922. This overlay image 320 is displayed in process 1036.

[0055] In process 1038, the distance 304 between the boom tip 106 and the refueling port 116 is tracked. In determination process 1040, it is determined whether the control to engage the aerial refueling boom 104 with the refueling port 116 falls within the range of safety parameter 306, based at least on the refueling port position 422 and the boom tip position 922. If it does not fall within the range of safety parameter 306, in process 1042, an alert 316 is generated based on the determination that the control to engage the aerial refueling boom 104 with the refueling port 116 is not within the range of safety parameter 306.

[0056] If the safety parameters are within range, in process 1044, the aerial refueling boom 104 is controlled to engage with the refueling port 116 based at least on the refueling port position 422. The control to engage the aerial refueling boom 104 with the refueling port 116 includes controlling the aerial refueling boom 104 to engage with the refueling port 116 based at least on the refueling port position 422 and the boom tip position 922. The flowchart 1000 then returns to process 1010.

[0057] Figure 11 is a flowchart 1100 illustrating an aerial refueling operation. In some embodiments, the process shown in Figure 11 is performed, at least in part, by one or more processors 1204 in the computer device 1200 shown in Figure 12 executing instructions. Process 1102 includes receiving a video stream containing a plurality of video frames, each video frame representing the aircraft to be refueled. Process 1104 includes determining the initial estimated position of the aircraft for each of the plurality of video frames, the initial estimated positions for the plurality of video frames constitute the estimated flight history of the aircraft.

[0058] Process 1106 includes the estimation correction unit identifying an estimated position that has been corrected to be temporally consistent for the aircraft, based on at least the estimated flight history of the aircraft. Process 1108 includes identifying the refueling port position of the aircraft, based on at least the corrected estimated position of the aircraft. Process 1110 includes controlling the aerial refueling boom to engage with the refueling port, based on at least the refueling port position.

[0059] Next, referring to Figure 12, a block diagram of a computer device 1200 suitable for implementing various aspects of this disclosure is shown. In some embodiments, the computer device 1200 includes one or more processors 1204, one or more presentation components 1206, and memory 1202. Embodiments of this disclosure relating to computer device 1200 can be implemented by a variety of computer devices, including, for example, personal computers, laptops, smartphones, mobile tablets, portable devices, consumer electronics, and dedicated computer devices. Categories such as “workstation,” “server,” “laptop,” and “portable device” are not distinguished and all fall within the scope assumed to be “computer device” as described in Figure 12 and herein. Embodiments of this disclosure can also be implemented in a distributed computing environment in which tasks are performed by remote devices connected via a communication network. Furthermore, although computer device 1200 is shown as a single device, in one embodiment, multiple computer devices may work together and share the illustrated resources. For example, in one embodiment, the memory 1202 may be distributed across multiple devices, and the processor 1204 may be included in different devices.

[0060] In one embodiment, memory 1202 includes any computer-readable medium described herein. For example, memory 1202 is used to store and access instructions 1202a configured to perform various operations described herein. In some embodiments, memory 1202 includes computer storage media such as volatile and / or non-volatile memory, removable or non-removable memory, a data disk for a virtual environment, or a combination thereof. In one embodiment, processor 1204 may include several processing units that read data from various elements such as memory 1202 and input / output (I / O) components 1210. Specifically, processor 1204 is programmed to execute computer-executable instructions to implement various aspects of this disclosure. In one embodiment, these instructions are executed by the processor, by multiple processors included in computer device 1200, or by a processor outside of computer device 1200. In some embodiments, processor 1204 is programmed to execute instructions as shown in the illustrated flowchart described later.

[0061] The presentation component 1206 presents data to the operator or other devices. In one embodiment, the presentation component 1206 includes a display device, speaker, printer, vibrator, etc. As those skilled in the art will understand, computer data can be presented in many different ways, for example, visually using a graphical user interface (GUI), audibly via a speaker, wirelessly via a connection to the computer device 1200, via a wired connection, etc. In one embodiment, the presentation component 1206 is not used when the processing and operation are sufficiently automated and human intervention is little or no need. The input / output port 1208 allows the computer device 1200 to be logically connected to other devices, including input / output components 1210, including those built into the computer device. Embodiments of the input / output component 1210 include, but are not limited to, a microphone, keyboard, mouse, joystick, gamepad, satellite receiver antenna, scanner, printer, wireless device, etc.

[0062] Computer device 1200 includes a bus 1216 that directly or indirectly connects memory 1202, one or more processors 1204, one or more presentation components 1206, input / output (I / O) ports 1208, input / output components 1210, power supply 1212, and network components 1214. Computer device 1200 should not be construed as including configurations or associated requirements that depend on any one or combination of components described herein. Bus 1216 represents one or more buses (e.g., an address bus, a data bus, or a combination thereof). In Figure 12, various blocks are shown with lines for clarity, but in some embodiments, the functional boundaries of components shown as different herein are ambiguous.

[0063] In some embodiments, the computer device 1200 is communicably connected to a network 1218 using a network component 1214. In some embodiments, the network component 1214 includes a network interface card and / or computer executable instructions (e.g., drivers) for operating the network interface card. In one embodiment, communication between the computer device 1200 and other devices is performed via a wired or wireless connection 1220 using any protocol or mechanism. In some embodiments, the network component 1214 is operable to perform public, private, or hybrid (public and private) data communication using a transport protocol, or to perform wireless data communication between devices using short-range communication technology (e.g., Near Field Communication (NFC), Bluetooth® standard communication, etc.), or to perform data communication combining these methods.

[0064] Furthermore, while the above description has described examples related to computer device 1200, embodiments of the present disclosure can also be implemented in a variety of other general-purpose or application-specific computer systems, environments, configurations, or devices. Well-known computer systems, computer environments, and / or computer configurations suitable for use in aspects of the present disclosure include, but are not limited to, smartphones, mobile tablets, mobile computer devices, personal computers, server computers, portable or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, and / or computer and communication devices embedded in wearable devices or accessory devices (e.g., watches, glasses, headsets, or earphones), network PCs, minicomputers, mainframe computers, distributed computer environments including any of the systems or devices described above, VR devices, holographic devices, and so on. Such systems and devices accept user input in any manner, such as input from input devices such as keyboards and pointing devices, gesture input, proximity input (e.g., by hovering), and / or voice input.

[0065] Embodiments of this disclosure are described in terms of the general concept of computer executable instructions, and for example, in terms of the concept of program modules executed by one or more computers or other devices implemented in software, firmware, hardware, or a combination thereof. In one example, computer executable instructions are configured as one or more components or modules that can be executed by a computer. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures for performing a particular task or implementing a particular abstract data type. In one example, implementation of an aspect of this disclosure may use any number of such components or modules in any configuration. For example, the aspects of this disclosure are not limited to the specific computer executable instructions or specific components or modules illustrated and described herein. Other embodiments of this disclosure include computer executable instructions with more or fewer functions than those illustrated and described herein. In embodiments using a general-purpose computer, the general-purpose computer is transformed into a computer device that realizes the special use of the aspects of this disclosure by configuring the general-purpose computer to execute the instructions described herein.

[0066] This is merely an example and not an limitation, but computer-readable media include computer storage media and computer communication media. Examples of computer storage media include volatile or non-volatile, removable or non-removable memory configured by any method or technique to store information such as computer-readable instructions, data structures, and program modules. Computer storage media are tangible and mutually exclusive with communication media. Computer storage media are hardware-implemented and exclude carrier waves and propagating signals. Computer storage media in this disclosure are not signals themselves. In one embodiment, the computer storage medium includes, for example, a hard disk, flash drive, fixed memory, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory and other memory technologies, compact disk read-only memory (CD-ROM), digital versatile disc (DVD) and other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disks and other magnetic storage devices, or any other non-transmission medium used to store information in a manner accessible by a computer device. On the other hand, the communication medium includes any information distribution medium, which embodies computer-readable instructions, data structures, program modules, etc., as modulated data signals such as carrier waves and other transmission mechanisms.

[0067] Several embodiments of this disclosure can be used for manufacturing and use applications, as illustrated and described in Figures 13 to 15. Therefore, embodiments of this disclosure will be described in relation to a method of manufacturing and using the apparatus shown in Figure 13, and an apparatus shown in Figure 14, 1400. Figure 14 shows a method of manufacturing and using the apparatus according to an embodiment of this disclosure. In one embodiment, in the pre-manufacturing stage, the method of manufacturing and using the apparatus 1300 includes specification and design 1302 of the apparatus 1400 shown in Figure 14, and material procurement 1304. In the manufacturing stage, the manufacturing 1306 of the components and subassemblies of the apparatus 1400 shown in Figure 14 and system integration 1308 are carried out. Subsequently, the apparatus 1400 shown in Figure 14 undergoes certification and delivery 1310 and enters a period of use 1312. During the period of customer use, the apparatus 1400 shown in Figure 14 is incorporated into routine maintenance and upkeep 1314. In one implementation, routine maintenance and upkeep include improvements, reconfigurations, modifications, and other maintenance or use subject to configuration control as described herein.

[0068] In one embodiment, each process of the method for manufacturing and using the apparatus 1300 is performed or carried out by a system integrator, a third party, and / or an operator. In such an embodiment, the operator is the customer. For illustrative purposes, the system integrator may include several apparatus manufacturers and major system subcontractors. The third party may include several sellers, subcontractors, and suppliers. Also in one embodiment, the operator may be the owner of the apparatus or group of apparatus, the manager responsible for the apparatus or group of apparatus, the user operating the apparatus, a leasing company, a military organization, a service organization, etc.

[0069] Next, referring to Figure 14, the device 1400 is presented. As shown in Figure 14, in one embodiment, the device 1400 is a flying device 1401, such as a spacecraft, aircraft, cargo plane, flying car, satellite, planetary probe, deep space probe, solar probe, etc. Also, as shown in Figure 14, in another embodiment, the device 1400 is a land transport device 1402, such as an automobile, truck, heavy machinery, construction equipment, boat, ship, submarine, etc. In yet another embodiment shown in Figure 14, the device 1400 is a modular device 1403 consisting of at least one or more of an aerial module, a payload module, and a ground module. The aerial module is a module that performs air transport or flight functions. The payload module is a module that performs cargo or living organisms (such as people or animals) transport functions. The ground module is a module that performs ground mobility functions. The solutions of this disclosure can be applied separately to each module, or to a set of multiple modules, such as a set of an air module and a payload module, a set of a payload module and a ground module, or a set of all modules.

[0070] Next, referring to Figure 15, a more specific example of a flight device 1401 that can suitably utilize an embodiment of the present disclosure is shown. In this embodiment, the flight device 1401 is an aircraft manufactured by the manufacturing and usage method 1300 shown in Figure 13, and includes a fuselage 1502 with a plurality of systems 1504 and interior 1506. Examples of the plurality of systems 1504 include one or more of the propulsion system 1508, electrical system 1510, hydraulic system 1512, and environmental system 1514. Any other systems may also be included. Although an example from the aerospace industry has been shown, other suitable embodiments can be applied to other industries, such as the automotive industry.

[0071] The embodiments of this disclosure are described in terms of general concepts of computer code and machine-executable instructions, examples of which include computer-executable instructions such as program components executed by machines other than computers, such as personal data assistants and other portable devices. Generally, program components refer to code for performing a specific task or realizing a specific abstract data type, such as routines, programs, objects, components, and data structures. The embodiments of this disclosure can be implemented in various system configurations, such as personal computers, laptops, smartphones, mobile tablets, portable devices, consumer electronics, and dedicated computer devices. The embodiments of this disclosure can also be implemented in a distributed computing environment in which tasks are performed by remote devices connected via a communication network.

[0072] A method for aerial refueling according to one embodiment includes receiving a video stream containing a plurality of video frames, each of which video frames indicates an aircraft to be refueled; identifying an initial estimated position for the aircraft for each of the plurality of video frames, wherein the initial estimated positions for the plurality of video frames constitute the estimated flight history of the aircraft; identifying a temporally consistent estimated position for the aircraft using an estimation correction unit, based at least on the estimated flight history of the aircraft; identifying the position of the refueling port of the aircraft based at least on the corrected estimated position for the aircraft; and controlling the aerial refueling boom to engage with the refueling port, based at least on the position of the refueling port.

[0073] A system for aerial refueling according to one embodiment comprises one or more processors and a memory storing instructions, wherein, when an instruction is executed by the one or more processors, the one or more processors are caused to perform the following processes: receive a video stream containing a plurality of video frames, each video frame indicating an aircraft to be refueled; identify an initial estimated position for the aircraft for each of the plurality of video frames, wherein the initial estimated positions for the plurality of video frames constitute the estimated flight history of the aircraft; identify a temporally consistent estimated position for the aircraft by an estimation correction unit based on at least the estimated flight history of the aircraft; identify the position of the aircraft's refueling port based on at least the corrected estimated position for the aircraft; and control the aerial refueling boom to engage with the refueling port based on at least the position of the refueling port.

[0074] A computer program product including a computer-readable medium incorporating computer-readable program code, wherein the computer-readable program code, when executed, is configured to perform a method of aerial refueling, the method comprising: receiving a video stream comprising a plurality of video frames, each video frame indicating an aircraft to be refueled; determining an initial estimated position for the aircraft for each of the plurality of video frames, wherein the initial estimated positions for the plurality of video frames constitute the estimated flight history of the aircraft; determining a temporally consistent estimated position for the aircraft by an estimation correction unit based on at least the estimated flight history of the aircraft; determining the position of the aircraft's refueling port based on at least the modified estimated position for the aircraft; and controlling an aerial refueling boom to engage with the refueling port based on at least the position of the refueling port.

[0075] Alternative or additional embodiments to those described herein include any combination of the following: - The video stream is supplied by a single camera. - The video stream is a monocular video stream. - Based at least the estimated flight history of the aircraft, the estimated correction unit determines estimated position correction parameters, the correction parameters include translational correction amounts and rotational correction amounts, and identifies the estimated position corrected to be consistent in time, based at least the estimated flight history and the estimated position correction parameters, to identify the estimated position corrected to be consistent in time. - The estimation correction unit includes the first NN. - The estimation correction unit includes an optimization unit. - Identify the position of the boom tip of the aerial refueling boom. - Controlling the aerial refueling boom to engage with the refueling port includes controlling the aerial refueling boom to engage with the refueling port based at least on the position of the refueling port and the position of the boom tip. - To generate an overlay image. - The overlay image includes at least an aircraft model and a model projection image based on the modified estimated position. - The overlay image includes the video frame showing the aircraft or the next video frame showing the aircraft. - Display the aforementioned overlay image. - The estimation correction unit corrects the estimated flight history according to known aircraft flight paths. - The boom control unit controls the aerial refueling boom to engage with the refueling port. - The estimation correction unit further includes a feature extraction unit. - The feature extraction unit extracts key points from the video stream. - The feature extraction unit includes a second neural network (NN). -The second NN includes a CNN. - The modified estimated position includes an estimated position with 6 degrees of freedom. -Each initial estimated position includes an estimated position with 6 degrees of freedom. - The rotation correction amount is converted from a rotation quaternion to a rotation matrix. - The video stream is a monocular video stream. -Filter the corrected estimated position. -The corrected estimated position is filtered using a Kalman filter. - After repeatedly generating the corrected estimated position, the position of the fuel filler port is determined. - Select the aircraft model based at least on the aircraft to be refueled. - Track the distance between the boom tip and the fuel inlet. - Based at least on the position of the fuel inlet and the position of the boom tip, it is determined whether the control to engage the aerial refueling boom with the fuel inlet falls within the range of safety parameters. - An alert is generated at least based on the determination that the control for engaging the aerial refueling boom with the refueling port is not within the range of safety parameters. - Training data for the first neural network is generated using a simulator that simulates the flight trajectory in an aerial refueling setting and generates pairs of aircraft position and ground truth aircraft data. -The aircraft's position is labeled using the aforementioned aircraft ground truth data. -The first neural network (NN) is trained using the aforementioned training data. - The optimization unit employs bundle adjustment.

[0076] Note 1. Method of aerial refueling, The system receives a video stream (200) containing multiple video frames (200a to 200d) in which each video frame indicates the aircraft (110) being refueled, For each of the plurality of video frames, an initial estimated position (502) for the aircraft is identified, wherein the initial estimated positions for the plurality of video frames constitute the estimated flight history (418) of the aircraft. Based at least on the estimated flight history of the aircraft, the estimated position (504) of the aircraft, which has been modified to be consistent in time, is identified by the estimated modification unit (430), Based on the modified estimated position of at least the aircraft, the location of the aircraft's refueling port (116) is determined, A method comprising controlling an aerial refueling boom (104) to engage with the refueling port, based at least on the position of the refueling port.

[0077] Note 2. The video stream is supplied by one camera, as described in Note 1.

[0078] Appendix 3. The method according to Appendices 1 to 2, further comprising determining estimated position correction parameters by the estimation correction unit based on at least the estimated flight history of the aircraft, wherein the correction parameters include a translational correction amount and a rotational correction amount, and identifying the time-consistently corrected estimated position includes identifying the time-consistently corrected estimated position based on at least the estimated flight history and the estimated position correction parameters.

[0079] Note 4. The estimation correction unit is the method described in Notes 1 to 3, including a first neural network (NN).

[0080] Note 5. The estimation correction unit is the method described in Notes 1 to 4, including the optimization unit.

[0081] Appendix 6. The method according to Appendix 1 to 5, further comprising determining the position of the boom tip of the aerial refueling boom and controlling the aerial refueling boom to engage with the refueling port, wherein controlling the aerial refueling boom to engage with the refueling port is based at least on the position of the refueling port and the position of the boom tip.

[0082] Note 7. Furthermore, the overlay image is, At least an aircraft model and a model projection image based on the modified estimated position, To generate an overlay image including the video frame showing the aircraft or the next video frame showing the aircraft, The method described in Appendix 1 to 6, which includes displaying the aforementioned overlay image.

[0083] Note 8. A system for aerial refueling, One or more processors, A memory storing instructions, and when an instruction is executed by one or more processors, the one or more processors, The system receives a video stream (200) containing multiple video frames (200a to 200d) in which each video frame indicates the aircraft (110) being refueled, For each of the plurality of video frames, an initial estimated position (502) for the aircraft is identified, wherein the initial estimated positions for the plurality of video frames constitute the estimated flight history (418) of the aircraft. Based at least on the estimated flight history of the aircraft, the estimated position (504) of the aircraft, which has been modified to be consistent in time, is identified by the estimated modification unit (430), Based on the modified estimated position of at least the aircraft, the location of the aircraft's refueling port (116) is determined, A system that performs a process including controlling an aerial refueling boom (104) to engage with the refueling port, based at least on the position of the refueling port.

[0084] Note 9. The video frames are supplied by a single camera, as described in Note 8.

[0085] Note 10. The system according to Notes 8-9, wherein the process further includes determining estimated position correction parameters by the estimation correction unit based on at least the estimated flight history of the aircraft, the correction parameters including translational correction amounts and rotational correction amounts, and identifying the time-consistently corrected estimated position includes identifying the time-consistently corrected estimated position based on at least the estimated flight history and the estimated position correction parameters.

[0086] Note 11. The estimation correction unit is the system described in Notes 8-10, including the first neural network (NN).

[0087] Note 12. The estimation correction unit is the system described in Notes 8 to 11, including the optimization unit.

[0088] Note 13. The system according to Notes 8 to 12, wherein the process further includes determining the position of the boom tip of the aerial refueling boom, and controlling the aerial refueling boom to engage with the refueling port includes controlling the aerial refueling boom to engage with the refueling port based at least on the position of the refueling port and the position of the boom tip.

[0089] Appendix 14. The system according to Appendix 8-13, further comprising a boom control unit that controls the aerial refueling boom and engages it with the refueling port.

[0090] Note 15. A computer program product including a computer-usable medium incorporating computer-readable program code, wherein the computer-readable program code is configured to perform a method of aerial refueling when executed, and the method is The system receives a video stream (200) containing multiple video frames (200a to 200d) in which each video frame indicates the aircraft (110) being refueled, For each of the plurality of video frames, an initial estimated position (502) for the aircraft is identified, wherein the initial estimated positions for the plurality of video frames constitute the estimated flight history (418) of the aircraft. Based at least on the estimated flight history of the aircraft, the estimated position (504) of the aircraft, which has been modified to be consistent in time, is identified by the estimated modification unit (430), Based on the modified estimated position of at least the aircraft, the location of the aircraft's refueling port (116) is determined, A computer program product comprising controlling an aerial refueling boom (104) to engage with the refueling port, based at least on the position of the refueling port.

[0091] Note 16. The video frames are supplied by a single camera, and are the computer program product described in Note 15.

[0092] Note 17. The computer program product according to Notes 15-16, wherein the method further includes determining estimated position correction parameters by the estimation correction unit based on at least the estimated flight history of the aircraft, the correction parameters including translational correction amounts and rotational correction amounts, and identifying the time-consistently corrected estimated position includes identifying the time-consistently corrected estimated position based on at least the estimated flight history and the estimated position correction parameters.

[0093] Note 18. The estimation correction unit is a computer program product as described in Notes 15-17, including a first neural network (NN).

[0094] Note 19. The estimation correction unit is the computer program product described in Notes 15 to 18, including the optimization unit.

[0095] Note 20. The computer program product according to Notes 15-19, wherein the method further includes determining the position of the boom tip of the aerial refueling boom, and controlling the aerial refueling boom to engage with the refueling port, which includes controlling the aerial refueling boom to engage with the refueling port based at least on the position of the refueling port and the position of the boom tip.

[0096] The various definite and indefinite articles used in introducing elements in aspects and embodiments of this disclosure mean that there is one or more of those elements. Furthermore, terms such as “equipped with,” “include,” and “possess” are used inclusively, meaning that there may be additional elements beyond those listed, and “embodiment” means “in one example / one embodiment.” Also, the expression “one or more of A, B, and C” means “at least one A, and / or at least one B, and / or at least one C.”

[0097] While the aspects of this disclosure have been described in detail above, various modifications and variations are possible without departing from the scope of the aspects of this disclosure as defined in the attached claims. The structures, products, and methods described above can be modified in various ways without departing from the scope of this disclosure, and everything contained in the above description and attached drawings is for illustrative purposes only and not to impose any limitations.

Claims

1. A method of aerial refueling, The system receives a video stream containing multiple video frames, each of which represents the aircraft being refueled. For each of the plurality of video frames, an initial estimated position of the aircraft is determined, wherein the initial estimated positions for the plurality of video frames constitute the estimated flight history of the aircraft. Based on the estimated flight history of the aircraft, the estimated position of the aircraft is identified by the estimation correction unit, which has been modified to be consistent in time. Based on the modified estimated position of at least the aircraft, the location of the aircraft's refueling port is determined, A method comprising controlling an aerial refueling boom to engage with the refueling port, based at least on the position of the refueling port.

2. The method according to claim 1, wherein the video stream is supplied by a single camera.

3. The method according to claim 1 or 2, further comprising determining estimated position correction parameters by the estimation correction unit based on at least the estimated flight history of the aircraft, wherein the correction parameters include a translational correction amount and a rotational correction amount, and identifying the time-consistently corrected estimated position includes identifying the time-consistently corrected estimated position based on at least the estimated flight history and the estimated position correction parameters.

4. The method according to claim 1 or 2, wherein the estimation correction unit includes a first neural network (NN).

5. The method according to claim 1 or 2, wherein the estimation correction unit includes an optimization unit.

6. The method according to claim 1 or 2, further comprising determining the position of the boom tip of the aerial refueling boom, and controlling the aerial refueling boom to engage with the refueling port, wherein controlling the aerial refueling boom to engage with the refueling port is based at least on the position of the refueling port and the position of the boom tip.

7. Furthermore, it is an overlay image, At least an aircraft model and a model projection image based on the modified estimated position, To generate an overlay image including the video frame showing the aircraft or the next video frame showing the aircraft, The method according to claim 1 or 2, comprising displaying the overlay image.

8. It is a system for aerial refueling. One or more processors, A memory storing instructions, and when an instruction is executed by one or more processors, the one or more processors, The system receives a video stream containing multiple video frames, each of which represents the aircraft being refueled. For each of the plurality of video frames, an initial estimated position of the aircraft is determined, wherein the initial estimated positions for the plurality of video frames constitute the estimated flight history of the aircraft. Based on the estimated flight history of the aircraft, the estimated position of the aircraft is identified by the estimation correction unit, which has been modified to be consistent in time. Based on the modified estimated position of at least the aircraft, the location of the aircraft's refueling port is determined, A system that performs a process including controlling an aerial refueling boom to engage with the refueling port, based at least on the position of the refueling port.

9. The system according to claim 8, wherein the video frames are supplied by a single camera.

10. The system according to claim 8 or 9, wherein the processing further includes determining estimated position correction parameters by the estimation correction unit based on at least the estimated flight history of the aircraft, the correction parameters including a translational correction amount and a rotational correction amount, and identifying the time-consistently corrected estimated position includes identifying the time-consistently corrected estimated position based on at least the estimated flight history and the estimated position correction parameters.

Images