Estimation of the position and attitude of fuel containers and boom tip for aerial refueling.

Abstract

To provide fuel receptacle and boom tip position and pose estimation for aerial refueling.SOLUTION: A video frame is received in which aircraft keypoints for an aircraft to be refueled are determined. Based on at least the aircraft keypoints, a position and a pose of a fuel receptacle on the aircraft are determined. In the video frame, a boom tip keypoint for a boom tip of an aerial refueling boom is also determined. Based on at least the boom tip keypoint, a position and a pose of the boom tip are determined. Based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip, the aerial refueling boom is controlled to engage the fuel receptacle. Some examples overlay projections of an aircraft model on displayed video for a human observer.SELECTED DRAWING: Figure 2B

Description

Background Art

[0004] , , ,

[0001] In-air refueling (air-to-air refueling) is typically performed manually by a trained operator with respect to a refueling boom. In some arrangements, the refueling boom and the aircraft being refueled are in view, and the operator is positioned behind a window. In this type of arrangement, a significant additional cost is required to accommodate the operator located behind the refueling platform.

[0002] In some arrangements, stereovision using dual cameras is used. In this case, the operator wears goggles that provide a three-dimensional (3D) view based on the view from the dual cameras. In some other arrangements, light detection and ranging (LIDAR) or radar is used to provide supplementary distance measurement to the operator. These latter types of arrangements require additional expensive components.

Summary of the Invention

[0003] The disclosed examples are described in detail below with reference to the accompanying drawings listed below. The following summary is provided to explain the examples or embodiments disclosed in this document. However, it is not meant to limit all examples to a specific sequence of configurations or steps.

[0004] The examples described in this book include solutions for estimating the position and attitude of fuel containers and boom tips for aerial refueling, which include receiving video frames, determining the aircraft keypoint of the aircraft being refueled within the video frames, determining the position and attitude of the aircraft's fuel container based at least on the aircraft keypoint, determining the boom tip keypoint of the boom tip of the aerial refueling boom within the video frames, determining the position and attitude of the boom tip based at least on the boom tip keypoint, and controlling the aerial refueling boom to engage with the fuel container based at least on the position and attitude of the fuel container and the boom tip. Some examples use only a single camera (monocular view) for video input. Some examples overlay a projection of an aircraft model onto the video displayed for operator or observer personnel. Some examples enable automated aerial refueling, such as aerial refueling, without requiring highly skilled refueling boom operators.

[0005] The disclosed examples are described in detail below with reference to the attached drawings listed below. [Brief explanation of the drawing]

[0006] [Figure 1A] As an example, configuration 100 is shown that uses estimation of the position and attitude of the fuel container and boom tip for aerial refueling. [Figure 1B] A block diagram of a computer vision (CV) architecture 150 that can be used in deployment 100 is shown, following the example. [Figure 2A] Following the example, 200 representative video frames from an aerial refueling operation are shown. [Figure 2B] An annotated version of video frame 200 is shown. [Figure 3] A block diagram of the aircraft position and attitude estimation pipeline 300 is shown, following the example. [Figure 4A] Following the example, a representative aircraft keypoint heatmap 400 for aircraft 110 is shown. [Figure 4B] Following the example, a representative boom tip keypoint heatmap 450 for boom tip 106 is shown. [Figure 5] Following the example, video frame 500 is shown in which the aircraft model projection 332 is overlaid on video frame 200. [Figure 6A] Following the example, a block diagram of the boom tip position and attitude estimation pipeline 600a is shown. [Figure 6B] A block diagram of the alternative boom tip position and attitude estimation pipeline 600b is shown, following the example. [Figure 7] A flowchart 700 is shown illustrating a method for estimating the position and attitude of the fuel container and boom tip for aerial refueling, which can be used in the arrangement 100 of Figure 1, following the example. [Figure 8] Flowchart 800 shows another method for estimating the position and attitude of the fuel container and boom tip for aerial refueling, which can be used in the arrangement 100 of Figure 1, following the example. [Figure 9] This is a block diagram of a computing device 900 suitable for implementing various aspects of the present disclosure, following an example. [Figure 10] This is a block diagram of a method 1000 for manufacturing and maintaining an apparatus using various embodiments of the present disclosure, following an example. [Figure 11] This is a block diagram of apparatus 1100 in which various aspects of the present disclosure may be advantageously used, as shown in the example. [Figure 12] This is a schematic perspective view of a specific flying device 1101, following the example. [Modes for carrying out the invention]

[0007] The corresponding reference letters indicate the corresponding parts in the entire drawing, as shown in the example.

[0008] Various examples will be described in detail with reference to the attached drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References made throughout this disclosure relating to specific examples and embodiments are provided for illustrative purposes only; conversely, not all embodiments are intended to be limited unless otherwise indicated.

[0009] The above overview and the following detailed descriptions of specific embodiments will be better understood when read in conjunction with the accompanying drawings. It should be understood that, as used herein, elements or steps described in the singular form and following the word "one ("a" or "an")" do not necessarily exclude multiple such elements or steps. Furthermore, references to embodiments or examples are not intended to be interpreted as excluding the existence of additional examples that also incorporate the listed features. Also, unless explicitly stated otherwise, embodiments that "include" or "have" one or more elements having a particular nature may include additional elements that do not possess that nature.

[0010] The embodiments and models described in this book relate to the estimation of the position and attitude of fuel containers and boom tips for aerial refueling, and include receiving video frames; determining the aircraft keypoint of the aircraft being refueled within the video frames; determining the position and attitude of the fuel container on the aircraft based at least on the aircraft keypoint; determining the boom tip keypoint of the boom tip of the aerial refueling boom within the video frames; determining the position and attitude of the boom tip based at least on the boom tip keypoint; and controlling the aerial refueling boom to engage with the fuel container based at least on the position and attitude of the fuel container and the position and attitude of the boom tip. Some examples use only a single camera (monocular view) for video input. Some examples overlay a projection of an aircraft model onto the video displayed for operator or observer personnel. Some examples enable automated aerial refueling, such as aerial refueling, without requiring highly skilled refueling boom operators.

[0011] Aspects of this disclosure have the technical effect of improving computer operation by, for example, reducing distance calculations, improving the efficiency of computing hardware, and providing better resource allocation compared to conventional systems that rely on many different measurement inputs. Aspects of this disclosure can estimate the position and orientation of a three-dimensional object (e.g., an aircraft fuel container) in a video stream collected by a single camera, such as in supporting autonomous aerial refueling operations and / or personnel-assisted aerial refueling operations. For example, aspects of this disclosure can identify the relative position and orientation (attitude) of an aircraft fuel container and a refueling boom on a refueling platform in order to automate the control of the refueling boom during refueling. In some examples, the position and orientation information is represented as six degrees of freedom (6DoF), including three-dimensional (3D) position (x, y, z coordinates) and orientation (roll, pitch, yaw).

[0012] Positioning is performed in stages such as detecting object bounding boxes in input 2D video frames, detecting 2D keypoints (object landmarks), and converting from 2D to 3D to determine 6DoF information for the tips of each fuel container and refueling boom. The multi-stage attitude estimation pipeline uses real-time deep learning-based detection algorithms, such as neural networks (NNs), including deep convolutional neural networks (CNNs), which can be residual neural networks (ResNets). This enables accurate detection and tracking in adverse weather and lighting conditions and can be used for autonomous air-to-air and / or refueling and operator feedback loops. Using a single camera reduces component failures and allows for easier integration with existing systems.

[0013] Referring more specifically to the drawings, Figure 1A shows configuration 100, which includes a refueling platform 102 and an aircraft 110 to be refueled. Both the refueling platform 102 and the aircraft 110 are examples of flight equipment 1101, which will be described in more detail in relation to Figures 11-12. In configuration 100, the refueling platform 102 uses an aerial refueling boom 104 to refuel the aircraft 110. A camera 108 provides a video stream 200a (shown in Figure 1B) for use in estimating the position and attitude of the fuel container and the boom tip.

[0014] Figure 1B shows a block diagram of a computer vision (CV) architecture 150 that performs position and orientation estimation of the fuel container and boom tip for arrangement 100. The components of architecture 150 are identified in Figure 1B, along with exemplary data and further details in Figures 2A-6B, and the operation of architecture 150 is described in more detail in relation to Figure 7 (showing flowchart 700). Architecture 150 receives a video stream 200a containing a video frame 200 and several additional video frames 200b. An exemplary video frame 200 is shown in a concise form in Figure 2A and in an annotated form in Figure 2B. The processing of video frame 200 is described. The processing of each of the several additional video frames 200b is similar to the processing of video frame 200.

[0015] Architecture 150 includes an aircraft position and attitude estimation pipeline 300 and a boom tip position and attitude estimation pipeline 600. The aircraft position and attitude estimation pipeline 300 is illustrated and described in more detail in relation to Figure 3. In some examples, the boom tip position and attitude estimation pipeline 600 is implemented as boom tip position and attitude estimation pipeline 600a, which is illustrated and described in more detail in relation to Figure 6A. In some examples, the boom tip position and attitude estimation pipeline 600 is alternatively implemented as boom tip position and attitude estimation pipeline 600b, which is illustrated and described in more detail in relation to Figure 6B.

[0016] The aircraft position and attitude estimation pipeline 300 outputs, for example, in 6DoF, the fuel tank position 330 (the position and attitude of the fuel tank on the aircraft). In some examples, the aircraft position and attitude estimation pipeline 300 also outputs an aircraft model projection 332. The boom tip position and attitude estimation pipeline 600 outputs a boom tip position 630 (the position and attitude of the boom). In some examples, the boom tip position and attitude estimation pipeline 600 also outputs a boom model projection 632. The fuel tank position 330 and the boom tip position 630 are provided to tracking logic 152 that determines the distance 154 between the boom tip 106 and the fuel tank 116 (both the boom tip 106 and the fuel tank 116 are shown in FIG. 2B). The tracking logic 152 determines boom control parameters 158, and the boom control parameters 15 are provided to a boom control unit 160. The boom control unit 160 autonomously moves the aerial refueling boom 104 and positions the boom tip 106 to engage the fuel tank 116. That is, the boom control unit 160 controls the aerial refueling boom 160 to engage the fuel tank 116. In some examples, the tracking logic 152 also determines whether it is within the range of safety parameters 156 to control the aerial refueling boom 104 to engage the fuel tank 116, and if not, generates a warning 166.

[0017] The boom control parameters 158, as used herein, include variables that describe how the boom 104 can move (e.g., roll, pitch, yaw, translation, docking shortening, extension, contraction, pivot, rotation, etc.), and may include limitations and speeds of such movements. The boom control parameters 158 can control the boom 104, for example, how to rotate the boom 104 (roll and pitch) and how to telescopically extend the boom 104 so that the 3D position of the boom tip 106 is projected onto the camera image 200 where the boom tip keypoint 452 is detected, given the constraints of the boom pivot position and the intrinsic and extrinsic parameters of the camera.

[0018] In some examples, the aircraft model projection 332 and / or the boom model projection 632 are provided to the video editing 162, and the video editing 162 overlays the aircraft model projection 332 and / or the boom model projection 632 onto the video frame 200 to generate an overlaid video frame 500. An exemplary video frame 500 is shown in FIG. 5. In some examples, the video frame 500 and / or the warning 166 are provided to the operator 164 via the presentation component 906 (e.g., by displaying the video frame 500 on a video monitor screen). In some examples, the operator 164 uses an input / output (I / O) component 910 (e.g., a joystick, mouse, keyboard, touch screen, keypad, and / or other input device) to provide boom control parameters 158 and control the aerial refueling boom 104 to position the boom tip 106 to engage the fuel receptacle 116.

[0019] FIG. 2A shows a representative video frame 200 from an aerial refueling operation. For clarity, FIG. 2A shows only a simplified version of the video frame 200. FIG. 2B shows an annotated version of the video frame 200. The video frame 200 shows the aircraft 110 outlined by the aircraft boundary box 210. As will be described below with respect to FIG. 3, the aircraft boundary box 210 is generated by an initial stage of the aircraft position and attitude estimation pipeline 300. The aircraft 110 has a fuel receptacle 116, which is outlined by a reference marker 118. The video frame 200 also shows the aerial refueling boom 104 having a boom tip 106 outlined by the boom tip boundary box 206. As will be described below with respect to FIG. 6, the boom tip boundary box 206 is generated by an initial stage of the boom tip position and attitude estimation pipeline 600. During the process, the aerial refueling boom 104 supplies fuel to the aircraft 110 by the boom tip 106 engaging the fuel receptacle 116. The reference marker 118 facilitates the placement of the fuel receptacle 116 on the aircraft 110.

[0020] Figure 3 shows a block diagram of the aircraft position and attitude estimation pipeline 300, including part of the architecture 150. A video frame 200 is provided to an aircraft bounding box detector 302 that determines the aircraft bounding box 210. In some examples, the aircraft bounding box detector 302 crops the video frame 200 to the area corresponding to the aircraft bounding box 210 to generate a cropped image 304. Cropping allows unnecessary sections of the video frame 200 to be ignored in later stages by taking only the content of the enclosed rectangle as input. Using only the relevant area helps reduce computation time and allows more computationally intensive algorithms to be used in later stages of the pipeline.

[0021] In some examples, filter 306 filters the video data using, for example, a Kalman filter. The Kalman filter uses a series of measurements observed over time, including statistical noise and other inaccuracies, to estimate a joint probability distribution across the variable in each time frame, thereby generating an estimate of an unknown variable that tends to be more accurate than one based on a single measurement alone. Thus, in some examples, filter 306 operates across video frames (e.g., video frame 200 and several additional video frames 200b).

[0022] In some examples, the aircraft bounding box detector 302 includes a NN (e.g., a deep CNN). The aircraft bounding box detector 302 is trained using the object model trainer 350, as described below. The output of the aircraft bounding box detector 302 (cropped and / or filtered in some examples) is provided to the aircraft keypoint detector 310. In some examples, the aircraft keypoint detector 310 includes a NN (e.g., a ResNet). The aircraft keypoint detector 310 is trained using the keypoint model trainer 352, as described below, and outputs an aircraft keypoint heatmap 400. Keypoint detection identifies the video frame locations of points on a 3D object that can be used for 6DOF attitude estimation. Keypoints can be selected as locations on a 3D object that can be consistently recognized, such as the wingtips of an aircraft. A heatmap is a graphical representation of the data that uses a system of color coding to represent various values. The pixel values ​​of the heatmap, for each keypoint, indicate the likelihood that a keypoint of the 3D object could be found at each pixel location in the image.

[0023] The aircraft keypoint heatmap 400 is described in more detail in relation to Figure 4A. In some examples, the aircraft keypoint heatmap 400 is filtered using a filter 316, which in some examples includes a Kalman filter (thus filtering the heatmap across video frames). In some examples, a threshold 318 is applied to estimate keypoints with low confidence levels.

[0024] The aircraft keypoint heatmap 400 (filtered and thresholded in some examples) is provided to the aircraft 2D / 3D conversion 320. In some examples, the aircraft 2D / 3D conversion 320 uses the perspective-n-point (PnP) algorithm. The PnP algorithm estimates the calibrated camera pose relative to the object by considering a set of N 3D points on the object and their corresponding 2D projections in the image collected by the camera. The PnP algorithm used utilizes the correspondence between the 2D pixel positions of the detected keypoints and the 3D keypoint positions on the object model to rotate and position the object in space so that the camera view of the 3D keypoints coincides with the 2D pixel positions.

[0025] The aircraft 2D / 3D conversion 320 determines the aircraft position 334 (position and attitude of the aircraft 110), from which the fuel container position 330 is determined. That is, once the aircraft position 334 is known, the fuel container position 330 at a given position on the aircraft 110 can be determined. In some examples, the aircraft position 334 and / or the fuel container position 330 are filtered using a filter 322. In some examples, the filter 322 also includes a Kalman filter (which filters over time across video frames). In some examples, the aircraft 2D / 3D conversion 320 also generates an aircraft model projection 332. The aircraft model projection 332 is determined by rendering a 3D aircraft model 346 according to the aircraft position 334.

[0026] Both the aircraft boundary box detector 302 and the aircraft keypoint detector 310 are initially trained offline, although in some examples continuous training may be used during deployment. The data preprocessing component 340 uses a simulator 342 to generate training images 344 on an object model trainer 350 using a 3D aircraft model 346. In some examples, the 3D aircraft model 346 includes a computer-aided design (CAD) model. In some examples, the training images 344 include approximately 20,000 labeled training images, which are generated by sweeping the entire range of 6DoF variations of the 3D aircraft model 346. To increase the variation of the dataset used for NN training, data augmentation is randomly applied to each image passed to the NN. These augmentations include brightness scaling, contrast scaling, image size scaling, and image transformation.

[0027] The data preprocessing component 340 can also label the training images 344 because the simulator 342 has ground truth data when generating the training images. The object model trainer 350 uses the training images 344 to train the aircraft bounding box detector 302. The training of the aircraft keypoint detector 310 is similar. The simulator 342 sweeps the 6DoF variance to generate the required number of training images 348. The keypoint model trainer 352 uses the training images 348 to train the aircraft keypoint detector 310. The pixel values ​​of the ground truth heatmap are assigned values ​​of a Gaussian probability distribution on 2D coordinates, the mean is equal to the 2D pixel location of the ground truth, and the covariance is left as a hyperparameter for training. The loss to be minimized during training is the mean squared error in pixels between the heatmap output of the neural network and the ground truth heatmap.

[0028] Figure 4A shows the aircraft keypoint heatmap 400 of aircraft 110. The aircraft keypoint heatmap 400 has a set of aircraft keypoints 402, including aircraft keypoint 402a, aircraft keypoint 402b, aircraft keypoint 402b, aircraft keypoint 402j, and other unlabeled aircraft keypoints (for clarity). Aircraft keypoints 402 are identifiable locations in a two-dimensional (2D) image that correspond to features of a three-dimensional (3D) aircraft, such as wingtips, sharp angles, seams, contact points of various features (e.g., canopy including fuselage), and reference markers (e.g., reference marker 118). In some examples, aircraft 110 may have 33 keypoints. However, a different number of keypoints can be used.

[0029] Figure 4B shows the boom tip keypoint heatmap 450 of the boom tip 106. In some examples, the boom tip keypoint heatmap 450 has only a single keypoint 452. In some examples, keypoints 402 and 452 in the aircraft keypoint heatmap 400 and boom tip keypoint heatmap 450 are represented as probability distributions rather than as binary points. In some examples, for keypoints 402 and 452, the position of the highest value pixel is retained, indicating that this is most likely to contain a keypoint of a 3D object. The value of that keypoint exceeds a adjusted threshold, and then the keypoint is considered detected.

[0030] That is, keypoints 402 and 452 each correspond to a pixel area, territory As the distance from the center of the region increases, the pixel value decreases according to the probability density function (pdf). In some examples, the maximum value of a pixel in a keypoint region reflects the confidence level of that keypoint. Therefore, the aircraft keypoint heatmap 400 and the boom tip keypoint heatmap 450 can not only convey the locations of keypoints 402 and 452, but also indicate their confidence levels. However, in some examples, only the highest value pixel is retained in each region.

[0031] Figure 5 shows video frame 500 with the aircraft model projection 332 overlaid on video frame 200 for display to worker 164. The aircraft model projection 332 is generated using a 3D aircraft model 346 and rendered according to position and attitude estimations from the aircraft position and attitude estimation pipeline 300. For clarity, only the outline of the aircraft model projection 332 is shown. In some examples, a boom model projection 632 is overlaid on video frame 200, either in addition to or instead of the aircraft model projection 332.

[0032] Figure 6A shows a block diagram of the boom tip position and attitude estimation pipeline 600a. In some examples, the boom tip position and attitude estimation pipeline 600a is used as the drivable boom tip position and attitude estimation pipeline 600 of architecture 150. Video frames 200 are provided to a boom tip bounding box detector 602 that determines the boom tip bounding box 206. In some examples, the boom tip bounding box detector 602 crops the video frames 200 to the area corresponding to the boom tip bounding box 206 to produce a cropped image 604. In some examples, a filter 606 filters the video data, for example, using a Kalman filter that operates across video frames (e.g., video frame 200 and several additional video frames 200b).

[0033] In some examples, the boom tip bounding box detector 602 includes a NN (e.g., a deep CNN). The boom tip bounding box detector 602 is trained using an object model trainer 650, as described below. The output of the boom tip bounding box detector 602 (which is cropped and / or filtered in some examples) is provided to the boom tip keypoint detector 610. In some examples, the boom tip keypoint detector 610 includes a NN (e.g., a ResNet). The boom tip keypoint detector 610 is trained using a keypoint model trainer 652, as described below, and outputs a boom tip keypoint heatmap 450. The boom tip keypoint heatmap 450 is described in relation to Figure 4B. In some examples, the boom tip keypoint heatmap 450 is filtered using a filter 616, which in some examples includes a Kalman filter (thus the heatmap is filtered across video frames). In some examples, a threshold 618 is applied to estimate keypoints with low confidence levels.

[0034] A boom tip keypoint heatmap 450 (filtered and thresholded in some examples) is provided to the boom tip 2D / 3D transform 620, which will be described in more detail in relation to Figure 6B. The boom tip 2D / 3D transform 620 determines the boom tip position 630, which in some examples is filtered by a filter 622. In some examples, the filter 622 also includes a Kalman filter (which filters over time across video frames). In some examples, the boom tip 2D / 3D transform 620 also generates a boom model projection 632. The boom model projection 632 is determined by rendering a 3D refueling boom model 646 according to the boom tip position 630.

[0035] The boom tip boundary box detector 602 and the boom tip keypoint detector 610 are both initially trained offline, although some examples may use continuous training during deployment. The data preprocessing component 640 uses the simulator 642 to generate training images 644 for the object model trainer 650 using the 3D refueling boom model 646. In some examples, the 3D refueling boom model 646 includes a computer-aided design (CAD) model. In some examples, the training images 644 include approximately 20,000 labeled training images, which are generated by sweeping the entire range of boom control parameters of the 3D refueling boom model 646.

[0036] The data preprocessing component 640 can also label the training images 644 because the simulator 642 has ground truth data when generating the training images. The object model trainer 650 uses the training images 644 to train the boom tip bounding box detector 602. The boom tip keypoint detector 610 is trained similarly. The simulator 642 sweeps for variability in the boom control parameters and generates the required number of training images 648. The keypoint model trainer 652 uses the training images 648 to train the boom tip keypoint detector 610.

[0037] Figure 6B shows a block diagram of the boom tip position and attitude estimation pipeline 600b. In some examples, the boom tip position and attitude estimation pipeline 600b is used as the drivable boom tip position and attitude estimation pipeline 600 of architecture 150. The boom tip position and attitude estimation pipeline 600b is similar to the boom tip position and attitude estimation pipeline 600a, except that the boom tip bounding box derivation unit 660 replaces the boom tip bounding box detector 602. The boom tip bounding box derivation unit 660 calculates the position of the boom tip bounding box 206, rather than detecting it from the image, using the boom model 646, camera parameters 664 (e.g., extrinsic and intrinsic parameters of camera 108), and boom control parameters 158. The boom model 662, camera parameters 664, and boom control parameters 158 are also input to the boom tip 2D / 3D conversion 620.

[0038] This method uses the known angle, extrinsicity, and geometric shape of the aerial refueling boom 104 relative to the camera 108 to determine the projection of the aerial refueling boom 104. By monitoring the pitch and roll state of the boom control unit 160 in each video frame, the pitch and roll of the aerial refueling boom 104 relative to the camera 108 can be determined at the time of capturing each image (video frame). Since the intrinsicity and position of camera 250 are known, the position of the aerial refueling boom 104 in the 2D pixel space of camera 108 can be determined.

[0039] Camera parameter information includes parameters used in the camera model to describe the mathematical relationship between the 3D coordinates of light-generating scene points and the 2D coordinates of their projection onto the image plane. Intrinsic parameters, also known as internal parameters, are parameters inherent to the camera itself, such as focal length and lens distortion. Extrinsic parameters, also known as external parameters or camera orientation, are parameters used to describe the transformation between the camera and its external environment. Camera extrinsic information, resolution, magnification, and other intrinsic information are known.

[0040] The boom tip 2D / 3D transformation 620 uses a method similar to the boom tip bounding box derivation unit 660 described above to capture its world position, using the known angles, extrinsicities, and geometric shapes of the object for each time example. However, instead of transforming the position into a 2D pixel coordinate space, it is transformed into an estimate of the boom control parameters.

[0041] Referring here to Figure 7, flowchart 700 illustrates a method for estimating the position and attitude of the fuel container and boom tip for aerial refueling. In some examples, the steps shown in Figure 7 are performed, at least in part, by executing instructions 902a (stored in memory 902) by one or more processors 904 of the computing device 900 in Figure 9. For example, any of the aircraft boundary box detector 302, aircraft keypoint detector 310, boom tip boundary box detector 602, and boom tip keypoint detector 610 can be trained in a first example of the computing device 900 and then deployed in a second (different) example of the computing device 900.

[0042] Steps 702-710 are performed before deployment to train the NN. Step 702 includes acquiring one or more 3D aircraft models (of the aircraft type expected to be refueled) and a 3D boom model, such as a 3D aircraft model 346 and a 3D boom model 646. Step 704 includes identifying points on the aircraft model that correspond to keypoints detectable in the 2D image. Step 706 includes generating a training image 344 of a first NN (e.g., located in an aircraft bounding box detector 302) using a simulator 342 that sweeps the 3D aircraft model 346 through various 6DoF values ​​to generate a set of aircraft images and aircraft ground truth data, and labeling the aircraft images using the aircraft ground. Step 706 also includes generating a training image 348 for a second NN (e.g., located in the aircraft keypoint detector 310) using an aircraft training heatmap (having keypoints based on identified points on the aircraft model) corresponding to a set of aircraft images, and labeling the aircraft training heatmap using aircraft ground truth data.

[0043] Step 708 includes generating a training image 644 for a third NN (e.g., located in a boom tip bounding box detector 602) using a simulator 642 that sweeps a 3D boom model 646 through various boom control parameter values ​​to generate a set of boom tip images and boom tip ground truth data, and labeling the boom tip images using the boom tip ground truth data. Step 708 also includes generating a training image 648 for a fourth NN (e.g., located in a boom tip keypoint detector 610) using a boom tip training heatmap (having keypoints based on the boom tip position) corresponding to a set of boom tip images, and labeling the boom tip training heatmap using the boom tip ground truth data. Step 710 includes training the NN using the training images 344, 348, 644, and 648.

[0044] The architecture 150 includes an aircraft position and attitude estimation pipeline 300 and a boom tip position and attitude estimation pipeline 600 (600a or 600b), which are deployed in step 712. Step 714 includes receiving video frames 200. In some examples, the video frames 200 are provided by a single camera 108. In some examples, the video frames 200 are monocular. In some examples, the process includes receiving a video stream 200a containing video frames 200 and several additional video frames 200b. Step 716 includes selecting a 3D aircraft model based on at least the aircraft to be refueled 110 (for example, selecting a 3D aircraft model 346 from a library of 3D aircraft models).

[0045] Step 718 includes determining the aircraft bounding box 210 of the aircraft being refueled 110 within video frame 200. In some examples, determining the aircraft bounding box 210 includes determining the aircraft bounding box 210 using a first NN, including a CNN. Step 720 includes cropping video frame 200 to the aircraft bounding box 210. Step 722 includes determining the aircraft keypoints 402 of the aircraft being refueled 110 within the (cropped) video frame 220. In some examples, determining the aircraft keypoints 402 includes determining the aircraft keypoints 402 using a second NN, including a ResNet. In some examples, determining the aircraft keypoints 402 includes determining the aircraft keypoints 402 within the aircraft bounding box 210. In some examples, determining the aircraft keypoints 402 includes generating an aircraft keypoint heatmap 400 of the aircraft keypoints 402. In some examples, generating the aircraft keypoint heatmap 400 includes determining a confidence value for each aircraft keypoint.

[0046] Step 724 includes filtering out aircraft keypoints 402 in the aircraft keypoint heatmap 400 that have confidence values ​​below a threshold. Step 726 includes determining the position and attitude 116 of the fuel container on the aircraft 110 (e.g., fuel container position 330) based at least on the aircraft keypoints 402. In some examples, the position and attitude of the fuel container represent 6DOF. In some examples, determining the position and attitude of the fuel container includes performing a 2D / 3D transformation 320 of the aircraft keypoints 402. In some examples, the 2D / 3D transformation 320 of the aircraft keypoints 402 uses a PnP algorithm. In some examples, determining the position and attitude of the fuel container 116 includes determining the position and attitude of the aircraft 110 (e.g., aircraft position 334). In some examples, determining the position and attitude of the fuel container 116 includes identifying aircraft keypoints associated with reference markers 118.

[0047] Step 728 includes determining the boom tip bounding box 206 of the boom tip 106 within the video frame 200. In some examples, determining the boom tip bounding box 206 includes determining the boom tip bounding box 206 using a third NN including a CNN (e.g., in a boom tip bounding box detector 602). In some examples, determining the boom tip bounding box 206 includes determining the boom tip bounding box 206 of the boom tip 106 using boom control parameters 158 and camera parameters 664 (e.g., camera calibration information). Step 730 includes cropping the video frame 200 to the boom tip bounding box 206.

[0048] Step 732 includes determining the boom tip keypoint 452 of the boom tip 106 of the aerial refueling boom 104 within the (cropped) video frame 200. In some examples, determining the boom tip keypoint 452 includes determining the boom tip keypoint 452 using a fourth NN, including ResNet (for example, in a boom tip keypoint detector 610). In some examples, determining the boom tip keypoint 452 includes determining the boom tip keypoint 452 within a boom tip bounding box 206. In some examples, determining the boom tip keypoint 452 includes generating a boom tip keypoint heatmap 450 of the boom tip keypoint 452. In some examples, generating the boom tip keypoint heatmap 450 includes determining the confidence value of the boom tip keypoint 452. Step 734 includes determining the position and orientation (boom tip position 630) of the boom tip 106 based on at least the boom tip keypoint 452. In some examples, the position and orientation of the boom tip 106 represent 6DOF. In some examples, determining the position and orientation of the boom tip 106 involves performing a 2D / 3D transformation 620 of the boom tip keypoint 452. In some examples, the position and orientation of the boom tip 106 may be determined using boom control parameters before it becomes necessary to perform boom tip keypoint detection. In such examples, steps 728-732 are not performed, and step 734 involves determining the position and orientation of the boom tip 106 of the aerial refueling boom 104.

[0049] Step 736 includes filtering at least one of the following with a Kalman filter: the aircraft bounding box 210, the aircraft keypoint heatmap 400, the position and attitude of the aircraft 110, the position and attitude of the fuel container 116, the boom tip bounding box 206, the boom tip keypoint heatmap 450, or the position and attitude of the boom tip 106. Each step receives a tracked version of the output of the previous step as input. Each filter tracks a point in multidimensional space at a certain velocity. The observed covariance matrix of each filter is a diagonal matrix having one identical variance for the values ​​of the multidimensional vectors and an identical variance one order of magnitude smaller for the vector velocities. The transition matrix of the filter adds the tracked velocity to each vector. For the bounding box detectors 302 and 602, the 2D position, aspect ratio, and height of the center of the box are tracked as a 4D vector at a certain velocity. The keypoint detectors 310 and 610 track 2D coordinates using separate filters for each keypoint. The 2D / 3D transform 320 and output filter 322 track the results of the translation and Rodriguez rotation vectors of the PnP algorithm, concatenated as a 6-dimensional (6D) vector. The 2D / 3D transform 620 estimates the 2D keypoint of the boom tip and resolves the boom control parameters given constraints on the boom pivot position and intrinsic and extrinsic parameters of the camera, such as how to rotate the boom (roll and pitch) and how to extend the boom telescope, so that, for example, the 3D position of the boom tip is projected onto the camera image 200 where the keypoint of the boom tip is detected.

[0050] The determination step 738 includes determining whether controlling the aerial refueling boom 104 to engage with the fuel container 116 is within the range of safety parameters 156, based at least on the position and orientation of the fuel container 116 and the position and orientation of the boom tip 106. In some examples, the safety parameters 156 include a range of safe angles and stable relative positions. If a warning is required, the step 740 includes generating a warning 166 based at least on the determination that controlling the aerial refueling boom 104 to engage with the fuel container 116 is not within the range of safety parameters 156, and displaying the warning 166 to the worker 164.

[0051] Alternatively, step 742 includes controlling the aerial refueling boom 104 to engage with the fuel container 116 based on at least the position and orientation of the fuel container 116 and the position and orientation of the boom tip 106. In some examples, controlling the aerial refueling boom 104 to engage with the fuel container 116 includes tracking the distance 154 between the boom tip 106 and the fuel container 116. In some examples, controlling the aerial refueling boom 104 to engage with the fuel container 116 includes determining boom control parameters 158 to close the distance 154 between the boom tip and the fuel container. In some examples, as part of step 742, the boom control unit 160 controls the aerial refueling boom 160 to engage with the fuel container 116.

[0052] Step 744 includes overlaying an aircraft model projection 332 onto the video frame 200 based on at least the position and attitude of the aircraft 110. In some examples, step 744 also or alternatively includes overlaying a boom model projection 632 onto the video frame 200 based on at least the position and attitude of the boom tip 106. One or both of these overlays generate the video frame 500. Step 746 includes displaying the video frame 500 having the aircraft model projection 332 overlay and / or the boom model projection 632 overlay to the worker 164.

[0053] Figure 8 shows a flowchart 800 illustrating a method for estimating the position and attitude of the fuel container and boom tip for aerial refueling. In some examples, the steps shown in Figure 8 are performed, at least in part, by executing instructions by one or more processors 904 of the computing device 900 in Figure 9. Step 802 includes receiving a video frame. Step 804 includes determining the aircraft keypoint of the aircraft being refueled within the video frame. Step 806 includes determining the position and attitude of the fuel container on the aircraft, at least based on the aircraft keypoint. Step 808 includes determining the boom tip keypoint of the boom tip of the aerial refueling boom within the video frame. Step 810 includes determining the position and attitude of the boom tip, at least based on the boom tip keypoint. Step 812 includes controlling the aerial refueling boom to engage with the fuel container, at least based on the position and attitude of the fuel container and the position and attitude of the boom tip.

[0054] Referring here to Figure 9, a block diagram of a computing device 900 suitable for implementing various aspects of this disclosure is illustrated. In some examples, the computing device 900 includes one or more processors 904, one or more presenting components 906, and memory 902. The disclosed examples relating to computing device 900 are implemented by a variety of computing devices, including personal computers, laptops, smartphones, mobile tablets, handheld devices, consumer electronics, and specialized computing devices. Categories such as “workstation,” “server,” “laptop,” and “handheld device” are not distinguished, as they are all assumed to be within the scope of Figure 9, and “computing device” is used throughout this document. The disclosed examples are also implemented in a distributed computing environment in which tasks are performed by remote-processing devices linked through a communication network. Furthermore, although computing device 900 is illustrated as a single device, in one example, several computing devices work together and share the illustrated device resources. In one example, memory 902 is distributed across several devices, and the provided one or more processors 904 are housed in different devices, etc.

[0055] In one example, memory 902 includes one of the computer-readable media described herein. In one example, memory 902 is used to store and access instructions 902a configured to perform various operations disclosed herein. In some examples, memory 902 includes computer storage media in the form of volatile and / or non-volatile memory, removable or non-removable memory, data disks in a virtual environment, or a combination thereof. In one example, one or more processors 904 include any number of processing units that read data from memory 902 or various entities such as input / output (I / O) components 910. In particular, one or more processors 904 are programmed to execute computer-executable instructions for implementing aspects of this disclosure. In one example, the instructions are executed by a processor, by several processors within a computing device 900, or by a processor outside of the computing device 900. In some examples, one or more processors 904 are programmed to execute instructions such as those described below and shown in the flowcharts provided in the accompanying drawings.

[0056] One or more presentation components 906 present data displays to an operator or another device. In one example, presentation components 906 include a display device, a speaker, a printing component, a vibration component, and the like. Those skilled in the art will understand and recognize that computer data may be presented in many ways (e.g., visually through a graphical user interface (GUI), aurally through a speaker, wirelessly between computing devices 900, via a wired connection, or otherwise). In one example, one or more presentation components 906 are not used when the process and operation are well automated and the need for human interaction is minimal or unnecessary. I / O ports 908 allow computing devices 900 to be logically coupled to other devices, some of which are integrated, including I / O components 910. Embodiments of I / O components 910 include, but are not limited to, a microphone, keyboard, mouse, joystick, gamepad, satellite receiver antenna, scanner, printer, wireless device, and the like.

[0057] The computing device 900 includes a bus 916 that directly or indirectly connects the following devices: memory 902, one or more processors 904, one or more presentation components 906, input / output (I / O) ports 908, I / O components 910, power supply 912, and network components 914. The computing device 900 should not be construed as having any dependencies or requirements relating to any single component or combination of components shown herein. Bus 916 represents one or more buses (such as an address bus, a data bus, or a combination thereof). The various blocks in Figure 9 are shown with lines for clarity, but some embodiments are unclear regarding the functionality of the various different components described herein.

[0058] In some examples, the computing device 900 is communicably coupled to a network 918 using a network component 914. In some examples, the network component 914 includes a network interface card and / or computer executable instructions (e.g., drivers) for operating the network interface card. In one example, communication between the computing device 900 and other devices is performed via a wired or wireless connection 920 using any protocol or mechanism. In some examples, the network component 914 is operable to communicate data wirelessly between devices using a transport protocol, via public, private, or hybrid (public and private), and using short-range communication technology (e.g., Near Field Communication (NFC), Bluetooth® brand communication, etc.), or a combination thereof.

[0059] Although described in relation to computing device 900, the examples of the disclosure can be implemented in many other general-purpose or dedicated computing system environments, configurations, or devices. Well-known embodiments of computing systems, environments, and / or configurations suitable for use in the manner of the disclosure include, but are not limited to, smartphones, mobile tablets, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, mobile computing and / or communication devices in wearable or accessory form factors (watches, glasses, headsets, earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, VR devices, holographic devices, etc. Such systems or devices accept user input in any way, from input devices such as keyboards and pointing devices, via gesture input, proximity input (such as hovering), and / or voice input.

[0060] Embodiments of this disclosure are described in the general context of computer executable instructions, such as program modules, which are executed by one or more computers or other devices in software, firmware, hardware, or a combination thereof. In one example, a computer executable instruction is organized into one or more computer executable components or modules. Generally, a program module includes, but is not limited to, routines, programs, objects, components, and data structures that perform a particular task or implement a particular abstract data type. In one example, embodiments of this disclosure are implemented with any number and organization of such components or modules. For example, embodiments of this disclosure are not limited to a particular computer executable instruction or a particular component or module shown in the figures and described herein. Other examples of this disclosure include different computer executable instructions or components having more or less functionality than those illustrated and described herein. In embodiments involving a general-purpose computer, embodiments of this disclosure, when configured to execute the instructions described herein, transform the general-purpose computer into a dedicated computing device.

[0061] Computer-readable media include, but are not limited to, computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable memories implemented in any method or technique for storing information such as computer-readable instructions, data structures, and program modules. Computer storage media are tangible and mutually exclusive with respect to communication media. Computer storage media are implemented in hardware and exclude carrier and propagating signals. Computer storage media for the purposes of this disclosure are not signals themselves. In one example, computer storage media include hard disks, flash drives, solid-state memory, phase-shifted random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or other non-transmission media used to store information for access by computing devices. In contrast, communication media typically include any information distribution medium that embodies computer-readable instructions, data structures, program modules, etc., in modulated data signals such as carrier waves or other transport mechanisms.

[0062] Some examples of the present disclosure are used in manufacturing and service applications, as illustrated and described in relation to Figures 10-12. Accordingly, the examples of the present disclosure are described in the context of the apparatus of the manufacturing and maintenance method 1000 shown in Figure 10 and the apparatus 1100 shown in Figure 11. Figure 11 shows a diagram illustrating the manufacturing and maintenance method 1000 of the apparatus according to an example. In one example, in the pre-manufacturing stage, the manufacturing and maintenance method 1000 of the apparatus includes the specification and design 1002 of the apparatus 1100 in Figure 11, and the procurement of materials 1104. In the manufacturing stage, the manufacturing 1006 of the components and subassemblies of the apparatus 1100 in Figure 11, and system integration 1008 are carried out. The apparatus 1100 in Figure 11 then undergoes authorization and delivery 1010 for use in operation 1012. During customer operation, the equipment 1100 in Figure 11 is scheduled for periodic maintenance and servicing 1014, which in one example includes modifications, reconfigurations, refurbishments, and other maintenance or servicing subject to the component management described herein.

[0063] In one example, each process of the equipment manufacturing and maintenance method 1000 is performed or carried out by a system integrator, a third party, and / or an operator. In these examples, the operator is the customer. For the purposes of this explanation, the system integrator includes any number of equipment manufacturers and subcontractors of the main system. The third party includes any number of vendors, subcontractors, and suppliers. In one example, the operator is the owner of the equipment or a fleet of equipment, the manager responsible for the equipment or a fleet of equipment, a user operating the equipment, a leasing company, a military organization, a service agency, etc.

[0064] Referring here to Figure 11, a device 1100 is provided. As shown in Figure 11, an example of a device 1100 is a flying device 1101 such as an aerospace vehicle, aircraft, air cargo, flying vehicle, satellite, planetary probe, deep space probe, solar probe. Further examples of a device 1100, also shown in Figure 11, are ground transport devices 1102 such as automobiles, trucks, heavy machinery, construction machinery, boats, ships, submarines, etc. A further example of a device 1100 shown in Figure 11 is a modular device 1103 including at least one or more of the following modules: an air module, a payload module, and a ground module. The air module provides airlift or flight functionality. The payload module provides functionality for transporting objects such as cargo or living organisms (people, animals, etc.). The ground module provides functionality for ground movement. The solutions disclosed herein may be applied to each module individually, or to groups such as an air module and a payload module, or a payload module and a ground module, or all modules.

[0065] Referring now to Figure 12, a more specific illustration of an aircraft 1101 in which embodiments of the present disclosure are advantageously used is shown. In this example, the aircraft 1101 is an aircraft manufactured by the method for manufacturing and maintaining the apparatus 1000 of Figure 10, and includes a fuselage 1202 with a plurality of systems 1204 and interior 1206. Embodiments of the plurality of systems 1204 include one or more of a propulsion system 1208, an electrical system 1210, a hydraulic system 1212, and an environmental system 1214. However, other systems are also candidates for inclusion. Although an aerospace example is shown, various advantageous examples are applicable to other industries such as the automotive industry.

[0066] The examples disclosed in this book are described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program components, which are executed by computers or other machines such as personal digital assistants or other handheld devices. Generally, program elements, including routines, programs, objects, elements, and data structures, refer to code that performs a specific task or code that implements a specific abstract data type. The disclosed examples are applicable to a variety of system components, including personal computers, laptops, smartphones, mobile tablets, handheld devices, consumer electronics, and specialized computing devices. The disclosed examples are also applicable to distributed computing environments where tasks are performed by remote processing devices linked through communication networks.

[0067] An exemplary method for estimating the position and attitude of fuel containers and boom tips for aerial refueling includes receiving video frames, determining the aircraft keypoint of the aircraft being refueled within the video frames, determining the position and attitude of the fuel containers on the aircraft based at least on the aircraft keypoint, and determining the position and attitude of the boom tip of the aerial refueling boom.

[0068] An exemplary system for estimating the position and attitude of fuel containers and boom tips for aerial refueling comprises one or more processors and a memory which, when executed by one or more processors, stores instructions causing one or more processors to perform operations including receiving a video frame, determining an aircraft keypoint of the aircraft being refueled within the video frame, determining the position and attitude of a fuel container on the aircraft based on at least the aircraft keypoint, and determining the position and attitude of the boom tip of an aerial refueling boom.

[0069] An exemplary computer program product includes a computer-readable medium in which computer-readable program code adapted to be executed for carrying out a method for estimating the position and attitude of a fuel container and boom tip for aerial refueling, the method including receiving a video frame, determining an aircraft keypoint of the aircraft being refueled within the video frame, determining the position and attitude of a fuel container on the aircraft based at least on the aircraft keypoint, and determining the position and attitude of the boom tip of an aerial refueling boom.

[0070] When presenting elements of the aspects of this disclosure or embodiments thereof, the articles “a, an” and “the, said” are intended to indicate that one or more of such elements exist. The words “comprising,” “including,” and “having” are intended to be comprehensive and mean that there may be further elements in addition to those listed. The word “implementation” is intended to mean “an example of.” The expression “one or more of the following: A, B, and C” means “at least one of A and / or at least one of B and / or at least one of C.”

[0071] Further illustrative and non-exclusive examples provided in this disclosure are set out in the following paragraphs. An example provided in this disclosure is a method for estimating the position and attitude of a fuel container and boom tip for aerial refueling, Receiving a video frame (200) (714), Within the video frame, determine the aircraft key point (402) of the aircraft being refueled (110) (722), Based at least on aircraft keypoints, the position and attitude (330) of the fuel container (116) on the aircraft (726) are determined. To determine the position and attitude (630) of the boom tip (106) of the aerial refueling boom (104) (734) There are methods that include this.

[0072] Optionally, this video frame is provided by a single camera.

[0073] Optionally, this method is Within a video frame, a neural network (NN), including a convolutional neural network (CNN), is used to determine the aircraft boundary box for the aircraft being refueled. Cropping video frames to an aircraft bounding box and This further includes determining aircraft keypoints, which includes determining aircraft keypoints within the aircraft boundary box.

[0074] Optionally, determining aircraft key points is possible. This includes using a convolutional neural network (CNN) to generate an aircraft keypoint heatmap of aircraft keypoints.

[0075] Optionally, this method is Controlling the aerial refueling boom to engage with the fuel container, based at least on the position and orientation of the fuel container and the position and orientation of the boom tip. It also includes.

[0076] Optionally, determining the position and orientation of the boom tip is possible. Within the video frame, the boom tip bounding box of the boom tip is determined using boom control parameters and camera calibration information. Within the boom tip boundary box, determine the boom tip keypoint at the boom tip, The position and orientation of the boom tip are determined based at least on the boom tip keypoint. Includes.

[0077] Optionally, the method is: Aircraft model projection overlay and / or boom model projection overlay accompanied by Displaying video frames It also includes.

[0078] Another example provided herein is a system for estimating the position and attitude of a fuel container and boom tip for aerial refueling, One or more processors, Memory, when executed by the one or more processors, the one or more processors Receiving a video frame (200) (714), Within the video frame, determine the aircraft key point (402) of the aircraft being refueled (110) (722), Based at least on aircraft keypoints, the position and attitude (330) of the fuel container (116) on the aircraft (726) are determined. To determine the position and attitude (630) of the boom tip (106) of the aerial refueling boom (104) (734) Memory that stores instructions to perform an action including It is a system that includes [this feature].

[0079] Optionally, this system, Boom control unit that controls the aerial refueling boom to engage with the fuel container. It is further equipped with [this feature].

[0080] Optionally, this operation is: Within a video frame, a neural network (NN), including a convolutional neural network (CNN), is used to determine the aircraft boundary box for the aircraft being refueled. Cropping video frames to an aircraft bounding box and This further includes determining aircraft keypoints, which includes determining aircraft keypoints within the aircraft boundary box.

[0081] Optionally, determining aircraft key points is possible. This includes using a convolutional neural network (CNN) to generate an aircraft keypoint heatmap of aircraft keypoints.

[0082] Optionally, determining the position and orientation of the fuel container is possible. Perform a 2D to 3D transformation of aircraft keypoints using the perspective-n-point (PnP) algorithm. Includes.

[0083] Optionally, determining the position and orientation of the boom tip is possible. Within the video frame, the boom tip bounding box of the boom tip is determined using boom control parameters and camera calibration information. Within the boom tip boundary box, determine the boom tip keypoint at the boom tip, The position and orientation of the boom tip are determined based at least on the boom tip keypoint. Includes.

[0084] Optionally, this operation is: Aircraft model projection overlay and / or boom model projection overlay accompanied by Displaying video frames It also includes.

[0085] An example provided herein is a computer program product including a computer-readable medium in which computer-readable program code adapted to be executed for performing a method for estimating the position and attitude of a fuel container and boom tip for aerial refueling, wherein the method is Receiving a video frame (200) (714), Within the received video frame, determine the aircraft keypoint (402) of the aircraft being refueled (110) (722), Based at least on aircraft keypoints, the position and attitude (330) of the fuel container (116) on the aircraft (726) are determined. To determine the position and attitude (630) of the boom tip (106) of the aerial refueling boom (104), The process involves generating a video frame (500) (744), Overlaying an aircraft model projection onto the generated video frame based at least on the aircraft's position and attitude (744), or Overlay a boom model projection onto the generated video frame, based at least on the position and orientation of the boom tip (744) To generate a video frame (500) including (744), Displaying video frames generated by an aircraft model projection overlay or a boom model projection overlay (746) This includes computer program products.

[0086] Optionally, this video frame is provided by a single camera.

[0087] Optionally, in one example, this method is Within a video frame, a neural network (NN), including a convolutional neural network (CNN), is used to determine the aircraft boundary box for the aircraft being refueled. Cropping video frames to an aircraft bounding box and This further includes determining aircraft keypoints, which includes determining aircraft keypoints within the aircraft boundary box.

[0088] Optionally, determining aircraft key points is possible. This includes using a convolutional neural network (CNN) to generate an aircraft keypoint heatmap of aircraft keypoints.

[0089] Optionally, determining the position and orientation of the fuel container is possible. Perform a 2D to 3D transformation of aircraft keypoints using the perspective-n-point (PnP) algorithm. Includes.

[0090] Optionally, determining the position and orientation of the boom tip is possible. Within the video frame, the boom tip bounding box of the boom tip is determined using boom control parameters and camera calibration information. Within the boom tip boundary box, determine the boom tip keypoint at the boom tip, The position and orientation of the boom tip are determined based at least on the boom tip keypoint. Includes.

[0091] While the aspects of this disclosure have been described in detail, it will be obvious that modifications and variations can be realized without departing from the scope of the aspects of this disclosure as defined in the accompanying claims. Since various modifications can be made to the above structures, products, and methods without departing from the scope of the aspects of this disclosure, all matters included in the above description and illustrated in the accompanying drawings are intended to be interpreted as illustrative and not limiting.

Claims

1. A method for estimating the position and attitude of a fuel container and boom tip for aerial refueling, Receiving a video frame (200) (714), Within the aforementioned video frame, the aircraft key point (402) of the aircraft being refueled (110) is determined (722), Determining the position and orientation (330) of the fuel container (116) on the aircraft based at least on the aircraft keypoint (726), Within the aforementioned video frame, the boom tip key point (452) of the boom tip (106) of the aerial refueling boom (104) is determined, The position and orientation (630) of the boom tip (106) of the aerial refueling boom (104) are determined (734) based at least on the boom tip key point (452) and Includes, Determining the aircraft keypoint (402) includes generating an aircraft keypoint heatmap (400) of the aircraft keypoint (402), Determining the boom tip key point (452) includes generating a boom tip key point heat map (450) of the boom tip key point (452), The aforementioned method further, Controlling the aerial refueling boom to engage with the fuel container, based at least on the position and orientation of the fuel container (116) and the position and orientation of the boom tip (106). Methods that include...

2. The method according to claim 1, wherein the video frame is provided by a single camera.

3. Within the video frame, a neural network (NN), including a convolutional neural network (CNN), is used to determine the aircraft boundary box for the aircraft being refueled. Cropping the aforementioned video frame onto the aircraft boundary box and The method according to claim 1 or 2, further comprising determining the aircraft keypoint, wherein determining the aircraft keypoint within the aircraft boundary box.

4. The method according to any one of claims 1 to 3, wherein a convolutional neural network (CNN) is used to generate the aircraft keypoint heatmap.

5. Each of the aircraft keypoints (402) corresponds to a region of pixels in the aircraft keypoint heatmap (400), the region of pixels has a pixel value, the value decreases according to a probability density function as the distance from the center of the region of pixels increases. The method according to any one of claims 1 to 4, wherein the boom tip keypoint (452) corresponds to a region of pixels in the boom tip keypoint heatmap (450), the region of pixels has a pixel value, and the value decreases according to a probability density function as the distance from the center of the region of pixels increases.

6. Determining the position and orientation of the boom tip is Within the aforementioned video frame, the boom tip bounding box of the boom tip is determined using boom control parameters and camera calibration information. Within the boom tip boundary box, the boom tip key point (452) is determined. The method according to any one of claims 1 to 5, including the method described in any one of claims 1 to 5.

7. Displaying the video frame with an overlay of an aircraft model projection and / or a boom model projection. The method according to any one of claims 1 to 6, further comprising:

8. A system for estimating the position and attitude of a fuel container and boom tip for aerial refueling, One or more processors, The memory that stores the instructions and The instruction is provided, and when the instruction is executed by the one or more processors, the one or more processors are provided with Receiving a video frame (200) (714), Within the aforementioned video frame, the aircraft key point (402) of the aircraft being refueled (110) is determined (722), Determining the position and orientation (330) of the fuel container (116) on the aircraft based at least on the aircraft keypoint (726), Within the aforementioned video frame, the boom tip key point (452) of the boom tip (106) of the aerial refueling boom (104) is determined, The position and orientation (630) of the boom tip (106) of the aerial refueling boom (104) are determined (734) based at least on the boom tip key point (452), and further, Controlling the aerial refueling boom to engage with the fuel container, based at least on the position and orientation of the fuel container and the position and orientation of the boom tip. Perform an action that includes this, Determining the aircraft keypoint (402) includes generating an aircraft keypoint heatmap (400) of the aircraft keypoint (402), Determining the boom tip key point (452) includes generating a boom tip key point heat map (450) of the boom tip key point (452). system.

9. Boom control unit that controls the aerial refueling boom to engage with the fuel container. The system according to claim 8, further comprising the above.

10. The aforementioned operation, Within the video frame, a neural network (NN), including a convolutional neural network (CNN), is used to determine the aircraft boundary box for the aircraft being refueled. Cropping the aforementioned video frame onto the aircraft boundary box and The system according to claim 8 or 9, further comprising, wherein determining the aircraft keypoint includes determining the aircraft keypoint within the aircraft boundary box.

11. The system according to any one of claims 8 to 10, wherein a convolutional neural network (CNN) is used to generate an aircraft keypoint heatmap of the aircraft keypoints.

12. Determining the position and orientation of the fuel container is Performing a 2D-to-3D transformation on the aircraft keypoints using a perspective n-point (PnP) algorithm. The system according to any one of claims 8 to 11, including the system described in any one of claims 8 to 11.

13. Determining the position and orientation of the boom tip is Within the aforementioned video frame, the boom tip bounding box of the boom tip is determined using boom control parameters and camera calibration information. Within the boom tip boundary box, the boom tip key point (452) is determined. The system according to any one of claims 8 to 12, including the system described in any one of claims 8 to 12.

14. The aforementioned operation, Displaying the video frame with an overlay of an aircraft model projection and / or a boom model projection. The system according to any one of claims 8 to 13, further comprising:

15. A computer program product comprising a computer-readable medium in which computer-readable program code adapted to be executed for performing a method for estimating the position and attitude of a fuel container and boom tip for aerial refueling, wherein the method Receiving a video frame (200) (714), In the received video frame, determine the aircraft key point (402) of the aircraft (110) being refueled (722), Determining the position and orientation (330) of the fuel container (116) on the aircraft (726) based at least on the aircraft key points (402), Within the aforementioned video frame (200), the boom tip key point (452) of the boom tip (106) of the aerial refueling boom (104) is determined, Based at least on the boom tip keypoint (452), the position and orientation (630) of the boom tip (106) of the aerial refueling boom (104) are determined (734), Includes, Determining the aircraft keypoint (402) includes generating an aircraft keypoint heatmap (400) of the aircraft keypoint (402), Determining the boom tip key point (452) includes generating a boom tip key point heat map (450) of the boom tip key point (452), The aforementioned method further, The process involves generating a video frame (500) (744), Overlaying an aircraft model projection onto the video frame (200) (744) based at least on the position and attitude of the aircraft, or Overlaying a boom model projection onto the video frame (200) based at least on the position and orientation of the boom tip (744) The process involves generating a video frame (500) (744) that includes, Displaying the video frame (500) generated with the overlay of the aircraft model projection or the overlay of the boom model projection (746) including, Computer program products.

Images