A system for refining the 6-degree-of-freedom attitude estimate of a target object.

Abstract

To provide a system for refining a six-degrees-of-freedom pose estimate of a target object based on a one-dimensional measurement.SOLUTION: A system includes a camera and a range-sensing device. The range-sensing device is configured to determine an actual distance measured between the range-sensing device and an actual point of intersection. The range-sensing device projects a line-of-sight that intersects with the target object at the actual point of intersection. The system also includes one or more processors in electronic communication with the camera and the range-sensing device and a memory coupled to the processors. The memory stores data and a program code into one or more databases. When executed by the processors, the program code causes the system to predict a six-degrees-of-freedom pose estimate of the target object. The system also determines a revised six-degrees-of-freedom pose estimate of the target object based on at least an absolute error.SELECTED DRAWING: None

Description

Technical Field

[0001] The present disclosure relates to a system and method for refining an estimated 6 - degree - of - freedom (6DOF) pose of a target object. More specifically, the present disclosure is directed to a system and method for refining an estimated 6DOF pose of a target object based on a single 1 - dimensional measurement value.

Background Art

[0002] Six degrees of freedom (6DOF) refers to the degrees of freedom of the movement of a rigid body in 3 - dimensional space. Specifically, a rigid body can not only move 3 - dimensionally on the x, y, and z axes, but can also change its orientation between the three axes by rotations called pitch, roll, and yaw.

[0003] Image - based pose estimation systems can estimate the 6DOF pose of an object. Furthermore, many image - based pose estimation systems also utilize a certain type of refinement process to correct an initial estimated 6DOF pose. In some types of pose estimation value refinement processes, a 3 - dimensional depth map is utilized, or instead, a large number of 2 - dimensional distance measurement values obtained using a laser rangefinder are utilized. However, both the 3 - dimensional depth map and the 2 - dimensional distance measurement values typically require significant processing and memory allocation requirements. Additionally, the laser rangefinder used in the 2 - dimensional distance measurement approach may require precisely manufactured moving parts to maintain consistent 2 - dimensional distance measurement values, thus increasing the cost of the system. Furthermore, in some types of pose estimation value refinement approaches, a special calibration pattern or corresponding marker may be required to register the scan lines of the laser rangefinder with corresponding features that are part of the model.

Summary of the Invention

[0004] In some embodiments, a system is disclosed for refining a six-degree-of-freedom pose estimate of a target object based on one-dimensional measurements. The system includes a camera configured to capture image data of the target object and a distance sensor configured to determine the actual distance measured between the distance sensor and the actual intersection. The distance sensor projects a line-of-sight that intersects the target object at the actual intersection. The system also includes one or more processors that electronically communicate with the camera and the distance sensor, as well as memory connected to the one or more processors. The memory stores data and program code to one or more databases, and when the program code is executed by one or more processors, the system predicts a six-degree-of-freedom pose estimate of the target object based on the image data of the target object. The system determines an estimated intersection that represents where the line-of-sight intersects the six-degree-of-freedom pose estimate of the target object. The system also determines an estimated distance measured between the distance sensor and the estimated intersection. The system calculates the absolute error associated with the six-degree-of-freedom pose estimate of the target object based on the difference between the actual distance and the estimated distance. Next, the system determines a modified 6-degree-of-freedom attitude estimate of the target object, based at least on absolute error.

[0005] In another embodiment, an aerial refueling system for a supply aircraft is disclosed. The aerial refueling system includes a boom assembly including a nozzle and a system for determining a modified 6-degree-of-freedom attitude estimate of a fuel receptacle located on a receiver aircraft. The nozzle of the boom assembly is configured to engage with the fuel receptacle of the receiver aircraft during a refueling operation. The system includes a camera configured to capture image data of the receiver aircraft and the fuel receptacle, and a distance sensing device configured to determine the actual distance measured between the distance sensing device and an actual intersection. The distance sensing device projects a line of sight that intersects the receiver aircraft at the actual intersection. The system also includes one or more processors that electronically communicate with the camera and the distance sensing device, as well as a memory coupled to the one or more processors. The memory stores data and program code into one or more databases, and when the program code is executed by one or more processors, the system predicts a 6-degree-of-freedom attitude estimate of the fuel receptacle located on the receiver aircraft based on the image data of the fuel receptacle located on the receiver aircraft. The system determines an estimated intersection point that represents the location where the line of sight intersects with the six-degree-of-freedom attitude estimate of the receiver aircraft. The system determines the estimated distance measured between the distance sensor and the estimated intersection point. Based on the difference between the actual distance and the estimated distance, the system calculates the absolute error associated with the six-degree-of-freedom attitude estimate of the fuel receptacle located on the receiver aircraft. The system then determines a corrected six-degree-of-freedom attitude estimate, at least based on the absolute error.

[0006] In yet another embodiment, a method for refining a six-degree-of-freedom pose estimate of a target object is disclosed. This method includes capturing image data of the target object by a camera. This method also includes determining an actual distance measured between a distance sensor and an actual intersection point by a distance sensor, the distance sensor projecting a line of sight that intersects the target object at the actual intersection point. This method also includes predicting a six-degree-of-freedom pose estimate of the target object based on the image data of the target object. This method further includes determining an estimated intersection point that represents where the line of sight intersects the six-degree-of-freedom pose estimate of the target object. This method further includes determining an estimated distance measured between the distance sensor and the estimated intersection point. This method also includes calculating an absolute error associated with the six-degree-of-freedom pose estimate of the target object based on the difference between the actual distance and the estimated distance. Finally, this method includes determining a corrected six-degree-of-freedom pose estimate based on at least the absolute error.

[0007] The described features, functions, and benefits may be achieved independently in various embodiments or in combination in other embodiments, and further details can be found by referring to the following description and drawings.

[0008] The drawings included herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram of a disclosed system for refining the 6-degree-of-freedom attitude estimate of a target object according to an exemplary embodiment, the system being deployed on a supply aircraft, and the target object being the fuel receptacle of a receiver aircraft. [Figure 2] This figure shows the 6-degree-of-freedom attitude estimates for the extension arm, receiver aircraft, and supply aircraft according to an exemplary embodiment. [Figure 3]This figure illustrates an exemplary approach to determining the 6-degree-of-freedom pose estimate of a target object based on multiple 2D keypoints and multiple 3D keypoints, according to an exemplary embodiment. [Figure 4A] This is a process flow diagram illustrating a method for refining the 6-degree-of-freedom attitude estimate of a target object according to an exemplary embodiment. [Figure 4B] This is a process flow diagram illustrating a method for refining the 6-degree-of-freedom attitude estimate of a target object according to an exemplary embodiment. [Figure 4C] This is a process flow diagram showing a method for determining the reprojection error. [Figure 5] Figure 1 shows a diagram of a computer system for the system disclosed in an exemplary embodiment. [Modes for carrying out the invention]

[0010] This disclosure relates to a system and method for refining a six-degree-of-freedom pose estimate of a target object based on a single one-dimensional measurement. The system includes a camera and a control module that electronically communicates with a distance sensor. The camera is configured to capture image data of the target object, and the distance sensor is configured to determine a one-dimensional measurement. The distance sensor determines the actual distance measured between the distance sensor and an actual intersection W'. Specifically, the actual intersection W' represents the location where the line of sight projected by the distance sensor intersects the target object. The system determines a six-degree-of-freedom pose estimate of the target object based on the image data captured by the camera. Next, the system determines an estimated intersection representing the location where the line of sight intersects the six-degree-of-freedom pose estimate of the target object. Next, the system determines an estimated distance measured between the distance sensor and the estimated intersection. The system calculates an absolute error based on the difference between the actual distance and the estimated distance. In one embodiment, the system also determines a reprojection error introduced by the six-degree-of-freedom pose estimate of the target object. Next, the system determines a corrected pose estimate of the target object based on the absolute error and, where possible, the reprojection error.

[0011] The following descriptions are illustrative in nature and are not intended to limit this disclosure, application, or use.

[0012] Referring to Figure 1, a system 10 for refining the 6-degree-of-freedom attitude estimate 8 (shown in Figure 2) of a target object 12 is shown. In the example shown in Figure 1, system 10 is part of an aerial refueling system 14 located in the tail section 16 of a tanker aircraft or supply aircraft 18. The aerial refueling system 14 includes a boom assembly 20 connected to the fuselage 22 of the supply aircraft 18 at an articulated joint 24. The boom assembly 20 includes a rigid section 26, a telescopic extension 28, and a nozzle 30. The nozzle 30 of the boom assembly 20 engages with a fuel receptacle 32 of a receiver aircraft 34 during a refueling operation. In the example shown, the receiver aircraft 34, and more specifically the fuel receptacle 32, represent the target object 12. Thus, system 10 refines the 6-degree-of-freedom attitude estimate 8 (Figure 2) of the fuel receptacle 32 and the receiver aircraft 34. The system 10 includes a control module 40 that electronically communicates with a camera 42 and a distance sensing device 44. The camera 42 is positioned to capture image data of the target object 12 (i.e., the fuel receptacle 32 and the receiver aircraft 34). Based on the image data captured by the camera 42, the control module 40 predicts an estimated 6-degree-of-freedom attitude 8 of the target object 12.

[0013] Referring to both Figures 1 and 2, the distance sensing device 44 is configured to determine the actual distance d between the distance sensing device 44 (shown in Figure 2) and the target object 12 (i.e., the fuel receptacle 32). The actual distance d represents a single one-dimensional measurement determined by the distance sensing device 44. As described below, the control module 40 of the system 10 determines the absolute error associated with the 6-degree-of-freedom attitude estimate 8 of the target object 12 based on the actual distance d. In one embodiment, the control module 40 determines a modified 6-degree-of-freedom attitude estimate of the target object 12 based on the absolute error. In another embodiment, also described below, the control module 40 also determines the reprojection error associated with the estimation of the 6-degree-of-freedom attitude estimate 8, and then determines a modified 6-degree-of-freedom attitude estimate based on both the reprojection error and the absolute error.

[0014] In the example shown in Figure 1, the control module 40 determines the position and orientation of the boom assembly 20 based on a modified 6-degree-of-freedom attitude estimate. However, it should be understood that Figure 1 is essentially illustrative and the system 10 is not limited to the aerial refueling system 14. In fact, the system 10 can be used in a variety of other applications where a 6-degree-of-freedom attitude estimate of a rigid object is estimated. As seen in Figure 2, the system 10 includes an extendable arm 38. The extendable arm 38 is represented by the boom assembly 20 shown in Figure 1, but it should be understood that the extendable arm 38 is not limited to the boom assembly 20. For example, in another embodiment, the extendable arm 38 is a robotic arm that grasps and manipulates an object. In this example, the control module 40 determines the position and orientation of the extendable arm 38 based on a modified 6-degree-of-freedom attitude estimate when the extendable arm 38 grasps and manipulates an object.

[0015] Camera 42 transmits a video or image feed to the control module 40. In the non-limiting embodiment shown in Figure 1, camera 42 is mounted on the underside 46 of the fuselage 22 of the supply aircraft 18. However, it should be understood that the position of camera 42 is not limited to a specific location on the receiver aircraft 34. Instead, camera 42 can be placed anywhere camera 42's field of view 50 captures the target object 12. For example, in the embodiment shown in Figure 1, camera 42 could be mounted along any number of locations along the underside 46 of the fuselage 22 of the supply aircraft 18, as long as camera 42's field of view 50 captures the fuel receptacle 32 and the receiver aircraft 34.

[0016] The distance sensing device 44 is any type of device for determining the distance to a specific target location without requiring physical contact. The distance sensing device 44 includes, but is not limited to, a laser rangefinder, an ultrasonic sensor, an infrared distance sensor, an optical detection and ranging (lidar) sensor, or a sonar sensor. In a non-limiting embodiment as shown in Figure 1, the distance sensing device 44 is statically mounted on the distal end 48 of the rigid part 26 of the boom assembly 20. In an example as seen in Figure 2, the distance sensing device 44 is also mounted on the distal end 52 of the extendable arm 38. As seen in both Figures 1 and 2, the line of sight L of the distance sensing device 44 is aligned with the longitudinal axis A-A of the extendable arm 38 (or boom assembly 20). Thus, the control module 40 determines the position and line of sight L of the distance sensing device 44 based on the movement of the extendable arm 38. For example, if the extendable arm 38 is a robotic arm, the control module 40 determines the position and line of sight L based on the joint angles of the robotic arm.

[0017] It should be understood that the distance detection device 44 can be located in various places other than the rigid part 26 of the boom assembly 20 as shown in Figure 1 or the extendable arm 38 as shown in Figure 2. In other words, the line of sight L of the distance detection device 44 does not have to be in a straight line with the longitudinal axis A-A of the extendable arm 38. Instead, the distance detection device 44 can be located at any location where the line of sight L of the distance detection device 44 intersects with the target object 12. For example, in an alternative embodiment, the distance detection device 44 is mounted immediately next to the camera 42 on the underside 46 of the fuselage 22 of the supply aircraft 18.

[0018] Referring to Figures 1 and 2, the position, orientation, and intrinsic parameters of camera 42 are determined in a preliminary offline camera calibration procedure, or the intrinsic parameters are stored in the memory 1034 of the control module 40 (Figure 5). Some examples of the intrinsic parameters of camera 42 include, but are not limited to, resolution and aspect ratio. A three-dimensional representation 54 of the target object 12 is shown by the dashed lines in Figure 2. The three-dimensional representation 54 is also stored in the memory 1034 of the control module 40. The control module 40 is configured to predict a 6-degree-of-freedom attitude estimate 8 of the target object 12 (i.e., the fuel receptacle 32 and the receiver aircraft 34) based on image data captured by camera 42, using any number of attitude estimation approaches. For example, in one non-limiting embodiment, the control module 40 determines the 6-degree-of-freedom attitude estimate 8 of the target object 12 based on a perspective-n-point algorithm.

[0019] Referring to both Figures 1 and 3, the perspective n-point algorithm estimates a 6-degree-of-freedom pose estimate 8 (Figure 2) of the target object 12 based on a plurality of 2D keypoints 60 and a plurality of 3D keypoints 62 (the 2D keypoints 60 are shown as circles and the 3D keypoints 62 are shown as crosses). Specifically, the perspective n-point algorithm requires three or more 3D keypoints 62 placed on the target object 12. The 3D keypoints 62 are detected by the control module 40 based on image data captured by the camera 42. The control module 40 detects three or more 3D keypoints 62 on the target object 12 (i.e., the fuel receptacle 32) in each image frame of the image feed received from the camera 42. The control module 40 then uses a deep neural network to predict the corresponding 2D keypoint 60 for each of the plurality of 3D keypoints 62. Next, the control module 40 aligns multiple 3D keypoints 62 with their corresponding 2D keypoints 60, and then predicts 6-degree-of-freedom attitude estimates based on the 3D keypoints 62.

[0020] While the perspective n-point algorithm is described, it should be understood that other pose estimation processes can also be used to determine a 6-degree-of-freedom pose estimate. For example, in an alternative approach, the 6-degree-of-freedom pose estimate is determined based on two or more point-tangent correspondences between a 3D keypoint 62 and a 2D keypoint 60. In another embodiment, the 6-degree-of-freedom pose estimate is determined by a deep neural network that determines the 6-degree-of-freedom pose estimate directly based on image data captured by the camera 42.

[0021] Returning to FIGS. 1 and 2, when the control module 40 determines the six-degree-of-freedom attitude estimated value 8, the control module 40 then aligns the longitudinal axis A-A of the extensible arm 38 in the direction towards the target object 12 (i.e., the receiver aircraft 34). The first six-degree-of-freedom attitude estimated value 8 as described above may be a locally rough estimated value, and it should be understood that the longitudinal axis A-A of the extensible arm 38 (and thus the line of sight L of the distance detection device 44) only needs to generally intersect the target object 12 (i.e., the receiver aircraft 34). In the example shown in FIGS. 1 and 2, since the nozzle 30 of the boom assembly 20 engages with the fuel receptacle 32 of the receiver aircraft 34 during the refueling operation, the distance detection device 44 projects the line of sight L towards the fuel receptacle 32.

[0022] The distance detection device 44 is configured to determine the actual distance d. Specifically, referring to FIG. 2, the actual distance d is measured between the distance detection device 44 and the actual intersection point W'. The line of sight L projected by the distance detection device 44 intersects the target object 12 (i.e., the fuel receptacle 32) at the actual intersection point W'. Therefore, the actual distance d represents a one-dimensional depth measurement value between the distance detection device 44 and the target object 12. It should be understood that before the attitude refinement process using the distance detection device 44, depth estimated values such as the estimated value of the distance d are associated with the largest amount of error when compared with the measured values of length and height. This is because the first six-degree-of-freedom attitude estimated value is based on the viewpoint of the camera 42 without a depth queue. Furthermore, it should also be understood that it is not necessary to know the position of the actual intersection point W' on the surface 70 of the receiver aircraft 34. Finally, it should be understood that the actual intersection point W' may be anywhere on the surface 70 of the receiver aircraft 34.

[0023] Specifically, referring to FIG. 2, next, the control module 40 determines an estimated intersection point W. The estimated intersection point W represents the location where the line of sight L intersects the 6-degree-of-freedom pose estimate 8 of the target object 12. As shown in FIG. 2, due to the coarseness of the initial 6-degree-of-freedom pose estimate 8, the estimated intersection point W is offset from the actual intersection point W'. Next, the control module 40 determines an estimated distance D measured between the distance detection device 44 and the estimated intersection point W. Next, the control module 40 calculates an absolute error associated with the 6-degree-of-freedom pose estimate 8 of the target object 12 based on the difference between the actual distance d and the estimated distance D. Specifically, the absolute error is expressed as follows in Equation 1. TIFF0007830123000001.tif8170Here, O represents the base of the extendable arm 38 shown in FIG. 2, and W' = (O + dL). In other words, Equation 1 can be expressed as TIFF0007830123000002.tif10170as

[0024] In addition to the absolute error, in one embodiment, the control module 40 also determines the reprojection error introduced by the 6-degree-of-freedom pose estimate 8. Specifically, the reprojection error represents the difference between a plurality of 2D pixel positions and a plurality of 2D keypoints 60 shown in Figure 3. The plurality of 2D pixel positions are determined by projecting a 3D keypoint 62 (Figure 2) into 2D space. It should be understood that the 3D keypoint 62 shown in Figure 3 is represented in camera space. Camera space refers to a 3D coordinate system with an origin represented by the center C of camera 42 (Figure 1), where the user defines three axes (i.e., x, y, and z). Thus, the 3D keypoint 62 shows how the target object 12 looks with respect to the perspective field of view of camera 42. For example, if the target object 12 is located 20 meters directly in front of camera 42, the z-coordinate of the resulting 3D keypoint 62 (assuming it is aligned with the line of sight of camera 42) is 20 meters. It should also be understood that when a 3D keypoint 62 is projected into 2D space to represent a 2D pixel position, the 3D keypoint 62 is flattened along the depth dimension. However, the distance sensing device 44 is advantageously aligned with the depth dimension and therefore adds information that would otherwise be missing from the 2D pixel position.

[0025] The reprojection error of the perspective n-point algorithm is expressed in Equation 2 as follows: TIFF0007830123000003.tif9170 Here, P represents the camera projection function of camera 42, V represents the coordinate transformation matrix, R represents the rotation matrix representing three of the six degree-of-freedom parameters (pitch, roll, yaw), X represents a matrix containing multiple 3D keypoints 62, t represents a vector representing the positional components (x, y, z) of the 6 degree-of-freedom parameters, and y' represents a 2D keypoint 60 (shown in Figure 3). The camera projection function of camera 42 transforms the 3D keypoints 62 represented in camera space into 2D space. The coordinate transformation matrix V transforms the 3D keypoints 62 represented in model space into camera space. Model space represents a 3D coordinate system with an origin 74 (seen in Figure 2) located at the center of the 3D representation 54. Vector t contains the positional components of the 6 degree-of-freedom parameters and defines the transformation between the origin 74 in model space and the center C in camera space. Similarly, the rotation matrix R includes the orientation component of the 6-degree-of-freedom parameter and defines rotations between axes defined in model space and axes defined in camera space.

[0026] In one embodiment, the control module 40 determines a modified 6-degree-of-freedom attitude estimate based solely on absolute error. In this embodiment, the control module 40 determines the minimum absolute error and then calculates a modified 6-degree-of-freedom attitude estimate that generates or results from the minimum absolute error. In other words, the control module 40 determines a value of the refined 6-degree-of-freedom attitude estimate associated with a minimum amount of absolute error. The minimum absolute error is expressed in Equation 3 as follows: TIFF0007830123000004.tif10170 Here, θ represents the estimated attitude of the target object 12 with 6 degrees of freedom, i.e., θ = [x, y, z, pitch, roll, yaw].

[0027] In another embodiment, the control module 40 determines a modified 6-degree-of-freedom attitude estimate 8 based on both absolute error and reprojection error. In one embodiment, the control module 40 determines a modified 6-degree-of-freedom attitude estimate by first determining the minimum value of a weighted sum combining the absolute error and reprojection error together. The weighted sum is expressed in Equation 4 as follows: TIFF0007830123000005.tif10170 Here, λ represents a user-defined scale factor. Changing the value of the scale factor λ yields specific embodiments for describing the relative accuracy of the distance sensor 44 and the 6-degree-of-freedom attitude estimate 8. The minimum weighted sum is determined based on a nonlinear least-squares algorithm. There are several types of nonlinear least-squares algorithms that can be used to determine the minimum weighted sum. Examples of nonlinear least-squares algorithms include, but are not limited to, gradient methods such as the Gauss-Newton method, the Levenberg-Marquardt method, and the conjugate gradient method, and direct search methods such as the Nelder-Mead simplex search.

[0028] Figures 4A to 4B are exemplary process flow diagrams illustrating method 200 for refining the 6-degree-of-freedom pose estimate 8 (Figure 2) of the target object 12. Generally referring to Figures 1 to 4A, method 200 begins in block 202. In block 202, camera 42 captures image data of the target object 12. Next, method 200 can proceed to block 204.

[0029] In block 204, the distance detection device 44 determines the actual distance d. As described above, the actual distance d is measured between the distance detection device 44 and the actual intersection point W' (shown in Figure 2), and the distance detection device 44 projects a line of sight L that intersects with the target object 12 at the actual intersection point W'. Next, method 200 can proceed to block 206.

[0030] In block 206, the control module 40 predicts a 6-degree-of-freedom pose estimate 8 of the target object 12 based on the image data of the target object 12. As described above, the 6-degree-of-freedom pose estimate 8 can be determined using any number of pose estimation approaches, such as the perspective n-point algorithm. Next, method 200 can proceed to block 208.

[0031] In block 208, the control module 40 determines an estimated intersection point W (Figure 2) that represents the location where the line of sight L intersects with the estimated 6-degree-of-freedom pose value 8 of the target object 12. Next, method 200 can proceed to block 210.

[0032] In block 210, the control module 40 determines the estimated distance D measured between the distance detection device 44 and the estimated intersection W. Next, method 200 can proceed to block 212.

[0033] In block 212, the control module 40 calculates the absolute error associated with the 6-degree-of-freedom attitude estimate 8 of the target object 12 based on the difference between the actual distance and the estimated distance. Next, method 200 can proceed to decision block 214.

[0034] In decision block 214, the modified 6-degree-of-freedom pose estimate is determined based on absolute error only, or, in other ways, on absolute error and reprojection error. If the control module 40 determines that the modified 6-degree-of-freedom pose estimate is determined based on absolute error only, the method proceeds to block 216.

[0035] In block 216, the control module 40 calculates the minimum absolute error. As described above, the control module 40 calculates the absolute error associated with the 6-degree-of-freedom attitude estimate 8 of the target object 12 based on the difference between the actual distance d and the estimated distance D (expressed in Equation 1). Next, method 200 can proceed to block 218.

[0036] In block 218, the control module 40 calculates a modified 6-degree-of-freedom attitude estimate that generates the minimum absolute error. Method 200 can then be terminated.

[0037] Returning to decision block 214, if the corrected 6-degree-of-freedom attitude estimate is not determined based solely on absolute error, method 200 proceeds to block 220 shown in Figure 4B. Specifically, if the control module 40 determines the corrected 6-degree-of-freedom attitude estimate based on both absolute error and reprojection error, method 200 proceeds to block 220.

[0038] In block 220, the control module 40 determines the reprojection error introduced by the 6-degree-of-freedom attitude estimate 8 of the target object 12. As explained above, the reprojection error represents the difference between multiple 2D pixel positions and multiple 2D keypoints 60 shown in Figure 3. It should be understood that the process flow diagram for determining the reprojection error is shown in Figure 4C. Next, method 200 can proceed to block 222.

[0039] In block 222, the control module 40 determines the minimum weighted sum of the absolute error and the reprojection error combined. The minimum weighted sum can be determined using various different approaches, such as the Levenberg-Marquardt algorithm. Method 200 can then proceed to block 224.

[0040] In block 224, the control module 40 calculates a modified 6-degree-of-freedom attitude estimate that generates the minimum weighted sum. In one embodiment, method 200 can then proceed to block 226.

[0041] In block 226, in one embodiment, the disclosed system 10 includes an extendable arm 38 (as shown in Figure 1 as boom assembly 20 and in Figure 2). Thus, in block 226, in response to determining the modified 6-degree-of-freedom attitude estimate, the control module 40 determines the position and orientation of the extendable arm 38 based on the modified 6-degree-of-freedom attitude estimate. The method 200 can then be terminated.

[0042] Referring to Figure 4C, a process flow diagram illustrating method 250 for determining the reprojection error is described. Referring to Figures 1, 3, and 4C, method 250 begins in block 252. In block 252, the control module 40 detects a number of three-dimensional keypoints 62 corresponding to the target object 12 based on image data captured by the camera 42. Method 250 can then proceed to block 254.

[0043] In block 254, the deep neural network predicts the corresponding two-dimensional keypoint 60 for each of the multiple three-dimensional keypoints 62. Next, method 250 can proceed to block 256.

[0044] In block 256, the control module 40 aligns multiple three-dimensional keypoints 62 with multiple two-dimensional keypoints 60. Next, method 250 can proceed to block 258.

[0045] In block 258, the control module 40 predicts a 6-degree-of-freedom attitude estimate 8 based on the 3D keypoint 62. Next, method 250 can proceed to block 260.

[0046] In block 260, the control module 40 determines multiple 2D pixel positions by projecting multiple 3D keypoints 62 into 2D space. Next, method 250 can proceed to block 262.

[0047] In block 262, the control module 40 determines the difference between a plurality of 2D pixel positions and a plurality of 2D keypoints 60, and the difference between the plurality of 2D pixel positions and the plurality of 2D keypoints 60 represents the reprojection error. After that, method 250 can be terminated.

[0048] Referring to the figures in general, the disclosed system offers various technical effects and benefits. Specifically, the disclosed system refines a 6-degree-of-freedom pose estimate by utilizing a single 1D measurement from a distance sensing device, as opposed to a 2D scan or alternatively a 3D depth map. Thus, the disclosed system does not require the considerable processing and memory allocation requirements, or a laser rangefinder with precisely manufactured moving parts, as is the case with some conventional systems currently available. Furthermore, unlike some conventional systems currently available, the disclosed system does not require any special calibration patterns or corresponding markers during the refinement process.

[0049] Referring to Figure 5, the control module 40 in Figure 1 may be implemented on one or more computer devices or systems, such as the exemplary computer system 1030. The computer system 1030 includes a processor 1032, memory 1034, mass storage device 1036, an input / output (I / O) interface 1038, and a human-machine interface (HMI) 1040. The computer system 1030 is operably connected to one or more external resources 1042 via a network 1026 or the I / O interface 1038. External resources may include, but are not limited to, servers, databases, mass storage devices, peripherals, cloud-based network services, or any other suitable computer resources that may be used by the computer system 1030.

[0050] The processor 1032 includes one or more devices selected from microprocessors, microcontrollers, digital signal processors, microprocessors, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other devices that operate signals (analog or digital) based on operation instructions stored in memory 1034. The memory 1034 includes, but is not limited to, a single memory device or multiple memory devices, including read-only memory (ROM), random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information. The mass storage device 1036 includes data storage devices such as hard drives, optical drives, tape drives, volatile or non-volatile solid-state devices, or any other devices capable of storing information.

[0051] Processor 1032 operates under the control of operating system 1046, which resides in memory 1034. Operating system 1046 manages computer resources so that computer program code, embodied as one or more computer software applications such as application 1048 located in memory 1034, can be directed by processor 1032 to execute instructions. In another example, processor 1032 could directly execute application 1048, in which case operating system 1046 could be omitted. One or more data structures 1049 also reside in memory 1034 and can be used by processor 1032, operating system 1046, or application 1048 to store or manipulate data.

[0052] The I / O interface 1038 provides a machine interface that enables the processor 1032 to be operationally connected to other devices and systems, such as the network 1026 or external resources 1042. This allows application 1048 to operate in cooperation with the network 1026 or external resources 1042 by communicating through the I / O interface 1038, providing a variety of features, functions, applications, processes, or modules, including examples of the present disclosure. Application 1048 also includes program code that depends on functions or signals executed by one or more external resources 1042, or otherwise provided by other systems or network components outside the computer system 1030. Indeed, given the virtually limitless hardware and software configurations possible, those skilled in the art will understand that examples of the present disclosure may include applications located outside the computer system 1030, applications distributed across multiple computers or other external resources 1042, or applications provided by computing resources (hardware and software) delivered as a service over the network 1026, such as cloud computing services.

[0053] The HMI 1040 is operably coupled to the processor 1032 of the computer system 1030 in a known manner that allows the user to interact directly with the computer system 1030. The HMI 1040 may include a video or alphanumeric display, a touchscreen, a speaker, and any other suitable audio and visual indicators that can provide data to the user. The HMI 1040 also includes input devices and control devices that can receive commands or inputs from the user and transmit the inputs to the processor 1032, such as an alphanumeric keyboard, a pointing device, a keypad, push buttons, control knobs, and a microphone.

[0054] Database 1044 can reside on mass storage device 1036 and can be used to collect and organize data used by the various systems and modules described herein. Database 1044 may include data and supporting data structures for storing and organizing the data. In detail, database 1044 can be arranged in any database configuration or structure, including but not limited to relational databases, hierarchical databases, network databases, or combinations thereof. Information or data stored in the records of database 1044 can be accessed in response to queries using a database management system in the form of a computer software application executed as instructions on processor 1032, and queries may be dynamically determined and executed by the operating system 1046, other applications 1048, or one or more modules.

[0055] Further illustrative and non-exclusive examples provided in this disclosure are described in the following paragraphs.

[0056] In one example provided in this disclosure, a system (10) for refining the 6-degree-of-freedom pose estimate (8) of a target object (12) based on one-dimensional measurements is provided. A camera (42) configured to capture image data of a target object (12), A distance detection device (44) configured to determine the actual distance (d) measured between the distance detection device (44) and the actual intersection point (W'), the distance detection device (44) projecting a line of sight (L) that intersects with the target object (12) at the actual intersection point (W'), One or more processors (1032) that communicate electronically with the camera (42) and the distance detection device (44), and The system (10) includes memory (1034) linked to one or more processors (1032), the memory (1034) stores data and program code for one or more databases (1044), and when the program code is executed by one or more processors (1032), it is sent to the system (10). Based on image data of the target object (12), predict the estimated 6-degree-of-freedom pose (8) of the target object (12) (206). Determine the estimated intersection point (W) that represents the location where the line of sight (L) intersects with the estimated pose (8) of the target object (12) with 6 degrees of freedom (208). Determining the estimated distance (D) measured between the distance detection device (44) and the estimated intersection point (W) (210), Based on the difference between the actual distance (d) and the estimated distance (D), calculate the absolute error associated with the 6-degree-of-freedom pose estimate (8) of the target object (12) (212), and The system is made to calculate (216) a modified 6-degree-of-freedom attitude estimate of the target object (12) based on at least absolute error.

[0057] Optionally, in the system (10) described in the previous paragraph, one or more processors (1032) Instructions for calculating the minimum absolute error (214), and Execute the instruction (216) to calculate a modified 6-degree-of-freedom attitude estimate that generates the minimum absolute error.

[0058] Optionally, in one of the systems (10) from the previous paragraph, one or more processors (1032) Instructions for determining the reprojection error (218) introduced by the 6-degree-of-freedom attitude estimate (8) of the target object (12), Instructions for determining the minimum weighted sum of absolute error and reprojection error (220), and Execute the instruction (222) to calculate the modified 6-degree-of-freedom attitude estimate that generates the minimum weighted sum.

[0059] Optionally, in one of the systems (10) from the previous paragraph, one or more processors (1032) determine the minimum value of the weighted sum based on a nonlinear least-squares algorithm.

[0060] Optionally, in one of the systems (10) from the previous paragraph, one or more processors (1032) Based on the perspective n-point algorithm, an instruction is executed to determine the estimated pose (8) of the target object (12) with 6 degrees of freedom.

[0061] Optionally, in one of the systems (10) from the previous paragraph, one or more processors (1032) Instructions for detecting (252) multiple three-dimensional keypoints (62) corresponding to a target object (12) based on image data captured by the camera (42), Instructions for predicting the corresponding 2D keypoint (60) using a deep neural network (254) for each of the multiple 3D keypoints (62), Instructions for aligning multiple 3D keypoints (62) with multiple 2D keypoints (62) (256), and An instruction (258) is executed to predict a 6-degree-of-freedom pose estimate (8) based on 3D keypoints (62).

[0062] Optionally, in one of the systems (10) from the previous paragraph, one or more processors (1032) Instructions for determining the positions of multiple 2D pixels (260) by projecting multiple 3D keypoints (62) onto a 2D space, and An instruction (262) is executed to determine (262) the difference between multiple 2D pixel positions and multiple 2D keypoints (60), which represents the reprojection error.

[0063] Optionally, one of the systems (10) from the previous paragraph further comprises an extendable arm (38) that defines a longitudinal axis, the longitudinal axis of the extendable arm (38) being in line with the line of sight (L) of the distance detection device (44).

[0064] Optionally, in one of the systems (10) from the previous paragraph, the extendable arm (38) is either the boom assembly (20) of an aerial refueling system (14) or a robotic arm.

[0065] Optionally, in one of the systems (10) from the previous paragraph, the distance sensing device (44) is statically attached to the extendable arm (38).

[0066] Optionally, in one of the systems (10) from the previous paragraph, one or more processors (1032) In response to determining the corrected 6-degree-of-freedom attitude estimate, an instruction (224) is executed to determine the position and orientation of the extendable arm (38) based on the corrected 6-degree-of-freedom attitude estimate.

[0067] Optionally, in one of the systems (10) in the preceding paragraph, the distance sensing device (44) is a laser rangefinder, an ultrasonic sensor, an infrared distance sensor, a light detection and distance measuring (lidar) sensor, or a sonar sensor.

[0068] Optionally, in one of the systems (10) from the previous paragraph, the actual distance (d) represents a one-dimensional depth measurement between the distance sensor (44) and the target object (12).

[0069] In another example provided in this disclosure, the aerial refueling system (14) of a supply aircraft (18) A boom assembly (20) including a nozzle (30), The system (10) comprises a system (10) for determining a modified 6-degree-of-freedom attitude estimate of a fuel receptacle (32) located on a receiver aircraft (34), wherein the nozzle (30) of the boom assembly (20) is configured to engage with the fuel receptacle (32) of the receiver aircraft (34) during a refueling operation, and the system (10) A camera (42) configured to capture image data (202) of the receiver aircraft (34) and fuel receptacle (32), A distance detection device (44) configured to determine (204) the actual distance (d) measured between the distance detection device (44) and the actual intersection point (W'), the distance detection device (44) projecting a line of sight (L) that intersects with the receiver aircraft at the actual intersection point (W'), One or more processors (1032) that communicate electronically with a camera (42) and a distance detection device (44), The system comprises one or more processors (1032) and a memory (1034) linked thereto, the memory (1034) storing data and program code for one or more databases (1044), and the program code, when executed by one or more processors (1032), is sent to the system (10). Based on image data of the fuel receptacle (32) located on the receiver aircraft (34), predict the 6-degree-of-freedom attitude estimate (8) of the fuel receptacle (32) located on the receiver aircraft (34) (206). Determine the estimated intersection point (W) that represents the location where the line of sight (L) intersects with the estimated attitude values ​​(8) of the receiver aircraft (34) in 6 degrees of freedom (208). Determining the estimated distance (D) measured between the distance detection device (44) and the estimated intersection point (W) (210), Based on the difference between the actual distance (d) and the estimated distance (D), calculate the absolute error associated with the 6-degree-of-freedom attitude estimate (8) of the fuel receptacle (32) located on the receiver aircraft (34) (212), and The system is instructed to calculate a corrected 6-degree-of-freedom attitude estimate based on at least the absolute error (216).

[0070] Optionally, in the aerial refueling system (14) of the previous paragraph, one or more processors (1032) Instructions for calculating the minimum absolute error (214), and Execute the instruction (216) to calculate a modified 6-degree-of-freedom attitude estimate that generates the minimum absolute error.

[0071] Optionally, in one of the aerial refueling systems (14) from the previous paragraph, one or more processors (1032) are configured Instructions for determining (218) the reprojection error introduced by the 6-degree-of-freedom attitude estimate (8) of the fuel receptacle (32) located on the receiver aircraft (34), Instructions for determining the minimum weighted sum of absolute error and reprojection error (220), and Execute the instruction (222) to calculate the modified 6-degree-of-freedom attitude estimate that generates the minimum weighted sum.

[0072] In another example provided herein, a method (200) for refining the 6-degree-of-freedom pose estimate (8) of a target object (12) is described as follows: The camera (42) captures image data of the target object (12) (202), The distance detection device (44) determines (204) the actual distance (d) measured between the distance detection device (44) and the actual intersection point (W'), wherein the distance detection device (44) projects and determines (204) the line of sight (L) that intersects with the target object (12) at the actual intersection point (W'). Based on image data of the target object (12), predict the estimated 6-degree-of-freedom pose (8) of the target object (12) (206). Determine the estimated intersection point (W) that represents the location where the line of sight (L) intersects with the estimated pose (8) of the target object (12) with 6 degrees of freedom (208). Determining the estimated distance (D) measured between the distance detection device (44) and the estimated intersection point (W) (210), Based on the difference between the actual distance (d) and the estimated distance (D), calculate the absolute error associated with the 6-degree-of-freedom pose estimate (8) of the target object (12) (212), and This includes calculating a corrected 6-degree-of-freedom attitude estimate based on at least absolute error (216).

[0073] Optionally, the method described in the previous paragraph is Calculating the minimum value of the absolute error (214), and This further includes calculating a modified 6-degree-of-freedom attitude estimate that generates the minimum absolute error (216).

[0074] You can choose one of the methods from the previous paragraph. The reprojection error introduced by the 6-degree-of-freedom attitude estimate (8) of the target object (12) is determined (218). Determining the minimum value of the weighted sum of the absolute error and the reprojection error combined (220), and This further includes calculating a modified 6-degree-of-freedom pose estimate that generates the minimum weighted sum (222).

[0075] Optionally, in one of the methods described in the previous paragraph, the 6-degree-of-freedom pose estimate (8) of the target object (12) is determined based on the perspective n-point algorithm.

[0076] The descriptions in this disclosure are essentially illustrative, and any modifications that do not deviate from the gist of this disclosure are intended to be within the scope of this disclosure. Such modifications should not be considered a deviation from the spirit and scope of this disclosure.

Claims

1. A system (10) for refining the 6-degree-of-freedom pose estimate (8) of a target object (12) based on one-dimensional measurements, wherein the system (10) A camera (42) configured to capture image data of the target object (12), A distance detection device (44) that projects a line of sight (L) that intersects with the target object (12) at an actual intersection (W'), wherein the distance detection device (44) is configured to determine the actual distance (d) measured between the distance detection device (44) and the actual intersection (W'), One or more processors (1032) that communicate electronically with the camera (42) and the distance detection device (44), and A memory (1034) connected to one or more processors (1032), The system (10) is provided with the memory (1034) storing data and program code for one or more databases (1044), and when the program code is executed by one or more processors (1032), the system (10) provides the following: Based on the image data of the target object (12), predict the estimated pose (8) of the six degrees of freedom of the target object (12) (206). Determine the estimated intersection point (W) which represents the location where the line of sight (L) intersects with the three-dimensional representation defined by the estimated 6 degrees of freedom of the target object (12) (208). Determining the estimated distance (D) measured between the distance detection device (44) and the estimated intersection point (W) (210), Based on the difference between the actual distance (d) and the estimated distance (D), calculate the absolute error associated with the estimated 6-degree-of-freedom pose (8) of the target object (12) (212), and To calculate the corrected 6-degree-of-freedom attitude estimate of the target object (12) such that it yields at least the minimum of the absolute error (216), A system (10) that causes this to happen.

2. The one or more processors (1032) described above, An instruction (214) for calculating the minimum value of the absolute error, and An instruction (216) to calculate the modified 6-degree-of-freedom attitude estimate that generates the minimum value of the absolute error, The system (10) according to claim 1, which performs the following:

3. The one or more processors (1032) described above, Commands for determining (218) the reprojection error introduced by the 6-degree-of-freedom attitude estimate (8) of the target object (12), Instructions for determining the minimum value of the weighted sum obtained by combining the absolute error and the reprojection error (220), and Instructions for calculating (222) the modified 6-degree-of-freedom attitude estimate that generates the minimum value of the weighted sum, A system (10) according to claim 1 or 2, which performs the following:

4. The system (10) according to claim 3, wherein one or more processors (1032) determine the minimum value of the weighted sum based on a nonlinear least squares algorithm.

5. The one or more processors (1032) described above, A system (10) according to any one of claims 1 to 4, which executes a command to determine the estimated pose (8) of the six degrees of freedom of the target object (12) based on a perspective n-point algorithm.

6. The one or more processors (1032) described above, An instruction (252) for detecting (252) a plurality of three-dimensional keypoints (62) which are feature points on the three-dimensional representation of the target object (12), based on the image data captured by the camera (42), Instructions for predicting (254) the corresponding 2D keypoint (60), which is the position point in the plane of the image data, using a deep neural network for each of the plurality of 3D keypoints (62), Commands for aligning the plurality of three-dimensional keypoints (62) with the plurality of two-dimensional keypoints (60) (256), and An instruction (258) to predict the 6-degree-of-freedom attitude estimate (8) based on the three-dimensional keypoint (62), The system (10) according to claim 5, which performs the following:

7. The one or more processors (1032) described above, Commands for determining the positions of a plurality of two-dimensional pixels (260) by projecting the plurality of three-dimensional keypoints (62) onto the two-dimensional space which is the plane of the image data, and An instruction (262) for determining the difference between the plurality of two-dimensional pixel positions and the plurality of two-dimensional keypoints (60) that represent the reprojection error, The system (10) according to claim 6, which performs the following:

8. The system (10) according to any one of claims 1 to 7, further comprising an extendable arm (38) that defines a longitudinal axis, wherein the longitudinal axis of the extendable arm (38) is in line with the line of sight (L) of the distance detection device (44).

9. The extendable arm (38) is a boom assembly (20) or robotic arm of an aerial refueling system (14), The system (10) according to claim 8, wherein the distance detection device (44) is statically attached to the extendable arm (38).

10. The one or more processors (1032) described above, The system (10) according to claim 8 or 9, which, in response to determining the modified six-degree-of-freedom attitude estimate, executes an instruction to determine (224) the position and orientation of the extendable arm (38) based on the modified six-degree-of-freedom attitude estimate.

11. The distance detection device (40) is a laser rangefinder, an ultrasonic sensor, an infrared distance sensor, a light detection and distance measuring (lidar) sensor, or a sonar sensor. The system (10) according to any one of claims 1 to 10, wherein the actual distance (d) represents a one-dimensional depth measurement between the distance detection device (44) and the target object (12).

12. A method (200) for refining the estimated pose (8) of a target object (12) with six degrees of freedom, The camera (42) captures image data of the target object (12) (202), The distance detection device (44) projects a line of sight (L) that intersects with the target object (12) at the actual intersection (W'), thereby determining the actual distance (d) measured between the distance detection device (44) and the actual intersection (W') (204). Based on the image data of the target object (12), predict the estimated pose (8) of the six degrees of freedom of the target object (12) (206). Determine the estimated intersection point (W) which represents the location where the line of sight (L) intersects with the estimated pose values ​​(8) of the six degrees of freedom of the target object (12) (208). Determining the estimated distance (D) measured between the distance detection device (44) and the estimated intersection point (W) (210), Based on the difference between the actual distance (d) and the estimated distance (D), calculate the absolute error associated with the estimated 6-degree-of-freedom pose (8) of the target object (12) (212), and Calculate the corrected 6-degree-of-freedom attitude estimate based at least on the absolute error (216), A method including (200).

13. Calculating the minimum value of the absolute error (214), and Calculate the modified 6-degree-of-freedom attitude estimate that generates the minimum value of the absolute error (216), The method according to claim 12 (200), further comprising:

14. Determining the reprojection error introduced by the 6-degree-of-freedom attitude estimate (8) of the target object (12) (218), Determining the minimum value of the weighted sum obtained by combining the absolute error and the reprojection error (220), and Calculate the modified 6-degree-of-freedom attitude estimate that generates the minimum value of the weighted sum (222), The method according to claim 12 or 13, further comprising (200).

15. The method according to any one of claims 12 to 14 (200), wherein the estimated pose value (8) of the six degrees of freedom of the target object (12) is determined based on a perspective n-point algorithm.

Images