Autonomous aerial refueling taper sleeve visual feature detection and pose measurement method and system

Abstract

The invention discloses a visual feature detection and pose measurement method for an autonomous aerial refueling taper sleeve, and the method comprises the steps: defining and calibrating the relative relation between required coordinate systems, and the coordinate systems comprise a world coordinate system, a camera coordinate system, an image coordinate system and a pixel coordinate system; a motion video of the taper sleeve is collected, key points of a taper sleeve picture are marked, and a taper sleeve data set is constructed; building a model infrastructure, and training to form a taper sleeve key point detection model for a sub-pixel-level key point pixel detection result of key point features on the non-cooperative target taper sleeve; fitting an elliptic equation of the inner and outer rings of the taper sleeve based on a sub-pixel-level key point detection coordinate result; based on the relative relation between the elliptic equation of the inner ring and the outer ring of the taper sleeve and the coordinate system, the initial value of the taper sleeve pose is solved through a double-space taper sleeve pose measurement method; the invention discloses a double-space circle pose measurement optimization method based on chamfering distance loss. A taper sleeve pose initial value is optimized through key point feature pose re-projection. The invention further discloses a corresponding system, electronic equipment and a computer readable storage medium.

Description

Technical Field

[0001] The present invention relates to the field of computer vision measurement technology, and in particular to a method and system for autonomous aerial refueling drogue visual feature detection and posture measurement. Background Art

[0002] Autonomous aerial refueling can significantly improve the aircraft's endurance and combat radius, enhance long-range strike capability, and improve air defense capabilities. Among them, the high-precision visual position measurement of the drogue is the key basis for successful docking.

[0003] At present, the methods of visual feature detection and pose measurement of the drogue are mainly used for drogues with active LED or color threshold passive optical cooperation targets, which are used to assist feature detection and visual measurement. However, the features of the above drogues are very different from those of drogues without optical cooperation targets in actual aerial scenes. Therefore, the existing methods have key problems such as difficult feature positioning and low measurement accuracy in actual aerial refueling scenes.

[0004] The feature detection method based on deep learning has the advantages of strong adaptability, high robustness and high precision, and can be applied to the visual feature detection of autonomous aerial refueling drogue. However, the feature detection method based on deep learning has problems such as complex calculation and long inference time. At the same time, autonomous aerial refueling mostly relies on airborne embedded devices or edge computing devices with limited computing resources. Therefore, how to realize the real-time detection and measurement of autonomous aerial refueling visual features on the airborne platform is also one of the key issues. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for visual feature detection and posture measurement of an autonomous aerial refueling drogue, wherein the visual feature detection method can realize real-time detection of sub-pixel key point features of a non-cooperative target drogue, and the posture measurement method can realize visual posture measurement based on the key point features of the drogue, thereby providing robust and accurate measurement results for the autonomous aerial refueling system and improving the efficiency of autonomous aerial refueling, thereby generally solving the problems of difficult positioning of key point features of non-cooperative targets, long reasoning time, and low measurement accuracy in the prior art, so as to achieve the purpose of robust and accurate measurement of non-cooperative target drogues.

[0006] The first aspect of the present invention is to provide a method for autonomous aerial refueling drogue visual feature detection and posture measurement, comprising:

[0007] S1, defining and calibrating the relative relationship between coordinate systems required for the position and posture measurement of the autonomous aerial refueling drogue, wherein the required coordinate systems include a world coordinate system, a camera coordinate system, an image coordinate system, and a pixel coordinate system;

[0008] S2, collecting motion video of the cone sleeve, marking key points of the cone sleeve image and constructing a cone sleeve dataset; wherein the motion video of the cone sleeve is collected at a fixed video frame rate and image resolution;

[0009] S3, building an infrastructure of a basic cone sleeve key point detection model, training the basic cone sleeve key point detection model to form a cone sleeve key point detection model based on deep learning, the cone sleeve key point detection model is used for sub-pixel key point pixel detection results of key point features on non-cooperative target cone sleeves;

[0010] S4, based on the sub-pixel key point detection coordinate results of the key point features output by the cone sleeve key point detection model, fitting the ellipse equations of the inner and outer circles of the cone sleeve; based on the relative relationship between the ellipse equations of the inner and outer circles of the cone sleeve and the coordinate system, solving the initial value of the cone sleeve posture by a dual-space cone sleeve posture measurement method;

[0011] S5, a dual-space circle pose measurement optimization method based on chamfer distance loss, optimizes the initial value of the cone sleeve pose by reprojecting the key point feature pose.

[0012] Preferably, the S1 comprises:

[0013] S11, define the coordinate origin and direction of the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system; the coordinate origin of the world coordinate system is defined at the position of the refueling pipe cone head of the oil receiving machine, and its three-axis coordinate directions are three basic coordinate axes of x-axis, y-axis and z-axis, which are perpendicular to each other and together constitute a rectangular coordinate system in three-dimensional space; among which: x-axis: usually represents the horizontal direction, with positive to the right and negative to the left; y-axis: usually represents the vertical direction, with positive downward and negative upward; z-axis: usually represents the depth direction, with positive forward and negative backward; the origin of the camera coordinate system is located at the optical center of the camera, and its three-axis coordinate directions are the camera optical axis perpendicular to the image plane and outward as Z c Axis, X c and Y c According to the right-hand coordinate system, they are parallel to the image plane respectively; the origin of the image coordinate system is the center of the image, and the directions of its two axes are that the positive direction of the x-axis is rightward and the positive direction of the y-axis is downward; the origin of the pixel coordinate system is the upper left corner of the image as the origin of the coordinate system, and the directions of its two axes are that the positive direction of the u-axis is rightward and the positive direction of the v-axis is downward;

[0014] S12, calibrating the relative position relationship among the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system; including:

[0015] (1) First, calibrate the camera’s internal parameters;

[0016] (2) Then, according to the camera intrinsic parameter imaging model principle and the spatial transformation relationship, the camera extrinsic parameters are calibrated based on the principle of minimizing the reprojection error between the two-dimensional detection points and the three-dimensional space points optimized by LM;

[0017] (3) Finally, the relative position relationship among the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system is determined based on formula (1), which is as follows:

[0018]

[0019] Wherein, R, T represent the relative position relationship between the world coordinate system and the camera coordinate system; u0 and v0 are the origin coordinates of the image coordinate system after the image coordinate system is converted to the pixel coordinate system; f x and f y are the normalized focal lengths of the x-axis and y-axis of the image coordinate system, u and v are the pixel plane coordinates, and x, y, and z are the world coordinate system coordinates.

[0020] Preferably, S2 includes:

[0021] S21, using a camera fixed on the nose of the receiving aircraft to collect multiple motion videos of the drogue and extract original images of the drogue, wherein the multiple motion videos include motion videos of multiple scenes, multiple scales and different drogue positions, and the camera collects the multiple motion videos at a fixed video frame rate and image resolution;

[0022] S22, marking key point features of the edges of the inner ring and the outer ring of the taper sleeve and generating a corresponding marking file;

[0023] S23, constructing a cone sleeve dataset based on the arrangement of the cone sleeve original images and the annotation files, wherein the cone sleeve dataset includes a training set, a validation set and a test set, and the proportions of the number of images in the training set, the validation set and the test set are 60%, 20% and 20% respectively.

[0024] Preferably, S3 includes:

[0025] S31, construct a basic cone sleeve key point detection model; the basic cone sleeve key point detection model includes 1 Steam, 4 Stages, 3 Downsample modules and 1 Neck module; the structure reparameterization technology is used to merge the convolution blocks into a 3*3 convolution, and the sub-pixel key point output head based on coordinate classification is used to encode and decode the key point coordinates. The specific formulas of key point encoding and decoding are shown in formula (2) and formula (3) respectively:

[0026] The key point encoding formula is shown in formula (2):

[0027]

[0028] Among them, σ is the standard deviation of the coordinate Gaussian distribution encoding, i and j are row vector index and column vector index respectively, are the horizontal and vertical coordinates of the key points before encoding, x i ,y j They are Gaussian distribution on the horizontal axis and Gaussian distribution on the vertical axis after encoding respectively.

[0029] The key point decoding formula is shown in formula (3):

[0030]

[0031] Where α is the scaling factor of the coordinate encoding and decoding, and argmax() is the function of the maximum value index in the output Gaussian distribution; x (i),o y (j) are the Gaussian distribution of the horizontal axis and the Gaussian distribution of the vertical axis before the cone sleeve key point detection model predicts the output decoding, They are the horizontal and vertical coordinates of the key points after decoding.

[0032] S32, constructing a loss function of the cone sleeve key point detection model, training the basic cone sleeve key point detection model based on the training set, the validation set and the loss function until convergence to obtain the cone sleeve key point detection model; the loss function includes a key point classification loss function, a KL divergence loss function, an MSE regression loss function and a chamfer distance loss function. The formulas of each loss function are shown in formulas (4)-(7):

[0033] The key point classification loss function is shown in formula (4):

[0034]

[0035] Among them, N is the number of training samples, M is the number of key point categories, and y ij is the label category of the jth key point of the i-th sample, p ij is the predicted category probability of the jth key point of the i-th sample.

[0036] The KL divergence loss function is shown in formula (5):

[0037]

[0038] Among them, M is the number of key point categories, Y(ij) is the labeled Gaussian distribution of the i-th key point in the j-th coordinate direction, and X(ij) is the predicted Gaussian distribution of the i-th key point in the j-th coordinate direction.

[0039] The MSE regression loss function is shown in formula (6):

[0040]

[0041] in, are the marked coordinates and predicted coordinates of the i-th key point on the inner ring ellipse of the cone sleeve, are the marked coordinates and predicted coordinates of the i-th key point on the outer ring ellipse of the cone sleeve.

[0042] The chamfer distance loss function is shown in formula (7):

[0043]

[0044] in, is the set of marked points and predicted points on the j-th ellipse of the cone sleeve, D chamfer is the chamfer distance loss function.

[0045] In summary, considering the above loss functions, the designed total loss function is shown in formula (8):

[0046]

[0047] Among them, L cls ,η1 is the classification loss and its corresponding weight, L KL ,η2 is the KL divergence loss and its corresponding weight, L MSE ,η3 is the MSE regression loss and its corresponding weight, L chamfer ,η4 is the chamfer distance loss and its corresponding weight, L total is the total classification loss;

[0048] S33, performing reasoning based on the cone sleeve key point detection model and outputting sub-pixel key point detection pixel coordinate results, including:

[0049] (1) using each frame of the test video in the test set as input to the cone sleeve key point detection model;

[0050] (2) The cone-set key point detection model performs inference and outputs the sub-pixel level key point detection coordinate results of the corresponding frame image.

[0051] Preferably, S4 includes:

[0052] S41, based on the determination that the inner and outer rings of the cone sleeve are annular and imaged as an ellipse on the image plane, and the sub-pixel key point detection coordinate results of the key point features output by the cone sleeve key point detection model, an ellipse fitting algorithm is used to calculate the ellipse equations of the inner and outer rings of the cone sleeve in the image plane pixel coordinate system; wherein the ellipse fitting algorithm is implemented based on the least squares method, and the ellipse equations of the inner and outer rings of the cone sleeve are shown in formula (9):

[0053] au2 +bv 2 +cuv+du+ev+f=0 (9);

[0054] Among them, u, v are pixel plane coordinates, a, b, c, d, e, f are the parameters of the general equation of the ellipse;

[0055] S42, based on the relative relationship between the ellipse equations of the inner and outer rings of the cone sleeve and the coordinate system, solving the initial value of the cone sleeve posture by a dual-space cone sleeve posture measurement method.

[0056] Preferably, the S42 includes:

[0057] (1) The general ellipse equation corresponding to the ellipse equations of the inner and outer rings of the tapered sleeve is transformed into an ellipse quadratic equation as shown in the following equation (10):

[0058] [uv 1]E[uv 1] T =0 (10)

[0059] Wherein, u, v are pixel plane coordinates, and E is the elliptic quadratic equation matrix; according to the general ellipse equation (9) and the elliptic quadratic equation (10), the elliptic quadratic equation matrix E is specifically shown in the following equation (11):

[0060]

[0061] Among them, a, b, c, d, e, and f are the parameters of the general equation of the ellipse;

[0062] (2) According to the camera imaging principle and perspective projection principle, the camera imaging formula corresponding to the mapping relationship between the three-dimensional point in space and the two-dimensional point in the plane is determined as shown in the following formula (12):

[0063]

[0064] Among them, K is the camera internal parameter matrix, f x ,f y , u0, v0 represent the relative position relationship between the camera coordinate system and the image coordinate system and the pixel coordinate system; u0 and v0 are the origin coordinates of the image coordinate system after converting the image coordinate system to the pixel coordinate system; f x and f y are the normalized focal lengths of the x-axis and y-axis of the image coordinate system, u and v are the pixel plane coordinates, and x, y, and z are the world coordinate system coordinates;

[0065] (3) According to the elliptic quadratic form formulas (10), (11) and the camera imaging formula (12), the spatial equation of the cone-in-cone in the camera coordinate system is obtained as shown in the following formula (13):

[0066] [xyz]KT EK[xyz]T=0 (13);

[0067] Among them, K is the camera internal parameter matrix, x, y, z are the coordinates of the world coordinate system, and E is the elliptic quadratic equation matrix; for the convenience of expression, define Q = K T EK;

[0068] (4) Determine the transformation matrix P and transform the view cone equation from the camera coordinate system to the standard coordinate system. The spatial equation of the view cone in the standard coordinate system is shown in the following equation (14):

[0069] P T QP=P -1 QP=diag{λ1,λ2,λ3} (14);

[0070] Where P is the transformation matrix, Q = K T EK, λ1, λ2, λ3 are matrices P T The characteristic value of QP;

[0071] (5) calculating the coordinates of the center point of the cone sleeve in the standard coordinate system and the plane normal vector based on the spatial equation of the viewing cone in the standard coordinate system;

[0072] The coordinates of the center point of the taper sleeve in the standard coordinate system are shown in the following formula (15):

[0073]

[0074] Among them, R is the radius of the cone sleeve space circle, λ1,λ2,λ3 are the matrix P T The characteristic value of QP, It is the coordinate of the center point of the cone sleeve space circle in the standard coordinate system.

[0075] The plane normal vector of the cone sleeve in the standard coordinate system is shown in the following formula (16):

[0076]

[0077] Among them, λ1,λ2,λ3 are the matrices P T The characteristic value of QP, is the plane normal vector of the cone sleeve space circle in the standard coordinate system;

[0078] (6) The center point coordinates and the plane normal vector are transformed from the standard coordinate system back to the camera coordinate system through the transformation matrix P. The center point coordinates and the plane normal vector of the cone sleeve in the camera coordinate system are shown in the following formula (17):

[0079]

[0080] Where P is the transformation matrix, is the coordinate of the center point of the cone sleeve space circle in the standard coordinate system, x, y, z are the coordinates of the center point of the cone sleeve space circle in the camera coordinate system; is the plane normal vector of the cone sleeve space circle in the standard coordinate system, n x ,n y ,n z is the plane normal vector of the cone sleeve space circle in the camera coordinate system;

[0081] At the same time, the plane normal vector of the cone sleeve in the standard coordinate system is converted into the pitch angle and yaw angle of the cone sleeve in the camera coordinate system as shown in formula (18):

[0082]

[0083] Among them, n x ,n y ,n z is the plane normal vector of the cone sleeve space circle in the camera coordinate system, φ is the pitch angle of the cone sleeve in the camera coordinate system, and β is the yaw angle of the cone sleeve in the camera coordinate system.

[0084] Preferably, S5 includes:

[0085] S51, based on the initial value of the cone sleeve posture obtained in S42, according to the camera perspective projection formula and the three-dimensional digital-analog point coordinates of the cone sleeve, calculate the pixel coordinates of the two-dimensional point of the cone sleeve posture reprojection on the pixel plane;

[0086] S52, constructing a minimization target optimization function based on the chamfer distance loss function of the two-dimensional detection point and the pose reprojection two-dimensional point, wherein the minimization target optimization function is shown in the following formula (19):

[0087]

[0088] Among them, P i is the pixel coordinate point set of the two-dimensional detection point, is the pixel coordinate point set of the pose reprojected two-dimensional point, are the optimized rotation matrix and translation vector, argmin() is the Levenberg-Marquardt optimization algorithm;

[0089] The chamfer distance loss function is shown in formula (20):

[0090]

[0091] Among them, P i is the pixel coordinate point set of the two-dimensional detection point, is the pixel coordinate point set of the pose reprojected two-dimensional point; |P i | is the number of corresponding point sets, For the two-dimensional detection point set Pi Middle detection point p ij The pixel coordinate point set of the two-dimensional point reprojected by the pose Center Projection Point The minimum distance between .

[0092] The second aspect of the present invention is to provide an autonomous aerial refueling drogue visual feature detection and posture measurement system, which is used to implement the method of the first aspect, including:

[0093] A system calibration module (101) is used to define and calibrate the relative relationship between coordinate systems required for the position and posture measurement of the autonomous aerial refueling drogue, wherein the required coordinate systems include a world coordinate system, a camera coordinate system, an image coordinate system and a pixel coordinate system;

[0094] An image acquisition module (102) is used to acquire motion video of the cone sleeve, mark key points of the cone sleeve image and construct a cone sleeve data set; wherein the motion video of the cone sleeve is acquired at a fixed video frame rate and image resolution;

[0095] A feature detection module (103) is used to build a basic framework of a basic cone sleeve key point detection model, train the basic cone sleeve key point detection model to form a cone sleeve key point detection model based on deep learning, and the cone sleeve key point detection model is used for sub-pixel key point pixel detection results of key point features on a non-cooperative target cone sleeve;

[0096] A posture measurement module (104) is used to fit the cone sleeve inner and outer circle ellipse equations based on the sub-pixel key point detection coordinate results of the key point features output by the cone sleeve key point detection model; and to solve the cone sleeve initial posture value by a dual-space cone sleeve posture measurement method based on the relative relationship between the cone sleeve inner and outer circle ellipse equations and the coordinate system;

[0097] A posture optimization module (105) is used for a dual-space circular posture measurement optimization method based on chamfer distance loss, and optimizes the initial value of the cone sleeve posture by reprojecting key point feature postures.

[0098] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is configured to read the instructions and execute the method described in the first aspect.

[0099] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor to execute the method described in the first aspect.

[0100] Beneficial effects of the method and system of the present invention:

[0101] The visual feature detection method can realize real-time detection of sub-pixel key point features of the non-cooperative target cone sleeve. At the same time, the pose measurement method can realize visual pose measurement based on the key point features of the cone sleeve, providing robust and accurate measurement results for the autonomous aerial refueling system and improving the efficiency of autonomous aerial refueling. It can generally solve the problems of difficult positioning of key point features of non-cooperative targets, long reasoning time, and low measurement accuracy in the existing technology, so as to achieve the purpose of robust and accurate measurement of non-cooperative target cone sleeves. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0103] Figure 1 A flow chart of a method for autonomous aerial refueling drogue visual feature detection and posture measurement according to an embodiment of the present invention;

[0104] Figure 2 This is a flow chart of step S1 of the autonomous aerial refueling drogue visual feature detection and posture measurement method provided according to an embodiment of the present invention;

[0105] Figure 3 A schematic diagram of the relative relationship between various coordinate systems of an aerial refueling measurement system provided according to an embodiment of the present invention;

[0106] Figure 4 This is a flow chart of step S2 of the autonomous aerial refueling drogue visual feature detection and posture measurement method provided according to an embodiment of the present invention;

[0107] Figure 5 This is a flow chart of step S3 of the autonomous aerial refueling drogue visual feature detection and posture measurement method provided according to an embodiment of the present invention;

[0108] Figure 6 A schematic diagram of a basic cone sleeve key point detection model provided according to an embodiment of the present invention;

[0109] Figure 7 A schematic diagram of key point encoding and decoding based on coordinate classification according to an embodiment of the present invention;

[0110] Figure 8 This is a flow chart of step S4 of the autonomous aerial refueling drogue visual feature detection and posture measurement method provided according to an embodiment of the present invention;

[0111] Fig. 9This is a flow chart of step S5 of the autonomous aerial refueling drogue visual feature detection and posture measurement method provided according to an embodiment of the present invention;

[0112] Fig.10 Autonomous aerial refueling drogue visual feature detection and posture measurement system architecture diagram;

[0113] Fig.11 A structural diagram of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0114] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0115] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0116] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0117] Embodiment 1

[0118] like Figure 1 As shown, this embodiment provides a method for autonomous aerial refueling drogue visual feature detection and posture measurement, including:

[0119] S1, defining and calibrating the relative relationship between coordinate systems required for the position and posture measurement of the autonomous aerial refueling drogue, wherein the required coordinate systems include a world coordinate system, a camera coordinate system, an image coordinate system, and a pixel coordinate system;

[0120] like Figure 2As shown, as a preferred embodiment, the S1 includes:

[0121] S11, defining the coordinate origin and direction of the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system;

[0122] In this embodiment, the coordinate origin of the world coordinate system is defined at the position of the refueling pipe cone of the receiving aircraft, and its three-axis coordinate directions are the three basic coordinate axes of the x-axis, y-axis and z-axis. These three coordinate axes are perpendicular to each other and together constitute a three-dimensional rectangular coordinate system. Among them: the x-axis: usually represents the horizontal direction, rightward is positive, leftward is negative; the y-axis: usually represents the vertical direction, downward is positive, upward is negative; the z-axis: usually represents the depth direction, forward is positive, backward is negative;

[0123] The origin of the camera coordinate system is located at the optical center of the camera, and the three-axis coordinate direction is the camera optical axis perpendicular to the image plane and outward as Z c Axis, X c and Y c Following the right-hand coordinate system, they are parallel to the image plane;

[0124] The origin of the image coordinate system is the center of the image, and the directions of the two axes are that the positive direction of the x-axis is to the right, and the positive direction of the y-axis is downward;

[0125] The coordinate origin of the pixel coordinate system is the upper left corner of the image, and the coordinate directions of the two axes are that the positive direction of the u axis is rightward and the positive direction of the v axis is downward.

[0126] like Figure 3 As shown, point p in the image coordinate system is the projection point of point P in the camera coordinate system, and the corresponding relationship between the image coordinate system and the pixel coordinate system is as follows Figure 3 As shown, u0 and v0 are the origin coordinates of the image coordinate system after the image coordinate system is converted to the pixel coordinate system.

[0127] S12, calibrating the relative position relationship among the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system.

[0128] In this embodiment, the S12 includes:

[0129] (1) First, the camera’s internal parameters are calibrated. The specific solution method is described in detail in Zhang Zhengyou’s article “A flexible new technique for camera calibration [R]. Microsoft Corporation, NSRTR-98-71, 1998”.

[0130] (2) Then, according to the principle of the camera intrinsic parameter imaging model and the spatial transformation relationship, the camera extrinsic parameters are calibrated based on the principle of minimizing the reprojection error between the two-dimensional detection points and the three-dimensional space points optimized by LM.

[0131] (3) Finally, the relative position relationship among the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system is determined based on formula (1), which is as follows:

[0132]

[0133] Wherein, R, T represent the relative position relationship between the world coordinate system and the camera coordinate system; u0 and v0 are the origin coordinates of the image coordinate system after the image coordinate system is converted to the pixel coordinate system; f x and f y are the normalized focal lengths of the x-axis and y-axis of the image coordinate system, u and v are the pixel plane coordinates, and x, y, and z are the world coordinate system coordinates.

[0134] S2, collecting motion video of the cone sleeve, marking key points of the cone sleeve image and constructing a cone sleeve dataset; wherein the motion video of the cone sleeve is collected at a fixed video frame rate and image resolution;

[0135] See also Figure 4 As a preferred implementation, S2 includes:

[0136] S21, using a camera fixed on the nose of the receiving aircraft to collect multiple motion videos of the drogue and extract original images of the drogue, wherein the multiple motion videos include motion videos of multiple scenes, multiple scales and different drogue positions, and the camera collects the multiple motion videos at a fixed video frame rate and image resolution;

[0137] S22, marking key point features of the edges of the inner ring and the outer ring of the taper sleeve and generating a corresponding marking file;

[0138] Specifically, in this embodiment, the inner ring of the cone sleeve includes 12 edge key points, and the outer ring includes 36 edge key points. All key points are evenly distributed on the edges of the inner ring and outer ring of the cone sleeve according to angles to form edge key point features;

[0139] S23, constructing a cone sleeve dataset based on the arrangement of the cone sleeve original images and the annotation files, wherein the cone sleeve dataset includes a training set, a validation set and a test set, and the proportions of the number of images in the training set, the validation set and the test set are 60%, 20% and 20% respectively.

[0140] S3, building an infrastructure of a basic cone sleeve key point detection model, training the basic cone sleeve key point detection model to form a cone sleeve key point detection model based on deep learning, the cone sleeve key point detection model is used for sub-pixel key point pixel detection results of key point features on non-cooperative target cone sleeves;

[0141] As a preferred embodiment, Figure 5 As shown, the S3 includes:

[0142] S31, ref. Figure 6 The basic cone sleeve key point detection model schematic diagram shown in the figure is used to construct the basic cone sleeve key point detection model; the basic cone sleeve key point detection model includes 1 Steam, 4 Stages, 3 Downsample modules and 1 Neck module; it is used to realize the extraction and fusion of underlying details, high-level semantics and multi-scale features; among them, in order to realize lightweight model and real-time processing, the structural re-parameterization technology is adopted to merge the convolution blocks into a 3*3 convolution, which effectively reduces the number of parameters and the computational cost of the model, and improves the inference speed and real-time performance of the detection model in airborne embedded devices and edge computing devices; at the same time, see Figure 7 In order to improve the detection accuracy of cone-set key points, a sub-pixel key point output head based on coordinate classification is used to encode and decode the key point coordinates. By converting the coordinate regression task into a coordinate classification task, the one-dimensional Gaussian distribution reduces the amount of output head calculation compared to the two-dimensional heat map method of Heatmap, and facilitates the training convergence of the cone-set key point detection model; at the same time, when the coordinate encoding and decoding scaling factor is greater than 1, the key point encoding and decoding accuracy based on coordinate classification can reach the sub-pixel level; the specific formulas for key point encoding and decoding are shown in equations (2) and (3), respectively:

[0143] The key point encoding formula is shown in formula (2):

[0144]

[0145] Among them, σ is the standard deviation of the coordinate Gaussian distribution encoding, i and j are row vector index and column vector index respectively, are the horizontal and vertical coordinates of the key points before encoding, x i ,y j They are the Gaussian distributions of the horizontal and vertical axes after encoding, respectively.

[0146] The key point decoding formula is shown in formula (3):

[0147]

[0148] Where α is the scaling factor of the coordinate encoding and decoding, and argmax() is the function of the maximum value index in the output Gaussian distribution; x (i),oy (j) are the Gaussian distributions of the horizontal and vertical coordinates before the cone sleeve key point detection model predicts the output decoding, They are the horizontal and vertical coordinates of the key points after decoding.

[0149] S32, constructing a loss function of the cone sleeve key point detection model, training the basic cone sleeve key point detection model based on the training set, the validation set and the loss function until convergence to obtain the cone sleeve key point detection model; the loss function includes a key point classification loss function, a KL divergence loss function, an MSE regression loss function and a chamfer distance loss function. The formulas of each loss function are shown in formulas (4)-(7):

[0150] The key point classification loss function is shown in formula (4):

[0151]

[0152] Among them, N is the number of training samples, M is the number of key point categories, and y ij is the label category of the jth key point of the i-th sample, p ij is the predicted category probability of the jth key point of the i-th sample.

[0153] The KL divergence loss function is shown in formula (5):

[0154]

[0155] Among them, M is the number of key point categories, Y(ij) is the labeled Gaussian distribution of the i-th key point in the j-th coordinate direction, and X(ij) is the predicted Gaussian distribution of the i-th key point in the j-th coordinate direction.

[0156] The MSE regression loss function is shown in formula (6):

[0157]

[0158] in, are the marked coordinates and predicted coordinates of the i-th key point on the inner ring ellipse of the cone sleeve, are the marked coordinates and predicted coordinates of the i-th key point on the outer ring ellipse of the cone sleeve.

[0159] The chamfer distance loss function is shown in formula (7):

[0160]

[0161] in, is the set of marked points and predicted points on the j-th ellipse of the cone sleeve, D chamfer is the chamfer distance loss function.

[0162] In summary, considering the above loss functions, the designed total loss function is shown in formula (8):

[0163]

[0164] Among them, L cls ,η1 is the classification loss and its corresponding weight, L KL ,η2 is the KL divergence loss and its corresponding weight, L MSE ,η3 is the MSE regression loss and its corresponding weight, L chamfer ,η4 is the chamfer distance loss and its corresponding weight, L total is the total classification loss;

[0165] S33, performing inference based on the cone sleeve key point detection model and outputting sub-pixel key point detection pixel coordinate results;

[0166] In this embodiment, the S33 includes:

[0167] (1) using each frame of the test video in the test set as input to the cone sleeve key point detection model;

[0168] (2) The cone-set key point detection model performs inference and outputs the sub-pixel level key point detection coordinate results of the corresponding frame image.

[0169] S4, based on the sub-pixel key point detection coordinate results of the key point features output by the cone sleeve key point detection model, fitting the ellipse equations of the inner and outer circles of the cone sleeve; based on the relative relationship between the ellipse equations of the inner and outer circles of the cone sleeve and the coordinate system, solving the initial value of the cone sleeve posture by a dual-space cone sleeve posture measurement method;

[0170] As a preferred embodiment, Figure 8 As shown, the S4 includes:

[0171] S41, based on the determination that the inner and outer rings of the cone sleeve are annular and imaged as an ellipse on the image plane, and the sub-pixel key point detection coordinate results of the key point features output by the cone sleeve key point detection model, an ellipse fitting algorithm is used to calculate the ellipse equations of the inner and outer rings of the cone sleeve in the image plane pixel coordinate system; wherein the ellipse fitting algorithm is implemented based on the least squares method, and the ellipse equations of the inner and outer rings of the cone sleeve are shown in formula (9):

[0172] au 2 +bv 2 +cuv+du+ev+f=0 (9);

[0173] Among them, u and v are pixel plane coordinates, and a, b, c, d, e, and f are the parameters of the general equation of the ellipse.

[0174] S42, based on the ellipse equations of the inner and outer rings of the cone sleeve and the relative relationship between the coordinate systems, solving the initial value of the cone sleeve posture by a dual-space cone sleeve posture measurement method, including:

[0175] (1) The general ellipse equation corresponding to the ellipse equations of the inner and outer rings of the tapered sleeve is transformed into an ellipse quadratic equation as shown in the following equation (10):

[0176] [uv 1]E[uv 1] T =0 (10)

[0177] Wherein, u, v are pixel plane coordinates, and E is the elliptic quadratic equation matrix. According to the general ellipse equation (9) and the elliptic quadratic equation (10), the elliptic quadratic equation matrix E is specifically shown in the following equation (11):

[0178]

[0179] Among them, a, b, c, d, e, and f are the parameters of the general equation of the ellipse.

[0180] (2) According to the camera imaging principle and perspective projection principle, the camera imaging formula corresponding to the mapping relationship between the three-dimensional point in space and the two-dimensional point in the plane is determined as shown in the following formula (12):

[0181]

[0182] Among them, K is the camera internal parameter matrix, f x ,f y , u0, v0 represent the relative position relationship between the camera coordinate system and the image coordinate system and the pixel coordinate system; u0 and v0 are the origin coordinates of the image coordinate system after converting the image coordinate system to the pixel coordinate system; f x and f y are the normalized focal lengths of the x-axis and y-axis of the image coordinate system, u and v are the pixel plane coordinates, and x, y, and z are the world coordinate system coordinates.

[0183] (3) According to the elliptic quadratic form formulas (10), (11) and the camera imaging formula (12), the spatial equation of the cone-in-cone in the camera coordinate system is obtained as shown in the following formula (13):

[0184] [xyz]K T EK[xyz] T =0 (13);

[0185] Among them, K is the camera internal parameter matrix, x, y, z are the coordinates of the world coordinate system, and E is the elliptic quadratic equation matrix; for the convenience of expression, define Q = K T EK.

[0186] (4) According to the matrix principle, the matrix Q is a real symmetric matrix. Therefore, there exists a transformation matrix P that can diagonalize the matrix Q. At the same time, in order to facilitate matrix calculation and solution, the transformation matrix P can transform the view cone equation from the camera coordinate system to the standard coordinate system. The spatial equation of the view cone in the standard coordinate system is shown in the following formula (14):

[0187] P T QP=P -1 QP=diag{λ1,λ2,λ3} (14);

[0188] Where P is the transformation matrix, Q = K T EK, λ1, λ2, λ3 are matrices P T Characteristic value of QP.

[0189] (5) calculating the coordinates of the center point of the cone sleeve in the standard coordinate system and the plane normal vector based on the spatial equation of the viewing cone in the standard coordinate system;

[0190] The coordinates of the center point of the taper sleeve in the standard coordinate system are shown in the following formula (15):

[0191]

[0192] Among them, R is the radius of the cone sleeve space circle, λ1,λ2,λ3 are the matrix P T The characteristic value of QP, It is the coordinate of the center point of the cone sleeve space circle in the standard coordinate system.

[0193] The plane normal vector of the cone sleeve in the standard coordinate system is shown in the following formula (16):

[0194]

[0195] Among them, λ1,λ2,λ3 are the matrices P T The characteristic value of QP, is the plane normal vector of the cone sleeve space circle in the standard coordinate system;

[0196] (6) The center point coordinates and the plane normal vector are transformed from the standard coordinate system back to the camera coordinate system through the transformation matrix P. The center point coordinates and the plane normal vector of the cone sleeve in the camera coordinate system are shown in the following formula (17):

[0197]

[0198] Where P is the transformation matrix, is the coordinate of the center point of the cone sleeve space circle in the standard coordinate system, x, y, z are the coordinates of the center point of the cone sleeve space circle in the camera coordinate system; is the plane normal vector of the cone sleeve space circle in the standard coordinate system, n x ,ny ,n z is the plane normal vector of the cone sleeve space circle in the camera coordinate system.

[0199] At the same time, the plane normal vector of the cone sleeve in the standard coordinate system is converted into the pitch angle and yaw angle of the cone sleeve in the camera coordinate system as shown in formula (18):

[0200]

[0201] Among them, n x ,n y ,n z is the plane normal vector of the cone sleeve space circle in the camera coordinate system, φ is the pitch angle of the cone sleeve in the camera coordinate system, and β is the yaw angle of the cone sleeve in the camera coordinate system.

[0202] The technical effects of implementing step S42 include: the key points of the inner ring and the outer ring of the cone sleeve are not on the same spatial plane, so the pose solution based on the inner ring and the outer ring of the cone sleeve can effectively avoid the ambiguity of the measurement result.

[0203] S5, a dual-space circle pose measurement optimization method based on chamfer distance loss, optimizes the initial value of the cone sleeve pose by reprojecting the key point feature pose.

[0204] like Fig. 9 As shown, as a preferred implementation, the S5 includes:

[0205] S51, based on the initial value of the cone sleeve posture obtained in S42, according to the camera perspective projection formula and the three-dimensional digital-analog point coordinates of the cone sleeve, calculate the pixel coordinates of the two-dimensional point of the cone sleeve posture reprojection on the pixel plane.

[0206] S52, constructing a minimization target optimization function based on the chamfer distance loss function of the two-dimensional detection point and the pose reprojection two-dimensional point, wherein the minimization target optimization function is shown in the following formula (19):

[0207]

[0208] Among them, P i is the pixel coordinate point set of the two-dimensional detection point, is the pixel coordinate point set of the pose reprojected two-dimensional point, are the optimized rotation matrix and translation vector, argmin() is the Levenberg-Marquardt optimization algorithm;

[0209] The chamfer distance loss function is shown in formula (20):

[0210]

[0211] Among them, P iis the pixel coordinate point set of the two-dimensional detection point, is the pixel coordinate point set of the pose reprojected two-dimensional point; |P i | is the number of corresponding point sets, For the two-dimensional detection point set P i Middle detection point p ij The pixel coordinate point set of the two-dimensional point reprojected by the pose Center Projection Point The minimum distance between .

[0212] Technical effect of S5: After the initial value of the cone sleeve pose is calculated based on the dual-space cone sleeve pose measurement method, the dual-space circular pose measurement optimization method based on chamfer distance loss is adopted. It improves the cone sleeve pose measurement accuracy by optimizing the key point feature reprojection and outputs the cone sleeve pose measurement optimization result.

[0213] Embodiment 2

[0214] like Fig.10 As shown, this embodiment provides an autonomous aerial refueling drogue visual feature detection and posture measurement system, which is used to implement the method of embodiment 1, including:

[0215] A system calibration module 101 is used to define and calibrate the relative relationship between coordinate systems required for the posture measurement of the autonomous aerial refueling drogue, wherein the required coordinate systems include a world coordinate system, a camera coordinate system, an image coordinate system, and a pixel coordinate system;

[0216] An image acquisition module 102 is used to acquire motion video of the cone sleeve, mark key points of the cone sleeve image and construct a cone sleeve data set; wherein the motion video of the cone sleeve is acquired at a fixed video frame rate and image resolution;

[0217] A feature detection module 103 is used to build an infrastructure of a basic cone sleeve key point detection model, train the basic cone sleeve key point detection model to form a cone sleeve key point detection model based on deep learning, and the cone sleeve key point detection model is used for sub-pixel key point pixel detection results of key point features on non-cooperative target cone sleeves;

[0218] The posture measurement module 104 is used to fit the ellipse equations of the inner and outer circles of the cone sleeve based on the sub-pixel key point detection coordinate results of the key point features output by the cone sleeve key point detection model; and solve the initial value of the cone sleeve posture through a dual-space cone sleeve posture measurement method based on the relative relationship between the cone sleeve inner and outer circle ellipse equations and the coordinate system;

[0219] The posture optimization module 105 is used for the dual-space circle posture measurement optimization method based on chamfer distance loss, and optimizes the initial value of the cone sleeve posture by reprojecting the key point feature posture.

[0220] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method as in the first embodiment.

[0221] like Fig.11 As shown, the present invention also provides an electronic device, including a processor 301 and a memory 302 connected to the processor 301, the memory 302 stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the method as in the first embodiment.

[0222] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for autonomous aerial refueling drogue visual feature detection and posture measurement, characterized in that: include: S1, defining and calibrating the relative relationship between coordinate systems required for the position and posture measurement of the autonomous aerial refueling drogue, wherein the required coordinate systems include a world coordinate system, a camera coordinate system, an image coordinate system, and a pixel coordinate system; S2, collecting motion video of the cone sleeve, marking key points of the cone sleeve image and constructing a cone sleeve dataset; wherein the motion video of the cone sleeve is collected at a fixed video frame rate and image resolution; S3, building an infrastructure of a basic cone sleeve key point detection model, training the basic cone sleeve key point detection model to form a cone sleeve key point detection model based on deep learning, the cone sleeve key point detection model is used for sub-pixel key point pixel detection results of key point features on non-cooperative target cone sleeves; S4, fitting the ellipse equations of the inner and outer circles of the cone sleeve based on the sub-pixel key point detection coordinate results of the key point features output by the cone sleeve key point detection model; Based on the relative relationship between the ellipse equations of the inner and outer rings of the cone sleeve and the coordinate system, the initial value of the cone sleeve posture is solved by a double-space cone sleeve posture measurement method; S5, a dual-space circle pose measurement optimization method based on chamfer distance loss, optimizes the initial value of the cone sleeve pose by reprojecting the key point feature pose.

2. The method for autonomous aerial refueling drogue visual feature detection and posture measurement according to claim 1 is characterized in that: The S1 includes: S11, define the coordinate origin and direction of the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system; the coordinate origin of the world coordinate system is defined at the position of the refueling pipe cone head of the oil receiving machine, and its three-axis coordinate directions are three basic coordinate axes of x-axis, y-axis and z-axis, which are perpendicular to each other and together constitute a rectangular coordinate system in three-dimensional space; among which: x-axis: usually represents the horizontal direction, with positive to the right and negative to the left; y-axis: usually represents the vertical direction, with positive downward and negative upward; z-axis: usually represents the depth direction, with positive forward and negative backward; the origin of the camera coordinate system is located at the optical center of the camera, and its three-axis coordinate directions are the camera optical axis perpendicular to the image plane and outward as Z axis. c Axis, X c and Y c According to the right-hand coordinate system, they are parallel to the image plane respectively; the origin of the image coordinate system is the center of the image, and the directions of its two axes are that the positive direction of the x-axis is rightward and the positive direction of the y-axis is downward; the origin of the pixel coordinate system is the upper left corner of the image as the origin of the coordinate system, and the directions of its two axes are that the positive direction of the u-axis is rightward and the positive direction of the v-axis is downward; S12, calibrating the relative position relationship among the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system; including: (1) First, calibrate the camera’s internal parameters; (2) Then, according to the camera intrinsic parameter imaging model principle and the spatial transformation relationship, the camera extrinsic parameters are calibrated based on the principle of minimizing the reprojection error between the two-dimensional detection points and the three-dimensional space points optimized by LM; (3) Finally, the relative position relationship among the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system is determined based on formula (1), which is as follows: Wherein, R, T represent the relative position relationship between the world coordinate system and the camera coordinate system; u0 and v0 are the origin coordinates of the image coordinate system after the image coordinate system is converted to the pixel coordinate system; f x and f v are the normalized focal lengths of the x-axis and y-axis of the image coordinate system, u and v are the pixel plane coordinates, and x, y, and z are the world coordinate system coordinates.

3. The method for autonomous aerial refueling drogue visual feature detection and posture measurement according to claim 2 is characterized in that: The S2 includes: S21, using a camera fixed on the nose of the receiving aircraft to collect multiple motion videos of the drogue and extract original images of the drogue, wherein the multiple motion videos include motion videos of multiple scenes, multiple scales and different drogue positions, and the camera collects the multiple motion videos at a fixed video frame rate and image resolution; S22, marking key point features of the edges of the inner ring and the outer ring of the taper sleeve and generating a corresponding marking file; S23, constructing a cone sleeve dataset based on the arrangement of the cone sleeve original images and the annotation files, wherein the cone sleeve dataset includes a training set, a validation set and a test set, and the proportions of the number of images in the training set, the validation set and the test set are 60%, 20% and 20% respectively.

4. The method for autonomous aerial refueling drogue visual feature detection and posture measurement according to claim 3 is characterized in that: The S3 includes: S31, construct a basic cone sleeve key point detection model; the basic cone sleeve key point detection model includes 1 Steam, 4 Stages, 3 Downsample modules and 1 Neck module; the convolution block is merged into a 3*3 convolution by using the structure reparameterization technology; the sub-pixel key point output head based on coordinate classification is used to encode and decode the key point coordinates, and the specific formulas of key point encoding and decoding are shown in formula (2) and formula (3) respectively: The key point encoding formula is shown in formula (2): Among them, σ is the standard deviation of the coordinate Gaussian distribution encoding, i and j are row vector index and column vector index respectively, are the horizontal and vertical coordinates of the key points before encoding, x i ,y j They are Gaussian distribution of horizontal axis and Gaussian distribution of vertical axis after encoding respectively; The key point decoding formula is shown in formula (3): Where α is the scaling factor of the coordinate encoding and decoding, and argmax() is the function of the maximum value index in the output Gaussian distribution; x (i),o y (j) are the Gaussian distribution of the horizontal axis and the Gaussian distribution of the vertical axis before the cone sleeve key point detection model predicts the output decoding, They are the horizontal and vertical coordinates of the key points after decoding; S32, constructing a loss function of the cone sleeve key point detection model, training the basic cone sleeve key point detection model based on the training set, the validation set and the loss function until convergence to obtain the cone sleeve key point detection model; the loss function includes a key point classification loss function, a KL divergence loss function, an MSE regression loss function and a chamfer distance loss function. The formulas of each loss function are shown in formulas (4)-(7): The key point classification loss function is shown in formula (4): Among them, N is the number of training samples, M is the number of key point categories, and y ij is the label category of the jth key point of the i-th sample, p ij is the predicted category probability of the jth key point of the i-th sample; The KL divergence loss function is shown in formula (5): Where M is the number of key point categories, Y(ij) is the annotated Gaussian distribution of the i-th key point in the j-th coordinate direction, and X(ij) is the predicted Gaussian distribution of the i-th key point in the j-th coordinate direction; The MSE regression loss function is shown in formula (6): in, are the marked coordinates and predicted coordinates of the i-th key point on the inner ring ellipse of the cone sleeve, are the marked coordinates and predicted coordinates of the i-th key point on the outer ring ellipse of the cone sleeve; The chamfer distance loss function is shown in formula (7): Among them, P j , is the set of marked points and predicted points on the j-th ellipse of the cone sleeve, D chamfer is the chamfer distance loss function; In summary, considering the above loss functions, the designed total loss function is shown in formula (8): Among them, L cls ,η1 is the classification loss and its corresponding weight, L KL ,η2 is the KL divergence loss and its corresponding weight, L MSE ,η3 is the MSE regression loss and its corresponding weight, L chamfer ,η4 is the chamfer distance loss and its corresponding weight, L total is the total classification loss; S33, performing reasoning based on the cone sleeve key point detection model and outputting sub-pixel key point detection pixel coordinate results, including: (1) using each frame of the test video in the test set as input to the cone sleeve key point detection model; (2) The cone-set key point detection model performs inference and outputs the sub-pixel level key point detection coordinate results of the corresponding frame image.

5. The method for autonomous aerial refueling drogue visual feature detection and posture measurement according to claim 4 is characterized in that: The S4 includes: S41, based on the determination that the inner and outer rings of the cone sleeve are annular and imaged as an ellipse on the image plane, and the sub-pixel key point detection coordinate results of the key point features output by the cone sleeve key point detection model, an ellipse fitting algorithm is used to calculate the ellipse equations of the inner and outer rings of the cone sleeve in the image plane pixel coordinate system; wherein the ellipse fitting algorithm is implemented based on the least squares method, and the ellipse equations of the inner and outer rings of the cone sleeve are shown in formula (9): oh 2 +by 2 +cuυ+du+eu+f=0 (9); Among them, u, v are pixel plane coordinates, a, b, c, d, e, f are the parameters of the general equation of the ellipse; S42, based on the relative relationship between the ellipse equations of the inner and outer rings of the cone sleeve and the coordinate system, solving the initial value of the cone sleeve posture by a dual-space cone sleeve posture measurement method.

6. The method for autonomous aerial refueling drogue visual feature detection and posture measurement according to claim 5 is characterized in that: The S42 includes: (1) The general ellipse equation corresponding to the ellipse equations of the inner and outer rings of the tapered sleeve is transformed into an ellipse quadratic equation as shown in the following equation (10): [uv 1]E[uv 1] T =0 (10); Wherein, u, v are pixel plane coordinates, and E is the elliptic quadratic equation matrix; according to the general ellipse equation (9) and the elliptic quadratic equation (10), the elliptic quadratic equation matrix E is specifically shown in the following equation (11): Among them, a, b, c, d, e, and f are the parameters of the general equation of the ellipse; (2) According to the camera imaging principle and perspective projection principle, the camera imaging formula corresponding to the mapping relationship between the three-dimensional point in space and the two-dimensional point in the plane is determined as shown in the following formula (12): Among them, K is the camera internal parameter matrix, f x ,f y , u0, v0 represent the relative position relationship between the camera coordinate system and the image coordinate system and the pixel coordinate system; u0 and v0 are the origin coordinates of the image coordinate system after converting the image coordinate system to the pixel coordinate system; f x and f y are the normalized focal lengths of the x-axis and y-axis of the image coordinate system, u and v are the pixel plane coordinates, and x, y, and z are the world coordinate system coordinates; (3) According to the elliptic quadratic form formulas (10), (11) and the camera imaging formula (12), the spatial equation of the cone-in-cone in the camera coordinate system is obtained as shown in the following formula (13): [xyz]K T EK[xyz] T =0 (13); Among them, K is the camera internal parameter matrix, x, y, z are the coordinates of the world coordinate system, and E is the elliptic quadratic equation matrix; for the convenience of expression, define Q = K T EK; (4) Determine the transformation matrix P and transform the view cone equation from the camera coordinate system to the standard coordinate system. The spatial equation of the view cone in the standard coordinate system is shown in the following equation (14): P T QP=P -1 QP=diag{λ1,λ2,λ3} (14); Where P is the transformation matrix, Q = K T EK, λ1, λ2, λ3 are matrices P T The characteristic value of QP; (5) calculating the coordinates of the center point of the cone sleeve in the standard coordinate system and the plane normal vector based on the spatial equation of the viewing cone in the standard coordinate system; The coordinates of the center point of the taper sleeve in the standard coordinate system are shown in the following formula (15): Among them, R is the radius of the cone sleeve space circle, λ1,λ2,λ3 are the matrix P T The characteristic value of QP, is the coordinate of the center point of the cone sleeve space circle in the standard coordinate system; The plane normal vector of the cone sleeve in the standard coordinate system is shown in the following formula (16): Among them, λ1,λ2,λ3 are the matrices P T The characteristic value of QP, is the plane normal vector of the cone sleeve space circle in the standard coordinate system; (6) The center point coordinates and the plane normal vector are transformed from the standard coordinate system back to the camera coordinate system through the transformation matrix P. The center point coordinates and the plane normal vector of the cone sleeve in the camera coordinate system are shown in the following formula (17): Where P is the transformation matrix, is the coordinate of the center point of the cone sleeve space circle in the standard coordinate system, x, y, z are the coordinates of the center point of the cone sleeve space circle in the camera coordinate system; is the plane normal vector of the cone sleeve space circle in the standard coordinate system, n x ,n y ,n z is the plane normal vector of the cone sleeve space circle in the camera coordinate system; At the same time, the plane normal vector of the cone sleeve in the standard coordinate system is converted into the pitch angle and yaw angle of the cone sleeve in the camera coordinate system as shown in formula (18): Among them, n x ,n y ,n z is the plane normal vector of the cone sleeve space circle in the camera coordinate system, φ is the pitch angle of the cone sleeve in the camera coordinate system, and β is the yaw angle of the cone sleeve in the camera coordinate system.

7. The method for autonomous aerial refueling drogue visual feature detection and posture measurement according to claim 6 is characterized in that: The S5 includes: S51, based on the initial value of the cone sleeve posture obtained in S42, according to the camera perspective projection formula and the three-dimensional digital-analog point coordinates of the cone sleeve, calculate the pixel coordinates of the two-dimensional point of the cone sleeve posture reprojection on the pixel plane; S52, constructing a minimization target optimization function based on the chamfer distance loss function of the two-dimensional detection point and the pose reprojection two-dimensional point, wherein the minimization target optimization function is shown in the following formula (19): Among them, P i is the pixel coordinate point set of the two-dimensional detection point, is the pixel coordinate point set of the pose reprojected two-dimensional point, are the optimized rotation matrix and translation vector, argmin() is the Levenberg-Marquardt optimization algorithm; The chamfer distance loss function is shown in formula (20): Among them, P i is the pixel coordinate point set of the two-dimensional detection point, is the pixel coordinate point set of the pose reprojected two-dimensional point; |P i | is the number of corresponding point sets, For the two-dimensional detection point set P i Middle detection point p ij The pixel coordinate point set of the two-dimensional point reprojected by the pose Center Projection Point The minimum distance between .

8. An autonomous aerial refueling drogue visual feature detection and posture measurement system, used to implement the method described in any one of claims 1 to 7, characterized in that: include: A system calibration module (101) is used to define and calibrate the relative relationship between coordinate systems required for the position and posture measurement of the autonomous aerial refueling drogue, wherein the required coordinate systems include a world coordinate system, a camera coordinate system, an image coordinate system and a pixel coordinate system; An image acquisition module (102) is used to acquire motion video of the cone sleeve, mark key points of the cone sleeve image and construct a cone sleeve data set; wherein the motion video of the cone sleeve is acquired at a fixed video frame rate and image resolution; A feature detection module (103) is used to build a basic framework of a basic cone sleeve key point detection model, train the basic cone sleeve key point detection model to form a cone sleeve key point detection model based on deep learning, and the cone sleeve key point detection model is used for sub-pixel key point pixel detection results of key point features on a non-cooperative target cone sleeve; A posture measurement module (104) is used to output the sub-pixel key point detection coordinate results of the key point features based on the cone sleeve key point detection model, and fit the ellipse equations of the inner and outer circles of the cone sleeve; Based on the relative relationship between the ellipse equations of the inner and outer rings of the cone sleeve and the coordinate system, the initial value of the cone sleeve posture is solved by a double-space cone sleeve posture measurement method; A posture optimization module (105) is used for a dual-space circular posture measurement optimization method based on chamfer distance loss, and optimizes the initial value of the cone sleeve posture by reprojecting key point feature postures.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor is used to read the instructions and execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the plurality of instructions can be read by a processor to execute the method according to any one of claims 1 to 7.

Images