A vision-based spatial pose estimation method for bionic flapping-wing UAVs

Abstract

The present invention discloses a vision-based method for estimating the spatial pose of a bionic flapping-wing UAV, which comprises the following steps: obtaining an image training data set; training the image training data set and annotation information using a deep learning detection algorithm; obtaining internal and external parameter information of an image acquisition device, and obtaining flight images of a bionic flapping-wing UAV; testing the flight images of the bionic flapping-wing UAV according to the deep learning detection algorithm to obtain test results of a key point image coordinate system and the wing profile state of the bionic flapping-wing UAV; measuring the three-dimensional coordinates and wing profile states of key points in the coordinate system of the bionic flapping-wing UAV under different wing profile states; and obtaining the spatial position and attitude results of the bionic flapping-wing UAV through calculation. The present invention defines wing profile states according to different flight attitudes, performs calculations after deep learning training and detection, and obtains the spatial position and attitude, thereby solving the problem that the spatial pose of a UAV is difficult to determine through external images due to its complex structure and diverse wing profile states.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle (UAV) spatial pose estimation, and more specifically, to a vision-based method for estimating the spatial pose of a bionic flapping-wing UAV. Background Art

[0002] Vision-based detection, tracking, and pose estimation are key countermeasures against drones. Using a monocular RGB sensor to capture multiple target images, image detection techniques are used to extract key points on the target drone, and geometric relationships are then used to calculate the target's position and pose. However, in practice, due to the complex background of trees, utility poles, and buildings in low-lying urban areas, coupled with frequent interference from foreign objects, traditional image detection and pose estimation methods often suffer from missed detections, false detections, and low pose estimation accuracy.

[0003] Existing technologies disclose a monocular vision-based drone pose estimation method. This method uses keypoint detection and a localization network to obtain the 2D coordinates of a target and calculates the target drone's pose based on its 3D keypoints and internal parameters of the image acquisition device. A limitation of this type of drone pose estimation method is that the targets are mostly fixed-wing or rotary-wing aircraft, which are nearly rigid bodies. This method does not account for the deformable keypoints of bionic drones during flight, similar to bird wings. This deformation significantly increases the error in pose estimation, making it difficult to meet the pose estimation requirements for bionic flapping-wing aircraft. Summary of the Invention

[0004] In order to solve the problems of the above-mentioned deficiencies and defects in the existing technologies, the present invention provides a vision-based bionic flapping-wing UAV spatial posture estimation method, which can effectively solve the problem that the existing bionic flapping-wing UAV spatial position and posture information source is concentrated in its own inertial navigation sensor, and lacks external information to estimate its spatial position and posture, thereby improving the accuracy and comprehensiveness of bionic flapping-wing UAV status monitoring.

[0005] In order to achieve the above-mentioned purpose of the present invention, the technical solutions adopted are as follows:

[0006] A vision-based method for estimating the spatial pose of a bionic flapping-wing UAV comprises the following steps:

[0007] The acquired bionic flapping-wing UAV images are annotated, and the annotated content includes the image coordinate key points and wing state information of the bionic flapping-wing UAV, thereby obtaining a bionic flapping-wing UAV image training dataset;

[0008] Use the deep learning detection algorithm to train the bionic flapping wing drone image training data set and annotation information to obtain training weights, update the deep learning detection algorithm, and obtain a trained deep learning detection algorithm;

[0009] Calibrate the internal and external parameters of the image acquisition device to obtain the internal and external parameter information of the image acquisition device, and obtain the flight image of the bionic flapping-wing UAV through the image acquisition device;

[0010] The flight images of the bionic flapping-wing UAV are tested based on the trained deep learning detection algorithm to obtain the test results of the key point image coordinate system and the wing profile state of the bionic flapping-wing UAV;

[0011] According to the airfoil state classification, the three-dimensional coordinate information of the key points of the bionic flapping-wing drone's own body coordinate system under different airfoil states is measured, and the airfoil state is recorded;

[0012] According to the internal and external parameter information of the image acquisition device, the test results of the key point image coordinate system, the airfoil state of the bionic flapping-wing UAV, and the three-dimensional coordinate information of the key points of the bionic flapping-wing UAV's own body coordinate system under different airfoil states; the spatial position and attitude estimation algorithm is used to solve the problem and obtain the spatial position and attitude results of the bionic flapping-wing UAV.

[0013] Preferably, the image coordinate key points of the bionic flapping-wing drone are respectively the nose of the bionic flapping-wing drone, the wingtip of the left wing, the wingtip of the right wing, the wingtip of the left tail wing, and the wingtip of the right tail wing;

[0014] The airfoil states are divided into three categories: a first airfoil state when the wing is above the horizontal line of the fuselage, a second airfoil state when the wing is at the horizontal line of the fuselage, and a third airfoil state when the wing is below the horizontal plane of the fuselage.

[0015] Furthermore, the image coordinate key points and wing state of the bionic flapping wing UAV image are annotated as follows:

[0016] First, use a rectangular box to calibrate the position of the image coordinate key point, and then calculate the center coordinates of the rectangular box to obtain the two-dimensional image coordinates of the image coordinate key point; the position of the rectangular box is represented by the xy coordinates of the upper left corner and the xy coordinates of the lower right corner;

[0017] The airfoil states are marked according to the flight posture of the bionic flapping-wing drone in the bionic flapping-wing drone image and are divided into three categories;

[0018] Finally, the calibration data of each image consists of 5 image coordinate key points, 2D image coordinates corresponding to the 5 image coordinate key points, and 1 airfoil state, a total of 16 sets of data;

[0019] After calibration, the bionic flapping-wing UAV image training dataset is obtained.

[0020] Preferably, the deep learning detection algorithm adopts the yolov5 target detection algorithm, adds an output dual channel on the basis of the yolov5 target detection algorithm, and performs feature training on the image coordinate key points and airfoil status.

[0021] Furthermore, two additional branches are introduced into the network architecture of the one-stage yolov5 object detection algorithm based on deep learning;

[0022] One of the branches is used to process the image coordinate key point detection task of the bionic flapping-wing drone and generate the image coordinates of the image coordinate key points on the bionic flapping-wing drone;

[0023] Another branch is used for the classification of the airfoil state of the bionic flapping-wing UAV, which is used to output the probability of the bionic flapping-wing UAV being in different airfoil states. Based on the output probability value, the motion state with the highest probability is selected as the airfoil state of the bionic flapping-wing UAV.

[0024] Preferably, the image acquisition device is calibrated for internal and external parameters to obtain internal and external parameter information of the image acquisition device, specifically as follows: using Zhang Zhengyou's chessboard calibration method, the chessboard is calibrated to be composed of squares of size L1 and alternating black and white squares of size L2, the chessboard is used as a calibration reference, and the image is captured by changing the chessboard orientation multiple times to obtain rich coordinate information;

[0025] Then, the corner points are extracted for calibration calculation. The calibration results are plotted through the average error bar chart, and images with average errors higher than the threshold are deleted. Finally, the Export Camera Parameters calculation is performed to obtain the internal and external parameter information of the image acquisition device.

[0026] The spatial position and attitude estimation algorithm is used to solve the problem and obtain the spatial position and attitude results of the bionic flapping-wing drone. Specifically, the three-dimensional coordinates of the key points of the image coordinates in the world coordinate system are calculated, and then the attitude cosine matrix and position matrix representing the bionic flapping-wing drone's posture information are solved through the perspective projection model.

[0027] Furthermore, the three-dimensional coordinates of the key points of the image coordinates in the world coordinate system are calculated using the following formula:

[0028]

[0029] Where P is the world 3D coordinate of the key point of the bionic flapping-wing drone image coordinate to be solved; P′ is the corresponding image coordinate, K is the intrinsic parameter matrix of the image acquisition device; α and β are the scaled focal lengths; s is the skew parameter; (u0, v0) is the optical center; w is the scale factor of the image point; R and It is the external parameter information of the image acquisition device, indicating the required 3D rotation and 3D translation.

[0030] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described above are implemented.

[0031] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described above.

[0032] The beneficial effects of the present invention are as follows:

[0033] The present invention provides a vision-based method for estimating the spatial position and attitude of a bionic flapping-wing drone. This method does not require the construction of a CAD model of an unfamiliar object. Instead, it uses a predefined bionic flapping-wing drone airfoil state and the coordinate positions of key points of its body system. Key points on the image are found through a deep learning detection algorithm, and image coordinate key point matching and PnP calculation are performed to estimate the spatial position and attitude of the bionic flapping-wing drone. The present invention labels and defines the airfoil state according to the different flight attitudes of the bionic flapping-wing drone, performs calculations after deep learning training and detection, and obtains its spatial position and attitude. This solves the problem that the bionic flapping-wing drone's spatial position and attitude are difficult to solve using external images due to its complex body structure and diverse airfoil states. Therefore, the present invention can adapt to more application scenarios and complex aircraft models, and has better attitude estimation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 The present invention provides a flow chart of the method for estimating the spatial pose of a bionic flapping-wing UAV.

[0035] Figure 2 It is a schematic diagram for establishing the coordinate system of the flapping-wing aircraft.

[0036] Figure 3 This is another schematic diagram for establishing the body coordinate system of an ornithopter.

[0037] Figure 4 This is another schematic diagram for establishing the coordinate system of the flapping-wing aircraft.

[0038] Figure 5 It is a schematic diagram of selecting key points of image coordinates.

[0039] Figure 6 It is a schematic diagram of the first airfoil state.

[0040] Figure 7 It is another schematic diagram of the first airfoil state.

[0041] Figure 8It is a schematic diagram of the second airfoil state.

[0042] Figure 9 It is another schematic diagram of the second airfoil state.

[0043] Figure 10 It is a schematic diagram of the third airfoil state.

[0044] Figure 11 It is another schematic diagram of the third airfoil state.

[0045] Figure 12 It is a schematic diagram of the network model of the target detection algorithm.

[0046] Figure 13 It is a diagram of a chessboard.

[0047] Figure 14 is a schematic diagram of deleting images where the average error is higher than a threshold.

[0048] Figure 15 This is the actual key point detection effect diagram. DETAILED DESCRIPTION

[0049] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0050] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0051] Example 1

[0052] like Figure 1 As shown, a vision-based method for estimating the spatial pose of a bionic flapping-wing UAV comprises the following steps:

[0053] S1: Annotate the acquired bionic flapping-wing UAV image, including the image coordinate key points and wing state information of the bionic flapping-wing UAV, to obtain the bionic flapping-wing UAV image training dataset;

[0054] S2: Use the deep learning detection algorithm to train the bionic flapping wing drone image training data set and annotation information to obtain training weights, update the deep learning detection algorithm, and obtain a trained deep learning detection algorithm;

[0055] S3: Calibrate the internal and external parameters of the image acquisition device to obtain the internal and external parameter information of the image acquisition device, and obtain the flight image of the bionic flapping-wing UAV through the image acquisition device;

[0056] S4: Test the flight image of the bionic flapping-wing UAV based on the trained deep learning detection algorithm to obtain the key point image coordinate system test results and the wing state of the bionic flapping-wing UAV;

[0057] S5: Classify the wing states and measure the three-dimensional coordinate information of the key points of the bionic flapping-wing drone's own body coordinate system under different wing states, and record the wing states;

[0058] S6: Based on the internal and external parameter information of the image acquisition device, the test results of the key point image coordinate system, the airfoil state of the bionic flapping-wing UAV, and the three-dimensional coordinate information of the key points of the bionic flapping-wing UAV's own body coordinate system under different airfoil states, a spatial position and attitude estimation algorithm is used to solve the problem and obtain the spatial position and attitude results of the bionic flapping-wing UAV.

[0059] In a specific embodiment, the image of the bionic flapping-wing drone is obtained as follows: a plurality of shooting devices (such as mobile phones, image acquisition devices, iPads and other shooting devices with camera functions) are used to collect data. After controlling the bionic flapping-wing drone to take off, the ground shooting personnel use the shooting equipment to collect data from the bionic flapping-wing drone. During the shooting process, the bionic flapping-wing drone is kept in the center of the image and the image acquisition device is kept stable to obtain high-quality data as much as possible. The data collection is divided into multiple video shootings, each shooting lasting 3 to 5 minutes, and the video clarity is selected as 1080P. After the shooting is completed, the shot video is divided into image sets.

[0060] Establish the body coordinate system of the flapping-wing aircraft, as follows Figure 2 、 Figure 3 、 Figure 4 As shown, the origin of the aircraft system is selected on the fixing buckle on the back of the flapping-wing aircraft.

[0061] The key points of the image coordinates of the bionic flapping-wing drone are selected, which are the nose, the wingtip of the left wing, the wingtip of the right wing, the wingtip of the left tail, and the wingtip of the right tail. Figure 5 shown.

[0062] According to the position of the wings of the bionic flapping wing drone, its airfoil state can be divided into three categories: when the wings are above the horizontal line of the fuselage, it is the first airfoil state ( Figure 6 、 Figure 7 ), when the wings are at the horizontal line of the fuselage, it is the second airfoil state ( Figure 8 、 Figure 9 ), when the wings are below the horizontal plane of the fuselage, it is the second airfoil state ( Figure 10 、 Figure 11 ).

[0063] Under the three airfoil states, the coordinates of the key points of the image coordinates in the body coordinate system of the flapping-wing aircraft are as follows:

[0064] left_tail=[-0.29 -0.1 -0.05]

[0065] right_tail=[-0.29 0.1 -0.05]

[0066] left_wing_up = [0 -0.18 0.17]

[0067] right_wing_up = [0 0.18 0.17]

[0068] left_wing_mid=[0 -0.26 0.17]

[0069] right_wing_mid=[0 0.26 0.17]

[0070] left_wing_down=[0 -0.18 -0.17]

[0071] right_wing_down=[0 0.18 -0.17]

[0072] head = [0.13 0 0]

[0073] Among them, left_tail represents Figure 5 The coordinates of the key point 4 in the body coordinate system, right_tail represents Figure 5 The coordinates of the key point 5 in the body coordinate system, left_wing_up represents Figure 5 The coordinates of key point 2 in the first wing state in the body coordinate system, right_wing_up represents Figure 5 The coordinates of the key point 3 in the first wing state in the body coordinate system, left_wing_mid represents Figure 5 The coordinates of key point 2 in the second wing state in the body coordinate system, right_wing_mid represents Figure 5 The coordinates of key point 3 in the second wing state in the body coordinate system, left_wing_down represents Figure 5The coordinates of key point 2 in the third wing state in the body coordinate system, right_wing_down represents Figure 5 The coordinates of key point 3 in the third airfoil state in the body coordinate system.

[0074] In a specific embodiment, the image coordinate key points and wing profile states of the bionic flapping wing UAV image are annotated as follows: data annotation uses labelimg data calibration software, and the annotation format is yolo.

[0075] First, use a rectangular box to calibrate the position of the image coordinate key point, and then calculate the center coordinates of the rectangular box to obtain the two-dimensional image coordinates of the image coordinate key point; the position of the rectangular box is represented by the xy coordinates of the upper left corner and the xy coordinates of the lower right corner;

[0076] The airfoil states are marked according to the flight posture of the bionic flapping-wing drone in the bionic flapping-wing drone image and are divided into three categories;

[0077] Finally, the calibration data of each image consists of 5 image coordinate key points, 2D image coordinates corresponding to the 5 image coordinate key points, and 1 airfoil state, a total of 16 sets of data;

[0078] After calibration, the bionic flapping-wing UAV image training dataset is obtained.

[0079] In a specific embodiment, the deep learning detection algorithm is modified and optimized as follows: the deep learning detection algorithm adopts the yolov5 target detection algorithm, adds an output dual channel on the basis of the yolov5 target detection algorithm, and performs feature training on the image coordinate key points and airfoil status.

[0080] Introducing two additional branches into the network architecture of the one-stage YOLOv5 object detection algorithm based on deep learning;

[0081] One of the branches is used to process the image coordinate key point detection task of the bionic flapping-wing drone and generate the image coordinates of the image coordinate key points on the bionic flapping-wing drone;

[0082] Another branch is used for the classification of the airfoil state of the bionic flapping-wing UAV, which is used to output the probability of the bionic flapping-wing UAV being in different airfoil states. Based on the output probability value, the motion state with the highest probability is selected as the airfoil state of the bionic flapping-wing UAV.

[0083] Through these improvements, the ability of the Yolov5 target detection algorithm will be enhanced, providing more accurate and comprehensive information for key point detection and airfoil state estimation of bionic flapping wing drones. The improved Yolov5 network structure is as follows Figure 12 shown.

[0084] The training process for the deep learning detection algorithm consists of three main steps: environment setup, model training, and model testing. Environment setup requires configuring the deep learning algorithm environment based on the hardware model. Within this configured environment, the flapping-wing aircraft detection dataset is divided into a training set and a test set. The training set is used to train the deep learning detection algorithm under different parameters. After training, the trained deep learning detection algorithm is tested and verified using the test set. The model with the highest performance accuracy is selected as the final detection model.

[0085] The improved Yolov5 target detection algorithm includes a feature extraction network CSP-Darknet53, a multi-scale feature fusion network FPN, and an improved prediction network Head. The feature extraction network consists of a downsampling Focus module, four basic convolutional network modules Conv modules, four feature extraction modules C3, and a pooling module SPPF. The improved prediction network Head includes a feature point category prediction network, a position prediction network, and an airfoil state prediction network. The feature point category prediction network and the position prediction network are represented in one output channel, while the airfoil state prediction network is represented by a PAN module in another output channel.

[0086] The training set was fed into the improved YOLOv5 object detection algorithm for training. The number of iterations was set to 120, the learning rate was 1.2e-4, and the Adam optimizer was used. The training decay rate for the CSP-Darknet53 network was 1e-4, and the decay rate for the FPN and Head networks was 1.2e-5. The training batch size was 32. The loss function consisted of classification loss, localization loss, and confidence loss. The classification loss included feature point classification loss and airfoil state classification loss. Training ended after the specified number of iterations, resulting in a bionic flapping-wing aircraft detection network.

[0087] The detection process of the yolov5 target detection algorithm is as follows: each frame image is scaled to 640x640x3 by up and down sampling, first passes through a Focus sampling module, then alternately passes through the convolution Conv module and the feature extraction module C3, alternating 4 times, and then passes through a pooling module SPPF module to obtain the image global feature map; the image global feature map is input into the multi-scale feature fusion network FPN, and the dimensional feature map of the three channels is output by down sampling, with dimensions of 80x80x(3x10), 40x40x(3x10), and 20x20x(3x10), where 8 0, 40, and 20 are the width and height of the feature map respectively. 3 means that each network presets three anchor boxes of different sizes. 10 means that each anchor box needs to predict 5 feature point category information, target box (x, y, w, h) position information, and confidence, a total of 10 values. The three-dimensional feature map is divided into two channels for processing. One channel directly predicts the category and position information of the bionic flapping-wing aircraft feature points for the three-dimensional feature maps. The other channel performs upsampling feature fusion on the three-dimensional feature maps and adds them in the third dimension to obtain an 80x80x3 feature map. The probabilities of the three airfoil states of the bionic flapping-wing aircraft are predicted on this feature map.

[0088] In a specific embodiment, the internal and external parameters of the image acquisition device are calibrated to obtain the internal and external parameter information of the image acquisition device, specifically as follows: Zhang Zhengyou's chessboard calibration method is used to calibrate the chessboard to a square with a size of L1 = 82.5 mm and a black and white square grid with a size of L2 = 10x7 mm ( Figure 13 ), using a chessboard as a calibration reference, which is easier to handle than complex three-dimensional objects. By capturing images by changing the chessboard orientation multiple times, rich coordinate information can be obtained;

[0089] After that, the corner points are extracted for calibration calculation. The calibration results are plotted through the average error bar graph, and the images with average errors higher than the threshold are deleted ( Figure 14 ), and finally perform Export Camera Parameters calculation to obtain the internal and external parameter information of the image acquisition device.

[0090] In this embodiment, the test image of the bionic flapping wing drone is obtained as follows: first, the position of the image acquisition device is fixed, the position of the image acquisition device in the world coordinate system is defined, and then data is collected. Then, the trained target detection algorithm is used to detect the key points and wing state of the bionic flapping wing drone on the collected data. Figure 15 The middle one is the actual key point detection effect diagram.

[0091] In a specific embodiment, the internal and external parameter information of the image acquisition device, the test results of the key point image coordinate system, the airfoil state of the bionic flapping-wing UAV, and the three-dimensional coordinate information of the key points of the bionic flapping-wing UAV's own body coordinate system under different airfoil states are input into the spatial position and attitude estimation algorithm for solution, and the output is obtained, that is, the spatial position and attitude of the UAV relative to the image acquisition device. The external parameters of the image acquisition device obtained by calibration are then used to obtain the position and attitude of the image acquisition device in the spatial coordinates, and coordinate conversion is performed to finally obtain the position and attitude of the UAV in the spatial coordinates.

[0092] Use the spatial position and attitude estimation algorithm (such as the EPnP attitude estimation algorithm) to solve and obtain the spatial position and attitude results of the bionic flapping-wing drone. Specifically, calculate the three-dimensional coordinates of the key points of the image coordinates in the world coordinate system, and then use the perspective projection model to solve the attitude cosine matrix and position matrix representing the bionic flapping-wing drone's attitude information.

[0093] In this embodiment, the three-dimensional coordinates of the key points of the image coordinates in the world coordinate system are calculated using the following formula:

[0094]

[0095] Where P is the world 3D coordinate of the key point of the bionic flapping-wing drone image coordinate to be solved; P′ is the corresponding image coordinate, K is the intrinsic parameter matrix of the image acquisition device; α and β are the scaled focal lengths; s is the skew parameter; (u0, v0) is the optical center; w is the scale factor of the image point; R and is the external parameter information of the image acquisition device, indicating the required 3D rotation and 3D translation; (u, v) is the coordinate of the key point of the image in the pixel coordinate system; r 11 ~r 33 is the camera extrinsic rotation matrix, [t x ,t y ,t z ] is the camera extrinsic translation matrix, and [x, y, z] is the three-dimensional coordinate of the image key point in the world coordinate system.

[0096] To solve P, the solution algorithm adopts the EPnP method, and the solution process is as follows:

[0097] D1: Select four points that are not in the same plane as the control points c in the world coordinate system (represented by w in the table above) w , to uniquely represent the coordinates p of the selected image coordinate key point w ,Right now: in:

[0098]

[0099] in, Represents the 4 control points in the world coordinate system, Represents the selected image key points, α ij Represents the weight matrix.

[0100] D2: In the image acquisition device coordinate system, the weight remains unchanged, that is:

[0101] in, Represents the 4 control points in the camera coordinate system.

[0102] D3: The obtained key point image coordinates are used as the input of the EPnP algorithm. For any key point in the image coordinates Consider the transformation relationship from the image acquisition device system to the pixel coordinate system:

[0103]

[0104] in, Represents the coordinate of the key point in the z-axis direction in the camera coordinate system, f x , f y represents the normalized focal length of the camera, Represents the coordinates of the control point in the camera coordinate system.

[0105] Expand it to:

[0106]

[0107]

[0108] in, Represents the coordinate of the control point in the z-axis direction in the camera coordinate system, [u i ,v i ] represents the coordinates of the key point in the pixel coordinate system.

[0109] Selecting 5 key points of image coordinates will result in 10 equations, which can be expressed in matrix form as follows:

[0110] M 2n×12 x 12×1 =0

[0111] in: M is the coordinate of the control point in the image acquisition device coordinate system. 2n×12 is the coefficient matrix of the characteristic equation, The coordinates of each control point to be solved in the image coordinate system.

[0112] D4: The solution for x can be transformed into the problem of solving the zero eigenvalue vector of M, which can also be transformed into the problem of solving the matrix M TThe problem of the eigenvector corresponding to the zero eigenvalue of M.

[0113]

[0114] Where N represents the number of control points, β i is the inverse matrix of M, v i is the zero vector matrix.

[0115] D5: After obtaining the coordinates of the key points of the image coordinate system in the image acquisition device system, the singular value decomposition (SVD) will be used to solve the optimal posture matrix and position matrix.

[0116] make: H=B T A, where

[0117] in, Represents the transposed matrix of the key points in the world coordinate system, Represents the transposed matrix of the key points in the camera coordinate system, A and B are decomposition matrices, Represents the coordinates of the key points in the camera coordinate system, Represents the coordinates of the key point in the world coordinate system.

[0118] D6: Decompose H by singular value to get:

[0119] H=U∑V T

[0120] Among them, U and V represent decomposition matrices.

[0121] D7: Get the attitude cosine matrix:

[0122] R=UV T

[0123] D8: Further obtain the displacement vector:

[0124]

[0125] D9: Solving Euler Angles

[0126] The Euler angle representation method uses (φ, θ, ψ) to represent the attitude of the bionic flapping-wing aircraft, where φ is the roll angle, θ is the pitch angle, ψ is the yaw angle, and the rotation order is φ→θ→ψ.

[0127] The conversion relationship from Euler angle to attitude cosine is known to be:

[0128]

[0129] The above conversion relationship can be reversed to obtain the Euler angle:

[0130]

[0131] D10: Then, according to the position of the fixed image acquisition device relative to the world coordinate system when acquiring the image, coordinate transformation is performed, and finally the position information of the flapping-wing aircraft relative to the world coordinate system is output.

[0132] In this embodiment, the image acquisition device includes but is not limited to mobile phones, cameras, iPads and other shooting devices.

[0133] In this embodiment, the target detection algorithm may adopt the yolov5 target detection algorithm, or may be replaced by other detection algorithms, including but not limited to Faster RCNN, DETR, etc.

[0134] In this embodiment, the spatial position and posture estimation algorithm includes but is not limited to the EPnP method, the DLT method, the RPnP method, and a method based on the Gauss-Newton optimization method that can optimize the EPnP result.

[0135] Example 2

[0136] A computer system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method described in Example 1 are implemented.

[0137] Example 3

[0138] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in Example 1.

[0139] Obviously, the above embodiments of the present invention are merely examples for the purpose of illustrating the present invention, and are not intended to limit the embodiments of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A vision-based method for estimating the spatial pose of a bionic flapping-wing UAV, characterized by: The method comprises the following steps: The acquired bionic flapping-wing UAV images are annotated, and the annotated content includes the image coordinate key points and wing state information of the bionic flapping-wing UAV, thereby obtaining a bionic flapping-wing UAV image training dataset; Use the deep learning detection algorithm to train the bionic flapping wing drone image training data set and annotation information to obtain training weights, update the deep learning detection algorithm, and obtain a trained deep learning detection algorithm; Calibrate the internal and external parameters of the image acquisition device to obtain the internal and external parameter information of the image acquisition device, and obtain the flight image of the bionic flapping-wing UAV through the image acquisition device; The flight images of the bionic flapping-wing UAV are tested based on the trained deep learning detection algorithm to obtain the test results of the key point image coordinate system and the wing profile state of the bionic flapping-wing UAV; According to the airfoil state classification, the three-dimensional coordinate information of the key points of the bionic flapping-wing drone's own body coordinate system under different airfoil states is measured, and the airfoil state is recorded; Based on the internal and external parameter information of the image acquisition device, the test results of the key point image coordinate system, the airfoil state of the bionic flapping-wing UAV, and the three-dimensional coordinate information of the key points of the bionic flapping-wing UAV's own body coordinate system under different airfoil states, the spatial position and attitude estimation algorithm is used to solve the problem and obtain the spatial position and attitude results of the bionic flapping-wing UAV.

2. The method for estimating the spatial pose of a bionic flapping-wing UAV based on vision according to claim 1, characterized in that: The image coordinate key points of the bionic flapping-wing drone are respectively the nose, the wingtip of the left wing, the wingtip of the right wing, the wingtip of the left tail wing, and the wingtip of the right tail wing of the bionic flapping-wing drone; The airfoil states are divided into three categories: a first airfoil state when the wing is above the horizontal line of the fuselage, a second airfoil state when the wing is at the horizontal line of the fuselage, and a third airfoil state when the wing is below the horizontal plane of the fuselage.

3. The method for estimating the spatial pose of a bionic flapping-wing UAV based on vision according to claim 2, characterized in that: The image coordinate key points and wing state of the bionic flapping wing drone image are marked as follows: First, use a rectangular box to calibrate the position of the image coordinate key point, and then calculate the center coordinates of the rectangular box to obtain the two-dimensional image coordinates of the image coordinate key point; the position of the rectangular box is represented by the xy coordinates of the upper left corner and the xy coordinates of the lower right corner; The airfoil states are marked according to the flight posture of the bionic flapping-wing drone in the bionic flapping-wing drone image and are divided into three categories; Finally, the calibration data of each image consists of 5 image coordinate key points, 2D image coordinates corresponding to the 5 image coordinate key points, and 1 airfoil state, a total of 16 sets of data; After calibration, the bionic flapping-wing UAV image training dataset is obtained.

4. The method for estimating the spatial pose of a bionic flapping-wing UAV based on vision according to claim 1, characterized in that: The deep learning detection algorithm adopts the YOLOv5 target detection algorithm, adds an output dual channel on the basis of the YOLOv5 target detection algorithm, and performs feature training on the key points of image coordinates and airfoil states.

5. The method for estimating the spatial pose of a bionic flapping-wing UAV based on vision according to claim 4, characterized in that: Introducing two additional branches into the network architecture of the one-stage YOLOv5 object detection algorithm based on deep learning; One of the branches is used to process the image coordinate key point detection task of the bionic flapping-wing drone and generate the image coordinates of the image coordinate key points on the bionic flapping-wing drone; Another branch is used for the classification of the airfoil state of the bionic flapping-wing UAV, which is used to output the probability of the bionic flapping-wing UAV being in different airfoil states. Based on the output probability value, the motion state with the highest probability is selected as the airfoil state of the bionic flapping-wing UAV.

6. The method for estimating the spatial pose of a bionic flapping-wing UAV based on vision according to claim 1, characterized in that: The image acquisition device is calibrated for internal and external parameters to obtain its internal and external parameter information. Specifically, Zhang Zhengyou's chessboard calibration method is used to calibrate a chessboard with a square size of L1 and a black and white square grid of size L2. The chessboard is used as a calibration reference. The image is captured by changing the chessboard orientation multiple times to obtain rich coordinate information. Then, the corner points are extracted for calibration calculation. The calibration results are plotted through the average error bar chart, and images with average errors higher than the threshold are deleted. Finally, the Export Camera Parameters calculation is performed to obtain the internal and external parameter information of the image acquisition device.

7. The method for estimating the spatial pose of a bionic flapping-wing UAV based on vision according to claim 1, characterized in that: The spatial position and attitude estimation algorithm is used to solve the problem and obtain the spatial position and attitude results of the bionic flapping-wing drone. Specifically, the three-dimensional coordinates of the key points of the image coordinates in the world coordinate system are calculated, and then the attitude cosine matrix and position matrix representing the bionic flapping-wing drone's posture information are solved through the perspective projection model.

8. The method for estimating the spatial pose of a bionic flapping-wing UAV based on vision according to claim 7, characterized in that: The three-dimensional coordinates of the key points of the image coordinates in the world coordinate system are calculated as follows: Where P is the world 3D coordinate of the key point of the bionic flapping-wing drone image coordinate to be solved; P′ is the corresponding image coordinate, K is the intrinsic parameter matrix of the image acquisition device; α and β are the scaled focal lengths; s is the skew parameter; (u0, v0) is the optical center; w is the scale factor of the image point; R and It is the external parameter information of the image acquisition device, indicating the required 3D rotation and 3D translation.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images