Multi-UAV pose tracking methods, devices and equipment

Abstract

This application relates to a method, apparatus, and device for multi-UAV pose tracking. The method includes: initializing a particle filter based on the initial pose of the UAVs to obtain the UAV particle distribution; predicting the UAV skeleton and component detection based on the initial UAV pose using a human migration network model; judging the similarity of the UAV particle distribution between two time points based on the UAV skeleton and component detection, and then matching them using a greedy algorithm to obtain the detection result; obtaining the pose distribution of each UAV at the previous time point, inputting the pose distribution of each UAV at the previous time point into a uniform motion model for attitude distribution prediction to obtain the attitude distribution of each UAV at the current time point; updating the weights of the attitude distribution of each UAV at the current time point based on the detection result to obtain the pose tracking result. This invention avoids the problem of UAVs being unable to be identified due to mutual occlusion, and realizes 6D pose tracking of UAVs.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) tracking technology, and in particular to a method, apparatus, and device for tracking the pose of multiple UAVs. Background Technology

[0002] 6D pose tracking is increasingly being used in augmented reality, robotics, and drone swarming, aiming to fully utilize information from sequential images to improve pose calculation accuracy and address challenges such as occlusion. In drone swarming, due to formation and the similarity in shape among multiple drones, the wingman tracking the lead drone may experience mutual occlusion within its field of view due to the similar shapes of the drones. To enable the wingman to continuously track the lead drone, we propose a robust 6D pose tracking method for multiple drones based on optical flow pose similarity and particle filtering.

[0003] Currently, mainstream 6D pose tracking methods fall into three categories: keypoint-based, edge-based, and region-based. These methods require manual feature design and meticulous hyperparameter tuning, and lack generalization ability. In recent years, some end-to-end methods have been proposed to improve the accuracy of 6D pose estimation, but their regression accuracy is limited. To address the pose prediction problem of symmetrical objects, poseRBPF combines Rao–Blackwellized particle filtering with a learned autoencoder network to effectively track the 6D pose of any type of symmetrical object. Minimally invasive robots use an elliptic matching observation model as the observation model for particle filtering, combined with motion information output by the robotic arm, to perform 6D pose tracking of unmarked surgical needles. However, these methods cannot be directly applied to multi-drone formation tracking scenarios, mainly because they cannot solve problems such as occlusion between drones with similar shapes. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, and device for multi-UAV pose tracking that can effectively distinguish multiple UAVs in a formation and continuously track them, in order to address the above-mentioned technical problems.

[0005] A method for multi-UAV pose tracking, the method comprising:

[0006] Obtain the initial pose of the UAV, and then initialize the particle filter based on the initial pose to obtain the UAV particle distribution; and

[0007] Based on the initial pose of the UAV, the UAV skeleton and component detection are obtained by prediction through the human migration network model;

[0008] Based on the detection of the drone skeleton and the components, the similarity of the drone particle distribution between two time points is judged, and then a greedy algorithm is used for matching to obtain the detection result; wherein, when a drone with a certain trajectory is detected by similarity judgment of drone particle distribution in the previous few time points, but the drone is not detected in the current time point, the drone skeleton that can be detected in the previous few time points is propagated to the current time point to supplement it.

[0009] The pose distribution of each UAV at the previous moment is obtained, and the pose distribution of each UAV at the previous moment is input into the uniform motion model to predict the pose distribution, so as to obtain the pose distribution of each UAV at the current moment.

[0010] Based on the detection results, the attitude distribution of each UAV at the current moment is updated with weights to obtain the pose tracking result.

[0011] In one embodiment, the initial pose of the UAV is obtained, and then a particle filter is initialized based on the initial pose of the UAV to obtain the UAV particle distribution, including:

[0012] Estimate the initial 6D pose of each UAV within the field of view and determine the initialization variance;

[0013] Based on the initial 6D pose and the initial variance, a number of particles are randomly selected as initial particles, and then the initial weights of each particle are calculated.

[0014] The particle distribution of the UAV is obtained by processing the initial weights of each particle.

[0015] In one embodiment, the initial weight expression is:

[0016]

[0017] In the formula, x is an m×6 dimensional vector representing the initial particle; μ is the mean vector represented by the initial pose; ∑=Cov pose This indicates the initialization of the variance matrix.

[0018] In one embodiment, the component detection is represented as:

[0019]

[0020] In the formula, The center position of the drone is indicated by (b) x ,b y ) with equal width and height (b w =b h Key bounding boxes; Conf represents confidence level; class k k = 1...K+1 represents the component category and the drone category;

[0021] The UAV skeleton includes a predicted target bounding box and skeleton endpoints, wherein:

[0022] The target bounding box prediction is represented as:

[0023] P i =[p x ,p y ,p w ,p h ,Conf,class1,class2,...,class K+1 ,(x1,y1),...,(x K ,y K )];

[0024] The skeleton endpoints are represented as follows:

[0025]

[0026] In the formula, P i The predicted center location of the drone is (p) x ,p y The width and height are divided into (p) w ,p h The bounding box of the drones; i = 1, 2, ..., n represents the number of drones; (x k ,y k ) indicates the coordinate position of the skeleton endpoint on the image.

[0027] In one embodiment, based on the detection of the UAV skeleton and the components, a similarity judgment is made on the distribution of UAV particles between two time points, and then a greedy algorithm is used for matching to obtain the detection result, including:

[0028] Based on the detection of the UAV skeleton and the components, the similarity of the UAV particle distribution between two time points is judged by optical flow pose similarity to obtain the matching result;

[0029] Then, a greedy algorithm is used to calculate the similarity of each matching result to each category of drone, and the drone is assigned a category based on the similarity, thus obtaining the detection result.

[0030] In one embodiment, the optical flow pose similarity expression is:

[0031]

[0032] In the formula, This indicates that the optical flow field is used to... r Time is transmitted to I l A drone instance at a given moment; OKS represents calculating the object key similarity between two drones.

[0033] In one embodiment, the uniform motion model is represented as:

[0034]

[0035] In the formula, (v xk ,v yk ) represents the 2D point of the k-th skeleton endpoint; Intr represents the camera intrinsic parameter matrix; P k R represents a 3D point; R represents the rotation matrix of the unmanned object; t represents the translation matrix.

[0036] In one embodiment, the pose distribution of each UAV at the current moment is updated with weights based on the detection results to obtain the pose tracking result, including:

[0037] Based on the detection results, the attitude distribution of each UAV at the current moment is updated using a weight update matrix to obtain the optimal pose tracking result.

[0038] The weight update matrix is ​​represented as follows:

[0039]

[0040] The pose tracking result is represented as follows:

[0041]

[0042] In the formula, It refers to the weight of the m-th particle of the i-th drone at time t-1; It refers to the weight of the m-th particle of the i-th UAV at time t; Let represent the Euclidean distance on the image between the endpoint of the m-th particle skeleton of the i-th UAV and the endpoint of the predicted skeleton of the network; This represents the pose of the m-th particle of the i-th UAV at time t-1. Let represent the pose of the m-th particle of the i-th UAV at time t.

[0043] A multi-UAV pose tracking device, the device comprising:

[0044] An initialization module is used to acquire the initial pose of the UAV, and then, based on the initial pose, initialize a particle filter to obtain the UAV particle distribution; and

[0045] The skeleton prediction module is used to predict the drone skeleton and component detection based on the initial pose of the drone using a human migration network model.

[0046] The similarity matching module is used to determine the similarity between the drone particle distributions at two time points based on the drone skeleton and the component detection, and then use a greedy algorithm to perform matching to obtain the detection result; wherein, when a drone with a certain trajectory is detected by similarity judgment based on the drone particle distributions at previous time points, but the drone is not detected at the current time point, the drone skeleton that can be detected at previous time points is propagated to the current time point to supplement it;

[0047] The pose tracking module is used to obtain the pose distribution of each UAV at the previous moment, input the pose distribution of each UAV at the previous moment into the uniform motion model to predict the pose distribution, and obtain the pose distribution of each UAV at the current moment; based on the detection results, the pose distribution of each UAV at the current moment is updated with weights to obtain the pose tracking result.

[0048] A computer device includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program as the steps of any of the methods described above.

[0049] The aforementioned multi-UAV pose tracking method, apparatus, and device acquire the initial pose of the UAVs, then initialize particle filtering based on the initial pose to obtain the UAV particle distribution; and based on the initial pose of the UAVs, predict the UAV skeleton and component detection using a human migration network model; based on the UAV skeleton and component detection, perform similarity judgment on the UAV particle distribution between two time points, and then use a greedy algorithm for matching to obtain the detection result; wherein, when a UAV with a certain trajectory is detected by similarity judgment based on the UAV particle distribution of previous time points, but not detected at the current time point, the skeleton of the UAV that could be detected in previous time points is propagated to the current time point to supplement it; acquire the pose distribution of each UAV at the previous time point, input the pose distribution of each UAV at the previous time point into a uniform motion model for attitude distribution prediction to obtain the attitude distribution of each UAV at the current time point; and update the weights of the attitude distribution of each UAV at the current time point based on the detection result to obtain the pose tracking result.

[0050] This invention provides UAV skeleton detection results through a human migration network model, while simultaneously tracing back several moments for similarity matching and supplementing any missing data in the network detection. The detection results are then used as the observation model for particle filtering, and a uniform motion model is proposed as the prediction model for particle filtering. The predicted attitude distribution of each UAV at the previous moment and the detection results are used as updated particle weights to obtain the optimal pose tracking result. This avoids the problem of UAVs with similar shapes occluding each other and failing to be recognized, achieving 6D pose tracking of UAVs with good robustness and the ability to continuously track. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating a multi-UAV pose tracking method in one embodiment;

[0052] Figure 2 This is a schematic diagram of the flowchart of a multi-UAV pose tracking method in one embodiment;

[0053] Figure 3 This is a schematic diagram of the pose similarity matching and supplementation process based on optical flow in one embodiment;

[0054] Figure 4 This is a schematic diagram of the reprojection of the tracking results in one embodiment;

[0055] Figure 5 This is a structural block diagram of a multi-UAV pose tracking device in one embodiment;

[0056] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] In implementing this solution, the inventors encountered limitations in the accuracy of 6D pose estimation for multiple UAVs in formation, particularly given the issue of mutual occlusion among UAVs. The solution primarily utilizes a human skeleton detection network, Kapao, to predict the UAV skeleton. To differentiate between similarly occluded UAVs during prediction, a more refined similarity matching method—based on optical flow pose similarity—is employed, taking into account the relative pose changes between time steps. Furthermore, to compensate for missed detections by the human skeleton detection network when UAVs are occluded, optical flow is used as a supplementary method. A uniform motion model is employed as the prediction model for the particle filter, predicting the current pose based on the previous optimal pose. The detection and similarity matching results are then used as the observation model for the particle filter framework, with the predicted pose and the object space error between the detection results serving as the updated particle weights.

[0059] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0060] In one embodiment, such as Figure 1 As shown, a multi-UAV pose tracking method is provided, including the following steps:

[0061] Step 202: Obtain the initial pose of the UAV, then initialize the particle filter based on the initial pose of the UAV to obtain the particle distribution of the UAV; and based on the initial pose of the UAV, make predictions through the human migration network model to obtain the detection of the UAV skeleton and components.

[0062] It is understandable that the initial pose of n UAVs within the field of view at the first moment is obtained by using a single-time 6D pose estimation method. At the same time, the parameters of the particle filter are initialized based on the initial pose, including M particles and corresponding weights.

[0063] The particle distribution of the UAV, including its pose distribution, is obtained by initializing the particle filter. Each particle represents a possible state of the UAV, and the distribution of these particles in the state space reflects the initial state estimate.

[0064] Step 204: Based on the detection of the drone skeleton and components, the similarity of the drone particle distribution between two time points is judged, and then a greedy algorithm is used for matching to obtain the detection result; wherein, when a drone with a certain trajectory is detected by similarity judgment of the drone particle distribution in the previous few time points, but the drone is not detected in the current time point, the drone skeleton that can be detected in the previous few time points is propagated to the current time point to supplement it.

[0065] It is understandable that the Kapao human skeleton detection network is used to simultaneously predict the drone skeleton and component detection. The midpoint of the drone component detection and the endpoint of the drone skeleton are theoretically coincident. Based on the principle of proximity, the midpoint of the detected component is used to correct the endpoint position of similar components in the skeleton.

[0066] On the other hand, to determine the type of drone within the field of view, a similarity judgment is first performed, followed by matching based on a greedy algorithm. For example, if the Kapao network can detect a drone with a certain trajectory in the previous three time steps, but misses it in the current time step, optical flow can be used to propagate the skeletons that can be detected in the previous three time steps to the current time step to supplement the detection.

[0067] Step 206: Obtain the pose distribution of each UAV at the previous moment, input the pose distribution of each UAV at the previous moment into the uniform motion model to predict the pose distribution, and obtain the pose distribution of each UAV at the current moment.

[0068] Step 208: Based on the detection results, update the weights of the attitude distribution of each UAV at the current moment to obtain the pose tracking results.

[0069] It is understandable that by using a constant velocity model to predict the pose distribution at the current moment based on the optimal pose at the previous moment, the detection results of the human skeleton detection network or the optical flow supplementation results are used as the observation model to update the particle weights, thereby obtaining the optimal pose tracking result. Finally, the pose at the current moment is updated based on the optimal pose tracking result.

[0070] like Figure 2 The diagram shows a flowchart of a multi-UAV pose tracking method. As can be seen, the method mainly consists of two modules. First, the particle filter observation model acquires the UAV skeleton and component detections, supplements these detections using optical flow, and then obtains the detection results through similarity matching. Second, the particle filter prediction model predicts the current pose distribution of each UAV based on the pose distribution of each UAV at the previous moment using a uniform motion model. Then, the detection results are used as the observation model to update the pose distribution weights of each UAV at the current moment, yielding the current optimal pose tracking result.

[0071] The aforementioned multi-UAV pose tracking method acquires the initial pose of the UAVs, then initializes a particle filter based on the initial pose to obtain the UAV particle distribution; and predicts the UAV skeleton and component detection using a human migration network model based on the initial UAV pose. Based on the detected UAV skeleton and components, it performs a similarity judgment on the UAV particle distribution between two time points, and then uses a greedy algorithm for matching to obtain the detection result. Specifically, if a UAV with a certain trajectory is detected when the similarity judgment is performed on the UAV particle distribution of previous time points, but not detected at the current time point, the skeleton of that UAV, which could be detected at previous time points, is propagated to the current time point to supplement the detection. It acquires the pose distribution of each UAV at the previous time point, inputs the pose distribution of each UAV at the previous time point into a uniform motion model for attitude distribution prediction, and obtains the attitude distribution of each UAV at the current time point. Based on the detection result, it updates the weights of the attitude distribution of each UAV at the current time point to obtain the pose tracking result.

[0072] This invention provides UAV skeleton detection results through a human migration network model, while simultaneously tracing back several moments for similarity matching and supplementing any missing data from the human skeleton detection network. The detection results are then used as the observation model for particle filtering, and a uniform motion model is proposed as the prediction model for particle filtering. The predicted attitude distribution of each UAV at the previous moment and the detection results are used as updated particle weights to obtain the optimal pose tracking result, achieving 6D pose tracking of UAVs with good robustness and the ability to continuously track.

[0073] In one embodiment, the initial pose of the UAV is obtained, and then a particle filter is initialized based on the initial pose of the UAV to obtain the UAV particle distribution. This includes: estimating the initial 6D pose of each UAV within the field of view and determining the initial variance; randomly selecting a number of particles as initial particles according to the initial 6D pose and the initial variance, and then calculating the initial weight of each particle; and processing according to the initial weight of each particle to obtain the UAV particle distribution.

[0074] Specifically, initially, based on the single-time 6D pose estimation method, the 6D poses of n UAVs within the field of view are estimated respectively. Subscript 0 represents the starting time. More specifically, the skeleton of the UAV is first detected using the Kapao network for human skeleton detection. Given the 3D point cloud model of the UAV and the corresponding camera intrinsic parameters of the image at a single time, the 2D-3D correspondence between the ten endpoints of the skeleton is established. Then, the Epnp method is used to obtain the initial 6D pose.

[0075] Based on the initial 6D pose, an initial particle filter is performed. More specifically, the initial 6D poses of n UAVs are first... Convert to quaternion The first three columns represent the initial translation matrix, and the remaining columns represent the quaternions of the rotation matrix, which are the attitude expressions that need to be updated. However, when predicting the rotation matrix, we initialize and predict it in the rotation angle space, and the rotation angle has a clear physical spatial meaning and continuity. Therefore, we initialize the variance Cov. pose It has 6 degrees of freedom:

[0076]

[0077] With the initial pose as the mean, Cov pose To initialize the variance, M particles are randomly selected from a 6-dimensional normal distribution, each particle representing a pose. The probability density of each particle is then calculated as the initial weight value, and the formula is as follows:

[0078]

[0079] In the formula, x is an m×6 dimensional vector representing the initial particle; μ is the mean vector represented by the initial pose; ∑=Cov pose This indicates the initialization of the variance matrix.

[0080] By normalizing the initial weights of each particle, the particle distribution of the UAV is obtained, as follows:

[0081] The initial weights of particle M are normalized:

[0082]

[0083] Then, by weighted summation based on the initial weights, the initial pose representation is obtained as follows:

[0084]

[0085] The above steps can be used to obtain the particle distribution representing the UAV pose at time 0.

[0086] In one embodiment, since the drone skeleton is obtained using the Kapao human skeleton detection network, the drone skeleton and component detection are predicted simultaneously in this part.

[0087] Among them, component detection O k This is an adaptation of the traditional object representation. Therefore, component detection is represented as:

[0088]

[0089] In the formula, The center position of the drone is indicated by (b) x ,b y ) with equal width and height (b w =b h Key bounding boxes; Conf represents confidence level; class k k = 1...K+1 represents the component category and the drone category. Considering that the shape, texture, and color information of the drones in formation are basically the same, a human skeleton detection network is not used to distinguish the drones, and they are all uniformly recorded as one type of drone.

[0090] UAV skeleton prediction is an extension of traditional object representation, including target bounding box prediction and skeleton endpoints, where:

[0091] The target bounding box prediction is represented as:

[0092] P i =[p x ,p y ,p w ,p h ,Conf,class1,class2,...,class K+1 ,(x1,y1),...,(x K ,y K )];

[0093] The skeleton endpoints are represented as follows:

[0094]

[0095] In the formula, P i The predicted center location of the drone is (p) x ,p yThe width and height are divided into (p) w ,p h The bounding box of the drones; i = 1, 2, ..., n represents the number of drones; (x k ,y k ) indicates the coordinate position of the skeleton endpoint on the image.

[0096] It is worth noting that the midpoint of the UAV component detection and the endpoint of the UAV skeleton are theoretically coincident. Considering that the target detection positioning results are relatively accurate, when fusing component detection and skeleton results, the endpoint positions of similar components in the skeleton are corrected based on the principle of proximity.

[0097] In one embodiment, based on the detection of the UAV skeleton and components, the similarity of the UAV particle distribution between two time points is determined, and then a greedy algorithm is used for matching to obtain the detection results, including:

[0098] Based on the detection of the UAV skeleton and the aforementioned components, the similarity of the UAV particle distribution between two time points is judged by optical flow pose similarity to obtain the matching result; then, the similarity of each matching result to each category of UAV is calculated by a greedy algorithm, and the UAV category is assigned according to the similarity, thereby obtaining the detection result.

[0099] It is understandable that Object Keypoint Similarity (OKS) is used to calculate the skeleton endpoint distance between two instances, given the i-th... r The i-th drone instance in the time-lapse image and the I l The j-th drone instance in the time-lapse image Then, based on the similarity of optical flow pose, it can be expressed as:

[0100]

[0101] In the formula, This indicates that the optical flow field is used to... r Time is transmitted to I l Real-time drone examples; This represents the result from the human skeleton detection network. If the j-th drone can be detected within the first three time steps, but the network cannot predict it at present... Then directly This serves as a supplement to the human skeleton detection network, which may miss some detections.

[0102] OKS represents the similarity of object key points between two drones, expressed as:

[0103]

[0104] In the formula, d represents the Euclidean distance between the corresponding skeleton endpoints; v kIndicates whether the k-th endpoint of the skeleton is visible; δ is used to filter out visible endpoints.

[0105] Next, a greedy algorithm is used to calculate the similarity of the matching results to each category of drone. The drone category is then assigned based on the similarity, thus obtaining the detection result.

[0106] In one embodiment, a kinematic model is used to propagate the attitude distribution from the previous time t-1 to the current time t, and a constant velocity model is used to propagate the 3D translation:

[0107]

[0108] In the formula, This represents a multidimensional normal distribution with mean μ and variance ∑; α is a hyperparameter of the constant velocity model.

[0109] When propagating the attitude distribution, the attitude is converted into a three-degree-of-freedom attitude angle Θ for propagation, expressed as:

[0110]

[0111] The rotation prediction distribution is achieved by convolving the previous rotation distribution with a three-dimensional Gaussian kernel.

[0112] Assuming the number of particles is M = 5000, we can obtain the pose distribution at time t through the kinematic model. To update the particle weights, we use the previous detection results as the observation model for our particle filtering, and update it using the object spatial collinearity error. Then the two-dimensional point (v) at the k-th skeleton endpoint... xk ,v yk ) is through the corresponding 3D point P k Projected onto the model, it can be described as follows:

[0113]

[0114] Where Intr represents the camera intrinsic parameter matrix, the object spatial collinearity error can be expressed as:

[0115]

[0116] Where k represents the k-th 2D-3D correspondence, the subscript m represents the m-th particle, and e km This represents the error at the k-th skeleton endpoint of the m-th pose. (R) m ,t m () indicates the prediction of the m-th pose. It is the line-of-sight projection matrix, represented as:

[0117]

[0118] In the formula, This represents the pre-determining of 2D points (v) in the camera coordinate system. xk ,v yk 3D camera rays.

[0119] According to e at time t km,t The coefficients are used to update the weights of the particles, and the coefficient update matrix is ​​as follows:

[0120]

[0121] In the formula, R represents the rotation matrix without human intervention; t represents the displacement matrix. It refers to the weight of the m-th particle of the i-th drone at time t-1; It refers to the weight of the m-th particle of the i-th UAV at time t; Let represent the Euclidean distance on the image between the endpoint of the m-th particle skeleton of the i-th UAV and the endpoint of the predicted skeleton of the network; This represents the pose of the m-th particle of the i-th UAV at time t-1. Let represent the pose of the m-th particle of the i-th UAV at time t. H is the error weighting coefficient. The error matrix of the m-th particle of the i-th UAV at time t, with dimensions equal to the particle dimension, is composed of the errors e at each skeleton endpoint. km,t constitute.

[0122] Next, the particle pose is updated according to the new weights, and the pose of the m-th particle of the i-th UAV at time t is denoted as pose. i m,t =[T x,t ,T y,t ,T z,t ,q x,t ,q y,t ,q z,t ,q t ] T Then we have:

[0123]

[0124] in This is the final prediction result of particle filtering, which is the optimal pose tracking result of the i-th UAV. The pose at the current moment is updated using the optimal pose tracking result.

[0125] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0126] In one embodiment, such as Figure 5 As shown, a multi-UAV pose tracking device is provided, including: an initialization module 402, a skeleton prediction module 404, a similarity matching module 406, and a pose tracking module 408, wherein:

[0127] Initialization module 402 is used to acquire the initial pose of the UAV, and then initialize the particle filter based on the initial pose of the UAV to obtain the UAV particle distribution; and

[0128] The skeleton prediction module 404 is used to predict the drone skeleton and component detection based on the initial pose of the drone using a human migration network model.

[0129] The similarity matching module 406 is used to determine the similarity between the drone particle distributions at two time points based on the drone skeleton and the component detection, and then use a greedy algorithm to perform matching to obtain the detection result; wherein, when a drone with a certain trajectory is detected by similarity judgment based on the drone particle distributions at previous time points, but the drone is not detected at the current time point, the drone skeleton that can be detected at previous time points is propagated to the current time point to supplement it.

[0130] The pose tracking module 408 is used to acquire the pose distribution of each UAV at the previous moment, input the pose distribution of each UAV at the previous moment into the uniform motion model to predict the pose distribution, and obtain the pose distribution of each UAV at the current moment; based on the detection results, the pose distribution of each UAV at the current moment is updated with weights to obtain the pose tracking result.

[0131] Specific limitations regarding the multi-UAV pose tracking device can be found in the limitations of the multi-UAV pose tracking method described above, and will not be repeated here. Each module in the aforementioned multi-UAV pose tracking device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0132] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores multi-UAV pose tracking data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a multi-UAV pose tracking method.

[0133] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:

[0135] Step 202: Obtain the initial pose of the UAV, then initialize the particle filter based on the initial pose of the UAV to obtain the particle distribution of the UAV; and based on the initial pose of the UAV, make predictions through the human migration network model to obtain the detection of the UAV skeleton and components.

[0136] Step 204: Based on the detection of the drone skeleton and components, the similarity of the drone particle distribution between two time points is judged, and then a greedy algorithm is used for matching to obtain the detection result; wherein, when a drone with a certain trajectory is detected by similarity judgment of the drone particle distribution in the previous few time points, but the drone is not detected in the current time point, the drone skeleton that can be detected in the previous few time points is propagated to the current time point to supplement it.

[0137] Step 206: Obtain the pose distribution of each UAV at the previous moment, input the pose distribution of each UAV at the previous moment into the uniform motion model to predict the pose distribution, and obtain the pose distribution of each UAV at the current moment.

[0138] Step 208: Based on the detection results, update the weights of the attitude distribution of each UAV at the current moment to obtain the pose tracking results.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0140] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A multi-UAV pose tracking method, characterized in that, The method includes: Obtain the initial pose of the UAV, and then initialize the particle filter based on the initial pose to obtain the UAV particle distribution; and Based on the initial pose of the UAV, the UAV skeleton and component detection are obtained by prediction through the human migration network model; Based on the detection of the drone skeleton and the components, the similarity of the drone particle distribution between two time points is judged, and then a greedy algorithm is used for matching to obtain the detection result; wherein, when a drone with a certain trajectory is detected by similarity judgment of drone particle distribution in the previous few time points, but the drone is not detected in the current time point, the drone skeleton that can be detected in the previous few time points is propagated to the current time point to supplement it. The pose distribution of each UAV at the previous moment is obtained, and the pose distribution of each UAV at the previous moment is input into the uniform motion model to predict the pose distribution, so as to obtain the pose distribution of each UAV at the current moment. Based on the detection results, the attitude distribution of each UAV at the current moment is updated with weights to obtain the pose tracking result; Obtain the initial pose of the UAV, and then initialize the particle filter based on the initial pose of the UAV to obtain the UAV particle distribution, including: Estimate the initial 6D pose of each UAV within the field of view and determine the initialization variance; Based on the initial 6D pose and the initial variance, a number of particles are randomly selected as initial particles, and then the initial weights of each particle are calculated. The particle distribution of the UAV is obtained by processing the initial weights of each particle. Based on the detection results, the attitude distribution of each UAV at the current moment is updated with weights to obtain the pose tracking results, including: Based on the detection results, the attitude distribution of each UAV at the current moment is updated using a weight update matrix to obtain the optimal pose tracking result. The weight update matrix is ​​represented as follows: ； The pose tracking result is represented as follows: ； In the formula, It means The first moment The first drone The weight of each particle; It refers to the first Time of the first The first drone The weight of each particle; Indicates the first The first drone Euclidean distance on the image between the endpoints of the particle skeleton and the endpoints of the network-predicted skeleton; Indicates the first Time of the first The first drone The pose of a particle; Indicates the first Time of the first The first drone The pose of a particle.

2. The multi-UAV pose tracking method according to claim 1, characterized in that, The initial weight expression is: ； In the formula, yes The dimension vector represents the initial particle; It is the mean vector represented by the initial pose; This indicates the initialization of the variance matrix.

3. The multi-UAV pose tracking method according to claim 1 or 2, characterized in that, The component detection is represented as follows: ； In the formula, Indicates the center position of the drone. It has equal width and height Keypoint bounding boxes; Indicates confidence level; Indicate the component category and the drone category; The UAV skeleton includes a predicted target bounding box and skeleton endpoints, wherein: The target bounding box prediction is represented as: ； The skeleton endpoints are represented as follows: ； In the formula, Indicates the location of the drone prediction center. It has width and height divided into The bounding box of the drone; Indicates the number of drones; This indicates the coordinate position of the skeleton endpoints on the image.

4. The multi-UAV pose tracking method according to claim 3, characterized in that, Based on the detection of the UAV skeleton and the components, a similarity judgment is made on the UAV particle distribution between two time points. Then, a greedy algorithm is used for matching to obtain the detection results, including: Based on the detection of the UAV skeleton and the components, the similarity of the UAV particle distribution between two time points is judged by optical flow pose similarity to obtain the matching result; Then, a greedy algorithm is used to calculate the similarity of each matching result to each category of drone, and the drone is assigned a category based on the similarity, thus obtaining the detection result.

5. The multi-UAV pose tracking method according to claim 4, characterized in that, The optical flow pose similarity expression is: ； In the formula, This indicates that the optical flow field will be used to... Transmitted in real time A drone instance at a given moment; OKS represents calculating the object key similarity between two drones.

6. The multi-UAV pose tracking method according to claim 4 or 5, characterized in that, The uniform motion model is represented as follows: ； In the formula, Indicates the first Two-dimensional points of the skeleton endpoints; Represents the camera intrinsic parameter matrix; Represents 3D points; Represents the rotation matrix without an operator; This represents the displacement matrix.

7. A multi-UAV pose tracking device, characterized in that, The apparatus employing the multi-UAV pose tracking method according to any one of claims 1 to 6 comprises: An initialization module is used to acquire the initial pose of the UAV, and then, based on the initial pose, initialize a particle filter to obtain the UAV particle distribution; and The skeleton prediction module is used to predict the drone skeleton and component detection based on the initial pose of the drone using a human migration network model. The similarity matching module is used to determine the similarity between the drone particle distributions at two time points based on the drone skeleton and the component detection, and then use a greedy algorithm to perform matching to obtain the detection result; wherein, when a drone with a certain trajectory is detected by similarity judgment based on the drone particle distributions at previous time points, but the drone is not detected at the current time point, the drone skeleton that can be detected at previous time points is propagated to the current time point to supplement it; The pose tracking module is used to obtain the pose distribution of each UAV at the previous moment, input the pose distribution of each UAV at the previous moment into the uniform motion model to predict the pose distribution, and obtain the pose distribution of each UAV at the current moment; based on the detection results, the pose distribution of each UAV at the current moment is updated with weights to obtain the pose tracking result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images