An unmanned aerial vehicle flight system based on visual target tracking

Abstract

This invention relates to the field of unmanned aerial vehicle (UAV) control technology, specifically to a UAV flight system based on visual target tracking, comprising: an image acquisition module for acquiring synchronized image frames; a 3D positioning module for obtaining the 3D position sequence of the tracked target in the world coordinate system; a trajectory prediction module for constructing a predicted displacement vector that changes linearly in the next moment and a predicted displacement vector that changes non-linearly in the next moment; a weight distribution generation module for obtaining the predicted trajectory weight distribution of the tracked target; a tracking quality evaluation module for determining the degree to which the bounding box center of each UAV deviates from the image center and determining the trajectory coverage index of each UAV; a leader UAV setting module for dynamically setting a leader UAV from a multi-UAV swarm; and a path planning module for planning the flight path of the UAV swarm. This invention can enhance the stability and reliability of multi-UAV cooperative target tracking.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) control technology, and more specifically to a UAV flight system based on visual target tracking. Background Technology

[0002] Visual target tracking-based unmanned aerial vehicle (UAV) flight systems refer to intelligent flight systems that acquire images in real time through onboard visual sensors and use target tracking algorithms to achieve autonomous identification, positioning, and continuous following of specific targets. They play an irreplaceable and crucial role in fields such as security monitoring, emergency rescue, infrastructure inspection, and aerial filming.

[0003] In current mainstream practices, visual target tracking based on a single drone is easily affected by obstructions, and the field of view of a single drone is limited and cannot be adjusted quickly. Targets are easily lost when they are obscured for extended periods or when their movement patterns suddenly change. To address this problem, multi-drone collaboration is considered an effective solution. The "leader-follower" method is widely used due to its simplicity, efficiency, and high formation accuracy; however, it is highly dependent on the reliability of the lead drone, and a centralized architecture can concentrate the communication burden on the lead node. If the lead drone malfunctions or its tracking performance deteriorates, it will affect the control and tracking performance of the entire drone swarm.

[0004] While technologies for dynamically switching navigator drones exist, these adjustments are primarily triggered only after the drone's battery status deteriorates or the navigator drone loses its tracking target, making them passive control measures. By the time these situations occur, tracking performance has already been affected, and switching navigators may be too late, easily leading to target loss. Summary of the Invention

[0005] This invention provides a visual target tracking-based unmanned aerial vehicle (UAV) flight system to solve existing problems.

[0006] The present invention provides a visual target tracking-based unmanned aerial vehicle (UAV) flight system, which adopts the following technical solution: One embodiment of the present invention provides a visual target tracking-based unmanned aerial vehicle (UAV) flight system, comprising: The image acquisition module is used to acquire synchronized image frames from multiple drone clusters targeting the same tracking target. The 3D positioning module is used to obtain the 3D position sequence of the tracked target in the world coordinate system based on the synchronous image frames through the intersection and fusion of multiple UAV lines of sight; The trajectory prediction module is used to determine the displacement vector sequence and the direction change sequence based on the three-dimensional position sequence, and to construct the predicted displacement vector for the next moment as a linear change and the predicted displacement vector for the next moment as a non-linear change based on the displacement vector sequence and the direction change sequence. The weight distribution generation module is used to determine the weights of the linearly changing predicted displacement vectors and the non-linearly changing predicted displacement vectors. Based on the linearly changing and non-linearly changing predicted displacement vectors, as well as the weights of the linearly changing and non-linearly changing predicted displacement vectors, the module obtains the weight distribution of the predicted tracking target trajectory through Gaussian superposition. The tracking quality assessment module is used to determine the degree to which the bounding box center of each drone deviates from the image center, and to determine the trajectory coverage index of each drone based on the predicted tracking target trajectory weight distribution. The pilot drone setting module is used to obtain a score for each drone as a candidate pilot drone based on the degree of deviation of the bounding box center from the image center and the trajectory coverage index, and to dynamically set the pilot drone from the multi-drone cluster based on the score. The path planning module is used to determine the area that the lead drone can guarantee tracking coverage based on the predicted target trajectory weight distribution, and to plan the flight path of the drone swarm within the area.

[0007] Furthermore, the three-dimensional positioning module is specifically used for: The synchronous image frames are input into the target detector based on a deep convolutional network, and the bounding box and confidence score of the tracked target in the synchronous image frames are output. Based on the camera's spatial position and attitude recorded by the onboard integrated navigation system of each UAV, and combined with the camera's installation parameters and camera intrinsic parameters in the body coordinate system, the centroid of the bounding box in the synchronized image frame is transformed into a three-dimensional ray that originates from the corresponding camera and points towards the tracking target, so as to unify the tracking target information identified by each camera position into the same three-dimensional space. For each three-dimensional ray corresponding to a drone, calculate the intersection residual between the three-dimensional ray and each of the other three-dimensional rays, and take the average of the multiple intersection residuals as the average residual of the three-dimensional ray. Sort the average residuals of all three-dimensional rays in ascending order to determine the lower quartile, upper quartile, and interquartile range; where the interquartile range is the difference between the upper quartile and the lower quartile. Three-dimensional rays with a residual mean greater than a preset residual threshold are identified as abnormal rays and removed; where the preset residual threshold is the sum of the upper quartile and a preset multiple multiplied by the interquartile range; For each remaining 3D ray after removal, the confidence level output by the target detector is used as the weight, and the least squares triangulation method is used for fusion. The position of the fusion result is used as the position of the tracking target. The positions of the tracked target at each moment within the preset first time window are arranged in chronological order to obtain the three-dimensional position sequence of the tracked target in the world coordinate system; wherein, the preset first time window consists of historical moments before the current moment.

[0008] Furthermore, the trajectory prediction module is specifically used for: For each pair of adjacent positions in the three-dimensional position sequence of the tracked target in the world coordinate system, subtract the previous position from the next position to obtain the displacement vector of that pair of adjacent positions. Arrange all displacement vectors according to time sequence to form a displacement vector sequence; For each pair of adjacent displacement vectors in the displacement vector sequence, the direction angle of the latter displacement vector is subtracted from the direction angle of the former displacement vector to obtain the change in direction of the pair of adjacent displacement vectors. Arrange all directional changes according to time sequence to form a sequence of directional changes; By performing linear trend fitting on the sequence of direction changes, the predicted direction change at the next moment can be obtained; Calculate the magnitude of each displacement vector in the displacement vector sequence to obtain the displacement length sequence; By performing linear trend fitting on the displacement length sequence, the predicted displacement length at the next moment can be obtained; Add the predicted change in direction of the displacement vector at the current moment to the predicted direction angle at the next moment to obtain the predicted direction angle at the next moment. Construct a predicted displacement vector that changes non-linearly at the next moment based on the predicted direction angle and the predicted displacement length at the next moment. Construct a predicted displacement vector that changes linearly at the next moment based on the current displacement vector direction angle and the predicted displacement length at the next moment.

[0009] Furthermore, the specific steps in the weight distribution generation module for determining the weights of the linearly changing predicted displacement vector and the weights of the non-linearly changing predicted displacement vector are as follows: Calculate the mean of all displacement lengths in the displacement length sequence, and use it as the average displacement length; Divide the predicted displacement length at the next moment by the average displacement length to obtain the normalized predicted displacement length at the next moment. Calculate the mean of the absolute values ​​of all directional changes in the sequence of directional changes, and use it as the average directional change. Divide the absolute value of the predicted direction change at the next time step by the average direction change to obtain the normalized predicted direction change at the next time step. The weight of the predicted displacement vector that changes linearly in the next moment is calculated by using the normalized predicted displacement length of the next moment as the numerator and the sum of the normalized predicted displacement length of the next moment and the change in the predicted direction of the next moment as the denominator. The weight of the predicted displacement vector that changes nonlinearly at the next time step is calculated by using the normalized predicted direction change at the next time step as the numerator and the sum of the normalized predicted displacement length at the next time step and the normalized predicted direction change at the next time step as the denominator.

[0010] Furthermore, the weight distribution generation module obtains the predicted tracking target trajectory weight distribution through Gaussian superposition based on the linearly changing predicted displacement vector and the non-linearly changing predicted displacement vector, as well as the weights of the linearly changing predicted displacement vector and the non-linearly changing predicted displacement vector. The specific steps include the following: The predicted displacement vectors that change linearly in the next moment and those that change non-linearly in the next moment are respectively transformed into the two-dimensional image plane of each UAV through the perspective transformation of the camera projection relationship, so as to obtain the corresponding linear pixel displacement vectors and non-linear pixel displacement vectors. Centered on the endpoints of the linear and non-linear pixel displacement vectors, the weights of the predicted displacement vectors that change linearly in the next time step and the weights of the predicted displacement vectors that change non-linearly in the next time step are used as the amplitudes. Gaussian superposition is performed with a preset bandwidth as the Gaussian kernel width to obtain the predicted tracking target trajectory weight distribution in the two-dimensional image plane of each UAV. The preset bandwidth is determined based on the linear trend fitting result of the displacement length sequence. Specifically, the displacement length sequence is fitted with a linear trend, and the mean of the absolute value of the residual between the actual displacement length sequence and the fitted linear trend is calculated. This mean value is used as the preset bandwidth.

[0011] Furthermore, the tracking quality assessment module is specifically used for: For each drone in a multi-drone cluster, the center coordinates of the target bounding box tracked by the drone in each synchronized image frame within a preset second time window are obtained, and the bounding box center coordinate sequence is constructed in chronological order. For each center coordinate in the bounding box center coordinate sequence, calculate the first distance between that center coordinate and the diagonal intersection point of the corresponding synchronized image frame of the UAV; All first distances are weighted and summed according to a preset weighting rule to obtain the weighted total distance. The preset weighting rule is as follows: the weight of the first distance corresponding to the current time is 1, the weight of the first distance corresponding to the earliest time in the preset second time window is 0, and the weight of the first distance corresponding to other times increases linearly with time. Divide the sum of weighted distances by the sum of weights to obtain the weighted average distance; Divide the weighted average distance by half the diagonal length of the corresponding synchronized image frame of the UAV to obtain the degree to which the center of the normalized bounding box of the UAV deviates from the center of the image. According to the predicted target trajectory weight distribution, the weighted centroid is calculated by weighting each pixel position in the synchronous image frame corresponding to the UAV with its corresponding weight value. Calculate the second distance between the weighted centroid and the diagonal intersection point of the corresponding synchronized image frame of the UAV; Divide the second distance by half the diagonal length of the corresponding synchronized image frame of the UAV to obtain the normalized trajectory coverage index of the UAV.

[0012] Furthermore, the pilot drone setting module obtains a score for each drone as a candidate pilot drone based on the degree to which the bounding box center deviates from the image center and the trajectory coverage index. The specific steps include the following: The sum of the degree to which the center of the normalized bounding box of the drone deviates from the center of the image and 1 is used as the first sum value; The sum of the normalized trajectory coverage index of the drone and 1 is used as the second sum. The product of the first sum and the second sum is used as the score for the drone to be considered as a candidate lead drone; the lower the score, the more suitable the drone is as a candidate lead drone.

[0013] Furthermore, the pilot drone setting module dynamically sets a pilot drone from a multi-drone cluster based on a score, including the following specific steps: Obtain the score of the leading drone at the current moment within a preset second time window, and construct a score sequence in chronological order; Calculate the mean and standard deviation of the rating series; For each drone in a multi-drone swarm, the drone's leading position is obtained by subtracting its score as a candidate leader drone from its mean score. The drone's leading position is then divided by the standard deviation to obtain the standardized value of its leading position. If the standardized value of the drone's leading position is greater than or equal to a preset multiple threshold, then the drone is identified as a candidate leading drone for the next moment. If there is no drone with a standardized leading value greater than or equal to a preset multiple threshold, then the current leading drone will remain unchanged. Obtain the switching evaluation value for each candidate lead drone; whereby the switching evaluation value is determined based on the three-dimensional spatial distance and leading degree between the candidate lead drone and the current lead drone; The candidate drone with the highest switching evaluation value will be selected as the drone for the next moment.

[0014] Furthermore, the specific steps for obtaining the switching evaluation value for each candidate lead drone are as follows: For each candidate lead drone, calculate the three-dimensional spatial distance between the candidate lead drone and the lead drone at the current moment; Obtain the maximum three-dimensional spatial distance between all candidate lead drones and the lead drone at the current moment, and use it as the maximum spatial distance; Divide the three-dimensional spatial distance between the candidate lead drone and the current lead drone by the maximum spatial distance to obtain the normalized spatial distance of the candidate lead drone. Divide the leading position of the candidate lead drone by the sum of the normalized spatial distance of the candidate lead drone and 1 to obtain the switching evaluation value of the candidate lead drone.

[0015] Furthermore, the path planning module plans the flight path of the UAV swarm within the region, including the following specific steps: Plan the flight path of the lead drone within the area that the lead drone can guarantee tracking coverage; Based on the flight path of the lead drone and the preset tracking array, plan the flight paths of each follower drone.

[0016] The beneficial effects of the technical solution of the present invention are as follows: The embodiments of the present invention propose a UAV flight system based on visual target tracking. It predicts the trajectory by analyzing the three-dimensional behavioral trajectory of the tracked target, and dynamically arranges the lead UAV according to the predicted trajectory and the monitoring status and coverage of the predicted trajectory of each UAV in a short period of time, thereby planning the flight path of each UAV. This enables predictive and proactive switching of the lead UAV, which can prevent problems before they occur and effectively avoid the problem of multi-UAV cluster control and tracking being affected by accidents or poor tracking performance of the lead UAV. Compared with the existing methods, the present invention has better tracking continuity and switching smoothness in the target tracking process, which can significantly improve tracking accuracy and tracking effect, and enhance the stability and reliability of multi-UAV collaborative target tracking. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a structural block diagram of a visual target tracking-based unmanned aerial vehicle (UAV) flight system according to the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a visual target tracking-based unmanned aerial vehicle (UAV) flight system proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] The purpose of this invention is to provide a target tracking method for a multi-UAV swarm that actively and dynamically switches between lead UAVs based on tracking performance, thereby improving the tracking continuity and switching smoothness of multi-UAV swarm target tracking, and thus enhancing tracking accuracy and performance. The scenario addressed by this invention is that existing "leader-follower" methods heavily rely on the reliability of the lead UAV in target tracking, and centralized architectures tend to concentrate the communication burden on the lead node; if the lead node experiences an accident or poor tracking performance, it will affect the control and tracking performance of the entire multi-UAV swarm.

[0022] A complete unmanned aerial vehicle (UAV) system consists of a fuselage structure module, a power system module, a flight control system module, a perception and navigation system module, a communication and data link module, and a mission payload module. Specifically: the fuselage structure module (frame, arms, wings, and landing gear, etc.) provides aerodynamic shape and physical support; the power system module (brushless motors, electronic speed controllers, propellers, batteries, etc.) provides lift and thrust; the flight control system module, as the core control unit (including the flight control motherboard, gyroscopes, accelerometers, barometers, and other sensors), calculates flight attitude in real time, maintains fuselage stability, resists airflow disturbances, and executes various flight commands; the perception and navigation system module (GPS, millimeter-wave radar, and ultrasonic sensors, etc.) is used for environmental perception, autonomous obstacle avoidance, and stable hovering in satellite-free environments; the communication and data link module (remote controller, onboard image / data transmission module, and antenna, etc.) is used for data transmission; and the mission payload module varies depending on specific operational requirements and may include a gimbal and camera, thermal imager, multispectral camera, loudspeaker, lidar, drop device, or agricultural spraying equipment, etc.

[0023] The following description, in conjunction with the accompanying drawings, details a specific solution for a visual target tracking-based unmanned aerial vehicle (UAV) flight system provided by the present invention.

[0024] Please see Figure 1The diagram illustrates a structural block diagram of a visual target tracking-based unmanned aerial vehicle (UAV) flight system according to an embodiment of the present invention. The system includes the following modules: The image acquisition module 100 is used to acquire synchronized image frames from multiple drone clusters targeting the same tracking target.

[0025] It should be noted that each drone in the multi-drone swarm is equipped with an airborne visual acquisition system, which includes a visible light camera and a three-axis stabilized gimbal. The visible light camera is mounted on the three-axis stabilized gimbal, which is fixed to a shock-absorbing mount on the underside or front of the drone's fuselage to reduce the impact of aircraft vibration on imaging stability. Each drone's visible light camera acquires image sequences of the tracked target and its surrounding environment in real time, recording the timestamp of each frame. Then, the image data acquired by each drone is aligned to the same time according to the timestamps, resulting in synchronized image frames for the same tracked target from the multi-drone swarm. Optionally, the synchronized image frames undergo preprocessing such as grayscale conversion and noise reduction enhancement for subsequent target detection and localization.

[0026] The 3D positioning module 200 is used to obtain the 3D position sequence of the tracked target in the world coordinate system by multi-UAV line-of-sight intersection and fusion based on synchronized image frames.

[0027] It's important to note that each drone acquires image sequences of the target and its surrounding environment via an onboard visible light camera. Simultaneously, the onboard integrated navigation system provides the drone's spatial position and attitude angle for each frame. This data is inherently fragmented and viewpoint-dependent; a single camera image only reflects the target's two-dimensional projection from a specific viewpoint. In real-world scenarios, targets are rarely isolated and static against an open background; they may move through complex environments such as urban blocks, forest paths, or undulating terrain. For example, a moving target on the ground might be surrounded by obstructions such as trees, lampposts, buildings, vehicles, or temporary debris. When the target passes an obstruction, if a drone happens to be near its rear or front, its camera's line of sight may be partially blocked by the tree canopy, resulting in only a portion of the target's outline appearing in the image, or even its complete disappearance. In such cases, relying on a single drone could easily lead to losing track of the target. In a multi-drone swarm, due to the different spatial distributions of the drones, their line-of-sight directions differ. When the target being tracked is obstructed by an obstacle, it usually only affects some camera positions, while the others can still track the target from different angles. By utilizing the orientation and position of each drone's camera, the target can be located. Furthermore, by integrating the target positions obtained at consecutive moments, the three-dimensional movement path of the target can be obtained.

[0028] The 3D positioning module 200 is specifically used for: Step S2001: Input the synchronized image frame into the target detector based on the deep convolutional network, and output the bounding box and confidence score of the tracked target in the synchronized image frame.

[0029] It should be noted that for each drone in a multi-drone swarm, the synchronized image frames acquired at the current moment are input into a pre-trained target detector based on a deep convolutional network. The target detector can employ a model pre-trained on a general target category and fine-tuned on the task target category to improve the accuracy and robustness of target recognition. After processing the input synchronized image frames, the target detector outputs the bounding box and confidence score of the tracked target in that synchronized image frame; the bounding box represents the position and extent of the tracked target in the image, and the confidence score represents the degree of confidence that the tracked target exists within the bounding box. Optionally, the bounding box can be represented using the coordinates of the center point, width, and height, or it can be represented using the coordinates of the top left and bottom right corners. The confidence score can be normalized to a value between 0 and 1. The target detector based on a deep convolutional network can consist of a backbone network and a detection head. The backbone network is used to extract multi-scale features from the input synchronized image frames, and can employ structures such as residual networks, dark networks, or high-efficiency networks. The detection head is used to predict the bounding box and confidence score of the tracked target in the image based on the feature map output by the backbone network, and can employ a single-stage or two-stage detection structure. By performing the above operations on the synchronized image frames of each UAV, the bounding box and confidence score of the tracked target corresponding to each UAV can be obtained.

[0030] Step S2002: Based on the camera spatial position and attitude recorded by the onboard integrated navigation system of each UAV, and combined with the camera's installation parameters and camera intrinsic parameters in the body coordinate system, the centroid of the bounding box in the synchronized image frame is transformed into a three-dimensional ray originating from the corresponding camera and pointing towards the tracking target, so as to unify the tracking target information identified by each camera position into the same three-dimensional space.

[0031] Specifically, for each UAV, the pixel coordinates of its centroid in the synchronized image frame are first extracted from the output bounding box. Then, using the UAV camera's intrinsic parameter matrix, the pixel coordinates of the centroid are back-projected onto the normalized camera coordinate system to obtain the camera coordinate system direction vector pointing from the camera's optical center towards the tracking target. The camera intrinsic parameter matrix can be obtained through pre-calibration and includes parameters such as focal length and principal point coordinates. Next, combined with the camera's installation parameters in the body coordinate system, the camera coordinate system direction vector is transformed into the body coordinate system. The installation parameters include the camera's rotation matrix and translation vector relative to the UAV body, which can be obtained through assembly calibration. Then, based on the UAV's current spatial position and attitude angle recorded by the UAV's onboard integrated navigation system, the direction vector in the body coordinate system is transformed into the world coordinate system to obtain the direction vector in the world coordinate system; simultaneously, based on the UAV's spatial position, attitude angle, and camera installation parameters, the position of the camera's optical center in the world coordinate system is determined. Finally, starting from the position of the camera's optical center in the world coordinate system and using the direction vector in the world coordinate system as the direction, a three-dimensional ray is constructed, originating from the corresponding camera and pointing towards the tracking target. This process is repeated for each drone in the multi-drone swarm, thus obtaining the corresponding three-dimensional ray for each drone and unifying the tracking target information identified by each camera position into a single three-dimensional space within the same world coordinate system.

[0032] Step S2003: For each UAV corresponding to a three-dimensional ray, calculate the intersection residual between the three-dimensional ray and each of the other three-dimensional rays, and take the average of the multiple intersection residuals as the average residual of the three-dimensional ray.

[0033] Specifically, for each drone in a multi-drone swarm, there is a corresponding 3D ray. For any given 3D ray, the intersection residual between that ray and each of the other 3D rays is calculated. The intersection residual measures the degree of consistency between two 3D rays in space: if both 3D rays point to the same tracking target and are accurately detected and calibrated, they should intersect near the target's true location, resulting in a small intersection residual; if one ray deviates in direction due to misidentification or calibration error, the intersection point will significantly deviate from the target's true location, resulting in a large intersection residual. For two 3D rays, the closest distance between them can be calculated as the intersection residual. Since two 3D rays may not strictly intersect in space, the intersection residual can be obtained by finding the closest pair of points on the two rays and using the distance between these points as the intersection residual; alternatively, the midpoint of this pair of points can be taken as the intersection point of the two rays, and the perpendicular distance from this intersection point to each ray can be calculated, with the larger or average distance taken as the intersection residual. The unit of the intersection residual is consistent with the length unit of the 3D spatial coordinate system. For the i-th 3D ray, the intersection residuals between it and each of the other 3D rays are calculated separately, resulting in multiple intersection residuals. The average of these residuals is then taken as the average residual of the i-th 3D ray. The average residual reflects the overall consistency between this 3D ray and all other 3D rays: the smaller the average residual, the better the intersection of this 3D ray with other rays, and the more reliable the detection result; the larger the average residual, the worse the consistency of this 3D ray with other rays, and the more likely the detection result is a misidentification or has a large error. Optionally, when calculating the intersection residual between two 3D rays, if the two 3D rays are approximately parallel, their intersection point may be located at infinity, leading to unstable intersection residual calculations. When the angle between the directions of the two 3D rays is small, the intersection residual calculation of this pair of rays is directly ignored. Performing the above operation on each drone in a multi-drone swarm yields the average residual of each 3D ray.

[0034] Step S2004: Sort the average residuals of all three-dimensional rays in ascending order, and determine the lower quartile, upper quartile, and interquartile range; where the interquartile range is the difference between the upper quartile and the lower quartile.

[0035] Specifically, the residual averages of all 3D rays are sorted in ascending order to obtain a residual average sequence. Each value in the residual average sequence corresponds to a 3D ray. Then, the lower quartile, upper quartile, and interquartile range are determined based on the sorted residual average sequence. The lower quartile refers to the value at the 25th percentile in the sorted residual average sequence, and the upper quartile refers to the value at the 75th percentile. The interquartile range reflects the distribution range of the middle 50% of the data in the residual average sequence.

[0036] Step S2005: Three-dimensional rays with an average residual value greater than a preset residual threshold are identified as abnormal rays and removed; wherein, the preset residual threshold is the sum of the upper quartile and a preset multiple multiplied by the interquartile range.

[0037] It should be noted that the preset multiplier can be set according to the strictness of anomalous ray removal in the actual scene. Preferably, the preset multiplier is 1.5 to identify mild anomalies; it can also be 3 to identify extreme anomalies. If the average residual of a certain 3D ray is greater than the preset residual threshold, then the 3D ray is determined to be an anomalous ray and is removed from the 3D ray set. Anomalous rays are usually caused by target misidentification, excessive camera calibration error, time synchronization deviation, or severe line of sight obstruction. By removing these anomalous rays, their adverse effects on the 3D position fusion results of the tracking target can be avoided. Optionally, if the number of remaining 3D rays after removing anomalous rays is less than three, effective triangulation fusion cannot be performed, and the tracking target localization at that moment can be abandoned.

[0038] Step S2006: For each remaining 3D ray after removal, the least squares triangulation method is used to fuse them, with the confidence level output by the target detector as the weight. The position of the fused result is used as the position of the tracking target.

[0039] Specifically, for each 3D ray remaining after removing anomalous rays, the position of the camera's optical center in the world coordinate system, the unit direction vector of the 3D ray, and the confidence level output by the target detector corresponding to the 3D ray are obtained. The confidence level is used as the weight of the 3D ray in the fusion process. A higher confidence level indicates a more reliable detection result for the ray, and thus a larger weight in the fusion; a lower confidence level indicates a less reliable detection result for the ray, and thus a smaller weight in the fusion. Then, a weighted least squares objective function is constructed: finding a 3D point in the world coordinate system such that the sum of the squares of the perpendicular distances from this 3D point to each remaining 3D ray, multiplied by the weight corresponding to that ray, is minimized. The perpendicular distance refers to the shortest distance from the 3D point to the 3D ray, i.e., the distance between the 3D point and the nearest point on the line containing the ray. Since the direction and origin of each 3D ray are known, this perpendicular distance can be obtained by projecting the line connecting the 3D point and the ray's origin onto a plane perpendicular to the ray's direction. Solving the weighted least squares objective function yields the 3D point that minimizes the objective function. Specifically, the objective function can be differentiated with respect to the 3D point coordinates and the derivative set to zero to obtain a system of linear equations. Solving this system yields the optimal 3D point. The coefficients of the linear equations are determined by the direction vectors and weights of each 3D ray, while the right-hand side is determined by the starting position, direction vector, and weight of each 3D ray. The resulting 3D point is the position of the fused target, which is used as the position of the tracking target in the world coordinate system at the current moment. Optionally, if the sum of the weights of the remaining 3D rays is zero, or if the linear equations are close to singular, the fused positioning result for that moment is discarded. By performing steps S2001 to S2006 for each moment, the tracking target position at each moment can be obtained.

[0040] Step S2007: Arrange the tracking target positions at each moment within the preset first time window in chronological order to obtain the three-dimensional position sequence of the tracking target in the world coordinate system; wherein, the preset first time window consists of historical moments before the current moment.

[0041] Specifically, the duration of the preset first time window can be set according to the target's movement speed, sampling frequency, and onboard computing power; the faster the target moves, the shorter the window can be; the slower the target moves, the longer the window can be. For example, the preset first time window is 5 seconds before the current moment. If the target position cannot be effectively tracked at a certain moment due to insufficient remaining 3D rays or other reasons, the position point at that moment can be marked as missing, and that moment can be skipped.

[0042] The trajectory prediction module 300 is used to determine the displacement vector sequence and the direction change sequence based on the three-dimensional position sequence, and to construct the predicted displacement vector for the next moment as a linear change and the predicted displacement vector for the next moment as a non-linear change based on the displacement vector sequence and the direction change sequence.

[0043] It should be noted that while the previous steps have already obtained the 3D position sequence of the target, in a target tracking scenario, tracking solely based on the target's current position often leads to a lag in the tracking system's response when the target's motion state changes. This can easily cause tracking failure when the target's motion trend changes drastically. Therefore, it is necessary to identify the target's motion trend from the 3D position sequence over a period of time and make a certain degree of prediction to achieve high-quality continuous tracking and maintain a favorable position in subsequent tracking. Based on the 3D position sequence of the target, the motion trend of the target can be identified: in a continuous 3D position sequence, the motion state of the target at each moment is reflected in the relationship between the positions of adjacent moments: the magnitude of the displacement vector between adjacent moments reflects the target's movement amplitude within that time period, the change in direction of adjacent displacement vectors reflects the degree of change in the target's direction of travel, and the accumulation of multiple consecutive changes in direction characterizes the overall curvature of the trajectory. The motion trend of the target in a real-world scenario can generally be divided into two categories: linear motion and non-linear motion. When the target is moving linearly, the projection of its 3D position sequence onto the horizontal plane is roughly distributed around a straight line, and the directions of the displacement vectors at adjacent moments are relatively consistent. When a target exhibits a decrease in displacement length or a sustained change in direction, it is likely overcoming inertia or completing a directional adjustment in preparation for an upcoming turn. Conversely, without these signs, the target typically will not turn. Similarly, when a target moves non-linearly, the changes in direction of adjacent displacement vectors are significant, and the trajectory exhibits a marked curvature. In this case, the motion trend predicted by the change in displacement length is exactly the opposite of that in linear motion: when the target shows a trend of increasing displacement length during non-linear motion, or a significant decrease in the sustained magnitude of the change in direction, it indicates that the target has likely completed a turn, detour, or obstacle avoidance, and is about to transition to stable linear motion; if these signs are absent, it indicates that the target will maintain its current motion trend. Overall, a decreasing trend in displacement length or an increasing trend in the change in direction indicates that the target's motion trend will change in the next moment (non-linear motion); conversely, an increasing trend in displacement length and a decreasing trend in the change in direction means that the target's motion trend will be retained to some extent in the next moment (linear motion).

[0044] The trajectory prediction module 300 is specifically used for: Step S3001: For each pair of adjacent positions in the three-dimensional position sequence of the tracking target in the world coordinate system, subtract the previous position from the next position to obtain the displacement vector of the pair of adjacent positions.

[0045] Specifically, for a sequence of three-dimensional positions of the target in the world coordinate system, adjacent positions are selected sequentially in chronological order. The three-dimensional coordinates of the subsequent position are subtracted from the three-dimensional coordinates of the preceding position to obtain the displacement vector corresponding to that pair of adjacent positions. The displacement vector is a three-dimensional vector whose direction indicates the direction of the target's movement from the preceding position to the subsequent position, and its magnitude indicates the distance the target moves within that time interval. For example, for the k-th and (k+1)-th positions in the three-dimensional position sequence, the three-dimensional coordinates of the (k+1)-th position are subtracted from the three-dimensional coordinates of the k-th position to obtain the k-th displacement vector. Performing this operation on each pair of adjacent positions sequentially yields multiple displacement vectors.

[0046] Step S3002: Arrange all displacement vectors according to the time sequence to form a displacement vector sequence.

[0047] Specifically, each displacement vector corresponds to a pair of adjacent position points in the three-dimensional position sequence. The displacement vectors are arranged sequentially according to the order of these adjacent position points in the three-dimensional position sequence. In the arranged displacement vector sequence, each displacement vector has a clear time sequence index, and the displacement vectors in the sequence represent the direction and distance of movement of the tracked target in each time interval.

[0048] Step S3003: For each pair of adjacent displacement vectors in the displacement vector sequence, subtract the direction angle of the previous displacement vector from the direction angle of the subsequent displacement vector to obtain the change in direction of the pair of adjacent displacement vectors.

[0049] Specifically, for the displacement vector sequence, adjacent displacement vectors are taken sequentially in chronological order. For each displacement vector, its orientation angle is first determined. The orientation angle is defined as the angle between the projection of the displacement vector onto the horizontal plane and a preset reference direction, with clockwise as the positive direction, and its value ranges from 0 to 360 degrees. The preset reference direction can be set according to the actual coordinate system, such as due east or due north. If the projection of a displacement vector onto the horizontal plane is zero, that is, the tracking target is moving or stationary in the vertical direction, the calculation of the direction change corresponding to that displacement vector can be skipped. Then, the orientation angle of the subsequent displacement vector is subtracted from the orientation angle of the previous displacement vector to obtain the angle difference between the two adjacent displacement vectors. Since the orientation angle ranges from 0 to 360 degrees, direct subtraction may result in a value greater than 180 degrees or less than -180 degrees. Therefore, the angle difference is normalized to fall between -180 degrees and 180 degrees. If the angle difference is greater than 180 degrees, 360 degrees is subtracted; if the angle difference is less than -180 degrees, 360 degrees is added. The normalized angle difference represents the change in direction for that pair of adjacent displacement vectors. A positive change in direction indicates a clockwise deflection of the target's direction of travel, while a negative change indicates a counter-clockwise deflection. The absolute value of the change in direction represents the magnitude of the deflection; a larger absolute value indicates a more severe deflection. By performing this operation on each pair of adjacent displacement vectors in the displacement vector sequence, multiple changes in direction can be obtained.

[0050] Step S3004: Arrange all directional changes according to time sequence to form a sequence of directional changes.

[0051] Specifically, each directional change corresponds to a pair of adjacent displacement vectors in the displacement vector sequence. The directional changes are arranged sequentially according to the order of these adjacent displacement vectors in the displacement vector sequence. In the arranged sequence of directional changes, each directional change has a clear time sequence index, and the directional changes in the sequence represent the directional deflection of the tracked target in each time interval.

[0052] Step S3005: Perform linear trend fitting on the direction change sequence to obtain the predicted direction change at the next moment.

[0053] Specifically, the portion of the direction change sequence falling within a preset first time window is selected. Using the time index corresponding to each direction change within this preset first time window as the independent variable and the numerical value of the direction change as the dependent variable, a linear trend fitting is performed to obtain a fitted straight line representing the trend of direction change over time. The linear trend fitting can employ the least squares method, i.e., finding a straight line that minimizes the sum of squared residuals from each direction change within the preset first time window to this line. The fitted straight line has a slope and an intercept, where the slope indicates the trend of direction change increasing or decreasing over time: a positive slope indicates an increasing trend, and a negative slope indicates a decreasing trend. Then, the direction change at the next moment is extrapolated based on the fitted straight line: substituting the time index corresponding to the next moment into the fitted straight line, the direction change at the next moment is calculated and used as the predicted direction change at the next moment. The predicted direction change at the next moment represents the expected deflection of the tracking target's direction of travel from the current moment to the next moment.

[0054] Step S3006: Calculate the magnitude of each displacement vector in the displacement vector sequence to obtain the displacement length sequence.

[0055] Specifically, the above operation is performed sequentially on each displacement vector in the displacement vector sequence to obtain multiple displacement lengths, which are then arranged in the same time order as the displacement vector sequence to form a displacement length sequence. Each displacement length in the displacement length sequence represents the distance the tracked target moves within the corresponding time interval, and can be used to describe the changes in the magnitude of the tracked target's movement.

[0056] Step S3007: Perform linear trend fitting on the displacement length sequence to obtain the predicted displacement length at the next moment.

[0057] Specifically, the portion of the displacement length sequence falling within a preset first time window is selected. Using the time index corresponding to each displacement length within this preset first time window as the independent variable and the numerical value of the displacement length as the dependent variable, a linear trend fitting is performed to obtain a fitted straight line representing the trend of displacement length change over time. The linear trend fitting can employ the least squares method, i.e., finding a straight line that minimizes the sum of squared residuals from each displacement length within the preset first time window to this line. The fitted straight line has a slope and an intercept, where the slope indicates the trend of displacement length increasing or decreasing over time: a positive slope indicates an increasing trend in the movement amplitude of the tracked target, i.e., the tracked target is accelerating; a negative slope indicates a decreasing trend in the movement amplitude of the tracked target, i.e., the tracked target is decelerating. Then, the displacement length for the next moment is extrapolated based on the fitted straight line: the time index corresponding to the next moment is substituted into the fitted straight line to calculate the displacement length for the next moment, which is then used as the predicted displacement length for the next moment.

[0058] Step S3008: Add the predicted direction change of the displacement vector at the current moment to the predicted direction angle at the next moment to obtain the predicted direction angle at the next moment. Construct a predicted displacement vector that changes non-linearly at the next moment based on the predicted direction angle at the next moment and the predicted displacement length at the next moment.

[0059] Specifically, if the summed angle exceeds 360 degrees, 360 degrees are subtracted; if it is less than 0 degrees, 360 degrees are added, ensuring the predicted direction angle for the next moment falls within the range of 0 to 360 degrees. A Cartesian coordinate system is pre-established in the horizontal plane, with a horizontal axis and a vertical axis. The positive direction of the horizontal axis aligns with the preset reference direction, and the positive direction of the vertical axis is perpendicular to the positive direction of the horizontal axis. Based on the predicted direction angle for the next moment, the unit direction vector in the horizontal plane is determined using trigonometric functions: the cosine of the predicted direction angle for the next moment is used as the component of the unit direction vector on the horizontal axis, and the sine of the predicted direction angle for the next moment is used as the component of the unit direction vector on the vertical axis, thus obtaining the unit direction vector in the horizontal plane. The magnitude of the unit direction vector is 1, and its direction is consistent with the predicted direction angle for the next moment. Then, the unit direction vector in the horizontal plane is multiplied by the predicted displacement length for the next moment to obtain the displacement component in the horizontal plane for the next moment. For the vertical direction, one of the following two methods can be used: First, the vertical component of the displacement vector at the current moment can be used as the vertical displacement component at the next moment; second, the vertical displacement component at the next moment can be set to zero. The displacement components in the horizontal plane and the vertical displacement components are then combined to obtain a predicted displacement vector that changes non-linearly at the next moment. This non-linear predicted displacement vector represents the possible displacement at the next moment if the tracked target continues to deflect according to the recent directional change trend. Optionally, if the absolute value of the predicted directional change at the next moment is greater than a preset maximum deflection threshold, the predicted directional change at the next moment is limited to the preset maximum deflection threshold, while maintaining its positive or negative sign. The preset maximum deflection threshold can be set according to the type of tracked target, sampling frequency, and actual motion scenario, preferably between 10° and 30°, for example, 20°. If the sampling frequency is high, the threshold can be appropriately reduced; if the sampling frequency is low, the threshold can be appropriately increased.

[0060] Step S3009: Construct a predicted displacement vector that changes linearly at the next moment based on the displacement vector direction angle at the current moment and the predicted displacement length at the next moment.

[0061] Specifically, based on the current displacement vector direction angle, the unit direction vector in the horizontal plane is determined using trigonometric functions: the cosine of the current displacement vector direction angle is used as the component of the unit direction vector on the horizontal axis, and the sine of the current displacement vector direction angle is used as the component of the unit direction vector on the vertical axis, thus obtaining the unit direction vector in the horizontal plane. The magnitude of the unit direction vector is 1, and its direction is consistent with the current displacement vector direction angle. Then, the unit direction vector in the horizontal plane is multiplied by the predicted displacement length for the next moment to obtain the displacement component in the horizontal plane for the next moment. For the vertical direction, one of the following two methods can be used: first, the vertical component of the current displacement vector is used as the vertical displacement component for the next moment; second, the vertical displacement component for the next moment is set to zero. The displacement components in the horizontal plane and the vertical displacement components are combined to obtain the predicted displacement vector that changes linearly for the next moment. The linearly changing predicted displacement vector indicates that if the tracked target maintains its current direction of motion, but only its movement amplitude changes according to the trend of the recent period, then the displacement that may occur at the next moment is...

[0062] The weight distribution generation module 400 is used to determine the weights of the linearly changing predicted displacement vectors and the non-linearly changing predicted displacement vectors, and to obtain the predicted tracking target trajectory weight distribution by Gaussian superposition based on the linearly changing and non-linearly changing predicted displacement vectors, as well as the weights of the linearly changing and non-linearly changing predicted displacement vectors.

[0063] The specific steps involved in determining the weights of the linearly changing predicted displacement vector and the non-linearly changing predicted displacement vector in the weight distribution generation module 400 are as follows: Step S4001: Calculate the mean of all displacement lengths in the displacement length sequence as the average displacement length.

[0064] Step S4002: Divide the predicted displacement length at the next moment by the average displacement length to obtain the normalized predicted displacement length at the next moment.

[0065] Specifically, the normalized predicted displacement length for the next moment is a dimensionless value, representing the multiple of the predicted movement amplitude relative to the average movement amplitude. If the normalized predicted displacement length for the next moment is greater than 1, it indicates that the predicted movement amplitude for the next moment is greater than the average movement amplitude, and the tracked target has an accelerating trend; if it is less than 1, it indicates that the predicted movement amplitude for the next moment is less than the average movement amplitude, and the tracked target has a decelerating trend; if it is equal to 1, it indicates that the predicted movement amplitude for the next moment is comparable to the average movement amplitude. If the average displacement length is less than the preset lower limit, or the average displacement length is zero, the average displacement length can be replaced with the preset lower limit before performing the division operation to avoid the normalization result being abnormally large due to an excessively small denominator. The preset lower limit can be set according to the type of tracked target, sampling frequency, and actual motion scenario, preferably 0.01 meters.

[0066] Step S4003: Calculate the mean of the absolute values ​​of all directional changes in the directional change sequence, and use it as the average directional change.

[0067] It should be noted that if there are missing direction changes in the direction change sequence due to positioning failure, these missing direction changes can be skipped and not included in the sum of the absolute values ​​of direction changes or the number of direction changes. If the number of direction changes is zero, meaning there are no valid direction changes, the average direction change calculated at the previous time step can be used.

[0068] Step S4004: Divide the absolute value of the predicted direction change at the next moment by the average direction change to obtain the normalized predicted direction change at the next moment.

[0069] Specifically, the normalized predicted direction change at the next time step is a dimensionless value, representing the multiple of the predicted direction change magnitude relative to the average direction change magnitude at the next time step. If the normalized predicted direction change magnitude at the next time step is greater than 1, it indicates that the predicted direction change magnitude at the next time step is greater than the average direction change magnitude, and the turning trend of the tracked target is strengthened; if it is less than 1, it indicates that the predicted direction change magnitude at the next time step is less than the average direction change magnitude, and the turning trend of the tracked target is weakened; if it is equal to 1, it indicates that the predicted direction change magnitude at the next time step is comparable to the average direction change magnitude. By dividing the absolute value of the predicted direction change magnitude at the next time step by the average direction change magnitude, the influence of differences in the turning frequencies of different tracked targets and different sampling frequencies on the weight calculation can be eliminated, making the trend of displacement length change comparable to the trend of direction change magnitude change. Optionally, if the average direction change magnitude is less than a preset lower limit value, or the average direction change magnitude is zero, the average direction change magnitude can be replaced with the preset lower limit value before subsequent division operations to avoid the normalization result being abnormally large due to an excessively small denominator. The preset lower limit of change can be set according to the type of the tracked target, the sampling frequency and the actual motion scenario, and is preferably 0.5 degrees.

[0070] Step S4005: Using the normalized predicted displacement length at the next time step as the numerator, and the sum of the normalized predicted displacement length at the next time step and the normalized predicted direction change at the next time step as the denominator, calculate the weight of the predicted displacement vector that changes linearly at the next time step.

[0071] It should be noted that the weight of the predicted displacement vector changing linearly in the next moment represents the probability that the tracked target will maintain its current direction of motion and only change its magnitude of movement according to a trend in the prediction of the next moment. When the normalized predicted displacement length in the next moment is large and the normalized predicted direction change in the next moment is small, it indicates that the magnitude of the tracked target's movement is increasing while the turning trend is not obvious. In this case, the probability of linear movement is high, and the weight of the predicted displacement vector changing linearly in the next moment is correspondingly large. Conversely, when the normalized predicted displacement length in the next moment is small and the normalized predicted direction change in the next moment is large, it indicates that the tracked target's turning trend is obvious while the magnitude of movement does not change much. In this case, the probability of linear movement is low, and the weight of the predicted displacement vector changing linearly in the next moment is correspondingly small.

[0072] Step S4006: Using the normalized predicted direction change at the next time step as the numerator, and the sum of the normalized predicted displacement length at the next time step and the normalized predicted direction change at the next time step as the denominator, calculate the weight of the predicted displacement vector that changes non-linearly at the next time step.

[0073] Specifically, the weight of the predicted displacement vector for a non-linear change in the next moment represents the probability that the tracked target will deflect according to the recent trend of directional change, i.e., undergo non-linear motion, in the prediction of the next moment. When the normalized predicted directional change in the next moment is large and the normalized predicted displacement length in the next moment is small, it indicates that the tracked target's turning trend is obvious but the change in movement amplitude is not significant. In this case, the probability of non-linear motion is high, and the weight of the predicted displacement vector for a non-linear change in the next moment is correspondingly large. Conversely, when the normalized predicted directional change in the next moment is small and the normalized predicted displacement length in the next moment is large, it indicates that the tracked target's movement amplitude is increasing but the turning trend is not obvious. In this case, the probability of non-linear motion is low, and the weight of the predicted displacement vector for a non-linear change in the next moment is correspondingly small. Through the above weight calculation, the relative importance of the linear motion assumption and the non-linear motion assumption in the prediction can be adaptively adjusted according to the trend of the tracked target's displacement length change and the trend of its directional change, thereby improving the accuracy of predicting the future movement trend of the tracked target.

[0074] The weight distribution generation module 400 obtains the predicted target trajectory weight distribution by Gaussian superposition based on the predicted displacement vectors with linear and non-linear changes, as well as the weights of the linear and non-linear predicted displacement vectors. The specific steps include the following: Step S4011: Transform the predicted displacement vector that changes linearly in the next moment and the predicted displacement vector that changes non-linearly in the next moment into the two-dimensional image plane of each UAV through perspective transformation based on camera projection relationship, to obtain the corresponding linear pixel displacement vector and non-linear pixel displacement vector.

[0075] Specifically, for each drone in a multi-drone swarm, the current position of the tracked target in the world coordinate system is obtained as the current world position. The current world position is then added to both the predicted displacement vector for a linear change and the predicted displacement vector for a non-linear change at the next moment, resulting in two predicted positions: the linear predicted position and the non-linear predicted position. The two-dimensional image plane refers to the pixel plane composed of the synchronized image frames of each drone at the current moment, consisting of horizontally and vertically arranged pixels, each with corresponding x and y coordinates. Then, using the drone's camera intrinsic and extrinsic parameters, the current world position, the linear predicted position, and the non-linear predicted position are projected onto the drone's two-dimensional image plane, respectively, to obtain the current pixel, the linear predicted pixel, and the non-linear predicted pixel. The camera extrinsic parameters are determined by the camera's spatial position and attitude recorded by the drone's onboard integrated navigation system, while the camera intrinsic parameters are obtained through pre-calibration. Next, the linear predicted pixel is subtracted from the current pixel to obtain the drone's linear pixel displacement vector; the non-linear predicted pixel is subtracted from the current pixel to obtain the drone's non-linear pixel displacement vector. Performing the above operations on each drone in a multi-drone swarm yields the linear and non-linear pixel displacement vectors for each drone. The units for both linear and non-linear pixel displacement vectors are pixels, representing the expected pixel offset and direction of movement in the image plane from the current pixel point, respectively, if the tracked target moves according to the linear motion assumption or the non-linear motion assumption.

[0076] Step S4012: Using the endpoints of the linear pixel displacement vectors and non-linear pixel displacement vectors as centers, and the weights of the predicted displacement vectors that change linearly in the next moment and the predicted displacement vectors that change non-linearly in the next moment as amplitudes, Gaussian superposition is performed with a preset bandwidth as the Gaussian kernel width to obtain the predicted tracking target trajectory weight distribution in the two-dimensional image plane of each UAV.

[0077] Specifically, in the two-dimensional image plane of the UAV, a first Gaussian kernel is constructed with the endpoint of the linear pixel displacement vector as the center, the weight of the predicted displacement vector that changes linearly in the next time step as the amplitude, and a preset bandwidth as the width of the Gaussian kernel. A second Gaussian kernel is constructed with the endpoint of the non-linear pixel displacement vector as the center, the weight of the predicted displacement vector that changes non-linearly in the next time step as the amplitude, and a preset bandwidth as the width of the Gaussian kernel. Then, the first and second Gaussian kernels are superimposed on the two-dimensional image plane. That is, for each pixel position in the two-dimensional image plane, the value of that pixel position in the first Gaussian kernel is added to the value in the second Gaussian kernel to obtain the predicted weight value for that pixel position. The predicted weight values ​​of all pixel positions together constitute the predicted tracking target trajectory weight distribution in the two-dimensional image plane of the UAV. The Gaussian kernel is a two-dimensional Gaussian function, its center position corresponds to the predicted pixel endpoint, its amplitude corresponds to the corresponding assumed weight, and its width is determined by the preset bandwidth. The preset bandwidth, measured in pixels, represents the dispersion of the Gaussian kernel. A larger preset bandwidth results in a more dispersed Gaussian kernel and a smoother distribution of predicted trajectory weights; a smaller preset bandwidth results in a more concentrated Gaussian kernel and a sharper distribution of predicted trajectory weights. In the predicted target trajectory weight distribution, the pixel with the larger weight value indicates a higher probability that the target will appear at that position in the next moment. Since linear motion and non-linear motion hypotheses correspond to different pixel endpoints and each has its own weight, the superimposed weight distribution can simultaneously reflect the probability of both motion hypotheses and adaptively favor the more likely motion trend based on the weights.

[0078] The preset bandwidth is determined based on the linear trend fitting result of the displacement length sequence. Specifically, the displacement length sequence is fitted with a linear trend, and the mean of the absolute value of the residual between the actual displacement length sequence and the fitted linear trend is calculated. This mean value is used as the preset bandwidth.

[0079] It should be noted that: a linear trend fitting is performed on the displacement length sequence to obtain a fitted straight line representing the trend of displacement length change over time. This fitted straight line was calculated in step S3007 to predict the displacement length at the next moment, and can be directly reused here without refitting. Then, for each displacement length in the displacement length sequence, the difference between that displacement length and the fitted value at the corresponding moment on the fitted straight line is calculated, and the absolute value of this difference is taken to obtain the absolute residual value corresponding to that displacement length. The absolute residual values ​​corresponding to all displacement lengths in the displacement length sequence are summed, and then divided by the number of displacement lengths to obtain the mean of the absolute residual values. The mean of the absolute residual values ​​is the length value in the world coordinate system, representing the average fluctuation amplitude of the displacement length sequence around the fitted straight line. The larger the mean of the absolute residual values, the more drastic the fluctuation of the displacement length sequence, the greater the uncertainty of the prediction, and the more dispersed the Gaussian kernel should be; the smaller the mean of the absolute residual values, the more stable the displacement length sequence, the more reliable the prediction, and the more concentrated the Gaussian kernel should be. Since Gaussian stacking is performed pixel-wise on the two-dimensional image plane of each drone, the mean of the absolute values ​​of the residuals needs to be converted from world coordinate system length units to pixel units to obtain the preset bandwidth. Specifically, the local scaling factor can be determined based on the ratio between the world displacement near the current position of the tracked target and the corresponding pixel displacement; the preset bandwidth in pixels is obtained by multiplying the mean of the absolute values ​​of the residuals by the local scaling factor. The local scaling factor can be obtained by projecting a small world displacement near the current position of the tracked target onto the image plane, calculating its corresponding pixel displacement, and removing the pixel position to obtain the world displacement. Optionally, if the mean of the absolute values ​​of the residuals is zero, or the preset bandwidth obtained after conversion is less than the preset minimum bandwidth value, then the preset minimum bandwidth value is used as the preset bandwidth. The preset minimum bandwidth value can be set according to the image resolution, preferably 1% of the image width.

[0080] The tracking quality assessment module 500 is used to determine the degree to which the center of the bounding box of each UAV deviates from the center of the image, and to determine the trajectory coverage index of each UAV based on the predicted tracking target trajectory weight distribution.

[0081] It should be noted that once the predicted movement trend of the target is obtained, a tracking strategy for a multi-UAV swarm can be formulated. In the "lead-follow" strategy, typically one lead UAV is responsible for continuously tracking the target, while the other UAVs follow the lead UAV to achieve multi-view tracking and blind spot coverage. However, when the target suddenly turns or enters an obstructed blind spot, the lead UAV may lose sight of the target, causing the entire swarm to fall into a brief period of coordination disorder, affecting the tracking effect and efficiency. For example, during tracking, the lead UAV is directly in front of the target and has good observation conditions; but when the target suddenly turns, the line-of-sight angle between the lead UAV and the target increases rapidly, and the target may quickly move out of the image field of view boundary, or the detection box may become unstable due to a rapid change in projection scale. At this time, the lead UAV itself may lose sight of the target, and the entire swarm will fall into a brief period of coordination disorder, which may result in tracking interruption. Therefore, a dynamic lead UAV mechanism is introduced, which designates the UAV with the most suitable viewing angle and current movement conditions as the real-time lead UAV, and the other UAVs follow the lead UAV to perform multi-view tracking and blind spot coverage of the target, thereby improving the effect and stability of target tracking in a multi-UAV swarm. In multi-drone collaborative tracking, each drone's observation of the target is not always continuous and reliable. When the target passes through an obstacle, the line of sight of some drones may be temporarily obstructed, causing jumps in the detection bounding box and a sudden drop in positioning quality indicators before recovering. Alternatively, if a drone's flight attitude adjustment is not timely, the target may move to the edge of the image, resulting in truncated detection bounding boxes and increased positioning errors. While these brief observational anomalies may not completely disable the drone's tracking capability, they will leave obvious fluctuations in its corresponding positioning quality time series. Drones that have been frequently affected by environmental factors or target movement in the recent past are clearly unsuitable as lead drones. Conversely, even if a drone's current positioning quality is not optimal, if its positioning quality time series is relatively stable and it provides better coverage of the target's movement trend direction compared to the current lead drone, it may be a more suitable candidate for a lead drone.

[0082] The Tracking Quality Assessment Module 500 is specifically used for: Step S5001: For each drone in the multi-drone cluster, obtain the center coordinates of the target bounding box tracked by the drone in each synchronized image frame within a preset second time window, and construct a sequence of bounding box center coordinates in chronological order.

[0083] Specifically, the second time window is preset to be a period of time prior to the current moment. The preset second time window can be the same as or different from the preset first time window in step S2007; preferably, they are the same, both taking 5 seconds prior to the current moment, so that trajectory prediction and tracking quality assessment use observation data within the same time period, simplifying window maintenance for the airborne system. The bounding box center coordinates are output by the target detector in step S2001, including the x and y coordinates of the center point. The acquired bounding box center coordinates are arranged sequentially according to the chronological order of each synchronized image frame, forming a bounding box center coordinate sequence. Each center coordinate in the bounding box center coordinate sequence has a clear chronological index, sequentially representing the position of the tracking target bounding box center in the corresponding synchronized image frame. If no tracking target is detected in a synchronized image frame, i.e., there are no valid bounding box center coordinates in that frame, then that frame can be skipped and not included in the bounding box center coordinate sequence.

[0084] Step S5002: For each center coordinate in the bounding box center coordinate sequence, calculate the first distance between the center coordinate and the diagonal intersection of the corresponding synchronous image frame of the UAV.

[0085] Specifically, for each center coordinate in the bounding box center coordinate sequence, the corresponding synchronized image frame is determined. The intersection of the diagonals of the synchronized image frame is the geometric center of that frame, with its x-coordinate being half the image width and its y-coordinate being half the image height. Then, the Euclidean distance between the x-coordinate of this center coordinate and the intersection of the diagonals is calculated, which is the first distance. The unit of the first distance is pixels, representing the pixel distance of the tracking target bounding box center from the image center in that synchronized image frame. By performing the above operation on each center coordinate in the bounding box center coordinate sequence, multiple first distances can be obtained.

[0086] Step S5003: Perform a weighted summation on all first distances according to the preset weighting rules to obtain the weighted total distance; wherein, the preset weighting rules are: the weight of the first distance corresponding to the current time is 1, the weight of the first distance corresponding to the earliest time in the preset second time window is 0, and the weight of the first distance corresponding to other times increases linearly with time.

[0087] Specifically, all first distances are arranged in the same time sequence as the bounding box center coordinate sequence. A weight is assigned to each first distance according to a preset weighting rule. If a preset second time window contains m first distances, numbered sequentially from 1 to m, then the weight of the 1st first distance is 0, the weight of the mth first distance is 1, and the weight of the hth first distance is (h-1) / (m-1), where h is an integer between 1 and m. Therefore, the closer a first distance is to the current time, the greater its weight; the farther away a first distance is from the current time, the smaller its weight. Then, each first distance is multiplied by its corresponding weight to obtain multiple weighted distances; these weighted distances are summed to obtain a total weighted distance. The total weighted distance represents the cumulative weighted deviation of the target bounding box center from the image center within the preset second time window. Since the weight is greatest at the current time, the total weighted distance effectively reflects the recent tracking performance.

[0088] Step S5004: Divide the sum of weighted distances by the sum of weights to obtain the weighted average distance.

[0089] Specifically, the weighted sum is the sum of the weights corresponding to all first distances participating in the weighted summation. The weighted average distance represents the weighted average pixel distance between the center of the tracking target bounding box and the image center within a preset second time window. The larger the weighted average distance, the more frequently the center of the tracking target bounding box has deviated from the image center recently, and the worse the tracking performance of the drone; the smaller the weighted average distance, the closer the center of the tracking target bounding box has been to the image center recently, and the better the tracking performance of the drone. Optionally, if the weighted sum is zero, that is, there are no valid first distances participating in the weighted summation, the weighted average distance is set to a preset first default value. The preset first default value can be set according to the image resolution, for example, set to half the length of the image diagonal.

[0090] Step S5005: Divide the weighted average distance by half the diagonal length of the corresponding synchronized image frame of the UAV to obtain the degree to which the center of the normalized bounding box of the UAV deviates from the center of the image.

[0091] Specifically, the diagonal length of the synchronized image frame is the square root of the sum of the squares of the width and height of the image frame. Dividing the diagonal length by 2 yields half of the diagonal length of the synchronized image frame. The degree to which the center of the normalized bounding box deviates from the image center is a dimensionless value, ranging from 0 to 1. The closer the value is to 0, the closer the center of the target bounding box is to the image center within a preset second time window, indicating better recent tracking performance of the drone; the closer the value is to 1, the more frequently the target bounding box deviates from the image center, indicating worse recent tracking performance of the drone. If half the diagonal length of the synchronized image frame is zero, or less than a preset first lower limit, it is replaced with the preset first lower limit before division to avoid an abnormally large normalization result due to an excessively small denominator. The preset first lower limit can be set according to the image resolution, for example, 1 pixel.

[0092] Step S5006: According to the predicted tracking target trajectory weight distribution, each pixel position in the synchronous image frame corresponding to the UAV is weighted by its corresponding weight value to calculate the weighted centroid.

[0093] Specifically, for each pixel location in the synchronized image frame, the x-coordinate and y-coordinate of that pixel location, as well as the corresponding weight value, are obtained. The x-coordinate of each pixel location is multiplied by its weight value, and this sum is applied to all pixel locations to obtain a weighted sum of x-coordinates. Similarly, the y-coordinate of each pixel location is multiplied by its weight value, and this sum is applied to all pixel locations to obtain a weighted sum of y-coordinates. Simultaneously, the weight values ​​of all pixel locations are summed to obtain the total weight value. The weighted sum of x-coordinates is divided by the total weight value to obtain the x-coordinate of the weighted centroid; the weighted sum of y-coordinates is divided by the total weight value to obtain the y-coordinate of the weighted centroid. The weighted centroid is formed by these x-coordinates and y-coordinates. The weighted centroid represents the weighted center position of the target's expected trajectory within the synchronized image frame.

[0094] Step S5007: Calculate the second distance between the weighted centroid and the diagonal intersection point of the synchronous image frame corresponding to the UAV.

[0095] Specifically, the Euclidean distance between the intersection of the x-coordinate of the weighted centroid and its diagonal is calculated; this is the second distance. The second distance is measured in pixels and represents the pixel distance between the weighted centroid of the target's expected trajectory and the image center. A larger second distance indicates that the predicted target trajectory deviates further from the image center, resulting in poorer future trajectory coverage for the drone; a smaller second distance indicates that the predicted target trajectory is closer to the image center, resulting in better future trajectory coverage for the drone.

[0096] Step S5008: Divide the second distance by half the diagonal length of the corresponding synchronized image frame of the UAV to obtain the normalized trajectory coverage index of the UAV.

[0097] Specifically, the normalized trajectory coverage index is a dimensionless value ranging from 0 to 1. The closer the value is to 0, the closer the weighted centroid of the target's expected trajectory is to the image center, indicating better future trajectory coverage for the drone and making it more suitable as a candidate leader drone. Conversely, the closer the value is to 1, the further the weighted centroid of the target's expected trajectory is from the image center, indicating poorer future trajectory coverage for the drone and making it less suitable as a candidate leader drone.

[0098] The pilot drone setting module 600 is used to obtain a score for each drone as a candidate pilot drone based on the degree of deviation of the bounding box center from the image center and the trajectory coverage index, and to dynamically set the pilot drone from the multi-drone cluster based on the score.

[0099] The navigation drone setting module 600 obtains a score for each drone as a candidate navigation drone based on the degree to which the bounding box center deviates from the image center and the trajectory coverage index. The specific steps include the following: Step S6001: The sum of the degree to which the center of the normalized bounding box of the UAV deviates from the center of the image and 1 is used as the first sum value.

[0100] Specifically, the first sum ranges from 1 to 2. The closer the first sum is to 1, the smaller the deviation of the normalized bounding box center of the drone from the image center, meaning the better the recent tracking performance of the drone; the closer the first sum is to 2, the worse the recent tracking performance of the drone.

[0101] Step S6002: Use the sum of the normalized trajectory coverage index of the UAV and 1 as the second sum.

[0102] Specifically, the second sum ranges from 1 to 2. The closer the second sum is to 1, the smaller the normalized trajectory coverage index of the UAV, meaning the closer the weighted centroid of the target's expected movement trajectory is to the center of the image, and the better the UAV's future trajectory coverage capability; the closer the second sum is to 2, the worse the UAV's future trajectory coverage capability.

[0103] Step S6003: The product of the first sum and the second sum is determined as the score of the drone as a candidate lead drone; wherein, the lower the score, the more suitable the drone is as a candidate lead drone.

[0104] Specifically, since the first and second sums both range from 1 to 2, the score ranges from 1 to 4. The closer the score is to 1, the smaller the deviation of the drone's normalized bounding box center from the image center and the smaller its normalized trajectory coverage index, indicating that the drone has good recent tracking performance and good future trajectory coverage, making it more suitable as a candidate leader drone. Conversely, the closer the score is to 4, the worse the drone's recent tracking performance and poor future trajectory coverage, making it less suitable as a candidate leader drone.

[0105] It's important to note that after obtaining the scores of the candidate lead drones, the drone with the highest score cannot be simply selected as the lead drone. The reason is that when both drones A and B have scores close to 1, assuming drone A, with the closest score (closest to 1), is located at the edge of the formation, while drone B, with the second closest score (and roughly the same as drone A), is closer to the center of the formation and the original lead drone, selecting drone B as the lead drone would be more effective. Although this sacrifices the lead drone's own tracking performance to some extent, for the entire formation, selecting drone B significantly reduces the planning complexity of reorganizing the formation, thereby indirectly improving the tracking performance during lead drone switching. Furthermore, directly selecting the drone with the closest score (closest to 1) could lead to the lead drone role repeatedly switching between several drones with good tracking performance, causing continuous adjustments or even chaos in the drone swarm's formation, ultimately reducing tracking effectiveness.

[0106] The pilot drone setting module 600 dynamically sets the pilot drone from a multi-drone cluster based on a score. The specific steps include the following: Step S6011: Obtain the score of the pilot drone at the current moment within the preset second time window, and construct a score sequence in chronological order.

[0107] Specifically, the acquired scores are arranged sequentially according to their chronological order at each moment, forming a score sequence. Each score in the score sequence has a clear chronological index, representing the score of the navigation drone at the corresponding moment. If the score of the navigation drone at a certain moment cannot be obtained due to missing data, that moment can be skipped and not included in the score sequence.

[0108] Step S6012: Calculate the mean and standard deviation of the rating sequence.

[0109] Specifically, the mean score represents the average score of the lead drone at the current moment within a preset second time window, reflecting the recent average tracking performance of the lead drone. A mean score closer to 1 indicates better recent average tracking performance; a mean score closer to 4 indicates worse recent average tracking performance. The standard deviation represents the fluctuation in the lead drone's score within the preset second time window, reflecting the stability of its recent tracking performance. A larger standard deviation indicates more drastic fluctuations in the lead drone's recent score and less stable tracking performance; a smaller standard deviation indicates more stable recent score and more stable tracking performance.

[0110] Step S6013: For each drone in the multi-drone cluster, subtract the score of the drone as a candidate leader drone from the mean to obtain the drone's leading position, and divide the drone's leading position by the standard deviation to obtain the standardized value of the drone's leading position.

[0111] Specifically, since lower scores indicate better tracking performance, when a drone's score is lower than the average score of the current lead drone, its lead is positive, and a larger value indicates better average tracking performance relative to the current lead drone. Conversely, when a drone's score is higher than the average score of the current lead drone, its lead is negative, indicating worse average tracking performance relative to the current lead drone. The standardized lead value represents the multiple of the drone's lead relative to the fluctuation range of the current lead drone's score. If the standardized lead value is greater than or equal to 1, it means the drone's lead exceeds one standard deviation of the current lead drone's score fluctuation, indicating that the drone is significantly better than the current lead drone and is suitable as a candidate lead drone for the next moment. If the standardized lead value is less than 1, it means the drone's lead falls within the normal fluctuation range of the current lead drone's score, and switching the lead drone is not very meaningful.

[0112] Step S6014: If the standardized value of the drone's leading position is greater than or equal to a preset multiple threshold, then the drone is determined as a candidate leading drone for the next moment.

[0113] Specifically, a preset multiple threshold is used to determine whether the drone's lead over the current lead drone is significant. If the standardized value of a drone's lead is greater than or equal to the preset multiple threshold, it indicates that the drone's lead exceeds a preset multiple of the current lead drone's score fluctuation, and its tracking performance is significantly better than the current lead drone's recent average tracking performance. Therefore, this drone is identified as a candidate lead drone for the next moment. The preset multiple threshold can be set according to the sensitivity of lead drone switching in the actual scenario. Preferably, the preset multiple threshold is 1, that is, when the standardized value of a drone's lead is greater than or equal to 1, it is considered that its lead exceeds one standard deviation of the current lead drone's score fluctuation, and switching to the lead drone is statistically significant. If it is desired to reduce the switching frequency and improve the stability of switching, the preset multiple threshold can be appropriately increased, for example, to 1.5 or 2; if it is desired to improve switching sensitivity and respond promptly to rapid changes in the tracking target's movement trend, the preset multiple threshold can be appropriately decreased, for example, to 0.8 or 0.5.

[0114] Step S6015: If there is no drone with a standardized leading value greater than or equal to a preset multiple threshold, then the current leading drone remains unchanged.

[0115] Specifically, if the standardized lead values ​​of all drones are less than a preset multiple threshold, it means that no drone's lead over the current lead drone exceeds a preset multiple of the lead drone's score fluctuation, indicating that the advantages of other drones relative to the lead drone are not significant. In this case, forcibly switching the lead drone might only trigger a switch due to normal score fluctuations, causing the lead role to repeatedly switch between drones with similar tracking states. This would lead to continuous adjustments or even chaos in the drone swarm's formation, ultimately reducing tracking effectiveness. Therefore, in this situation, the current lead drone is maintained; that is, the current lead drone continues to lead in the next moment, while the other drones continue to follow and coordinate with it. This avoids meaningless lead drone switching, maintaining the stability of the formation and the continuity of the tracking process.

[0116] Step S6016: Obtain the switching evaluation value for each candidate lead drone; wherein the switching evaluation value is determined based on the three-dimensional spatial distance and leading degree between the candidate lead drone and the current lead drone.

[0117] Step S6016 further includes: Step S60161: For each candidate lead drone, calculate the three-dimensional spatial distance between the candidate lead drone and the lead drone at the current moment.

[0118] Specifically, the 3D position of the current lead drone in the world coordinate system and the 3D position of each candidate lead drone in the world coordinate system are obtained. The 3D positions can be provided by the onboard integrated navigation system of each drone or by the formation control module of a multi-drone swarm. For each candidate lead drone, its 3D position is subtracted from the current lead drone's 3D position to obtain a position difference vector; the magnitude of this position difference vector is calculated to obtain the 3D spatial distance between the candidate lead drone and the current lead drone. The unit of the 3D spatial distance is meters, representing the distance between the candidate lead drone and the current lead drone in the formation. A larger 3D spatial distance indicates that the candidate lead drone is farther from the current lead drone, and the formation adjustment range needs to be larger after switching to the lead drone; a smaller 3D spatial distance indicates that the candidate lead drone is closer to the current lead drone, and the formation adjustment range needs to be smaller after switching.

[0119] Step S60162: Obtain the maximum three-dimensional spatial distance between all candidate lead drones and the lead drone at the current moment, and use it as the maximum spatial distance.

[0120] Step S60163: Divide the three-dimensional spatial distance between the candidate lead drone and the current lead drone by the maximum spatial distance to obtain the normalized spatial distance of the candidate lead drone.

[0121] Specifically, the normalized spatial distance is a dimensionless value, ranging from 0 to 1. A smaller normalized spatial distance indicates that the candidate lead drone is closer to the current lead drone, and the formation adjustment range required after a switch is smaller; a larger normalized spatial distance indicates that the candidate lead drone is farther from the current lead drone, and the formation adjustment range required after a switch is larger. If there is only one candidate lead drone, its normalized spatial distance is 1.

[0122] Step S60164: Divide the leading degree of the candidate lead drone by the sum of the normalized spatial distance of the candidate lead drone and 1 to obtain the switching evaluation value of the candidate lead drone.

[0123] Specifically, the switching evaluation value comprehensively considers the candidate lead drone's leading position and its distance from the current lead drone: a greater leading position indicates that the candidate lead drone's tracking performance is better than the current lead drone; a smaller normalized spatial distance indicates that the candidate lead drone is closer to the current lead drone, and the smaller the range of formation adjustment required after the switch. Therefore, a larger switching evaluation value indicates that the candidate lead drone is more suitable as the lead drone for the next moment.

[0124] Step S6017: Select the candidate drone with the highest switching evaluation value as the drone for the next moment.

[0125] Specifically, the candidate lead drone with the highest switching evaluation value indicates that it performs best in terms of both leading position and formation spatial distance. This means the candidate lead drone has a significant advantage in tracking performance compared to the current lead drone, and is closer to the current lead drone, requiring less adjustment of the formation after the switch. This reduces formation reorganization costs while maintaining tracking performance. Optionally, if multiple candidate lead drones have the same or similar switching evaluation values, the candidate lead drone with the smaller normalized spatial distance can be prioritized to further reduce formation reorganization costs.

[0126] The path planning module 700 is used to determine the area that the lead drone can guarantee tracking coverage based on the predicted target trajectory weight distribution, and to plan the flight path of the drone swarm within the area.

[0127] It should be noted that the target trajectory prediction was completed in the previous steps, and a lead drone was set based on this prediction, combined with the tracking performance of each drone and the formation status. At this point, the system has a clear judgment of the target's movement trend, namely, the directions the target may move in the next period of time, how its speed may change, and the spatial range corresponding to these changes. To ensure tracking effectiveness, the lead drone should plan its own flight path to maintain stable observation of the target during the target's impending maneuvers. This flight path should also serve as a reference benchmark for other drones to follow and assist in observation. The lead drone's path planning must first ensure coverage of the target's expected path. Secondly, in the "lead-follow" formation architecture, since the lead drone's position and speed are the spatial benchmark for other drones to form a formation and adjust their own observation strategies, if the lead drone's flight path is too tortuous, involves frequent sharp turns, or has drastic speed changes, other drones will need to perform continuous large-scale maneuvers to maintain their relative position with the lead drone. This not only increases the control difficulty of the following drones but may also cause them to temporarily lose sight of the target due to excessive maneuvering. Therefore, the path of the lead drone should have moderate smoothness and predictability, allowing other drones to maintain formation with minimal control overhead. Thus, while ensuring tracking accuracy, formation-friendly operation must also be considered.

[0128] Specifically, the weighted center position and weighted dispersion radius of the predicted target trajectory weight distribution are calculated. The weighted center position is the weighted centroid of the predicted target trajectory weight distribution. The weighted dispersion radius is the average weighted distance from each position point in the predicted target trajectory weight distribution to the weighted center position. For each position point, the distance from that position point to the weighted center position is calculated, and this distance is summed using the weight value of that position point, then divided by the sum of all weight values ​​to obtain the weighted average distance, which is used as the weighted dispersion radius. Next, a circular area is drawn within the horizontal plane where the target is located at the current moment, with the weighted center position as the center and twice the weighted dispersion radius as the radius. This circular area is defined as the area that the navigation UAV can guarantee tracking coverage. Within this circular area, the probability of the target appearing in the next moment is relatively high, and the navigation UAV flying within this area can maintain a good observation effect on the target.

[0129] The path planning module 700 plans the flight paths of a drone swarm within a region, including the following specific steps: Step S7001: Within the area that the lead drone can guarantee tracking coverage, plan the flight path of the lead drone.

[0130] Specifically, the system acquires the pilot drone at the next moment and its current flight status data, including at least one of the pilot drone's current position, velocity, velocity direction, and attitude angle. The pilot drone's flight status data and the area guaranteed for tracking coverage are input into a pre-constructed digital twin model. This digital twin model simulates the pilot drone's flight process under different control methods. The digital twin model includes at least one of a six-degree-of-freedom rigid body dynamics model, an actuator and sensor response model, and a structural health and fluid-structure interaction model. Before executing the tracking task, the pilot drone's health status is synchronized to the digital twin model, including at least one of signal response speed and motor response characteristics. Based on the input flight status data, the digital twin model simulates flight control under various control methods, outputting multiple sets of pilot drone control parameters and their corresponding candidate flight paths. Each set of control parameters corresponds to one candidate flight path, and each candidate flight path includes multiple path points and the velocity corresponding to each path point. Then, candidate flight paths that would leave the area guaranteed for tracking coverage are excluded from the multiple candidate flight paths, retaining those that remain within the guaranteed tracking coverage area. For each retained candidate flight path, calculate the curvature of each path point within that path, and then calculate the mean curvature of all path points along that candidate flight path. The smaller the mean curvature, the smoother and less tortuous the candidate flight path. Select the candidate flight path with the smallest mean curvature as the flight path for the navigation drone.

[0131] Step S7002: Based on the flight path of the lead drone and the preset tracking array, plan the flight paths of each follower drone.

[0132] Specifically, the preset tracking array is a formation array that maintains good tracking performance while simultaneously achieving multi-angle target tracking and blind spot coverage, with the lead drone remaining unchanged. The preset tracking array includes the expected positions of each follower drone relative to the lead drone. For each follower drone, the expected array position is determined based on the preset tracking array. Then, the expected flight path and corresponding control parameters for the follower drone to switch from its current position to the expected array position are calculated using the aforementioned digital twin model. Each follower drone receives multiple candidate flight paths to reach its expected array position. After obtaining all candidate flight paths for the follower drones, distributed model predictive control combined with a spatiotemporal conflict detection method is used to optimize all candidate flight paths. Distributed model predictive control optimizes the flight paths of each follower drone while satisfying its dynamic and control constraints, enabling it to reach its expected array position as quickly as possible. Spatiotemporal conflict detection detects whether there are spatial or temporal conflicts between the candidate flight paths of the follower drones and eliminates conflicting path combinations. Through the above optimizations, a fastest and conflict-free overall solution is obtained. In this solution, each follower drone corresponds to a flight path and its control parameters, which constitute the flight path of each follower drone. Each follower drone flies according to its own flight path, achieving multi-angle tracking of the target and blind spot coverage.

[0133] The preceding steps have already planned tracking flight paths for each drone. The following processing is then performed: The original lead drone sends its tracking mission data to the new lead drone via the communication and data link module. The new lead drone, controlled by the flight control system module according to the newly planned flight path and control parameters from the preceding steps, combined with environmental perception information from the perception and navigation system module, drives the power system module. Through brushless motors, electronic speed controllers, propellers, etc., the drone's pitch angle, roll angle, yaw rate, and total thrust are adjusted, enabling it to fly along the planned path and track the target. If path planning and other calculations are performed by the base station, the control commands are transmitted from the base station to the flight control system module via the communication and data link module. The remaining follower drones follow a similar control method, flying along their respective planned flight paths to achieve multi-angle tracking and blind spot coverage of the target. During the tracking process, the mission payload module (i.e., the aerial gimbal and camera) continuously senses the target's movement and continues to arrange and plan the flight path of the lead drone according to the above steps, achieving proactive dynamic target tracking by the lead drone.

[0134] In summary, in this embodiment of the invention, trajectory prediction is performed by analyzing the three-dimensional behavioral trajectory of the target being tracked. Based on the predicted trajectory and the short-term monitoring status and coverage of the predicted trajectory by each UAV, a lead UAV is dynamically assigned, and the flight paths of each UAV are planned. This enables predictive and proactive switching of the lead UAV, preventing problems before they occur and effectively avoiding the impact on multi-UAV swarm control and tracking due to unexpected events or poor tracking performance of the lead UAV. Compared with existing methods, this invention offers superior tracking continuity and switching smoothness during target tracking, significantly improving tracking accuracy and performance, and enhancing the stability and reliability of multi-UAV collaborative target tracking.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual target tracking-based unmanned aerial vehicle (UAV) flight system, characterized in that, include: The image acquisition module is used to acquire synchronized image frames from multiple drone clusters targeting the same tracking target. The 3D positioning module is used to obtain the 3D position sequence of the tracked target in the world coordinate system based on the synchronous image frames through the intersection and fusion of multiple UAV lines of sight; The trajectory prediction module is used to determine the displacement vector sequence and the direction change sequence based on the three-dimensional position sequence, and to construct the predicted displacement vector for the next moment as a linear change and the predicted displacement vector for the next moment as a non-linear change based on the displacement vector sequence and the direction change sequence. The weight distribution generation module is used to determine the weights of the linearly changing predicted displacement vectors and the non-linearly changing predicted displacement vectors. Based on the linearly changing and non-linearly changing predicted displacement vectors, as well as the weights of the linearly changing and non-linearly changing predicted displacement vectors, the module obtains the weight distribution of the predicted tracking target trajectory through Gaussian superposition. The tracking quality assessment module is used to determine the degree to which the bounding box center of each drone deviates from the image center, and to determine the trajectory coverage index of each drone based on the predicted tracking target trajectory weight distribution. The pilot drone setting module is used to obtain a score for each drone as a candidate pilot drone based on the degree of deviation of the bounding box center from the image center and the trajectory coverage index, and to dynamically set the pilot drone from the multi-drone cluster based on the score. The path planning module is used to determine the area that the lead drone can guarantee tracking coverage based on the predicted target trajectory weight distribution, and to plan the flight path of the drone swarm within the area.

2. The UAV flight system based on visual target tracking according to claim 1, characterized in that, The three-dimensional positioning module is specifically used for: The synchronous image frames are input into the target detector based on a deep convolutional network, and the bounding box and confidence score of the tracked target in the synchronous image frames are output. Based on the camera's spatial position and attitude recorded by the onboard integrated navigation system of each UAV, and combined with the camera's installation parameters and camera intrinsic parameters in the body coordinate system, the centroid of the bounding box in the synchronized image frame is transformed into a three-dimensional ray that originates from the corresponding camera and points towards the tracking target, so as to unify the tracking target information identified by each camera position into the same three-dimensional space. For each three-dimensional ray corresponding to a drone, calculate the intersection residual between the three-dimensional ray and each of the other three-dimensional rays, and take the average of the multiple intersection residuals as the average residual of the three-dimensional ray. Sort the average residuals of all three-dimensional rays in ascending order to determine the lower quartile, upper quartile, and interquartile range; where the interquartile range is the difference between the upper quartile and the lower quartile. Three-dimensional rays with a residual mean greater than a preset residual threshold are identified as abnormal rays and removed; where the preset residual threshold is the sum of the upper quartile and a preset multiple multiplied by the interquartile range; For each remaining 3D ray after removal, the confidence level output by the target detector is used as the weight, and the least squares triangulation method is used for fusion. The position of the fusion result is used as the position of the tracking target. The positions of the tracked target at each moment within the preset first time window are arranged in chronological order to obtain the three-dimensional position sequence of the tracked target in the world coordinate system; wherein, the preset first time window consists of historical moments before the current moment.

3. The UAV flight system based on visual target tracking according to claim 2, characterized in that, The trajectory prediction module is specifically used for: For each pair of adjacent positions in the three-dimensional position sequence of the tracked target in the world coordinate system, subtract the previous position from the next position to obtain the displacement vector of that pair of adjacent positions. Arrange all displacement vectors according to time sequence to form a displacement vector sequence; For each pair of adjacent displacement vectors in the displacement vector sequence, the direction angle of the latter displacement vector is subtracted from the direction angle of the former displacement vector to obtain the change in direction of the pair of adjacent displacement vectors. Arrange all directional changes according to time sequence to form a sequence of directional changes; By performing linear trend fitting on the sequence of direction changes, the predicted direction change at the next moment can be obtained; Calculate the magnitude of each displacement vector in the displacement vector sequence to obtain the displacement length sequence; By performing linear trend fitting on the displacement length sequence, the predicted displacement length at the next moment can be obtained; Add the predicted change in direction of the displacement vector at the current moment to the predicted direction angle at the next moment to obtain the predicted direction angle at the next moment. Construct a predicted displacement vector that changes non-linearly at the next moment based on the predicted direction angle and the predicted displacement length at the next moment. Construct a predicted displacement vector that changes linearly at the next moment based on the current displacement vector direction angle and the predicted displacement length at the next moment.

4. The UAV flight system based on visual target tracking according to claim 3, characterized in that, The specific steps involved in determining the weights of the linearly changing predicted displacement vector and the non-linearly changing predicted displacement vector in the weight distribution generation module are as follows: Calculate the mean of all displacement lengths in the displacement length sequence, and use it as the average displacement length; Divide the predicted displacement length at the next moment by the average displacement length to obtain the normalized predicted displacement length at the next moment. Calculate the mean of the absolute values ​​of all directional changes in the sequence of directional changes, and use it as the average directional change. Divide the absolute value of the predicted direction change at the next moment by the average direction change to obtain the normalized predicted direction change at the next moment. The weight of the predicted displacement vector that changes linearly in the next moment is calculated by using the normalized predicted displacement length of the next moment as the numerator and the sum of the normalized predicted displacement length of the next moment and the change in the predicted direction of the next moment as the denominator. The weight of the predicted displacement vector that changes nonlinearly at the next time step is calculated by using the normalized predicted direction change at the next time step as the numerator and the sum of the normalized predicted displacement length at the next time step and the normalized predicted direction change at the next time step as the denominator.

5. The UAV flight system based on visual target tracking according to claim 4, characterized in that, The weight distribution generation module obtains the predicted target trajectory weight distribution through Gaussian superposition based on the linearly changing predicted displacement vector and the non-linearly changing predicted displacement vector, as well as the weights of the linearly changing and non-linearly changing predicted displacement vectors. The specific steps include the following: The predicted displacement vectors that change linearly in the next moment and those that change non-linearly in the next moment are respectively transformed into the two-dimensional image plane of each UAV through the perspective transformation of the camera projection relationship, so as to obtain the corresponding linear pixel displacement vectors and non-linear pixel displacement vectors. Centered on the endpoints of the linear and non-linear pixel displacement vectors, the weights of the predicted displacement vectors that change linearly in the next time step and the weights of the predicted displacement vectors that change non-linearly in the next time step are used as the amplitudes. Gaussian superposition is performed with a preset bandwidth as the width of the Gaussian kernel to obtain the predicted tracking target trajectory weight distribution in the two-dimensional image plane of each UAV. The preset bandwidth is determined based on the linear trend fitting result of the displacement length sequence. Specifically, the displacement length sequence is fitted with a linear trend, and the mean of the absolute value of the residual between the actual displacement length sequence and the fitted linear trend is calculated. This mean value is used as the preset bandwidth.

6. The UAV flight system based on visual target tracking according to claim 5, characterized in that, The tracking quality assessment module is specifically used for: For each drone in a multi-drone cluster, the center coordinates of the target bounding box tracked by the drone in each synchronized image frame within a preset second time window are obtained, and the bounding box center coordinate sequence is constructed in chronological order. For each center coordinate in the bounding box center coordinate sequence, calculate the first distance between that center coordinate and the diagonal intersection point of the corresponding synchronized image frame of the UAV; All first distances are weighted and summed according to a preset weighting rule to obtain the weighted total distance. The preset weighting rule is as follows: the weight of the first distance corresponding to the current time is 1, the weight of the first distance corresponding to the earliest time in the preset second time window is 0, and the weight of the first distance corresponding to other times increases linearly with time. Divide the sum of weighted distances by the sum of weights to obtain the weighted average distance; Divide the weighted average distance by half the diagonal length of the corresponding synchronized image frame of the UAV to obtain the degree to which the center of the normalized bounding box of the UAV deviates from the center of the image. According to the predicted target trajectory weight distribution, the weighted centroid is calculated by weighting each pixel position in the synchronous image frame corresponding to the UAV with its corresponding weight value. Calculate the second distance between the weighted centroid and the diagonal intersection point of the corresponding synchronized image frame of the UAV; Divide the second distance by half the diagonal length of the corresponding synchronized image frame of the UAV to obtain the normalized trajectory coverage index of the UAV.

7. The UAV flight system based on visual target tracking according to claim 6, characterized in that, The pilot drone setting module obtains a score for each drone as a candidate pilot drone based on the degree to which the bounding box center deviates from the image center and the trajectory coverage index. The specific steps include the following: The sum of the degree to which the center of the normalized bounding box of the drone deviates from the center of the image and 1 is used as the first sum value; The sum of the normalized trajectory coverage index of the drone and 1 is used as the second sum. The product of the first sum and the second sum is used as the score for the drone to be considered as a candidate lead drone; the lower the score, the more suitable the drone is as a candidate lead drone.

8. The UAV flight system based on visual target tracking according to claim 7, characterized in that, The pilot drone setting module dynamically selects a pilot drone from a multi-drone cluster based on a score. The specific steps include the following: Obtain the score of the leading drone at the current moment within a preset second time window, and construct a score sequence in chronological order; Calculate the mean and standard deviation of the rating series; For each drone in a multi-drone swarm, the drone's leading position is obtained by subtracting its score as a candidate leader drone from its mean score. The drone's leading position is then divided by the standard deviation to obtain the standardized value of its leading position. If the standardized value of the drone's leading position is greater than or equal to a preset multiple threshold, then the drone is identified as a candidate leading drone for the next moment. If there is no drone with a standardized leading value greater than or equal to a preset multiple threshold, then the current leading drone will remain unchanged. Obtain the switching evaluation value for each candidate lead drone; whereby the switching evaluation value is determined based on the three-dimensional spatial distance and leading degree between the candidate lead drone and the current lead drone; The candidate drone with the highest switching evaluation value will be selected as the drone for the next moment.

9. The UAV flight system based on visual target tracking according to claim 8, characterized in that, The specific steps for obtaining the switching evaluation value for each candidate lead drone are as follows: For each candidate lead drone, calculate the three-dimensional spatial distance between the candidate lead drone and the lead drone at the current moment; Obtain the maximum three-dimensional spatial distance between all candidate lead drones and the lead drone at the current moment, and use it as the maximum spatial distance; Divide the three-dimensional spatial distance between the candidate lead drone and the current lead drone by the maximum spatial distance to obtain the normalized spatial distance of the candidate lead drone. Divide the leading position of the candidate lead drone by the sum of the normalized spatial distance of the candidate lead drone and 1 to obtain the switching evaluation value of the candidate lead drone.

10. The UAV flight system based on visual target tracking according to claim 1, characterized in that, The path planning module plans the flight paths of the UAV swarm within the region, including the following specific steps: Plan the flight path of the lead drone within the area that the lead drone can guarantee tracking coverage; Based on the flight path of the lead drone and the preset tracking array, plan the flight paths of each follower drone.

Images