Multi-target tracking method and system under rapid motion visual angle of unmanned aerial vehicle

Abstract

The present application relates to the technical field of image processing, in particular to a multi-target tracking method and system under the rapid motion visual angle of a UAV, comprising the following steps: acquiring motion attitude data of a UAV and a sequence of continuous video frames taken during flight, combining target detection results of adjacent two frames and corresponding motion attitude data, acquiring spatial motion parameters of the UAV in the corresponding frames, and mapping target positions in a previous frame to a coordinate system corresponding to a current frame according to a unified similarity transformation function model; after unifying the coordinate system, correlating and matching the mapped target in the previous frame with the detection result of the current frame and updating the target trajectory state. The observation visual angle is corrected by estimating the motion of the UAV itself, and the matching problem is placed in a unified stable space. By combining the scale and position changes of the target group in the video, the inter-frame motion parameters of the UAV are inversely estimated, so as to map the targets in the previous and subsequent frames to the same coordinate system, and the accurate determination of the target correlation is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-target tracking method and system under rapid motion visual angle of unmanned aerial vehicle. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] The use of unmanned aerial vehicles (UAVs) for detecting and tracking ground targets such as motor vehicles, pedestrians, etc., has become an important technical means in the fields of traffic management, environmental monitoring, industrial inspection, and emergency rescue. However, during the execution of tasks, the UAVs usually accompany high-speed translation, frequent pitch angle adjustment, and rapid changes in flight height, resulting in significant background changes, target scale dynamic scaling, and position drift in the video sequences collected by the UAVs. Traditional target tracking algorithms are mostly based on the assumption of a relatively static scene or stable camera pose, and are difficult to cope with the problem of unstable feature expression under such dynamic visual angles, i.e., "feature drift". This phenomenon easily causes fluctuations in detection results, target association errors, and trajectory interruptions, thereby significantly reducing the stability and reliability of tracking. SUMMARY

[0004] The present application provides a multi-target tracking method and system under rapid motion visual angle of unmanned aerial vehicle, aiming to solve the problem of decreased tracking continuity and stability caused by dramatic visual angle changes and target size changes during rapid motion of the unmanned aerial vehicle.

[0005] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] The first aspect of the present application provides a multi-target tracking method under rapid motion visual angle of unmanned aerial vehicle, comprising the following steps:

[0007] Obtaining a sequence of continuous video frames taken during the flight of the unmanned aerial vehicle and motion attitude data of the unmanned aerial vehicle;

[0008] Performing target detection on the t-th frame of image to obtain target detection results containing target position and scale features;

[0009] According to the target detection results of the t-th frame and the t-1-th frame and the corresponding motion attitude data, obtaining the spatial motion parameters of the unmanned aerial vehicle in the corresponding frames, the spatial motion parameters at least including a scale change factor, a rotation angle, and a translation vector;

[0010] Establishing a unified similarity transformation function model based on the spatial motion parameters, and mapping the t-1-th frame coordinate system to the t-th frame coordinate system based on the unified similarity transformation function model;

[0011] In the unified coordinate system, the mapped targets of the t-1th frame are associated and matched with the detection results of the tth frame, and the target trajectory state is updated;

[0012] The target detection on the t+1th frame image is continuously performed until all video frames are processed.

[0013] Further, according to the target detection results of the tth frame and the t-1th frame, a scale change factor of the unmanned aerial vehicle between adjacent frames is calculated, specifically: pixel scale features of each target in the t-1th frame and the tth frame are extracted, pixel-scale features of all targets in the two frames are counted based on the known prior physical size of the target, the pixel-physical size conversion relationship of the two frames is calculated respectively, and the scale change factor between the two frames is obtained accordingly.

[0014] Further, when there is a static ground object with a known physical size in the video frame, an additional scale estimation is constructed using the known physical size and the detected pixel size of the static ground object, and the obtained scale change factor is weighted and fused to obtain an updated scale change factor.

[0015] Further, according to the target detection results of the tth frame and the t-1th frame, a rotation angle of the unmanned aerial vehicle between adjacent frames is calculated, specifically: based on the point set composed of all target center points in the tth frame and the t-1th frame, the principal direction vectors corresponding to the point sets of the two frames are calculated respectively through principal direction analysis, and the rotation angle between the frames is calculated according to the included angle between the principal direction vectors.

[0016] Further, according to the target detection results of the tth frame and the t-1th frame, a translation vector of the unmanned aerial vehicle between adjacent frames is calculated, specifically: on the basis of the known scale change factor and rotation angle, the point set of the tth frame is inversely transformed to the scale and direction of the t-1th frame, and the translation vector is obtained by calculating the centroid difference or kernel density cross-correlation of the point sets of the two frames.

[0017] Further, in the unified coordinate system, the mapped targets of the t-1th frame are associated and matched with the detection results of the tth frame, and the target trajectory state is updated; comprising the following steps:

[0018] The position of the existing target trajectory of the t-1th frame is mapped to the tth frame coordinate system by using a unified similarity transformation function model to obtain a predicted position;

[0019] According to the imaging regions of the t-1th frame and the tth frame, an overlapping region visible in both frames, a newly added region visible only in the tth frame, and a lost region visible only in the t-1th frame are calculated and divided, so as to screen out the predicted positions and the current detection positions located in different regions;

[0020] For the predicted position and the current detection position in the overlapping region, the correlation matching of the detection result and the updating of the target track state are realized by constructing a comprehensive cost matrix and solving a global optimal allocation.

[0021] For the current detection position in the new region which is not matched, it is marked as a new candidate track.

[0022] For the predicted position in the loss region which is not matched, it is marked as an exit candidate track.

[0023] Further, the constructed comprehensive cost matrix is preliminarily screened by a distance threshold, and at least includes a geometric distance cost between the predicted position and the current detection position.

[0024] Further, if the predicted position marked as the new candidate track is successfully correlated in the subsequent K consecutive frames, it is updated as the valid track, otherwise the predicted position is cleared.

[0025] Further, if the predicted position marked as the exit candidate track is re-correlated in the subsequent M frames in the overlapping region, it is restored as the active track, otherwise the track is terminated after the M frames.

[0026] The second aspect of the present application provides a multi-target tracking system under the rapid motion perspective of the unmanned aerial vehicle, comprising:

[0027] A data acquisition module configured to acquire a sequence of continuous video frames and motion attitude data of the unmanned aerial vehicle during flight.

[0028] A target detection module configured to perform target detection on the t-th frame of image to obtain a target detection result containing target position and scale features.

[0029] A motion parameter module configured to acquire spatial motion parameters of the unmanned aerial vehicle in the corresponding frame according to the target detection results of the t-th frame and the t-1-th frame and the corresponding motion attitude data, the spatial motion parameters at least including a scale change factor, a rotation angle and a translation vector.

[0030] A coordinate transformation module configured to establish a unified similarity transformation function model based on the spatial motion parameters, and map the t-1-th frame coordinate system to the t-th frame coordinate system based on the unified similarity transformation function model.

[0031] A track updating module configured to perform correlation matching between the mapped target of the t-1-th frame and the detection result of the t-th frame in the unified coordinate system, and update the target track state.

[0032] An iteration loop module configured to continue target detection on the t+1-th frame of image until all video frames are processed.

[0033] Compared with the prior art, the one or more technical solutions have the following beneficial effects:

[0034] 1. By estimating the motion of the unmanned aerial vehicle itself, the observation angle is corrected, so that the matching problem is placed in a unified stable space. At the same time, by using the size and position change of the target group in the video, the inter-frame motion parameters of the unmanned aerial vehicle are inversely estimated, and the targets of the front and rear frames are mapped to the same coordinate system to cope with the unstable feature expression problem caused by the dynamic scene.

[0035] 2. By mapping the front and rear frame targets to the same coordinate system, the interference caused by the motion of the unmanned aerial vehicle is eliminated, the problems of difficult trajectory association and trajectory interruption caused by rapid target movement and dramatic changes in the angle of view are effectively solved, and stable and continuous tracking under multiple flight attitudes such as high-speed translation, pitch and lift is realized.

[0036] 3. By region division and differentiated management strategy, accurate tracking of small targets is realized. In the overlapping area, global matching of fused geometric features is used to improve the association accuracy; in the field of view boundary, false targets generated by image jitter or false detection are filtered through the labeled new candidate trajectories, and the situation that small targets are misjudged as lost due to temporary disappearance is avoided through the labeled exit candidate trajectories, thereby systematically reducing the missed detection and false detection rate of small targets.

[0037] 4. By analyzing the size change of the target and the static object in the video sequence, the motion parameters can be estimated, different models of unmanned aerial vehicles and cameras can be adapted, and tedious pre-calibration work is not required, greatly improving the universality and ease of use of the method.

[0038] 5. Through the labeled new candidate trajectories and exit candidate trajectories, a "new birth confirmation mechanism" and a "boundary tolerance mechanism" are formed, which provides a cautious and fault-tolerant mechanism for target creation and termination, ensures that each stable trajectory has high reliability, and maximizes the integrity of the trajectory. The identification of the final output trajectory is more stable, the life cycle is more accurate, and high-quality data foundation is provided for subsequent behavior analysis, traffic statistics and other high-level applications. BRIEF DESCRIPTION OF DRAWINGS

[0039] The drawings accompanying the specification of this application form a part thereof, serve to provide further understanding of the present application, and together with the description of the exemplary embodiments of the present application and the explanation thereof serve to explain the present application, and do not constitute improper limitations on the present application.

[0040] Figure 1 is a multi-target tracking flowchart provided by one or more embodiments of the present application;

[0041] Figure 2 is a motion estimation schematic diagram provided by one or more embodiments of the present application. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] As described in the background section, during mission execution, UAVs often undergo high-speed translation, drastic pitch angle adjustments, and rapid changes in flight altitude. These movements result in captured video image sequences exhibiting phenomena such as drastic background changes, rapid target scaling, and significant positional drift. Traditional target detection algorithms typically rely on relatively stable scene assumptions, making it difficult to address the instability of feature representation caused by such dynamic scenes, i.e., "feature drift," which leads to a significant decrease in detection reliability.

[0045] Meanwhile, due to the limited flight altitude of drones, targets of interest (such as motor vehicles) in images generally occupy a small pixel area, appearing as "small targets." Existing conventional deep detection networks are typically designed for general scenarios, and their network structures are prone to losing weak details of small targets during downsampling, resulting in insufficient feature extraction capabilities for small targets. Especially during drone flight, the rapid movement of the drone further reduces the detection accuracy of small targets, leading to a high false negative rate.

[0046] Furthermore, existing multi-target tracking algorithms are mostly designed for fixed ground cameras or scenes with gentle movement. These algorithms fail to fully consider the dramatic disturbances in the field of view unique to drones during rapid movement, as well as the frequent disappearance and reappearance of targets due to occlusion or rapid entry / exit from the frame. This makes it difficult for the tracker to correctly identify the target after rapid movement or brief disappearance, resulting in problems such as interrupted target trajectory and frequent identity switching, which cannot guarantee the continuity and stability of target tracking.

[0047] Therefore, this solution presents a multi-target tracking method under the perspective of a rapidly moving UAV. It corrects the observation perspective by estimating the UAV's own motion, placing the matching problem within a unified and stable space. By utilizing the scale and position changes of target objects in the video, the inter-frame motion parameters of the UAV are estimated in reverse, thus mapping targets from consecutive frames to the same coordinate system. Based on this, by dividing the field of view and performing global optimal matching, accurate and robust determination of target relationships is achieved.

[0048] The multi-target detection process of the scheme can be understood as solving the problem of "matching between two frames of targets" in image data. The scheme details the motion estimation algorithm based on target scale change and the target tracking algorithm based on global minimum distance matching when solving the target matching problem.

[0049] In combination Figure 1 The overall detection process of the scheme is introduced, including the following steps:

[0050] Obtain the motion video and motion attitude data of the unmanned aerial vehicle;

[0051] Read the t frame data; for the t frame data, perform small target detection on the cut image to obtain the target detection result of the t frame;

[0052] The motion estimation algorithm based on target scale change is used to calculate the displacement of the unmanned aerial vehicle between the t frame and the t-1 frame;

[0053] A coordinate system is established based on the position of the unmanned aerial vehicle at the t frame, and the coordinate systems of the t-1 frame and the t frame are unified through coordinate mapping, and the targets of the two frames are combined into the same coordinate system;

[0054] The algorithm based on global minimum distance matching is used to complete the matching between the two frames of targets;

[0055] When t=t+1, return to the step of reading the t frame data to form a loop.

[0056] The detailed process of the motion estimation algorithm based on target scale change and the target tracking algorithm based on global distance minimum matching is introduced below.

[0057] The motion estimation algorithm based on target scale change: according to the target change between adjacent two frames during the motion of the unmanned aerial vehicle, the spatial coordinate change of the unmanned aerial vehicle between adjacent two frames is calculated, and the spatial coordinate change of the two frames of targets is converted, as shown in Figure 2 .

[0058] The motion estimation algorithm based on target scale change proposed in the scheme is suitable for the unmanned aerial vehicle vertical view of the traffic road scene, and can estimate the spatial motion parameters of the unmanned aerial vehicle between adjacent frames by statistical analysis of the scale and position change of the moving targets and static objects in the video continuous frames under the condition that the camera internal and external parameters are unknown, thereby establishing a unified coordinate reference system and providing a basis for subsequent target detection and tracking.

[0059] The scheme takes the unmanned aerial vehicle vertical view of the traffic road scene as an example to further introduce the specific multi-target detection process. The moving target in this scene is a vehicle, and the static object is a road.

[0060] The multi-target tracking method under the rapid motion view of the unmanned aerial vehicle includes the following steps:

[0061] Step 1: Object detection and scale extraction;

[0062] Step 2: Object scale statistics and pixel-meter conversion;

[0063] Step 3: Rotation angle estimation;

[0064] Step 4: Translation estimation;

[0065] Step 5: Establish a unified coordinate system;

[0066] Step 6: Based on the global minimum distance matching algorithm, complete the matching between two frames of objects.

[0067] Step 1: Object detection and scale extraction.

[0068] In the t - 1 frame and the t frame image, the traffic scene is detected or segmented to obtain the boundary box or segmentation mask of multiple vehicle targets. The scale feature of each target is calculated.

[0069] For example, take the length of the boundary box as the length of the long side, denoted as ; where the subscripts t - 1, i and t, j represent the corresponding belonging frame, represents the pixel scale.

[0070] Step 2: Object scale statistics and pixel-meter conversion.

[0071] Taking the vehicle as the target, the typical physical length prior of the vehicle (such as 4.8 meters) is used to statistically analyze the scale distribution of the target set. To obtain a representative pixel scale parameter, define and as the representative scale (median or mode) of the target set in the t - 1 frame and the t frame, respectively, calculated as follows:

[0072] ; ;

[0073] Where and i are the pixel lengths of the long side of the t-1 frame and the t frame, respectively, the j th and the th target detection rectangular boundary box.

[0074] - The pixel-meter conversion relationship between the t 1 frame and the t frame is obtained as follows: ; ;

[0075] Then estimate the scale change ratio of the drone's altitude or equivalent field of view: .

[0076] Step 3: Estimate the rotation angle.

[0077] All target center points in frame t are scaled according to the following ratios. After normalization, two point cloud sets are constructed. Principal component analysis is then performed on each point cloud set to extract the first principal component direction vector. , , which serves as the principal direction vector for each frame. The inter-frame rotation angle is estimated as follows:

[0078] ;

[0079] in, , , These represent the main direction vector of frame t-1, the transpose of the main direction vector of frame t-1, and the main direction vector of frame t, respectively.

[0080] Step 4: Translation estimation.

[0081] At a known scale With rotation angle In the case of the first t Inverse transform of frame point cloud to the 1st t-1 The frame scale and orientation are then used, and the translation vector is obtained by employing the point set centroid difference or kernel density cross-correlation method:

[0082] ;

[0083] in, These are the coordinates of the translation vector.

[0084] Step 5: Establish a unified coordinate system.

[0085] The relationship between two adjacent frames can be represented as a similarity transformation:

[0086] ;

[0087] in, It is a two-dimensional rotation matrix. Spatial scale factor, It is a translation vector.

[0088] Through this transformation, it can be t-1 The target coordinates in the frame are mapped to the first frame. t Under the frame reference frame, establish a unified coordinate system.

[0089] Step 6: Based on the global minimum distance matching algorithm, complete the matching between the targets in the two frames.

[0090] The target tracking algorithm based on global distance minimum matching uses the inter-frame similar transformation obtained by the above-mentioned "motion estimation algorithm based on target size change" The last frame target is projected to the current frame reference system, the overlapping region, the new region and the lost region in the current frame field of view are explicitly divided, and the global minimum total distance matching problem is constructed in the overlapping region to complete the ID association; the trajectories in the new and lost regions are processed according to the new and exit strategies respectively, so as to give consideration to the correct judgment of the introduction of new targets and the exit of field of view boundary targets.

[0091] In the target tracking algorithm based on global distance minimum matching, the input and symbol convention are as follows:

[0092] The last frame existing trajectory set: Each trajectory contains position information (including frame center coordinates and four vertex coordinates), size parameters, motion state information (such as velocity quantity and covariance matrix), and appearance feature cache, etc.

[0093] The current frame target detection set: Each detection result includes position parameters, size parameters, confidence information, and optional appearance embedding ;

[0094] Let the imaging regions corresponding to the adjacent two frames be the imaging region of the last frame and the imaging region of the current frame respectively.

[0095] System output: current frame target set with ID and trajectory state update result.

[0096] The algorithm specifically includes the following steps:

[0097] 1) Mapping prediction: through the inter-frame spatial transformation matrix , the last frame trajectory is projected to the current frame reference system. For any trajectory point , the predicted position of the trajectory point in the t th frame is:

[0098] ;

[0099] The predicted covariance matrix is updated at the same time:

[0100] ;

[0101] Wherein, is the predicted covariance matrix of the i th trajectory; if a Kalman filter is used, represents the prediction step function.

[0102] This step ensures consistent alignment between the "historical trajectory" and the current frame coordinate system.

[0103] 2) Region division: Based on the polygonal relationship of the field of view between two adjacent frames, the field of view is divided into three types of regions: overlapping region, newly added region, and lost region.

[0104] Overlapping area: , represents the visible area shared by the two frames, which is the effective working area for performing matching calculations.

[0105] New areas: This indicates a region that has newly entered the field of view in the current frame, and the corresponding detection target is regarded as a new candidate.

[0106] Lost area: This indicates a region that was visible in the previous frame but not in the current frame, and the corresponding trajectory is considered to be excluded from the candidate list.

[0107] Based on the divided regions, construct the corresponding trajectory and detection sets:

[0108] ; ; ;

[0109] ;

[0110] in, This is a set of predicted trajectories within the overlapping region, used for matching with the detection in the current frame;

[0111] This is the set of detection results within the overlapping region, used for matching calculations; This is the set of detection results within the newly added area, corresponding to the newly entered target. This represents the set of predicted trajectories within the lost area, corresponding to targets that may have disappeared from view.

[0112] By filtering the set, the "natural phenomenon" of additions and exits at the boundary is explicitly modeled, avoiding misinterpretation as missed or false detections in global matching.

[0113] 3) Candidate matching and threshold filtering within the overlapping region: Within the overlapping region Ω, predict each trajectory Finding matching detections A threshold screening strategy based on Mahalanobis distance is adopted, with the following screening criteria:

[0114] ;

[0115] in For trajectory i The predicted covariance matrix, γis the threshold value, which can be set according to the chi-square distribution or an empirical threshold value (for example, when the degree of freedom is 2, γ ≈9.21 corresponds to a confidence level of 0.99).

[0116] This step reduces the size of the cost matrix by excluding detection results with large spatial differences, improves matching efficiency, and reduces the risk of mismatch.

[0117] 4) Cost matrix construction in the overlapping region and global optimal matching: in the overlapping region Ω, for all candidate matching pairs (i, j) (where i represents the i-th track in the previous frame, i represents the j-th detection target in the current frame), the comprehensive matching cost is calculated. The comprehensive cost function is defined as: j j

[0118] ;

[0119] where the first term is the position information cost based on weighted Euclidean distance; the second term is the appearance feature similarity cost; and the third term is the target size difference cost. represents the detection target position observation, represents the track prediction position, is the prediction covariance matrix of the i-th track; i are the appearance feature vectors of the track and the detection target i , respectively, j are the bounding boxes corresponding to the track and the detection target, respectively. The scale difference function is defined as:

[0120] ;

[0121] where w and h represent the width and height of the target rectangular box, respectively, and this term is used to constrain the target size change amplitude between adjacent frames. Parameters , , are the weight coefficients of the position term, the appearance term, and the scale term (all are non-negative real numbers), respectively, which are used to adjust the relative importance of each part of the cost in the comprehensive matching.

[0122] After calculating the cost of all candidate matching pairs (i, j) , the cost matrix is constructed. In the overlapping part Ω of the target detection region, the set of predicted tracks in the previous frame is defined as , the set of detection results in the current frame is , and the matching process is performed in​​​ With between

[0123] Let the number of elements of n= | | and m= | | be , the matching relationship matrix is established, and the optimization goal of matching is to minimize the total cost function:

[0124] ;

[0125] wherein =1 indicates that the trajectory i successfully matches the detection target j , =0 indicates that it is not matched.

[0126] The constraint condition is: , ; to ensure that each trajectory and detection target is matched only once in the current frame.

[0127] The optimization can be solved by the Hungarian algorithm to obtain the globally optimal matching result. When a trajectory or a detection target has no feasible candidate in the current frame, the unmatched state can be maintained and re-associated in the subsequent frame. Through this globally optimal matching strategy, the local misalignment and identity drift problem can be effectively reduced, and the stability and overall matching accuracy of target tracking can be improved.

[0128] 5) Processing strategy for new area and lost area: for the targets appearing in the new area and the lost area, the system respectively adopts two strategies of new trajectory initialization and exiting trajectory management to maintain the dynamic consistency of the trajectory set.

[0129] (a) New area :

[0130] For the detection set in the new area , if the detection target is not matched with the trajectory of the last frame, it is initialized as a new candidate trajectory. The system sets a confirmation counter for the candidate trajectory, which represents the number of consecutive frames required to confirm the new trajectory. When the candidate trajectory is detected in consecutive K frames and the appearance feature remains consistent, it is determined to be a valid trajectory, otherwise it is automatically removed. This mechanism can suppress boundary flickering and reduce false detection.

[0131] (b) Lost area :

[0132] The lost area The system marks it as a candidate for exiting and includes it in the boundary keeping queue where represents the number of tolerated frames allowed after the trajectory exits. Within this time window, if the trajectory re-enters the field of view or appears in the overlap region Ω again, it is restored to the active state; if it exceeds , it is determined to be permanently lost. This design can effectively deal with short-term occlusion or edge jitter of the field of view.

[0133] (c) Unmatched processing in the overlap region:

[0134] For trajectories that are unmatched in the overlap region Ω, the system maintains a mismatch count where represents the number of consecutive unmatched frames. When the number of consecutive unmatched frames reaches the threshold, the trajectory is determined to be terminated. For unmatched detection results, they re-enter the confirmation process as candidate new-born trajectories. This processing method can reduce false deletions and improve tracking stability.

[0135] 6) State update (for matched targets): After completing the global matching in the overlap region, the state of the successfully matched target pair is updated to achieve dynamic maintenance and continuous recognition of the trajectory parameters.

[0136] Motion state update: Using the detection result , the predicted trajectory is corrected by filtering, and Kalman filtering or α-β filtering strategy is used to update the position, velocity, and covariance matrix of the target. This update process can achieve optimal estimation of the trajectory state and noise suppression.

[0137] Appearance update (optional): For the appearance description vector of the trajectory, the exponential moving average strategy (EMA) is used to fuse with the appearance features of the current detection target, and the update formula is:

[0138] ;

[0139] where, represents the appearance cache feature of the trajectory, represents the appearance feature of the detection target, and EMA operation is a weighted smoothing update, which is used to balance historical information and new observation features, thereby improving the robustness and temporal stability of the appearance representation.

[0140] Survival state management: maintain miss count and confirmation status for each matched track. If successfully matched in consecutive frames, it remains in "confirmed / active" state; if not matched in consecutive frames, it enters candidate lost state and terminates the track when exceeding threshold

[0141] Output and loop: after update, the system outputs the ID target state table under the current frame, including position, size, speed, covariance and appearance feature information. Then update the frame index to t <- t + 1, read the next frame image and repeat steps 1) to 6) to realize the whole process loop.

[0142] The scheme aims at the problems of unstable matching, track interruption and ID mismatch in multi-target detection and tracking under dynamic view of UAV, and proposes a group target matching method based on motion estimation and spatial coordinate unification. By establishing a UAV motion compensation and spatial transformation model between frames, the coordinates of all targets in the previous frame are mapped to the unified spatial reference system of the current frame, which fundamentally eliminates the projection drift caused by camera motion, attitude change or view angle offset. This method makes target matching no longer rely on image plane overlap relationship, but based on geometric space consistency, which significantly improves the matching stability and track continuity under high dynamic view.

[0143] Compared with traditional algorithms that only rely on IOU or nearest neighbor matching, the scheme estimates the camera motion between frames and completes coordinate alignment before matching, so that the same target maintains geometric consistency in consecutive frames, thereby avoiding visual mismatch caused by high-speed translation, pitch or steering of UAV. Traditional methods are prone to problems such as IOU close to zero, matching failure or track interruption when the field of view overlap decreases; while the scheme effectively offsets the target offset caused by camera pose change through spatial coordinate unification and motion compensation mechanism, so it can still maintain stable target association in complex background, dynamic lighting and non-planar scene. This improvement improves the robustness of the algorithm in high dynamic environment in principle, significantly reduces the risk of missed detection and false matching.

[0144] ​The scheme adopts a global cost matrix construction and a minimum total distance matching method, and realizes the overall optimal matching of the group level through the Hungarian algorithm. Compared with the traditional local matching strategy, this method can consider the global consistency and spatial constraint relationship between group targets at the same time, thereby effectively reducing the matching drift and ID confusion caused by the accumulation of local misfit. Combined with the region division mechanism, the scene under the change of field of view is divided into three categories: overlapping area, new area and lost area. The newly added target is verified through continuous frame confirmation and appearance consistency criterion, and the lost target is managed through boundary maintenance and exit control, realizing the orderly control of trajectory generation, maintenance and disappearance. This mechanism can still maintain continuous tracking under complex conditions such as short-time occlusion and partial target out-of-view, reducing the fragmentation of trajectories.

[0145] Unlike traditional methods that rely on camera extrinsic calibration or ground reference points, this scheme does not require external calibration and can realize spatial coordinate unification based only on target scale change estimation. This scheme has a simple structure, high computational efficiency, good versatility and portability, and can adapt to different models of unmanned aerial vehicles and multiple types of camera systems. It can maintain stable performance at various flight altitudes (30-120 meters) and various motion modes (horizontal flight, climbing, turning, etc.), and has excellent adaptive ability.

[0146] In terms of system implementation, the main computational load of this scheme is concentrated in the detection network and the Hungarian matching algorithm stage, with low overall computational complexity and high running efficiency. It can run in real time on an unmanned aerial vehicle edge computing platform (such as Jetson Xavier) or a ground workstation. The overall processing speed of the system can meet the application requirements of real-time video analysis and multi-target tracking tasks, and is widely applicable to traffic monitoring, industrial inspection, security and control, emergency search and rescue, sea target identification and environmental monitoring, etc. in various dynamic scenes.

[0147] In summary, this scheme introduces motion compensation and spatial coordinate unification mechanism, global matching optimization strategy and trajectory life cycle management method, which fundamentally solves the problems of target drift, unstable matching and trajectory interruption under the dynamic view of unmanned aerial vehicles. Compared with existing technologies, this scheme has a simpler structure, more efficient algorithm and wider applicability, not only improving the accuracy and stability of group target tracking, but also reducing the implementation complexity and system deployment cost. It has significant engineering application value and broad prospects for popularization.

[0148] In this embodiment, a multi-rotor unmanned aerial vehicle (UAV) equipped with a 4K resolution camera (e.g., DJI Mavic 2Pro model) is used as the data acquisition platform. The UAV collects video data at a flight height of about 80 meters, a horizontal flight speed of about 10 m / s, and a video acquisition frame rate of 25 frames per second. The shooting scene includes typical ground traffic elements such as motor vehicles, lane lines, and zebra crossings. The vehicles in the road are the main detection and tracking targets. The UAV camera has not undergone special internal and external parameter calibration. Under this condition, the system can still achieve stable image detection and target tracking, indicating that the scheme has strong robustness and environmental adaptability.

[0149] The computing device uses an NVIDIA GTX1660 graphics processing unit, equipped with 16GB of memory, and runs the Ubuntu20.04 operating system. The software environment includes Python 3.8, OpenCV 4.6, and the PyTorch 1.12 deep learning framework. In this embodiment, the target detection module uses the YOLOv5s model, and the input image size is 640x640, with a confidence threshold of 0.4.

[0150] After completing the image (video frame) acquisition, the system processing flow is as follows.

[0151] Step S1: Motion estimation and coordinate unification.

[0152] 1. In the t-1th frame and the tth frame images, respectively run target detection to obtain the vehicle detection box set:

[0153] ; ; ;

[0154] wherein, is the target detection set of the t-1th frame, is the target detection set of the tth frame, B is the target detection box rectangular frame, are the left side and length-width of the rectangular frame, respectively.

[0155] 2. Calculate the main shaft length of each vehicle , and take the histogram mode of all vehicle lengths , combined with the average physical length of the vehicle =4.8m, to obtain the pixel-meter conversion relationship: ;

[0156] and get the scale change ratio ;

[0157] wherein, Pixel-meter conversion relationship for t-1 frame, Pixel-meter conversion relationship for t frame.

[0158] 3. Perform principal component analysis on the target point cloud respectively to obtain the principal direction vectors of the two frames of point clouds , and calculate the rotation angle as follows:

[0159] ;

[0160] wherein, , are the principal direction vectors of the t-1 frame, the t frame, and the transpose of the t-1 frame principal direction vector, respectively.

[0161] 4. Inverse scale and inverse rotate the point cloud of the t frame to the t-1 coordinate system of the t-1 frame, and estimate the translation vector using the centroid difference method and the phase correlation method t = (Δx, Δy) .

[0162] 5. Establish a unified similarity transformation between the two frames:

[0163] ;

[0164] wherein, is a two-dimensional rotation matrix, is a spatial scale factor, is a translation vector.

[0165] Through the transformation, the target coordinates in the t-1 frame are mapped to the t frame reference system to establish a unified coordinate system.

[0166] Step S2: Region division.

[0167] Based on the field of view polygons of the two frames, divide the overlapping region, the new region and the lost region, and select the detection trajectories belonging to each region.

[0168] Step S3: Global matching in overlapping domain.

[0169] Predict and detect in Ω, and calculate the cost matrix:

[0170] ;

[0171] Use the Hungarian algorithm to solve the minimum total cost matching to obtain a one-to-one correspondence ;

[0172] Unmatched detection is a new candidate; and unmatched trajectory is marked as lost once.

[0173] Step S4: Add and loss processing.

[0174] For new track, need hit =3 frame after confirmation. For track in =5 frame, if re-enter, resume, otherwise terminate.

[0175] Step S5: State update and output.

[0176] For matched track, update motion state (Kalman filter) and appearance feature cache;

[0177] Output target detection result with ID .

[0178] The above only is the preferred embodiment of the present application, and is not used for limiting the present application, for the person skilled in the art, the present application can have various changes and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-target tracking method under the rapid motion visual angle of a UAV, characterized in that, The method comprises the following steps: acquiring a sequence of continuous video frames taken during the flight of the unmanned aerial vehicle and motion attitude data of the unmanned aerial vehicle; performing target detection on a t-th frame of image in the sequence of continuous video frames to obtain a target detection result containing target position and scale features; acquiring spatial motion parameters of the unmanned aerial vehicle in corresponding frames according to the target detection results of the t-th frame and a (t-1) -th frame and corresponding motion attitude data, the spatial motion parameters at least including a scale change factor, a rotation angle and a translation vector; establishing a unified similarity transformation function model based on the spatial motion parameters, and mapping a coordinate system of the (t-1) -th frame to a coordinate system of the t-th frame based on the unified similarity transformation function model; in the unified coordinate system, performing correlation matching on the mapped target of the (t-1) -th frame and the detection result of the t-th frame, and updating a target trajectory state; the method comprises the following steps: using the unified similarity transformation function model to map the position of an existing target trajectory of the (t-1) -th frame to the coordinate system of the t-th frame to obtain a predicted position; calculating and dividing an overlapping region commonly visible in the two frames, a newly added region visible only in the t-th frame and a lost region visible only in the (t-1) -th frame according to the imaging regions of the (t-1) -th frame and the t-th frame, so as to screen out the predicted position and the current detection position located in different regions; for the predicted position and the current detection position in the overlapping region, correlation matching of the detection result and updating of the target trajectory state are realized by constructing a comprehensive cost matrix and solving a global optimal allocation; for the current detection position in the newly added region which is not matched, the current detection position is marked as a new candidate trajectory; for the predicted position in the lost region which is not matched, the predicted position is marked as an exit candidate trajectory; continuing target detection on a (t+1) -th frame of image until all video frames are processed. 2.The method of claim 1, wherein, According to the target detection results of the t-th frame and the (t-1) -th frame, a scale change factor of the unmanned aerial vehicle between adjacent frames is calculated, specifically: pixel scale features of each target in the (t-1) -th frame and the t-th frame are extracted, pixel-scale features of all targets in the two frames are counted based on the known prior physical size of the target, pixel-physical size conversion relationships of the two frames are calculated respectively, and the scale change factor between the two frames is obtained accordingly. 3.The method of claim 2, wherein, When there is a static ground object with a known physical size in the video frame, an additional scale estimation is constructed by using the known physical size and the detected pixel size of the static ground object, and the obtained scale change factor is weighted and fused to obtain an updated scale change factor. 4.The method of claim 1, wherein, According to the target detection results of the t-th frame and the (t-1) -th frame, a rotation angle of the unmanned aerial vehicle between adjacent frames is calculated, specifically: based on a point set composed of all target center points in the t-th frame and the (t-1) -th frame, main direction vectors corresponding to the two frame point sets are calculated respectively through main direction analysis, and the frame interval rotation angle is calculated according to the included angle between the main direction vectors. 5.The method of claim 1, wherein, According to the target detection results of the t-th frame and the (t-1) -th frame, a translation vector of the unmanned aerial vehicle between adjacent frames is calculated, specifically: on the basis of the known scale change factor and rotation angle, the point set of the t-th frame is inversely transformed to the scale and direction of the (t-1) -th frame, and the translation vector is obtained by calculating the centroid difference or kernel density cross-correlation of the point sets of the two frames. 6.The method of claim 1, wherein, The constructed comprehensive cost matrix is pre-screened by a distance threshold, and at least includes a geometric distance cost between the predicted position and the current detected position.

7. The method of claim 1, wherein, The predicted position marked as "newborn candidate track" is updated as "valid track" if successfully associated in the subsequent K consecutive frames, otherwise the predicted position is cleared. 8.The method of claim 1, wherein, The predicted position marked as "exit candidate track" is restored as "active track" if re-associated in the overlapping area in the subsequent M frames, otherwise the track is terminated after M frames.

9. A multi-target detection system under the quick motion perspective of a UAV, characterized in that, The multi-target tracking method under rapid motion perspective of the unmanned aerial vehicle according to any one of claims 1-8 comprises: a data acquisition module configured to acquire a sequence of continuous video frames and motion attitude data of the unmanned aerial vehicle during flight; a target detection module configured to perform target detection on a t-th frame image in the sequence of continuous video frames to obtain a target detection result containing target position and scale features; a motion parameter module configured to obtain spatial motion parameters of the unmanned aerial vehicle in the corresponding frame according to the target detection results of the t-th frame and a (t-1) -th frame and the corresponding motion attitude data, the spatial motion parameters at least including a scale change factor, a rotation angle and a translation vector; a coordinate transformation module configured to establish a unified similarity transformation function model based on the spatial motion parameters, and map the (t-1) -th frame coordinate system to the t-th frame coordinate system based on the unified similarity transformation function model; a track updating module configured to associate and match the mapped target of the (t-1) -th frame with the detection result of the t-th frame in the unified coordinate system, and update the target track state; an iterative loop module configured to continue target detection on a (t+1) -th frame image until all video frames are processed.

Images