A multi-target tracking method and system for detecting loss and association failure

Abstract

The application provides a multi-target tracking method for detecting loss and association failure under a UAV platform, comprising: pre-processing real-time video traffic data; inputting the pre-processing result to a joint network to obtain a detection result; on the basis of the detection result, combining a track generated by multi-target tracking to generate a candidate target; and optimizing the candidate target to obtain a predicted lost detection target. The application can meet the demand of real-time online multi-target tracking task under a low-altitude UAV platform, can overcome the defects of reduced tracking performance caused by target occlusion, detection loss and UAV platform maneuvering in the prior art, has strong robustness, applicability and real-time performance, and enhances the visual perception ability of the UAV platform to the surrounding environment, and has wide application prospects in civil and military fields.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of object detection and tracking in computer vision, in particular, to a multi-target tracking method and system for detection loss and association failure. BACKGROUND

[0002] Multi-target tracking under the unmanned aerial vehicle platform can not only obtain the spatial position of the target, but also obtain the motion trajectory under the continuous time sequence, which assists in realizing the perception of the surrounding environment, and has been widely applied in military and civilian fields. In the military field, it can be used for battlefield reconnaissance, close-range air support escort; in the civilian field, it can be used for rapid distribution, pedestrian monitoring and disaster rescue tasks. Therefore, it has important research value and application value to study the multi-target tracking method under the unmanned aerial vehicle platform.

[0003] With the increasing demand for real-time performance, stability and full-process automation of target tracking methods in actual application scenarios, online tracking based on detection has gradually become the mainstream direction of related research. Generally, this framework includes target detection, association matching, trajectory updating and other links. First, the position of the target in the image is obtained through the target detection method, then the detected target and the trajectory are associated, and finally the motion trajectory of the target is generated, and the motion trajectory list of the multi-target can be dynamically updated and managed during the processing process.

[0004] Since the input of the trajectory association link depends on stable and reliable target detection results, the overall effect of the multi-target tracking method depends largely on the performance of the target detection method. In the face of complex scenes such as target occlusion that can easily lead to detection loss, effective step and correction methods need to be studied. In addition, due to the high mobility and flexibility of the unmanned aerial vehicle platform, the influence caused by the platform's own motion needs to be considered before multi-target tracking, and targeted strategies need to be adopted for the motion platform background such as unmanned aerial vehicles. SUMMARY

[0005] In view of the defects in the prior art, the purpose of the present application is to provide a multi-target tracking method and system for detection loss and association failure.

[0006] According to one aspect of the present application, a multi-target tracking method under an unmanned aerial vehicle platform for detection loss and association failure is provided, comprising:

[0007] Pretreating real-time video flow data;

[0008] Inputting the pretreatment result to a joint network to obtain a detection result;

[0009] On the basis of the detection result, candidate targets are generated in combination with the trajectories generated by the multi-target tracking;

[0010] Optimizing the candidate target to obtain a predicted missing detection target.

[0011] Preferably, the preprocessing comprises:

[0012] Extracting and saving images frame by frame from the original video stream data;

[0013] Applying image smoothing methods such as median filtering, Gaussian filtering or mean filtering to each frame of image to suppress noise;

[0014] Randomly cropping or center cropping;

[0015] Obtaining the preprocessing results of each frame of image.

[0016] The joint network takes the preprocessing results of each frame of image as input, uses a deep learning convolutional neural network as the basic network type, constructs a deep aggregation module and a high-resolution module to realize joint training of multi-target detection and feature extraction, and obtains the detection results and feature vectors of the target from the two output branches of the network respectively.

[0017] Preferably, the high-level semantic feature space of multiple targets in the image is divided into n preset subspaces, each corresponding to a target semantic category with a different ID.

[0018] Preferably, on the basis of the detection results, candidate targets are generated in combination with the trajectories generated by multi-target tracking.

[0019] Preferably, the optimization processing of the candidate target to obtain the missing detection target comprises:

[0020] Discriminating the confidence of the candidate target frame and filtering trajectories whose confidence is lower than a set confidence threshold;

[0021] Correcting the position of the candidate target frame.

[0022] Preferably, the position correction to obtain the target frame of the missing target comprises:

[0023] According to the centrality of the target detection branch, the position of the candidate target frame generated by trajectory prediction is corrected, and the center of the target is moved to the point with the maximum centrality.

[0024] Preferably, it further comprises:

[0025] Taking the association cost matrix of the trajectory and the candidate target as the basis for judging the motion state of the unmanned aerial vehicle platform;

[0026] Taking the spatial transformation matrix of adjacent two frames of image data as input, outputting the motion of the unmanned aerial vehicle platform whether a large displacement occurs;

[0027] On the basis of the camera motion discrimination model predicting that the camera has large motion, image registration technology is used to correct the camera motion, and the influence caused by the camera motion itself is eliminated.

[0028] Preferably, the lost detection target is obtained based on the above method, the trajectory consistency degree is taken as the reference, the target feature vector, the spatial position information and the overlapping degree of the target frame and the trajectory are comprehensively considered to obtain the final correlation cost matrix, and then the correlation matching of the trajectory and the candidate target is realized, and the trajectory list is managed and updated in real time.

[0029] According to a second aspect of the present application, a multi-target tracking system for detecting loss and correlation failure under a UAV platform is provided, comprising:

[0030] An image preprocessing module, which obtains real-time video stream data from the visual system of the UAV platform and saves, performs image smoothing and region clipping on each frame of image data respectively, and obtains the preprocessing result of each frame of image data;

[0031] A joint network module, which takes the preprocessing result of each frame of image data as input, takes a deep learning convolutional neural network as a basic network type, and obtains the detection result and the feature vector of the target from two output branches of the network respectively;

[0032] A target region proposal module, which uses the trajectory generated by the multi-target tracking to generate candidate targets on the basis of the joint network detection result, discriminates the confidence of the candidate target frame generated by the trajectory prediction, filters the trajectory output below the set confidence threshold, and corrects the position of the candidate target frame;

[0033] A camera motion discrimination and registration module, which takes the correlation cost matrix of the trajectory and the candidate target as the basis for discriminating the motion state of the UAV platform, takes the spatial transformation matrix of adjacent two frames of image data as input, and outputs whether the UAV platform has large displacement motion; on the basis of the camera motion discrimination model predicting that the camera has large motion, image registration technology is used to correct the camera motion, and the influence caused by the camera motion itself is eliminated;

[0034] A trajectory correlation module, which takes the trajectory consistency degree as the reference, calculates the correlation cost matrix of the target feature vector, the spatial position information and the overlapping degree of the target frame and the trajectory, realizes the correlation matching of the trajectory and the candidate target, and manages and updates the trajectory list in real time.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] The multi-target tracking method and system for target detection loss under the unmanned aerial vehicle platform in the embodiment of the application can meet the needs of real-time online multi-target tracking tasks under the low-altitude unmanned aerial vehicle platform, can overcome the defects of reduced tracking performance caused by target occlusion, detection loss and unmanned aerial vehicle platform maneuvering of existing methods, has strong robustness, applicability and real-time performance, and enhances the visual perception ability of the unmanned aerial vehicle platform to the surrounding environment;

[0037] The joint network module in the embodiment of the application fully utilizes the advantages of multi-task learning, can efficiently extract high-level semantic features of different ID targets in the multi-target tracking task, and improves the detector performance;

[0038] The target region proposal module in the embodiment of the application reduces the excessive dependence of the target tracking task on the detector performance, corrects the position of the candidate target frame generated by the trajectory prediction, and introduces the trajectory confidence to improve the reliability of the multi-target tracking result;

[0039] The camera motion discrimination and registration module in the embodiment of the application uses image registration technology to eliminate the influence of the motion of the unmanned aerial vehicle platform itself on the multi-target tracking task, increases the overlapping degree of the same ID target frame in adjacent two frames of images, and reduces the situation of association failure. BRIEF DESCRIPTION OF DRAWINGS

[0040] Other features, objects and advantages of the application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings:

[0041] Figure 1 is a multi-target tracking task scene schematic diagram of the unmanned aerial vehicle platform in an embodiment of the application;

[0042] Figure 2 is a multi-target tracking task implementation flowchart of the unmanned aerial vehicle in an embodiment of the application;

[0043] Figure 3 is a target detection and feature extraction joint network structure schematic diagram in an embodiment of the application;

[0044] Figure 4 is a predicted target position correction schematic diagram of the target region proposal module in an embodiment of the application;

[0045] Figure 5 is a method flowchart of the camera motion discrimination and registration module in an embodiment of the application;

[0046] Figure 6 is an implementation flowchart of the camera motion discrimination model in an embodiment of the application;

[0047] Figure 7is a schematic diagram of a registration result of a camera motion discrimination and registration model in an embodiment of the present application;

[0048] Figure 8 is an evaluation result of an IoU distribution of adjacent two frames of the same ID target frame in an embodiment of the present application;

[0049] Figure 9 is a video sequence tracking result in a low-altitude unmanned aerial vehicle scene in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The present application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made. These all belong to the protection scope of the present application.

[0051] Referring to Figure 1 , it is a low-altitude four-axle unmanned aerial vehicle platform application scene schematic diagram of an embodiment of the present application, which includes a tracked target 1, an unmanned aerial vehicle device 2, a visual system sensor 3 and a computer 4. Among them, the visual system sensor 3 is placed at the center of the unmanned aerial vehicle device 2 and is fixed; the visual system sensor 3 shoots a continuous image sequence containing the tracked target 1, and transmits the recorded video stream information to the computer 4; the computer 4 covers an image preprocessing module, a joint network module, a target region proposal module, a camera motion discrimination and registration module and a trajectory association module, thereby realizing multi-target tracking under the unmanned aerial vehicle platform.

[0052] As Figure 2 shown, it is a flow chart of a multi-target tracking method for detecting loss and association failure under an unmanned aerial vehicle platform in a preferred embodiment of the present application, which includes:

[0053] S100 image preprocessing module: pre-processing real-time video traffic data;

[0054] S200 joint network module: input the pre-processing result to the joint network to obtain a detection result containing target position, category information and target feature vector;

[0055] S300 target region proposal module: on the basis of the detection result, candidate targets are generated in combination with the trajectories generated by multi-target tracking;

[0056] S400 camera motion discrimination and registration module: the candidate targets are optimized to obtain a candidate target set with high position precision as the lost detection target;

[0057] The S500 trajectory association module combines the obtained lost detection targets with the trajectories generated by multi-target tracking to obtain the tracking results.

[0058] In a preferred embodiment, S100 includes: performing image preprocessing operations such as image smoothing and region cropping on the original image sequence frame by frame; using image smoothing algorithms such as Gaussian filtering, median filtering, or mean filtering to suppress and smooth noise components in the original image; and adapting the image size to the preset input size of the joint network model.

[0059] In a preferred embodiment, the joint network module in S200 uses a convolutional neural network as the basic network type. It achieves the fusion of feature maps at different scales through a deep aggregation module and a high-resolution module. The deep aggregation module performs multiple fusions and calculations on the feature maps, resulting in a network with higher classification accuracy and fewer parameters. The high-resolution module fuses shallow features with deep features one after another, suppressing the impact of increased downsampling times on the detection performance of small targets.

[0060] Specifically: The feature fusion calculation method used in the deep aggregation module is as follows:

[0061]

[0062] In the formula, σ is the activation function used, such as ReLU, GeLU, etc., Norm represents the normalization method used, such as BN, LN, etc., and W i and b i The parameter x represents the convolution kernel. i The features are the input.

[0063] The fusion method used in the high-resolution module is as follows:

[0064] x out =Conv 3x3 (TConv(Conv 1x1 (x i+j ))+x i )

[0065] In the formula, represents a convolution operation with a 1x1 kernel, represents a deconvolution operation, represents a convolution operation with a 3x3 kernel, and represents the input features.

[0066] The output of the joint network includes two branches: multi-object detection and feature extraction, such as... Figure 3As shown, respectively used for predicting the position category of the target in the image and the corresponding extracted high-level semantic feature. The position category is an intuitive result of the target detection in the image, representing the position of the target distribution in the image and the category of the target itself; from the subsequent tracking work, the position information is crucial for the motion feature acquisition of the image sequence, and it is generally believed that the spatial position change of the same target in adjacent two images will not be large, which is also an important basis for associating the targets in adjacent two images. The distribution of the feature vector output by the feature extraction branch in the feature space represents n different subspaces, i.e., different semantic categories, so that the feature extraction work of the network can not only distinguish different categories of targets, such as car and people, but also distinguish different targets in the same category as much as possible, such as car1 and car2. The n subspaces correspond to n different ID semantic categories.

[0067] In the embodiment, first, the joint network weight needs to be pre-trained. For the training of the multi-target detection branch, cross entropy, focal loss and GIoU are used as the loss functions for predicting the target center degree, category and spatial position in the image, respectively; for the training of the feature extraction branch, the metric learning method is used, the distance between the features with the same label in the given n IDs is reduced through network training, and the distance between the features with different labels is increased; in addition, the joint training of multi-target detection and feature extraction belongs to the category of multi-task learning, and the generality of feature extraction can be improved through the shared information between complementary tasks. In the construction of the joint training loss function, the adaptive learning strategy of the loss function weight is adopted, and the calculation process is as follows:

[0068]

[0069] In the formula, L detection represents the detection branch loss function, L feature represents the feature extraction branch loss function, W, σ1, σ2 represent the parameters to be optimized.

[0070] For the target detection result obtained in S200, the target detection result may be lost due to target occlusion in the image and the like, and the present application provides a preferred embodiment. The existing method only uses the prediction result of the detection network as the candidate target, resulting in that the existing method has a strong dependence on the target detection result. In this embodiment, target trajectory estimation is introduced, and the candidate target set is not only composed of the detection result by Kalman filtering prediction of the target trajectory. Its role is that the detection network part can be compensated by the candidate target generated by trajectory prediction in the case of detection loss caused by confidence lower than the threshold value (or occlusion and the like), reducing the dependence on the target detection result.

[0071] It is assumed that at t-1 moment, there is a trajectory The position of the target at time t can be predicted by, but not limited to, a Kalman filtering method, and is denoted as The target obtained by the target detector at time t is denoted as D det = {D1,..., D m}, wherein (x0, y0) and (x1, y1) are the coordinates of the upper left corner and the lower right corner of the target frame, respectively, and c is the target confidence; and the final candidate target set is D = D track ∪ D det The formula for obtaining the candidate target set is composed of the output results of the target detection network and the trajectory prediction results.

[0072] In order to obtain further more accurate candidate target frames, the application provides a preferred embodiment. In this embodiment, the candidate target frame is optimized in two steps.

[0073] In the first step, the target region proposal module in this embodiment introduces the confidence of the trajectory, and the confidence of the trajectory is updated in real time through the association result of the trajectory and the target. When the target is missed, the lost target can be re-tracked through the candidate target generated by the trajectory. However, in the case of long-time loss of the target, the candidate target predicted by the trajectory is not accurate, and therefore the confidence of the trajectory is decayed for the trajectory that has not been updated for a long time. In addition, low accuracy of the detector can cause the trajectory to be associated with a target with low confidence for a long time, and the associated target is often not a real target, and therefore the confidence of the trajectory also needs to be decayed. In summary, the confidence calculation formula of the trajectory is as follows:

[0074]

[0075] In the formula, is the confidence of the trajectory at time t, is the confidence of the trajectory at time t-1, s d is the confidence of the candidate target associated with the trajectory, d represents the candidate target associated with the trajectory, d ∈ D det ∪ D track , and a ∈ (0, 1) is a decay coefficient of the confidence of the trajectory. Then, according to a pre-set confidence threshold, the trajectories with low confidence are filtered out from the trajectory list.

[0076] It is worth mentioning that the calculation of the track confidence here is divided into two cases according to whether the candidate target is derived from the detection result or the track prediction, and alpha in the formula is a value less than 1, and 0.9 is used in the specific implementation. If the candidate target associated with the track comes from the track prediction result, the track confidence at time t will be alpha times the track confidence at time t-1, that is, the confidence decay occurs. If the candidate target comes from the detection result, the track confidence will be the larger one of the two calculated values, which is definitely greater than alpha times the confidence at time t-1. It can also be shown that for the source of the candidate target, the detection result is more inclined to be believed in the case of detection (detection is not lost); the candidate target generated by track prediction is a remedial measure in the case of detection loss.

[0077] Secondly, in view of the high uncertainty of the candidate target frame positioning based on track prediction in S300, the target region proposal module uses the centrality output of the target detection branch to correct the track prediction position. As shown in Figure 4 , it is assumed that the target scale does not change significantly in a short time, only the spatial position of the target changes, so the search is performed in the target prediction frame, and the target center is moved to the local maximum centrality point to prevent the target from drifting.

[0078] Since the unmanned aerial vehicle platform has strong maneuverability and flexibility, the camera motion itself is easy to bring significant position deviation to the image data, based on which the application provides a preferred embodiment for S400 camera motion discrimination and registration. The method flow chart of the camera motion discrimination and registration module used in this embodiment is shown in Figure 5 . Specifically,

[0079] Firstly, the association cost matrix of the track and the candidate target is used as the basis for discriminating the motion state of the unmanned aerial vehicle platform. When the unmanned aerial vehicle has no large motion, the minimum association cost corresponding to the track in the association matrix is small, and when the unmanned aerial vehicle has large motion, the association cost of the track and the candidate target will greatly increase or even cannot be associated;

[0080] Then, whether the camera has large motion is judged according to the association cost matrix of the current frame image data and the image data at the last time. The camera motion discrimination model is modeled as a binary classification problem, as shown in Figure 6 , the association cost matrix M of the track at time t-1 and the candidate target {D1,…,D m} at time t is calculated, M∈R nxm , considering that the dimensions of different times n and m may not be consistent, the association cost matrix is expanded to obtain M'∈R NxNThen, the maximum values of Softmax are taken in row and column respectively, the obtained vectors are spliced, and finally whether there is large motion in the adjacent two frames is predicted through a support vector machine or other binary classification model.

[0081] Finally, when it is judged that the UAV platform has large motion, the image registration technology is combined with the multi-target tracking task in the embodiment, the influence of the camera motion on the track association is eliminated by using the image registration technology; first, the point features and feature descriptors of the image are obtained by using the FAST, SIFT, SURF and other feature point detection methods, then the optimal image transformation matrix H is iteratively calculated and found by using the random sample consensus (RANSAC) method, and finally the candidate target position at t time is corrected according to the following formula:

[0082] x t =H t x t-1 +Δx t

[0083] x t 、x t-1 represent the positions of the target in the image plane at t and t-1 time, H t represents the image transformation matrix obtained by registration, and Δx t represents the motion of the target itself. Figure 7 is the registration result of the adjacent two frames of images in the UAV video sequence in a common scene, wherein Figure 7 (a) and Figure 7 (b) represent the current frame image and the image at the last time respectively, Figure 7 (c) represents the result image obtained after image registration. Figure 8 is the evaluation result of the IoU distribution of the adjacent two frames of the same ID target box, and it can be seen from the figure that after the influence of the camera motion is eliminated by using the image registration technology, the IoU distribution of the adjacent two frames of the same ID target box is right-shifted, that is, the association failure can be reduced.

[0084] Based on the above embodiment, the accurate missing target box is obtained, and after the candidate target box position is corrected by the camera motion discrimination and registration module, the application provides a preferred embodiment for track association. The track association module comprehensively considers the state information, position information and appearance information of the track and the candidate target box to obtain the final association cost matrix; wherein the Mahalanobis distance is used to calculate the association cost of the state information and the position information of the target box; the IoU is used to calculate the overlap degree of the target box and the track, and then multiple factors are integrated to take the minimum value of the respective calculated association cost to constitute the final association cost matrix. The KM method is used to associate and match the track and the candidate target box, the track list is updated and managed, and the multi-target tracking result under the UAV platform is obtained, Figure 9For the video sequence tracking result of the application in the low-altitude unmanned airport scene, it can be seen from the figure that even in the case of obvious occlusion, the embodiment can still accurately predict the target position and track, meeting the demand of multi-target tracking task under the low-altitude unmanned platform.

[0085] The multi-target tracking method of the unmanned platform under the complex scene of the above-mentioned embodiments of the application can meet the demand of real-time online multi-target tracking task under the low-altitude unmanned platform, can overcome the defects of the existing method due to target occlusion, detection loss and unmanned platform maneuvering, has strong robustness, applicability and real-time performance, enhances the visual perception ability of the unmanned platform to the surrounding environment, and has wide application prospect in civil and military fields.

[0086] Based on the same inventive concept, the multi-target tracking system for detection loss and association failure under the unmanned platform is also provided in other embodiments of the application, comprising:

[0087] An image preprocessing module, which obtains real-time video stream data from the visual system of the unmanned platform and saves, respectively performs image smoothing and region cropping on each frame of image data, and obtains the preprocessing result of each frame of image;

[0088] A joint network module, which takes the preprocessing result of each frame of image as input, takes a deep learning convolutional neural network as a basic network type, respectively obtains the detection result and the feature vector of the target from two output branches of the network, divides the high-level semantic feature space of the multi-target in the image into n preset subspaces, and respectively corresponds to n different ID target semantic categories;

[0089] A target region proposal module, which uses the track generated by the multi-target tracking to generate a candidate target based on the detection result of the joint network, discriminates the confidence of the candidate target frame generated by the track prediction, filters the low-confidence track output, and corrects the position of the candidate target frame;

[0090] A camera motion discrimination and registration module, which takes the association cost matrix of the track and the candidate target as the basis for discriminating the motion state of the unmanned platform, takes the spatial transformation matrix of the adjacent two frames of image data as input, and outputs whether the unmanned platform has a large displacement motion; on the basis of the motion discrimination model predicting that the camera has a large motion, the image registration technology is used to correct the camera motion, and the influence caused by the camera motion itself is eliminated;

[0091] The trajectory association module calculates an association cost matrix of the target feature vector, the spatial position information and the overlapping degree of the target frame and the trajectory based on the trajectory consistency degree, realizes the association matching of the trajectory and the candidate target, and manages and updates the trajectory list in real time.

[0092] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application.

Claims

1. A multi-target tracking method for unmanned aerial vehicle (UAV) platforms targeting detection loss and association failure, characterized in that, include: Preprocess real-time video traffic data; Input the preprocessing results into the joint network to obtain the detection results; Based on the detection results, candidate targets are generated by combining the trajectories produced by multi-target tracking; The candidate targets are optimized to obtain a set of candidate targets with high positioning accuracy as the lost detection targets; The joint network takes the preprocessing results of each frame image as input and uses a deep learning convolutional neural network as the basic network type to construct a deep aggregation module and a high-resolution module to achieve joint training of multi-target detection and feature extraction. The target detection results and feature vectors are obtained from the target detection branch and feature extraction branch of the network, respectively. The step of generating candidate targets based on the detection results and combining the trajectories generated by multi-target tracking includes: using the Kalman filter method to predict and update the trajectories of past image frames, which together with the detection results of the current frame constitute a candidate target set; The optimization process for candidate targets to obtain a high-precision candidate target set as the missing detection targets includes: The confidence level of candidate target boxes is judged, and trajectory outputs below a set confidence threshold are filtered out; The filtered candidate bounding boxes are corrected using their positions; The step of correcting the filtered candidate target box using location includes: Based on the centrality of the target detection branch of the joint network, the candidate target boxes generated by trajectory prediction are positionally corrected, and the target center is moved to the point with the local maximum centrality. Also includes: The correlation cost matrix between the trajectory and the candidate target is used as the basis for determining the motion state of the UAV platform; Using the spatial transformation matrix of two adjacent frames of image data as input, the output is whether the UAV platform has undergone displacement motion. After determining that the camera has moved, image registration is performed to correct the camera. It also includes calculating the correlation cost matrix of the feature vector, spatial location information and overlap between the target bounding box and the trajectory based on the obtained lost detection target, matching the trajectory and candidate target according to the correlation cost matrix, and managing and updating the trajectory list in real time.

2. The multi-target tracking method for unmanned aerial vehicle (UAV) platforms based on claim 1, characterized in that, The preprocessing includes: Extract and save images frame by frame from the raw video stream data; Image smoothing methods such as median filtering, Gaussian filtering, or mean filtering are used to suppress noise in each frame of the image; Perform random cropping or center cropping; The preprocessing results of each frame of the image are obtained.

3. The multi-target tracking method for unmanned aerial vehicle (UAV) platforms based on claim 1, characterized in that, The feature vector divides the high-level semantic feature space of multiple targets in the image into n predefined subspaces, each corresponding to a different semantic category of the target with a different ID.

4. A multi-target tracking system for unmanned aerial vehicle (UAV) platforms targeting detection loss and association failure, used to implement the multi-target tracking method for unmanned aerial vehicle platforms targeting detection loss and association failure as described in any one of claims 1-3, characterized in that, include: The image preprocessing module acquires and saves real-time video stream data from the vision system of the UAV platform, performs image smoothing and region cropping on each frame of image data, and obtains the preprocessing results of each frame of image. The joint network module takes the preprocessing results of each frame image as input and uses a deep learning convolutional neural network as the basic network type to construct a deep aggregation module and a high-resolution module to achieve joint training of multi-target detection and feature extraction. The target detection results and feature vectors are obtained from the target detection branch and feature extraction branch of the network, respectively. The target region proposal module, based on the joint network detection results, uses the trajectory generated by multi-target tracking to generate candidate targets, and at the same time judges the confidence of the candidate target boxes generated by trajectory prediction, filters out candidate target boxes below a set confidence threshold, and corrects the position of the candidate target boxes. The camera motion discrimination and registration module uses the correlation cost matrix between the trajectory and the candidate target as the basis for judging the motion state of the UAV platform, takes the spatial transformation matrix of two adjacent frames of image data as input, and outputs whether the UAV platform has undergone displacement. Based on the prediction of camera displacement, image registration technology is used to correct the camera motion and eliminate the influence caused by the camera's own motion. The trajectory association module calculates the association cost matrix based on the trajectory consistency degree, which includes the target feature vector, spatial location information, and the degree of overlap between the target box and the trajectory. This enables the association and matching of trajectories and candidate targets, and allows for real-time management and updating of the trajectory list.

Images