A multi-target collaborative tracking method based on high-altitude and ground perspectives

Abstract

The present invention discloses a multi-target collaborative tracking method based on high-altitude and ground perspectives, comprising the following steps: step s1, dividing two synchronized videos of air-ground collaborative perspectives into V small video segments of equal length; step s2, generating small track segments of the target under different perspectives for each small video segment according to the target detection result; step s3, calculating the similarity scores between different small track segments; step s4, using a joint optimization function to perform track association based on the similarity scores to generate short cross-view tracks; step s5, repeating steps s2 to s4, connecting the short cross-view tracks, so as to obtain a long cross-view track as the final tracking result; the method of the present invention has a higher matching accuracy than the existing method and obtains good cross-domain performance.

Description

Technical Field

[0001] The present invention belongs to the field of target tracking, and relates to target tracking in videos shot simultaneously from two perspectives by a wearable camera and an unmanned aerial vehicle, and in particular to a multi-target collaborative tracking method based on high-altitude and ground perspectives. Background Art

[0002] The background technologies involved in the present invention are:

[0003] (1) Pedestrian target detector (see reference [1]): As an important component of data preprocessing, pedestrian target detectors have been widely used in various fields such as target tracking, pedestrian relocation, and action recognition. Currently, commonly used pedestrian detectors can be divided into two categories: background modeling-based methods and statistical learning-based methods. The former must adapt to changes in the environment and is also limited by conditions such as image jitter caused by camera shake. The latter is a commonly used pedestrian detection method that constructs a pedestrian detection classifier by constructing a large number of samples. At present, the role of deep learning in pedestrian detection cannot be ignored. Among them, the YOLO detector is used in the present invention to provide the corresponding target detection frame.

[0004] (2) Air-ground collaborative perspective mobile camera network: Advances in motion camera technology have provided new perspectives for video surveillance. Drones can provide a bird's-eye view of a group of objects on the ground. Wearable cameras can provide a ground view of the same group of objects. This paper proposes a new air-ground collaborative perspective mobile camera network, where the bird's-eye view can provide global information about the target and the ground view can provide local details of the target. Therefore, the two can complement each other well, thereby providing better coverage and flexibility for outdoor surveillance. This complementary network can be effectively applied to tasks such as collaborative tracking and individual / group activity recognition.

[0005] (3) Cross-view multi-target association based on spatial distribution: Associating the top-down view with the ground view becomes very difficult due to the inability to utilize appearance and motion information. However, cross-view association based on spatial distribution can achieve association between targets by leveraging the spatial position distribution information between the aerial view and the ground view. This method can better collaboratively analyze data from different viewpoints, thus being applied to tasks such as target tracking, person re-identification, and action recognition. Summary of the Invention

[0006] The purpose of the present invention is to propose a multi-target tracking technology that combines global and local information for synchronized videos in the air-ground collaborative perspective of the same scene. With the help of wearable cameras and unmanned aerial vehicle equipment, a relatively simple and effective strategy is used to achieve collaborative tracking of targets in the air-ground collaborative perspective. The present invention can be used to simultaneously track targets in the ground and high-altitude perspectives in videos shot by cameras in the air-ground collaborative perspective in the same scene without limiting the number of targets in the perspective, and to match the targets in the two perspectives one-to-one.

[0007] The present invention is implemented by the following technical solutions:

[0008] A multi-target collaborative tracking method based on high-altitude and ground-based perspectives includes the following steps:

[0009] Step s1: Split the two synchronized videos of the air-ground collaborative perspective into V small video segments of equal length;

[0010] Step s2: Generate small target trajectories at different viewing angles for each small video segment based on the target detection results;

[0011] Step s3, calculating the similarity scores between different small segments of trajectories;

[0012] Step s4, using a joint optimization function to perform trajectory association based on the similarity score C to generate a short cross-view trajectory;

[0013] Step s5, repeating steps s2 to s4, connecting the short cross-view trajectories to obtain the cross-view long trajectory as the final tracking result; wherein: the short cross-view trajectory generation process:

[0014] The similarity measure between any trajectory segments across time and space is obtained through the similarity score C;

[0015] The similarity measure is solved using the following formula:

[0016]

[0017] Where: c ij represents the similarity between trajectory segments i and j, which can be calculated in step 3; a ij It is a binary variable with a value of 0-1, indicating whether two trajectory segments belong to the same target.

[0018] Furthermore, the similarity between the two small trajectories at the same time in different perspectives in step S3 is obtained by using the cross-perspective trajectory similarity:

[0019] 201. Using spatial distribution consistency

[0020] The matching results of the high-altitude and ground views are obtained using a method based on spatial distribution:

[0021] If the detection box B m With detection box B q If the matching result is the same person, then S(B m ,B q )=1; otherwise, S(B m ,B q )=0;High altitude view track B m With ground view track B q The similarity between them is expressed as follows:

[0022]

[0023] Where F is the number of overlapping frames, B m is the detection result under the top view, B q is the detection result under the ground view, |B m | is track B m The number of |B q | is track B q the number of

[0024] 202. Using Appearance Similarity

[0025] Use Siamese networks to measure the similarity between trajectories;

[0026] Calculate the average image of the two trajectories under the air-ground collaborative perspective and input it into the network;

[0027] Extract features

[0028] Calculate the Euclidean distance between features to get the similarity score

[0029] Combining the above similarity scores, the edge weights are calculated using linear combination:

[0030] Furthermore, the similarity score between two small segments of trajectories of the same view at different times in step S3 is generated by using the consistency of the trajectories across time:

[0031] 302. Using color histogram to measure the appearance similarity of single-view targets to obtain appearance trajectory consistency:

[0032] Compute the target color histogram for a single view track;

[0033] The median value Ψ(B) of the color histogram is used as the appearance descriptor of track B;

[0034] Computing appearance similarity between trajectories using histogram intersection

[0035] 302. Using constant velocity motion model to predict motion consistency and obtain action continuity:

[0036] Calculate the forward deviation error δ using the motion model p and reverse bias error δ n ;

[0037] By δ=α(δ p +δ n ) measure inter-trajectory variability;

[0038] Converting errors into similarities

[0039] Combining the above similarity scores, the edge weights are calculated using linear combination: Beneficial effects

[0040] This paper uses wearable cameras and drones, selects a simple and effective strategy, and independently designs a joint air-ground collaborative perspective multi-target tracking method. This method effectively solves the occlusion problem in multi-pedestrian motion scenes, with accurate results and efficient algorithms. Specifically, it has the following advantages:

[0041] a. High tracking accuracy

[0042] By using the video captured under the air-ground collaborative perspective, we can obtain the global motion trajectory and local appearance details of the target, which greatly alleviates the target loss caused by occlusion, thus achieving a higher tracking accuracy. The visualization of the collaborative tracking results can be seen in Figure 2 .

[0043] b. Low algorithm complexity and fast speed

[0044] This method is used for target tracking. Based on extensive experimental statistics, it runs at a speed of 4.24fps, outperforming most existing tracking methods. While maintaining accuracy, it significantly reduces time consumption.

[0045] c. Strong migration ability

[0046] The present invention does not require a large amount of data for training, does not rely on specific data features, and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 :Flowchart of multi-target tracking problem from air-ground collaborative perspective

[0048] Figure 2 :Schematic diagram of multi-target tracking results from air-ground collaborative perspective DETAILED DESCRIPTION

[0049] The present invention will be described in detail below with reference to the accompanying drawings:

[0050] The present invention provides a multi-target collaborative tracking method based on high-altitude and ground-level perspectives. First, the video in each view is divided into video segments of equal length, and trajectories are extracted in each video segment. Then, the trajectories of adjacent video segments in multiple views are used to establish spatiotemporal data association, thereby obtaining cross-view trajectories. Finally, the short trajectories are connected to obtain the cross-view long trajectory as the final tracking result. Figure 1 The specific process shown is:

[0051] (1) Step s1: Split the two synchronized videos from the air-ground collaborative perspective into V small videos of equal length; wherein:

[0052] Given a video from an aerial view and multiple videos from ground view, we synchronously split the two videos into segments of equal length.

[0053] Step s2: Generate target trajectories at different viewing angles for each short video segment based on the target detection results; Step s3: Calculate the similarity scores between different short trajectories;

[0054] The step of generating the similarity score between the two small segments of trajectories at the same time in different perspectives in step S3 by using the cross-perspective trajectory similarity is as follows:

[0055] 201. Using spatial distribution consistency

[0056] The matching results of the high-altitude and ground views are obtained using a method based on spatial distribution:

[0057] If the detection box B m With detection box B q If the matching result is the same person, then S(B m ,B q )=1; otherwise, S(B m ,B q )=0;High altitude view track B m With ground view track B q The similarity between them is expressed as follows:

[0058]

[0059] Where F is the number of overlapping frames, B m is the detection result under the top view, B q is the detection result under the ground view, |B m | is track B m The number of |B q | is track B q the number of

[0060] 202. Using Appearance Similarity

[0061] Use Siamese networks to measure the similarity between trajectories;

[0062] Calculate the average image of the two trajectories under the air-ground collaborative perspective and input it into the network;

[0063] Extract features;

[0064] Calculate the Euclidean distance between features to get the similarity score

[0065] Combining the above similarity scores, the edge weights are calculated using linear combination: The steps of generating similarity scores between two small trajectories of the same view at different times in step S3 by using the consistency of the trajectories across time are as follows:

[0066] 301. Using color histogram to measure the appearance similarity of single-view targets to obtain appearance trajectory consistency:

[0067] Compute the target color histogram for a single view track;

[0068] The median value Ψ(B) of the color histogram is used as the appearance descriptor of track B;

[0069] Computing appearance similarity between trajectories using histogram intersection

[0070] 302. Using constant velocity motion model to predict motion consistency and obtain action continuity:

[0071] Calculate the forward deviation error δ using the motion model p and reverse bias error δ n ;

[0072] By δ=α(δ p +δ n ) measure inter-trajectory variability;

[0073] Converting errors into similarities

[0074] Combining the above similarity scores, the edge weights are calculated using linear combination:

[0075] Step s4, using a joint optimization function to perform trajectory association based on the similarity score C to generate a short cross-view trajectory;

[0076] Step s5, repeating steps s2 to s4, connecting the short cross-view trajectories to obtain the cross-view long trajectory as the final tracking result; wherein: the short cross-view trajectory generation process:

[0077] The similarity measure between any trajectory segments across time and space is obtained through the similarity score C;

[0078] According to the similarity metric, the following formula is used to solve

[0079]

[0080] Where: c ij represents the similarity between trajectory segments i and j; a ij It is a binary variable with a value of 0-1, indicating whether two trajectory segments belong to the same target.

Claims

1. A multi-target collaborative tracking method based on high-altitude and ground perspectives, characterized in that: Step s1, splitting the two synchronized videos of the air-ground collaborative perspective into V small videos of equal length; Step s2, generating small segment trajectories of the target under different viewing angles for each small segment of video according to the target detection result; Step s3, calculating the similarity scores between different small segments of trajectories; Step s4, using a joint optimization function to perform trajectory association based on the similarity score C to generate a short cross-view trajectory; Step s5, repeating steps s2 to s4, connecting the short cross-view trajectories, thereby obtaining the cross-view long trajectory as the final tracking result; wherein: the short cross-view trajectory generation process: The similarity score C is used to obtain the similarity measure between any trajectory segments across time and space; According to the similarity metric, the following formula is used to solve function Where: c ij represents the similarity between trajectory segments i and j, which can be calculated by step 3; a ij It is a binary variable of 0-1, indicating whether two trajectory segments belong to the same target.

2. The multi-target collaborative tracking method based on high-altitude and ground perspectives according to claim 1, characterized in that: The step of obtaining the similarity score between two small segments of trajectories at the same time in different perspectives in step S3 by using the cross-perspective trajectory similarity is as follows:

201. Use spatial distribution consistency The matching results of the aerial and ground views are obtained using a method based on spatial distribution: If the detection box B m With detection box B q If the matching result is the same person, then S(B m ,B q )=1; Otherwise, S(B m ,B q )=0; High altitude view track B m With ground view track B q The similarity between them is shown as follows: Where F is the number of overlapping frames, B m is the detection result under the top view, B q is the detection result under the ground view, |B m | is track B m The number of |B q | is track B q the number of 202. Use appearance similarity Use Siamese networks to measure the similarity between trajectories; Calculate the average image of the two trajectories under the air-ground collaborative perspective and input them into the network; Extract features Calculate the Euclidean distance between features to get the similarity score Combining the above similarity scores, the weight of the edge is calculated using linear combination:

3. The multi-target collaborative tracking method based on high-altitude and ground perspectives according to claim 1, characterized in that: The steps of obtaining the similarity score between two small segments of trajectories of the same view at different times in step S3 by using the consistency of the trajectories across time are as follows:

301. Use color histogram to measure the appearance similarity of single-view targets to obtain appearance trajectory consistency: Compute the target color histogram for a single view track; The median value Ψ(B) of the color histogram is used as the appearance descriptor of track B; Computing the appearance similarity between trajectories using histogram intersection 302. Using constant speed motion model to predict motion consistency and obtain action continuity: Calculate the forward deviation error δ using the motion model p and reverse bias error δ n ; By δ = α (δ p +δ n ) measure the variability between trajectories; Converting Errors to Similarities Combining the above similarity scores, the weight of the edge is calculated using linear combination:

Images