Virtual video generation method for multi-target collaborative tracking from complementary perspectives

Through the Unity engine simulation, the generation of multi-view video data and automatically labeled, solving the problem of high data acquisition and labeling cost in multi-objective perceived video analysis tasks under multi-view angles, achieving fast and accurate data generation, and supporting video analysis and research.

CN114445540BActive Publication Date: 2024-11-19TIANJIN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210075589.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-22
Publication Date
2024-11-19
Estimated Expiration
2042-01-22

AI Technical Summary

Technical Problem

The existing video analysis technology based on deep learning requires a large number of video data acquisition and labeling in real scenes in multi-view multi-objective perceived video analysis tasks, resulting in high costs and huge manpower and material consumption.

Method used

Using 3D development tools such as the Unity engine, we use character models and scene modeling to generate multi-view video data simulation and automatically calculate the position and ID of rectangular annotation boxes to reduce the need for manual annotation and achieve rapid generation of rich training data.

Benefits of technology

It effectively reduces the cost of data set acquisition and labeling required for multi-objective perceived video analysis tasks in multiple perspectives, generates a large number of accurate training data, supports video analysis and research, and can adjust parameters to meet the needs of different scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114445540B_ABST
    Figure CN114445540B_ABST
Patent Text Reader

Abstract

The invention discloses a virtual video generation method for complementary-view multi-target collaborative tracking, the method comprising: 1) loading a character model and a scene modeling, creating a data storage path, randomly initializing and storing the initial positions of pedestrians and multi-view cameras; 2) setting the initial positions of pedestrians and cameras, controlling the movement of pedestrians and cameras by frame, and storing video data containing pedestrians and photographers shot by cameras of each view; 3) deleting the scene modeling, setting the rendering background of cameras of each view to pure black, setting the initial positions of pedestrians and cameras to be the same as step 2), displaying only one person each time, controlling the movement of pedestrians and cameras by frame, and storing the video data of the person shot by cameras of each view with a pure black background; 4) repeating step 3); 5) calculating the rectangular annotation frame positions and IDs of all persons in each frame according to all single-person pure black background videos obtained in step 4); 6) removing the annotation frames of severely obscured persons in each frame.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a virtual video generation method for complementary-viewing-angle multi-target collaborative tracking. Background Art

[0002] Compared with the limited coverage of single-view fixed cameras and the defects of single shooting angle, in scenes facing outdoor dense personnel monitoring, the use of multi-view cameras (especially the complementary perspectives of top view and horizontal view) to shoot videos and conduct collaborative analysis of data to achieve large-scale full-scene dynamic monitoring has important research significance and development prospects. Multi-target perception is a typical task in multi-person scene video analysis, mainly including: multi-target detection, association and time-series tracking, which has naturally been widely studied in academia.

[0003] However, existing deep learning-based video analysis technologies often require a large amount of training data sets. For the multi-view multi-target perception video analysis tasks mentioned above, it is very difficult to shoot and annotate multi-person scene video data in real scenes, which often requires a lot of manpower and material resources. Summary of the invention

[0004] The present invention provides a virtual video generation method for complementary view multi-target collaborative tracking. The present invention solves the problem of huge cost of data set collection and annotation required for multi-target perception video analysis tasks under multi-view. With the help of 3D development tools such as Unity engine, 3D character models and scene modeling are used to control the movement of people in the scene by writing scripts, and multi-view video data close to the real scene is simulated and generated. At the same time, rectangular annotation boxes of all targets in the video and their corresponding IDs are generated to quickly generate a large amount of rich training data. See the following description for details:

[0005] A virtual video generation method for complementary perspective multi-target collaborative tracking, the method comprising:

[0006] 1) Load the character model and scene modeling, create a data storage path, randomly initialize and store the initial positions of pedestrians and multi-view cameras;

[0007] 2) Set the initial positions of pedestrians and cameras, control the movement of pedestrians and cameras by frame, and store the video data including pedestrians and photographers captured by cameras at different viewing angles;

[0008] 3) Delete the scene modeling, set the rendering background of each camera to pure black, set the initial positions of pedestrians and cameras to the same as step 2), display only one person at a time, control the movement of pedestrians and cameras by frame, and store the video data of the person with pure black background taken by each camera;

[0009] 4) Repeat step 3) until all the single-person pure black background video data of all pedestrians and the photographer are stored;

[0010] 5) Calculate the position and ID of the rectangular annotation box of all persons in each frame of each perspective video obtained in step 2) based on all single-person pure black background videos obtained in step 4);

[0011] 6) Remove the labeling boxes of severely occluded characters in each frame of each perspective video obtained in step 5), and output the remaining labeling boxes.

[0012] The multi-view cameras include a top view camera and a horizontal view camera, and the horizontal view camera includes 4 types;

[0013] Among them, the 4 types are:

[0014] (1) The photographer stands outside the crowd and shoots towards the center of the crowd. The photographer does not move, and the camera's field of view does not move either.

[0015] (2) The photographer stands outside the crowd and shoots toward the center of the crowd. The camera moves left and right, and the photographer follows the lens.

[0016] (3) The photographer stands outside the crowd and shoots towards the center of the crowd. The photographer moves in a circle around the crowd. During the movement, the camera swings left and right, and the photographer follows the lens.

[0017] (4) The photographer stands inside the crowd and shoots away from the center of the crowd. The photographer does not move but moves with the camera.

[0018] Wherein, the moving speed of the pedestrian is:

[0019] n is the total number of characters in the scene

[0020] The character model is given a base rotation speed ω = 5° / frame and a maximum offset rotation speed Δω = 10° / frame, and the rotation speed of each person is agreed to be:

[0021] n is the total number of people in the scene.

[0022] Furthermore, the method for generating the position of the rectangular annotation box of all persons in each frame of the video of each viewing angle and their ID is specifically as follows:

[0023] After the virtual scene video is generated in the first pass, it will be run again (total number of pedestrians + total number of photographers) times, with IDs ranging from 0 to (total number of pedestrians + total number of photographers - 1), and each pass is a complete reproduction of the first pass;

[0024] During the first run, the position, action, and camera position data of the randomly generated person are stored to ensure that the subsequent runs are exactly the same as the first run. Starting from the second run, only one person is displayed each time, and a virtual video containing only the person from each perspective is obtained through one run.

[0025] After the first run, remove the scene file and set the rendering background of all cameras to pure black, so as to obtain a virtual video in which each character only contains a single character in all cameras and the other backgrounds are all black.

[0026] For a picture with a completely black background and only the person is not black, directly traverse all the pixels in the picture to find the minimum non-black pixel horizontal coordinate xmin, the minimum non-black pixel vertical coordinate ymin, the maximum non-black pixel horizontal coordinate xmax, and the maximum non-black pixel vertical coordinate ymax. Then (xmin, ymin) and (xmax, ymax) are the upper left and lower right coordinates of the person's annotation box.

[0027] The specific removal of the marked boxes of severely obscured characters in each frame of each viewing angle video obtained in step 5) is:

[0028] Sort all the annotation boxes in this frame by area from large to small, and traverse all the undeleted annotation boxes from left to right. During the traversal process, traverse all the undeleted annotation boxes on the right side of the currently selected annotation box. If the overlapping area of ​​the two annotation boxes exceeds 1 / 2 of the area of ​​the smaller annotation box, delete the smaller annotation box.

[0029] The beneficial effects of the technical solution provided by the present invention are:

[0030] 1. This method aims to solve the problem that the research on multi-view multi-target perception video analysis tasks requires a large amount of data support, while the collection and annotation of a large amount of real data is very expensive; this method can easily obtain the training data required for video analysis under multi-view cameras;

[0031] 2. The present invention can generate a large amount of rich data for multi-view (especially complementary view consisting of top view and horizontal view) multi-target perception video analysis tasks, and it consumes much less manpower and material resources than manual data annotation, and is faster and more accurate;

[0032] 3. The present invention is adjustable and can simulate scenes according to the requirements of tasks. It allows multiple parameters to be changed when generating virtual video data, such as the number of pedestrians, the type of horizontal viewing angle camera, the position of the mobile camera, the number of mobile cameras, etc. At the same time, the lighting in the scene also allows multiple parameters to be modified to create various lighting environments for video analysis tasks, which can provide sufficient and rich training data for video analysis research;

[0033] 4. The pedestrians in the virtual video data set generated in the present invention are all character models produced by a 3D development engine. These models fully simulate the appearance of pedestrians in reality, with different skin colors, ages, body shapes, hairstyles, clothing, etc. At the same time, some characters will also have suitcases, backpacks, glasses or hats, etc., but they do not have the appearance of real people, so security and privacy issues can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of a method for generating virtual video data for multi-view and multi-target perception;

[0035] Figure 2 The figure is a comparison chart before and after treatment;

[0036] Among them, (a) is the 5th frame of the original video; (b) is the 5th frame of one of the character videos after processing.

[0037] Figure 3 A schematic diagram of the result of keeping the annotation boxes for one frame in the video. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0039] In order to solve the above problems, an embodiment of the present invention proposes a virtual video generation method for multi-target perception, which generates annotations while generating large-scale multi-person scene multi-view video data, so as to provide sufficient and rich training data for the research of video analysis algorithms. The invention first loads the character model into the scene model, and performs the initialization operation by assigning each character a random starting position and orientation. Secondly, the type of multi-view camera is set to simulate the top view camera of the drone perspective and the head-up camera of the wearable camera perspective respectively. Then, the movement behavior of multiple people in the scene, including the movement trajectory, speed and posture of pedestrians, is controlled to simulate and generate multi-view multi-person scene video data close to the real scene. Finally, while generating video data, a single-person pure black background video of each person in the scene at each perspective and its corresponding ID are generated, so as to give the rectangular annotation box position and ID of each person in each video, so that manual annotation is not required, saving a lot of manpower and material resources.

[0040] The background technologies involved in the present invention include:

[0041] (1) Synthetic data (see references [1], [2]): Synthetic data refers to the acquisition of new controllable samples based on prior knowledge, specifically using simulated scenes to obtain specific images, videos and their annotation information. Synthetic data is generally created based on game engines such as Unreal, Blender and Unity. Synthetic data can be simply understood as "building a specific virtual world for a certain visual task, in which you have a controllable perspective". Synthetic data can effectively reduce the consumption of manpower and material resources, and obtain data that is difficult to obtain and annotate in reality for some tasks. At the same time, the controllability of synthetic data can be used to conduct qualitative and quantitative experimental analysis of problems in research, thereby conducting more in-depth and scientific research on some new problems. At present, the main starting point for using synthetic data for computer vision research is still to solve the "data problem", for example: the PersonX synthesis engine designed for pedestrian re-identification tasks based on the Unity engine proposed in CVPR2019 and the SAIL-VOS dataset constructed for Amodal semantic segmentation (segmenting each object instance including its invisible and occluded parts) on videos.

[0042] (2) Drones and wearable cameras (see reference [3]): Advances in motion camera technology have provided new perspectives for video surveillance. Drones (e.g., aerial drones) can provide a bird’s-eye view of a group of objects on the ground. Wearable cameras (e.g., Google Glass and GoPro) mounted on the wearer’s head can provide a horizontal view of the same group of objects. The present invention generates data for a multi-view multi-target perception video analysis task, where there is a top-view camera in the air and multiple horizontal cameras on the ground. Since the bird’s-eye view and the horizontal view can complement each other’s information well, it can provide better coverage and flexibility for outdoor surveillance and can be effectively applied to tasks such as collaborative tracking and individual / group activity recognition.

[0043] Example 1

[0044] The technical solution adopted in the embodiment of the present invention is:

[0045] A virtual video data generation method for multi-view multi-target perception based on 3D development tools, see Figure 1 , the method comprises the following steps:

[0046] Step s1: Load the character model and scene modeling, create a data storage path, randomly initialize and store the initial positions of pedestrians and multi-view cameras (and photographers);

[0047] Step s2: setting the initial positions of the pedestrian and the camera (and the photographer), controlling the movement of the pedestrian and the camera (and the photographer) by frame, and storing the video data including the pedestrian and the photographer taken by the camera at each viewing angle;

[0048] Step s3: Delete the scene modeling, set the rendering background of each camera to pure black, set the initial positions of the pedestrian and the camera (and the photographer) to be the same as step s2, display only one person at a time, control the movement of the pedestrian and the camera (and the photographer) by frame, and store the video data of the person with a pure black background taken by each camera;

[0049] Step s4: repeat step s3 until all the pedestrians and the photographer's single-person pure black background video data are stored;

[0050] Step s5: Calculate the position and ID of the rectangular annotation box of all persons in each frame of each perspective video obtained in step s2 according to all single-person pure black background videos obtained in step s4;

[0051] Step s6: remove the labeling boxes of severely occluded characters in each frame of each viewing angle video obtained in step s5, and the remaining labeling boxes are what is required.

[0052] In summary, the embodiment of the present invention solves the problem of huge cost of collecting and labeling data sets required for multi-objective perception video analysis tasks under multi-viewing angles through the above steps s1-s6.

[0053] Example 2

[0054] The solution in Example 1 is further introduced below with reference to specific examples, as described below for details:

[0055] 1. Initialization of pedestrian position

[0056] In the embodiment of the present invention, the initial positions of all pedestrians are randomly generated in a circular area. The inner diameter and outer diameter of the circular area can be set by oneself. By randomly generating a radius length between [inner diameter, outer diameter] and randomly generating a rotation angle between [0°, 360°), the initial position of each character can be calculated according to the polar coordinate transformation. Then, an initial orientation is randomly generated for each character between [0°, 360°), thus completing the operation of initializing the pedestrian position.

[0057] (II) Camera type and initialization

[0058] There are two types of cameras in the embodiment of the present invention, a top-view camera and a horizontal-view camera. The top-view camera simulates the image taken by a drone without the need for a photographer, while the horizontal-view camera simulates the image taken by a handheld camera or a wearable camera and requires a photographer. There are a total of four types of horizontal-view cameras to choose from: (1) The photographer stands outside the crowd and shoots toward the center of the crowd. The photographer does not move, and the camera field of view does not move. For this type of camera, the photographer and the camera do not need to be updated in each frame. (2) The photographer stands outside the crowd and shoots toward the center of the crowd. The camera will swing the lens left and right, and the photographer will follow the lens rotation. For this type of camera, when updating each frame, the offset angle and swing direction of the camera swing in this frame need to be recorded and stored in an array (because there may be multiple horizontal viewing angle cameras). At the same time, in order to prevent the camera from swinging too much, resulting in the situation where there are basically no pedestrians in the field of view, the swing range needs to be limited. Here, the swing range is limited to [-15°, 15°]. By setting the swing degree of the camera in each frame and the offset angle of the camera's current frame, the camera's swing direction and offset angle are calculated and updated, thereby limiting the camera's swing range to the specified angle. (3) The photographer stands outside the crowd and shoots toward the center of the crowd. At the same time, the photographer will move in a circle around the crowd. During the movement, the camera will swing left and right, and the photographer will follow the lens. For this type of camera, when updating each frame, the position of the camera and the photographer in this frame must first be calculated based on the photographer's rotation speed and the length of the circle radius, and then the camera and the photographer are updated to swing left and right. The method is the same as that of type (2) cameras, so it will not be repeated here. (4) The photographer stands inside the crowd and shoots away from the center of the crowd. The photographer does not move, but moves with the camera. For this kind of camera, there is no need to limit the camera's swing range when updating each frame. The camera can rotate 360 ​​degrees. It only needs to rotate the camera and the photographer in each frame according to the camera's swing speed.

[0059] In order to make these cameras shoot from positions with large viewing angle differences, the cameras and the corresponding photographers also need to be initialized. For the top view camera, it only needs to randomly generate a rotation angle of [0°, 360°) during initialization, that is, the camera is randomly rotated by an angle according to the vertical axis. For the cameras of type (1) and (2) under the horizontal view, it only needs to calculate the initial camera and photographer positions through polar coordinate transformation according to the set camera radius length and the randomly generated initial rotation angle during initialization; for the camera of type (3), it is necessary to reduce the camera radius length by a certain value each time a camera of type (3) is used during initialization, so as to avoid overlapping trajectories when the photographer moves; finally, since the camera of type (4) is to be generated inside the crowd, the camera radius should be smaller than the inner diameter of the crowd initialization circle during initialization. At the same time, the horizontal view camera can also be set to look up, look down, tilt left, and tilt right during initialization to meet the needs of video research and analysis under different viewing angles.

[0060] 3. Pedestrian trajectory control

[0061] The character model can change its physical position by changing its orientation in each frame and moving forward. Therefore, in order to generate pedestrians in the scene of the video with different movement trajectories, in addition to initializing these pedestrians at different positions, they can also have different rotation speeds and movement speeds. In the embodiment of the present invention, the character model is given the following parameters:

[0062] Base moving speed:

[0063]

[0064] Maximum displacement speed:

[0065] Δv = 0.05 * height of the character model

[0066] Therefore, the movement speed of each person is:

[0067] n is the total number of characters in the scene

[0068] In this way, each person in the scene has a different movement speed. Next, the character model is also given a base rotation speed ω = 5° / frame and a maximum offset rotation speed Δω = 10° / frame, and the rotation speed of each person is agreed to be:

[0069] n is the total number of people in the scene

[0070] In this way, each person in the scene has a different rotation speed, and the problem of pedestrian trajectory control is solved.

[0071] (IV) Calculate the position and ID of the rectangular annotation box of each person in each frame of the video

[0072] For the generated virtual video data, due to the interference of the scene background and other characters, it is impossible to directly obtain the annotation box of each character. Therefore, in order to generate the annotation boxes of all characters, after the embodiment of the present invention generates the virtual scene video in the first pass, it will run (total number of pedestrians + total number of photographers) times (ID from 0 to (total number of pedestrians + total number of photographers-1)), and each pass is a complete reproduction of the first pass. At the same time, in order to ensure that the content captured by the same camera in the same frame in each pass is exactly the same, it is necessary to store some randomly generated data during the first run, such as: the initial generation position and orientation of pedestrians, the initial position of the photographer and the camera, etc., and then use these saved data to re-initialize each time it runs, so as to ensure that the position, action and camera position of the character are exactly the same as the first time. At the same time, in order to eliminate the interference of other characters, starting from the second pass, only one person is displayed each time, so that a virtual video containing only the target person from each perspective can be obtained through one run.

[0073] Finally, in order to eliminate the interference of the scene and background, it is necessary to remove the scene file after the first run of the system, and set the rendering background of all shooting cameras to pure black. In this way, a virtual video can be obtained in which each character only contains a single character under all cameras and the other backgrounds are all black. Figure 2 It is a picture of a frame in the original video and the corresponding frame of one of the characters after being processed by this method.

[0074] Next, for the picture with a completely black background and only the person is not black, directly traverse all the pixels in the picture to find the minimum non-black pixel horizontal coordinate xmin, the minimum non-black pixel vertical coordinate ymin, the maximum non-black pixel horizontal coordinate xmax, and the maximum non-black pixel vertical coordinate ymax. Then (xmin, ymin) and (xmax, ymax) are the upper left and lower right coordinates of the person's annotation box.

[0075] (V) Remove the labeling boxes of severely obscured pedestrians

[0076] After finding the positions of all the character annotation boxes in a frame using the previous step, the occlusion problem still needs to be solved, that is, the character whose body is mostly blocked should not be labeled. The method is to sort all the annotation boxes in this frame by area from large to small, traverse all the undeleted annotation boxes from left to right, and traverse all the undeleted annotation boxes to the right of the currently selected annotation box during the traversal process. If the overlapping area of ​​the two annotation boxes exceeds 1 / 2 of the area of ​​the smaller annotation box, delete the smaller annotation box. Figure 3That is, the result after drawing the annotation boxes for all the characters in a frame of the video and removing the occlusions. It can be seen that except for the person on the far right whose annotation box is not retained because most of his body is blocked, the annotation boxes of all other people are retained and in the correct position.

[0077] References

[0078] [1]Sun

[0079] [2]Hu YT, Chen HS, Hui K, et al.SAIL-VOS: Semantic Amodal InstanceLevel Video Object Segmentation–A Synthetic Dataset and Baselines[C] / / 2019IEEE / CVF Conference on Computer Vision and Pattern Recognition.IEEE, 2019.

[0080] [3]Zhao J, Han R, Gan Y, et al.Human Identification and InteractionDetection in Cross-View Multi-Person Videos with Wearable Cameras[C] / / ACMMultimedia Conference 2020.ACM, 2020.

[0081] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0082] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A virtual video generation method for complementary view multi-target collaborative tracking, characterized in that: The method comprises: 1) Load the character model and scene modeling, create a data storage path, randomly initialize and store the initial positions of pedestrians and multi-view cameras; 2) Set the initial positions of pedestrians and cameras, control the movement of pedestrians and cameras by frame, and store the video data including pedestrians and photographers captured by cameras at different viewing angles; 3) Delete the scene modeling, set the rendering background of each camera to pure black, set the initial positions of pedestrians and cameras to the same as step 2), display only one person at a time, control the movement of pedestrians and cameras by frame, and store the video data of the person with pure black background taken by each camera; 4) Repeat step 3) until all the single-person pure black background video data of all pedestrians and the photographer are stored; 5) Calculate the position and ID of the rectangular annotation box of all persons in each frame of each perspective video obtained in step 2) based on all single-person pure black background videos obtained in step 4); 6) removing the annotation boxes of the severely occluded characters in each frame of the video of each viewing angle obtained in step 5), and outputting the remaining annotation boxes; The multi-view cameras include a top view camera and a horizontal view camera, and the horizontal view camera includes 4 types; Among them, the 4 types are: (1) The photographer stands outside the crowd and shoots towards the center of the crowd. The photographer does not move, and the camera's field of view does not move either. (2) The photographer stands outside the crowd and shoots toward the center of the crowd. The camera moves left and right, and the photographer follows the lens. (3) The photographer stands outside the crowd and shoots towards the center of the crowd. The photographer moves in a circle around the crowd. During the movement, the camera swings left and right, and the photographer follows the lens. (4) The photographer stands inside the crowd and shoots away from the center of the crowd. The photographer does not move but moves with the camera.

2. The method for generating virtual video for complementary view multi-target collaborative tracking according to claim 1, characterized in that: The moving speed of the pedestrian is: Among them, v is the reference moving speed, Δv is the maximum offset speed; The character model is given a base rotation speed ω = 5° / frame and a maximum offset rotation speed Δω = 10° / frame, and the rotation speed of each person is agreed to be:

3. The method for generating virtual video for complementary view multi-target collaborative tracking according to claim 1, characterized in that: The method for generating the position of the rectangular annotation box of all persons in each frame of the video of each viewing angle and their ID is specifically as follows: After the virtual scene video is generated in the first pass, it will be run again (total number of pedestrians + total number of photographers) times, with IDs ranging from 0 to (total number of pedestrians + total number of photographers - 1), and each pass is a complete reproduction of the first pass; During the first run, the position, action, and camera position data of the randomly generated person are stored to ensure that the subsequent runs are exactly the same as the first run. Starting from the second run, only one person is displayed each time, and a virtual video containing only the person from each perspective is obtained through one run. After the first run, remove the scene file and set the rendering background of all cameras to pure black, so as to obtain a virtual video in which each character only contains a single character in all cameras and the other backgrounds are all black. For a picture with a completely black background and only the person is not black, directly traverse all the pixels in the picture to find the minimum non-black pixel horizontal coordinate xmin, the minimum non-black pixel vertical coordinate ymin, the maximum non-black pixel horizontal coordinate xmax, and the maximum non-black pixel vertical coordinate ymax. Then (xmin, ymin) and (xmax, ymax) are the upper left and lower right coordinates of the person's annotation box.

4. The method for generating virtual video for complementary view multi-target collaborative tracking according to claim 1, characterized in that: The specific removal of the marked boxes of severely obscured characters in each frame of the video of each viewing angle obtained in step 5) is: Sort all the annotation boxes in this frame by area from large to small, and traverse all the undeleted annotation boxes from left to right. During the traversal process, traverse all the undeleted annotation boxes on the right side of the currently selected annotation box. If the overlapping area of ​​the two annotation boxes exceeds 1 / 2 of the area of ​​the smaller annotation box, delete the smaller annotation box.