A synthetic data generation method for multi-person relationship perception from complementary perspectives
By using 3D development tools such as Unity engine to generate multi-view multi-person relationship-aware video data, the problem of high data collection and labeling in multi-view multi-view tasks is solved, and the rapid and accurate generation of rich data sets is achieved, reducing the need for reducing the cost of data collection and labeling. Manpower and material resources are consumed.
Patent Information
- Application Number
- CN202210075601.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-22
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-01-22
AI Technical Summary
The existing video analysis technology based on deep learning requires the acquisition and labeling of multi-person scene video data in a multi-person relationship perception task from multiple perspectives, which is relatively expensive and consumes a lot of manpower and material resources.
Using 3D development tools such as the Unity engine, by loading character models and scene modeling, writing scripts to control the movement and behavior of people in the scene, simulate and generate multi-view video data, and generate the labeled box position, character ID and belonging of each frame in the video. Group ID and other information quickly generate a large amount of and rich training data.
It effectively reduces the acquisition and labeling cost of data sets required for multi-person relationship-aware video analysis tasks from multiple perspectives. The generated data is rich and adjustable, and is suitable for a variety of video analysis tasks, reducing the cost and time of human data acquisition.
Smart Images

Figure CN114494554B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a method for generating synthetic data for complementary perspective multi-person relationship perception. Background Art
[0002] Compared with the limited coverage of single-view fixed cameras and the defects of single shooting angle, in scenes facing outdoor dense personnel monitoring, the use of multi-view cameras (especially the complementary perspectives of top view and horizontal view) to shoot videos and conduct collaborative analysis of data to achieve large-scale full-scene dynamic monitoring has important research significance and development prospects. As a typical task in multi-person scene video analysis, multi-person relationship perception is mainly aimed at the evolution of relationships between different individuals and groups in multi-person scenes, and naturally has also been widely studied in academia.
[0003] However, existing deep learning-based video analysis technologies often require a large amount of training data. As for the multi-view multi-person relationship perception video analysis task mentioned above, it is very difficult to shoot and annotate multi-person scene video data in real scenes, which often requires a lot of manpower and material resources. Summary of the invention
[0004] The present invention provides a synthetic data generation method for multi-person relationship perception from complementary perspectives. The present invention solves the problem of huge cost of data collection and annotation required for multi-person relationship perception video analysis tasks under multi-perspectives. The present invention uses 3D development tools such as the Unity engine, 3D character models and scene modeling, and scripts to control the movement and behavior of people in the scene, simulate and generate multi-perspective video data close to the real scene, and simultaneously generate information on the evolution of all groups in the video. It can also generate information such as the position of the character's annotation box in each frame of the video, the character ID and the ID of the group to which it belongs, so as to quickly generate a large amount of rich training data, as described below:
[0005] A method for generating synthetic data for complementary perspective multi-person relationship perception, the method comprising:
[0006] Set up pedestrian class, group class and evolution class, and initialize them according to the requirements of video generation;
[0007] Load character models and scene modeling, create data storage paths, randomly initialize and store the initial positions of individuals, groups, and multi-view cameras;
[0008] Set the initial positions of individuals, groups and cameras, and add all initialized pedestrian and group objects to the pedestrian and group queues respectively;
[0009] Execute the group evolution function frame by frame. The evolution function is used to determine whether the evolution of the current frame has not started or has been completed to determine whether to execute the evolution. When executing the evolution, the individuals and groups involved are removed from the queue. After the evolution is completed, the newly generated individuals and groups are added to their respective queues.
[0010] Control the movement of all remaining individuals and groups in the pedestrian and group queues of the current frame frame by frame, and control the movement of the camera frame by frame, store the video data including pedestrians and photographers shot by cameras of different viewpoints; store the type, start frame, and end frame information of all group evolutions in the video.
[0011] Among them, if the position of the character's annotation box in each frame of the video, the character's ID and the ID information of the group to which he belongs are generated, it is necessary to store the position, direction and behavior action information of all characters in each frame.
[0012] The beneficial effects of the technical solution provided by the present invention are:
[0013] 1. This method aims to solve the problem that the research on multi-view multi-person relationship perception video analysis tasks requires a large amount of data support, while the collection and annotation of a large amount of real data is very costly; this method can easily obtain the training data required for video analysis under multi-view cameras;
[0014] 2. The present invention can generate a large amount of rich data for the video analysis task of multi-view (especially the complementary view consisting of top view and horizontal view) multi-person relationship perception, while the manpower and material resources spent are much less than those spent on manual data collection, and the annotation is also faster and more accurate;
[0015] 3. The present invention has the adjustability to simulate scenes according to the requirements of the task, and allows multiple parameters to be changed when generating synthetic video data, such as: the number of individuals at the beginning, the number of groups and the number of people in each group, the type and number of group evolution, the type and number of horizontal viewing angle cameras, etc. At the same time, the lighting in the scene also allows multiple parameters to be modified to create various lighting environments for video analysis tasks;
[0016] 4. The pedestrians in the synthetic video data set generated in the present invention are all character models produced by a 3D development engine. These models fully simulate the appearance of pedestrians in reality, with different skin colors, ages, body shapes, hairstyles, clothing, etc. At the same time, some characters also have suitcases, backpacks, glasses or hats, etc., but they do not have the appearance of real people, so security and privacy issues can be effectively reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 A flowchart of a method for generating synthetic video data for multi-view multi-person relationship perception;
[0018] Figure 2 A schematic diagram of the joining evolution;
[0019] Figure 3 Schematic diagram of the leaving evolution when type=1;
[0020] Figure 4 It is a schematic diagram of dissolution evolution when type=0;
[0021] Figure 5 is a schematic diagram of the aggregation evolution;
[0022] Figure 6 Schematic diagram of the merge evolution;
[0023] Figure 7 Schematic diagram of separation evolution. DETAILED DESCRIPTION
[0024] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.
[0025] The technical background involved in the embodiments of the present invention includes:
[0026] (1) Synthetic data (see references [1] and [2]): Synthetic data refers to the acquisition of new controllable samples based on prior knowledge, specifically using simulated scenes to obtain specific images, videos and their annotation information. Synthetic data is generally created based on game engines such as Unreal, Blender and Unity. Synthetic data can be simply understood as "building a specific virtual world for a certain visual task, in which you have a bird's-eye view". Synthetic data can effectively reduce the consumption of manpower and material resources, and obtain data that is difficult to obtain and annotate in reality for some tasks. At the same time, the controllability of synthetic data can be used to conduct qualitative and quantitative experimental analysis of problems in research, thereby studying some new problems more deeply and scientifically. At present, the main starting point for using synthetic data for computer vision research is still to solve the "data problem", such as the PersonX synthesis engine designed for pedestrian re-identification tasks based on the Unity engine proposed in CVPR2019 and the SAIL-VOS dataset constructed for Amodal semantic segmentation (segmenting each object instance including its invisible and occluded parts) on videos.
[0027] (2) Drones and wearable cameras (see reference [3]): Advances in motion camera technology have provided new perspectives for video surveillance. Drones (e.g., aerial drones) can provide a bird’s-eye view of a group of objects on the ground. Wearable cameras (e.g., Google Glass and GoPro) mounted on the wearer’s head can provide a horizontal view of the same group of objects. An embodiment of the present invention generates data for a multi-view multi-person relationship perception video analysis task, where there is a top-view camera in the air and multiple horizontal cameras on the ground. Since the bird’s-eye view and the horizontal view can complement each other’s information well, it can provide better coverage and flexibility for outdoor surveillance and can be effectively applied to tasks such as collaborative tracking and individual / group activity recognition.
[0028] In order to solve the problems existing in the background technology, an embodiment of the present invention proposes a method for generating synthetic videos for multi-person relationship perception, which generates annotations while generating multi-perspective video data of large-scale multi-person scenes, so as to provide sufficient and rich training data for the research of video analysis algorithms.
[0029] The embodiment of the present invention first loads the character model into the scene model, and performs the initialization operation by assigning each character (in the form of an individual or a group) a random starting position and orientation. Secondly, the type of multi-view camera is set to simulate the top view camera of the drone perspective and the head-up camera of the wearable camera perspective respectively. Then the motion behavior of multiple people in the scene is controlled, including: pedestrian movement trajectory, speed posture, interactive behavior (such as conversation), etc., to simulate and generate multi-view multi-person scene video data close to the real scene. At the same time, in addition to simulating pedestrian movement and internal interactive behavior of the group, the generation and change process of the group are also simulated, including: ① Join: multiple individuals join the generated group; ② Leave: multiple individuals leave the original participating group; ③ Merge: multiple groups merge into one group; ④ Separate: one group separates into multiple groups; ⑤ Aggregate: multiple individuals gather in a short time to generate a group; ⑥ Disband: The group disbands into multiple individuals in a short time. Finally, while generating the video data, the types of all evolutions in the video, the start frame, the end frame, and the position of the annotation box of the characters in each frame of the video, the character ID and the ID of the group to which they belong, etc., are also generated, so that there is no need for manual annotation, saving manpower and material resources.
[0030] Example 1
[0031] The embodiment of the present invention provides a method for generating synthetic video data for multi-view multi-person relationship perception based on a 3D development tool, see Figure 1 , the method comprises the following steps:
[0032] Step s1: Set the pedestrian class, group class and evolution class, and initialize them according to the requirements of video generation;
[0033] Step s2: Load the character model and scene modeling, create a data storage path, randomly initialize and store the initial positions of individuals, groups, and multi-view cameras (and photographers);
[0034] Step s3: Set the initial positions of individuals, groups and cameras (and photographers), and add all initialized pedestrian and group class objects to the pedestrian and group class queues respectively;
[0035] Step s4: Execute the group evolution function frame by frame. The evolution function is used to determine whether the evolution of the current frame has not started or has been completed to determine whether to execute the evolution. When executing the evolution, the individuals and groups involved will be removed from the queue. After the evolution is completed, the newly generated individuals and groups will be added to their respective queues.
[0036] Step s5: Control the movement of all remaining individuals and groups in the pedestrian and group queues of the current frame frame by frame, and control the movement of the camera (and the photographer) by frame, and store the video data including pedestrians and photographers shot by cameras of different viewing angles;
[0037] Step s6: Store the type, start frame, end frame and other information of all group evolutions in the video; if you also need to generate information such as the position of the character's annotation box in each frame of the video, the character ID and the ID of the group to which he belongs, you also need to store information such as the position, direction and behavior of all characters in each frame.
[0038] In summary, the embodiment of the present invention solves the problem of the huge cost of collecting and labeling the data set required for the multi-person relationship perception video analysis task under multi-perspective through the above steps s1 to s6, and meets various needs in practical applications.
[0039] Example 2
[0040] The scheme in Example 1 is further introduced below in combination with specific examples and formulas, as described below for details:
[0041] (I) Settings for pedestrians, groups, and evolutionary classes
[0042] Since 3D development tools such as the Unity engine update scenes and states on a frame-by-frame basis, each time a function that implements group evolution is called, only one frame will actually be run, and it is necessary to store the state information after each frame of the function is run. In the embodiment of the present invention, the person class, group class, and evolution class are mainly created to store this information.
[0043] 1. Person class: 4 variables are defined in this class.
[0044] (1) person_id: It records the pedestrian’s ID number. This ID number is the index of the pedestrian’s character model stored in the array after it is loaded into the scene. This ID number can be used to conveniently control various behaviors of the character model.
[0045] (2) distance: records the distance between the pedestrian and the center of the group to be joined. In the evolution of pedestrians joining the group and multiple pedestrians gathering into a group, it is necessary to judge whether the pedestrian has moved to the target position based on the value of distance.
[0046] (3) Atposition: records whether the pedestrian has joined the group. In the evolutionary behavior of pedestrians joining the group and multiple pedestrians gathering into a group, since each pedestrian joins the group at a different speed, it is necessary to determine whether all pedestrians have completed the evolutionary behavior based on the atposition value of each pedestrian.
[0047] (4) needrotate: records the remaining rotation angle of the pedestrian in the current frame. In order to make the rotation of the pedestrian in the generated video look smoother and more coherent, a global variable is set to limit the maximum rotation angle of the character in each frame. Therefore, when the angle of rotation of the character is large, it cannot be completed in one frame, and needrotate is needed to record the remaining rotation angle of the person at all times.
[0048] 2. Group class: Three variables are defined in this class.
[0049] (1) memberid: It is a List used to store the IDs of all people in the group. In the embodiment of the present invention, the group is set to have a central person and two circles of people on the outside. The storage order of the IDs in the group has the following specifications: memberid[0] stores the ID of the central person, Store the id of the first circle of pedestrians, Store the ids of the second circle of pedestrians, where n is the total number of people in the group.
[0050] (2)total_member: records the total number of people in the group (the total number of people in the group must be greater than or equal to 2).
[0051] (3) needrotate: It is also a List, which is used to record the remaining rotation angles of each person in the group in the current frame. The storage order corresponds to the storage order of memberid.
[0052] (4) group_id: records the id of the group, that is, the id of the group to which all pedestrians in the group belong.
[0053] 3. Activity class: Different group evolutions have different evolution classes. The following explains some of the common variables they have.
[0054] (1) start_frame: the starting frame of the evolution behavior. The group evolution will only start when the current frame = the starting frame.
[0055] (2) isend: indicates whether the evolution behavior is finished. If isend=true of the evolution class, the evolution function will not execute the corresponding evolution but will exit directly.
[0056] (3) join_person: is a List of Person type, used to store all individuals participating in the evolution, so as to facilitate direct operations on these individuals in the evolution function.
[0057] (4) join_group: is a List of Group type, which is used to store all groups participating in the evolution, making it convenient to directly operate on these groups in the evolution function.
[0058] 2. Initialization of individual and group positions
[0059] In the embodiment of the present invention, the initial positions of all pedestrians (including individuals and groups) are randomly generated in a circular ring area. The inner diameter and outer diameter of the circular ring can be set by oneself. By randomly generating a radius length between [inner diameter, outer diameter] and randomly generating a rotation angle between [0°, 360°), the initial position of each individual and the central person of the group can be calculated according to the polar coordinate transformation, and then an initial orientation is randomly generated between [0°, 360°), that is, the operation of initializing the position of each individual and the central person of the group is completed. After completing the initialization of the position of the central person of the group, the first circle and the second circle of pedestrians (with different radii) are divided according to the total number of people in the group, and the rotation angle θ centered on the central person of the group is randomly generated between [0°, 360°) for these pedestrians, then (x+r*cosθ, z+r*sinθ) is the initial position of each of these pedestrians, where (x, z) is the initial position of the central person of the group, and the initial orientation of each person in the group is consistent with the initial orientation of the central person of the group, so that the position initialization operation of the individual and the group is completed.
[0060] (III) Camera type and initialization
[0061] There are two types of cameras in the embodiment of the present invention, a top view camera and a horizontal view camera. The top view camera simulates the picture taken by the drone without placing the photographer, while the horizontal view camera simulates the picture taken by a handheld camera or a wearable camera and requires the placement of the photographer. That is, there are a total of 4 types of horizontal view cameras to choose from:
[0062] (1) The photographer stands outside the crowd and shoots towards the center of the crowd. The photographer does not move, and the camera field of view does not move either. For this type of camera, there is no need to update the photographer and the camera in each frame.
[0063] (2) The photographer stands outside the crowd and shoots toward the center of the crowd. The camera will swing left and right, and the photographer follows the lens. For this type of camera, when updating each frame, the offset angle and swing direction of the camera swing in this frame need to be recorded and stored in an array (because there may be multiple horizontal viewing angle cameras). At the same time, in order to prevent the camera from swinging too much, resulting in a situation where there are basically no pedestrians in the field of view, the swing range needs to be limited. Here, the swing range is limited to [-15°, 15°]. By setting the swing degree of each frame of the camera and the offset angle of the current frame of the camera, the camera swing direction and offset angle are calculated and updated, so that the camera swing range is limited to the specified angle.
[0064] (3) The photographer stands outside the crowd and shoots towards the center of the crowd. At the same time, the photographer moves in a circle around the crowd. During the movement, the camera swings left and right, and the photographer follows the lens. For this type of camera, when updating each frame, the position of the camera and the photographer in the frame must first be calculated based on the photographer's rotation speed and the length of the circle radius, and then the camera and the photographer are updated to swing left and right. The method is the same as that of the camera of type (2), so it will not be repeated here.
[0065] (4) The photographer stands inside the crowd and shoots away from the center of the crowd. The photographer does not move, but moves with the camera. For this kind of camera, there is no need to limit the camera's swing range when updating each frame. It can rotate 360 degrees. It only needs to rotate the camera and the photographer in each frame according to the camera's swing speed.
[0066] In order to make these cameras shoot from positions with large viewing angle differences, the cameras and the corresponding photographers also need to be initialized. For the top view camera, it only needs to randomly generate a rotation angle of [0°, 360°) during initialization, that is, the camera is randomly rotated by an angle according to the vertical axis (y axis). For the cameras of type (1) and (2) under the horizontal view, it is only necessary to calculate the initial camera and photographer positions through polar coordinate transformation according to the set camera radius length and the randomly generated initial rotation angle during initialization; for the camera of type (3), it is necessary to reduce the camera radius length by a certain value each time a camera of type (3) is used during initialization, so as to avoid overlapping trajectories when the photographer moves; finally, since the camera of type (4) is to be generated inside the crowd, the camera radius should be smaller than the inner diameter of the crowd initialization circle during initialization. At the same time, the horizontal view camera can also be set to look up, look down, tilt left, and tilt right during initialization to meet the needs of video research and analysis under different viewing angles.
[0067] (IV) Storage of pedestrians and groups
[0068] In the embodiment of the present invention, queues of Person type queue_person and queues of Group type queue_group are used to store all available individuals and groups in the current frame, that is, individuals and groups that are not currently participating in any evolution. At the same time, when the trajectory of individuals and groups is controlled in each frame, only all individuals and groups in queue_person and queue_group will be moved, and all evolution functions will control the individuals and groups participating in the evolution by themselves. The advantage of using a queue is that the evolution function will select individuals and groups participating in the evolution from the head of the team, and add the newly generated individuals and groups to the tail of the team after the evolution is completed, so that each individual and group has the opportunity to participate in the evolution. At the same time, the principle of last in and last out of the queue will also ensure that individuals and groups that have just participated in the evolution will not participate in the evolution again immediately, ensuring that other individuals and groups in the queue can participate in the evolution as soon as possible, making the generated video data more natural and real.
[0069] (V) Pedestrian trajectory control
[0070] The character model can change its physical position by changing its orientation in each frame and moving forward. Therefore, in order to generate different action trajectories for individuals and groups in the scene of the video, in addition to initializing these individuals and groups at different positions, they can also have different rotation speeds and movement speeds. In the embodiment of the present invention, a reference movement speed is given to the character model:
[0071]
[0072] Maximum displacement speed:
[0073] Δv = 0.05 * height of the character model
[0074] Therefore, the moving speed of each individual and group (the moving speed of each person in the group is equal to the moving speed of the central figure in the group) is:
[0075] n is the total number of characters in the scene
[0076] In this way, each individual and group in the scene has a different movement speed. Next, the character model is also given a base rotation speed ω = 5° / frame and a maximum offset rotation speed Δω = 10° / frame, and the rotation speed of each individual and group (the rotation speed of each person in the group is equal to the rotation speed of the central character in the group) is agreed to be:
[0077] n is the total number of people in the scene
[0078] In this way, each individual and group in the scene also has a different rotation speed, and the problem of pedestrian trajectory control is solved.
[0079] 6. Realization of Group Evolution
[0080] (1) Join Evolution:
[0081] Joining evolution means that multiple individuals join the generated group. The design is to first select a group to stop moving and turn to the center (the person at the center of the group turns to the second person in the group) to talk. Then a certain number of pedestrians are randomly selected, and their respective rotation angles and positions are determined according to their current positions and joined to the group to talk. After all pedestrians have joined, everyone in the group stops talking and starts moving together. The schematic diagram is as follows Figure 2 shown.
[0082] (2) Leaving and disbanding evolution:
[0083] Leaving evolution means that multiple individuals leave the group they originally participated in. Two implementation methods are designed for this evolution behavior, which are distinguished by the type variable in the evolution class. When type = 0, the group participating in the evolution will randomly select some individuals to leave, and these individuals will move away from the center of the group while moving with the group. When type = 1, all people in the group participating in the evolution will first stop moving and turn to the center (the person at the center of the group turns to the second person in the group) to talk, and then randomly select some individuals to leave and leave in the direction away from the center of the group after a random period of time. After another random period of time, all the remaining people in the group stop talking and start moving together. The schematic diagram is as follows Figure 3 shown.
[0084] Disbanding evolution means that a group disbands into multiple individuals in a short period of time. Two implementation methods are also designed for this evolution behavior, which are also distinguished by the type variable in the evolution class. The implementation process is basically the same as that of leaving evolution, except that the leaving individuals are no longer randomly selected, but all individuals must leave. I will not go into details here. Figure 4 shown.
[0085] (3) Aggregation evolution:
[0086] Aggregate evolution is when multiple individuals gather in a short period of time to form a group. The design is to first select two individuals and calculate the distance between them. If the distance is less than the threshold (the threshold is slightly larger than the distance between the two in the group), the individuals will continue to move until the distance is greater than the threshold. After that, the two individuals will move to the current center position o. At this time, a certain number of pedestrians will be randomly selected to move to position o. People who arrive at the target position will start talking until everyone joins the group. After a random period of time, all people in the group stop talking and start moving together. The schematic diagram is as follows Figure 5 shown.
[0087] (4) Merge evolution:
[0088] Merged evolution means that multiple groups are merged into one group. The design scheme is to first select multiple groups, one of which is used as the main group. The main group stops moving and all turn to the center (the person at the center of the group turns to the second person in the group) to talk. Each of the remaining groups selected calculates the distance from the center position o of the main group. If the distance is less than the preset threshold 1, the group moves away, otherwise the group moves toward the center position o until the distance from the center position is the preset threshold 2 (preset threshold 2 < preset threshold 1). At this time, the random position of each person in the group is calculated, and the group is disbanded and everyone moves to their respective positions. After arriving at the position, if the person is not facing the center position o, he needs to rotate toward the center and talk. After all the people in all the remaining groups join the main group, all the people in the group stop talking for a random period of time and start moving together. The schematic diagram is as follows Figure 6 shown.
[0089] (5) Separation evolution:
[0090] Separation evolution means that a group is separated into multiple groups. The design is to first select a group, which stops moving and all turns to the center (the person at the center of the group turns to the second person in the group) to talk. Then, the number of groups separated and the number of people in each group are randomly generated (ensuring that the number of people in each group is greater than or equal to 2), and each group is given a random starting frame and gathering position for separation. When the current frame = the separation start frame of the group, all people in the group leave the original group and gather at the location of the new group to talk. After all people in the group gather, all people in the group stop talking for a random period of time and start moving together. Finally, when all groups have completed separation, the separation evolution ends. The schematic diagram is as follows Figure 7 shown.
[0091] Example 3
[0092] In addition to steps s1 to s6 in the above embodiment, the embodiment of the present invention further includes the following steps:
[0093] Step s7: Delete the scene modeling, set the rendering background of each camera to pure black, set the initial positions of the pedestrian and the camera (and the photographer) to be the same as step s3, and display only one person at a time. Control the movement of the pedestrian and the camera (and the photographer) frame by frame according to the position, orientation, and behavior information of the person in each frame stored in step s6, and store the video data of the person with a pure black background shot by the camera at each viewpoint;
[0094] For the generated virtual video data, due to the presence of scene background and interference from other characters, it is impossible to directly obtain the annotation box of each character. Therefore, in order to generate the annotation boxes of all characters, after the embodiment of the present invention generates the virtual scene video in the first pass, it will run (total number of pedestrians + total number of photographers) times (character ID from 0 to (total number of pedestrians + total number of photographers-1)), and each pass is a complete reproduction of the first pass. At the same time, in order to ensure that the content captured by the same camera in the same frame in each pass is exactly the same, it is necessary to store the position, direction and behavior of all pedestrians in each frame and the initial position of the camera of each perspective during the first run. After that, these saved data are used for setting each time it runs, so as to ensure that the position, action and camera position of the character are exactly the same as the first time. At the same time, in order to eliminate the interference of other characters, starting from the second pass, only one person is displayed each time, so that a virtual video containing only the target person in each perspective can be obtained through one run.
[0095] Then, in order to eliminate the interference of the scene and background, it is necessary to remove the scene file after the first run of the system, and set the rendering background of all shooting cameras to pure black. In this way, a virtual video can be obtained in which each character only contains a single character under all cameras and the other backgrounds are all black.
[0096] Step s8: repeat step s7 until all pedestrians and the photographer's single-person pure black background video data are stored;
[0097] Step s9: Calculate the rectangular annotation box position, person ID and group ID of all persons in each frame of each perspective video obtained in step s5 according to all single-person pure black background videos obtained in step s8;
[0098] For images with a completely black background and only the person is not black, directly traverse all pixels in the image to find the minimum non-black pixel horizontal coordinate xmin, the minimum non-black pixel vertical coordinate ymin, the maximum non-black pixel horizontal coordinate xmax, and the maximum non-black pixel vertical coordinate ymax. Then (xmin, ymin) and (xmax, ymax) are the upper left and lower right coordinates of the person's annotation box. At the same time, for each person's ID, in the process of repeatedly running (total number of pedestrians + total number of photographers) times, it will be assigned a value between 0 and (total number of pedestrians + total number of photographers-1) according to the running order.
[0099] Finally, it is necessary to calculate the group ID to which each person belongs. For all individuals, there is no group to which they belong, so the group ID to which they belong is uniformly set to -1. For all groups, they can be divided into two categories: groups that are not participating in the evolution and are stored in the group queue, and groups that are not in the group queue and are participating in the evolution. For the first category, the group_id variable value of the group instance is the group ID of all people in the group. At the same time, a global variable next_groupid (starting from 0) is set to record the next available group ID number. Whenever a new group is generated, the group_id of the group instance is next_groupid. For the second category, the group ID to which each person in the group belongs is determined by the current execution status of the evolution and next_groupid.
[0100] Step s10: remove the labeling boxes of severely occluded people in each frame of each viewing angle video obtained in step s9, and the remaining labeling boxes are what is required.
[0101] After finding the positions of all the character annotation boxes in a frame using the previous step, the occlusion problem still needs to be solved, that is, the character whose body is mostly blocked should not be labeled. The method is to sort all the annotation boxes in this frame from large to small by area, traverse all the undeleted annotation boxes from left to right, and traverse all the undeleted annotation boxes on the right side of the currently selected annotation box during the traversal process. If the overlapping area of the two annotation boxes exceeds 1 / 2 of the area of the smaller annotation box, delete the smaller annotation box.
[0102] References
[0103] [1]Sun
[0104] [2]Hu YT, Chen HS, Hui K, et al.SAIL-VOS: Semantic Amodal InstanceLevel Video Object Segmentation–A Synthetic Dataset and Baselines[C] / / 2019IEEE / CVF Conference on Computer Vision and Pattern Recognition.IEEE, 2019.
[0105] [3]Zhao J, Han R, Gan Y, et al.Human Identification and InteractionDetection in Cross-View Multi-Person Videos with Wearable Cameras[C] / / ACMMultimedia Conference 2020.ACM, 2020.
[0106] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.
[0107] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A synthetic data generation method for complementary perspective multi-person relationship perception, characterized in that: The method comprises: Set up pedestrian class, group class and evolution class, and initialize them according to the requirements of video generation; Load character models and scene modeling, create data storage paths, randomly initialize and store the initial positions of individuals, groups, and multi-view cameras; Set the initial positions of individuals, groups and cameras, and add all initialized pedestrian and group objects to the pedestrian and group queues respectively; Execute the group evolution function frame by frame. The evolution function is used to determine whether the evolution of the current frame has not started or has been completed to determine whether to execute the evolution. When executing the evolution, the individuals and groups involved are removed from the queue. After the evolution is completed, the newly generated individuals and groups are added to their respective queues. Control the movement of all remaining individuals and groups in the current frame's pedestrian and group queues by frame, and control the movement of the camera by frame, store the video data including pedestrians and photographers shot by cameras of different viewpoints; store the type, start frame, and end frame information of all group evolutions in the video; The initial positions of the individuals and groups are randomly generated in a circular area, and the initial positions of each individual and the central figure of the group are calculated according to the polar coordinate transformation by generating a radius length between [inner diameter, outer diameter] and a rotation angle between [0°, 360°), and an initial orientation is generated between [0°, 360°), thereby completing the operation of initializing the positions of each individual and the central figure of the group; According to the total number of people in the group, the inner circle and the outer circle are divided. The pedestrians are randomly rotated by an angle θ between [0°, 360°) with the center person of the group as the center. Then (x+r*cosθ,z+r*sinθ) is the initial position of each pedestrian, where (x,z) is the initial position of the center person of the group. The initial orientation of each person in the group is consistent with the initial orientation of the center person of the group. The multi-view camera includes a top view camera and a horizontal view camera.
2. The method for generating synthetic data for complementary perspective multi-person relationship perception according to claim 1, characterized in that: If the position of the character's annotation box, the character's ID, and the ID information of the group to which the character belongs in each frame of the video are generated, it is necessary to store the position, orientation, and behavior information of all the characters in each frame.
3. The method for generating synthetic data for complementary perspective multi-person relationship perception according to claim 1, characterized in that: The created pedestrian class (Person), group class (Group) and evolution class (Activity) are used to store the state information after the group evolution function runs for each frame.
4. The method for generating synthetic data for complementary perspective multi-person relationship perception according to claim 1, characterized in that: The created pedestrian class and group class queues are used to store individuals and groups that do not participate in the evolution in the current frame.
5. The method for generating synthetic data for complementary perspective multi-person relationship perception according to claim 1, characterized in that: The benchmark moving speed of the character model is: The maximum displacement speed is: Δv = 0.05 * height of the character model The movement speed of each individual and group is:
6. The method for generating synthetic data for complementary perspective multi-person relationship perception according to claim 1, characterized in that: The rotation speed of each individual and the group is:
7. The method for generating synthetic data for complementary perspective multi-person relationship perception according to claim 1, characterized in that: The frame-by-frame execution of the population evolution function is specifically: (1) Joining evolution: multiple individuals join the generated group, one group is selected to stop moving and all turn to the center to talk, then a certain number of pedestrians are randomly selected, and their respective rotation angles and positions are determined according to their current positions and they join the group to talk. After all pedestrians have joined, all people in the group stop talking and start moving together; (2) Leaving and disbanding evolution: multiple individuals leave the group they originally participated in. When type = 0, the group participating in the evolution will randomly select some individuals to leave, and these individuals will move away from the center of the group while following the group. When type = 1, all people in the group participating in the evolution will first stop moving and turn to the center to talk. Then, some individuals who leave will be randomly selected to leave in the direction away from the center of the group after a random period of time. After another random period of time, all the remaining people in the group will stop talking and start moving together. Dissolution evolution means that a group dissolves into multiple individuals in a short period of time. The process implemented by the type variable in the evolution class is basically the same as the departure evolution. (3) Aggregation evolution: multiple individuals gather in a short period of time to form a group. The distance between individuals is calculated. If the distance is less than a threshold, the individual will continue to move until the distance is greater than the threshold. After that, the two individuals will move to the current center position. Then a certain number of pedestrians will be randomly selected to move to the center position. People who arrive at the target position will start talking until everyone joins the group. After a random period of time, all people in the group stop talking and start moving. (4) Merge evolution: The main group stops moving and all turn to the center to talk. Each of the remaining groups selected calculates the distance from the center of the main group. If the distance is less than the preset threshold 1, the group moves away. Otherwise, the group moves toward the center until the distance from the center reaches the preset threshold. At this time, the random arrival position of each person in the group is calculated, and the group is disbanded. Each person moves to their respective arrival position. After arriving at the position, if the person is not facing the center, he needs to rotate toward the center and talk. After all the people in the remaining groups join the main group, all people in the group stop talking for a random period of time and start moving; (5) Separation evolution: a group is separated into multiple groups. First, a group is selected. The group stops moving and all turn to the center to talk. Then, the number of groups separated and the number of people in each group are randomly generated, and each group is assigned a random starting frame and gathering position for separation. When the current frame = the separation starting frame of the group, all people in the group leave the original group and gather at the location of the new group to talk. After all people in the group gather, all people in the group stop talking and start moving for a random period of time. When all groups have completed separation, the separation evolution ends.
8. The method for generating synthetic data for complementary perspective multi-person relationship perception according to claim 1, characterized in that: The method further comprises: Stores the single-person black background video data of all characters shot by cameras at different viewing angles; The rectangular annotation box positions, character IDs, and group IDs of all people in each frame of each perspective video are calculated based on all the single-person pure black background videos; the annotation boxes of severely obscured people in each frame of each perspective video are removed.
