A dynamic 3D crowd reconstruction method for large scene videos

Abstract

The present invention discloses a dynamic three-dimensional crowd reconstruction method for large-scene videos, and relates to the field of three-dimensional vision technology. Compared with the previous large-scene static multi-person reconstruction method, the present invention solves the problem of posture feature loss caused by dynamic occlusion; based on a top-down multi-person reconstruction framework, the robust three-dimensional position, posture and shape of the crowd are obtained by iterative optimization, and a crowd grouping optimization paradigm is designed to cluster individuals with similar motion trajectories, and the unoccluded sequence is used to adaptively repair the occluded sequence through the unoccluded score of the character's motion, and an asynchronous motion consistency loss is designed for joint optimization, and the time warping technology is used to find the best alignment between posture sequences to solve the motion loss problem caused by dynamic occlusion; the human motion prior model is trained by using a variational autoencoder, which effectively enhances the realism and smoothness of the motion.

Description

Technical Field

[0001] The invention belongs to the field of three-dimensional vision technology and relates to a dynamic three-dimensional crowd reconstruction method for large-scene videos. Background Art

[0002] In a large 3D scene, the reconstruction of the three-dimensional position, posture and shape of dynamic crowds is of great significance to public safety, emergency management, sports and other fields. By accurately reconstructing dynamic crowds, crowd behavior can be monitored and analyzed to identify potential security threats and abnormal situations, or to perform accident replay. In addition, in sports events and large-scale activities, this technology can also provide accurate analysis of on-field behavior or audience behavior, helping to improve sports strategies and optimize on-field arrangements.

[0003] Although various methods have been developed to reconstruct multi-person poses and shapes from small and medium-sized scene videos, they are difficult to cope with dynamic human reconstruction in large scenes due to the variation of individual scales and differentiated camera spaces. Specifically, the existing techniques first track the local position of each person in the image and then perform single-person dynamic reconstruction, but this method relies on the weak perspective projection assumption and discards the key position information of the individual. Or an end-to-end method is used to estimate the SMPL model, but it is difficult to apply to large scenes because scaling large scene images to the network input resolution will cause the loss of most small and medium-sized individuals. Crowd3D and GroupRec are two state-of-the-art methods for large-scene static multi-person reconstruction, which can predict the 3D position, pose and shape of hundreds of individuals in a unified camera coordinate system from a single large scene image. However, when applied to each frame of a large scene video, these methods usually produce temporally unstable and non-smooth 3D motion, and often lose objects due to severe occlusion or complete occlusion. In large scene videos, the density of moving individuals is high and occlusion occurs frequently, resulting in spatiotemporal discontinuity, which seriously affects the accuracy of dynamic human reconstruction in large scenes.

[0004] In order to solve the above problems, the present invention proposes a dynamic three-dimensional crowd reconstruction method for large-scene videos; a framework of three-dimensional position, posture and shape of dynamic crowds is reconstructed from large-scene videos, which solves the loss of posture features caused by dynamic occlusion; individuals with similar motion trajectories are clustered by designing a crowd grouping optimization paradigm, and low-quality sequences are adaptively repaired using high-quality unoccluded sequences, which solves the problem of motion loss caused by dynamic occlusion; an asynchronous motion consistency loss is designed, and the time warping technology is used to find the best alignment between posture sequences, so as to achieve an occluded sequence guided by aligned high-quality posture sequences; a human motion prior model is trained by using a variational autoencoder (VAE), which effectively enhances the realism and smoothness of the motion; and a new synthetic dataset VirtualCrowd is created to verify and advance research in the field of large-scale dynamic crowd reconstruction. Summary of the invention

[0005] The purpose of the present invention is to propose a dynamic three-dimensional crowd reconstruction method for large scene videos to solve the problems raised in the background technology:

[0006] Existing methods are unable to obtain dynamic 3D reconstruction results of crowds in large scenes that are coherent, realistic, robust to occlusion, and harmoniously interact with the ground.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A dynamic three-dimensional crowd reconstruction method for large scene videos includes the following steps:

[0009] S1, dividing the large scene image into several image blocks by an adaptive cropping method, and scaling the image blocks to a uniform size while maintaining the original aspect ratio of the image;

[0010] S2, based on the image blocks obtained after the processing in S1, detect the two-dimensional information, obtain the two-dimensional information of all characters in the large scene image through merging and deduplication operations, and select several two-dimensional key points of the characters to estimate the camera parameters and ground equations;

[0011] S3, based on the two-dimensional information and camera parameters of all characters obtained in S2, estimate the initialization parameter model SMPL of each character in the large scene and the virtual interaction point of the three-dimensional human scene, and then obtain the motion trajectory of each character through the matching tracking method combining detection and prediction, and use the two-dimensional key points of the characters to preliminarily optimize the initialization SMPL model;

[0012] S4, based on the preliminary optimization results of S3, use the human posture prior optimizer to optimize the position and posture of the SMPL model of each frame;

[0013] S5. Use variational autoencoders to train a human motion prior model, consider the posture changes of people during movement, and further optimize the SMPL model of each frame;

[0014] S6. Design a crowd grouping optimization paradigm to divide the crowd in a large scene into several groups according to their trajectories, introduce asynchronous motion consistency loss for constraints, use the unoccluded sequence to guide the occluded sequence, and finally obtain the crowd dynamic reconstruction result.

[0015] Preferably, the S1 is as follows:

[0016] S101, according to the principle that characters in large scenes are larger when they are closer and smaller when they are farther away, the image block size is set in proportion to the size of the characters in the vertical direction;

[0017] S102, unify the low-resolution image to (n,n) by bilinear interpolation, and fill the insufficient part with 0.

[0018] Preferably, the step of selecting a number of two-dimensional key points of a person to estimate camera parameters and ground equations in S2 is as follows:

[0019]

[0020] z' s ×p' s =K(z a ×K -1 p a +h×N)

[0021] Where K represents the camera parameter matrix; N represents the ground normal; D represents the constant term of the ground equation; L cos is the cosine distance; λ angle and λ mod is the weight of the corresponding loss term; p s and p a They are the center points of the two shoulders and the center points of the two ankles of the two-dimensional key points of the human body; p' s is the predicted shoulder center estimated using the camera parameters and the ground equation; ||·||2 is the second norm; z a is the depth of the ankle center; z' s is the predicted depth of the center of the shoulder; h is the assumed height of the person.

[0022] Preferably, S3 is specifically as follows:

[0023] S301, estimating the initialization parameter model SMPL model and the three-dimensional human-scene virtual interaction point in the local coordinate system according to the image blocks and the two-dimensional information of all characters;

[0024] S302, performing 3D reconstruction by combining bottom-up detection with top-down prediction, providing reconstructed 3D position information according to 3D human-scene virtual interaction points, and matching the posture, position and appearance of adjacent persons in space across time;

[0025] S303: Preliminarily optimize the posture and motion of the SMPL model using the human body motion trajectory information and the detected two-dimensional key points.

[0026] Preferably, the S4 is specifically as follows:

[0027] S401, using a human posture priori optimizer to perform root optimization, and adjusting the rotation matrix and translation matrix of the root node of the initially optimized SMPL model;

[0028] S402, using a human posture prior optimizer to perform SMPL optimization, and further adjust the rotation matrix of each joint point of the SMPL model.

[0029] Preferably, in S5, a human motion prior model is trained through a variational autoencoder (VAE) structure of an encoder-decoder structure; during optimization, latent variable encoding is extracted from the motion, the latent variable encoding is used as the optimization variable, and then decoded to obtain the optimized SMPL model.

[0030] Preferably, the S6 is specifically as follows:

[0031] S601. Design a new paradigm for crowd grouping, divide the movement sequence of the crowd in the global space into several segments, and then perform cluster analysis on the movement sequence to divide the crowd in the large scene into several groups with similar movement patterns;

[0032] S602, according to the two-dimensional key point detection confidence and joint importance, the unobstructed score of each character movement is calculated, and the occluded sequence and unobstructed sequence to be repaired are adaptively selected, as well as the corresponding optimization weight of the unobstructed sequence;

[0033] S603, introducing asynchronous motion consistency loss for joint optimization, using time warping to align pose sequences, so that the unoccluded sequence guides the reconstruction of the occluded sequence.

[0034] Compared with the prior art, the present invention provides a dynamic three-dimensional crowd reconstruction method for large-scene videos, which has the following beneficial effects:

[0035] The present invention reconstructs the framework of the three-dimensional position, posture and shape of dynamic crowds from large-scene videos. By inputting dynamic large-scene monocular videos, it is possible to obtain crowd dynamic reconstruction results that are coherent, realistic, robust to occlusion, and harmonious with the interaction with the ground in large scenes. By designing a crowd grouping optimization paradigm to cluster individuals with similar motion trajectories, and adaptively using high-quality unoccluded sequences to repair low-quality sequences, the motion loss problem caused by dynamic occlusion is solved. An asynchronous motion consistency loss is designed to use time warping technology to find the optimal alignment between posture sequences, and an occluded sequence guided by aligned high-quality posture sequences is achieved. At the same time, a human motion prior model is trained using a variational autoencoder, which effectively enhances the realism and smoothness of the motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a multi-person reconstruction framework diagram proposed in Example 1 of the present invention;

[0037] Figure 2 This is a diagram showing a virtual data set proposed in Example 2 of the present invention;

[0038] Figure 3 This is a schematic diagram of the reconstruction result in a real scene proposed in Example 2 of the present invention;

[0039] Figure 4 This is a schematic diagram of the reconstruction result in a virtual scene proposed in Example 2 of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Embodiment 1:

[0042] See also Figure 1 The present invention proposes a dynamic three-dimensional crowd reconstruction method for large scene videos, comprising the following steps:

[0043] S1. The large scene image is divided into several image blocks by an adaptive cropping method, and the image blocks are scaled to a uniform size for subsequent processing while maintaining the original aspect ratio of the image, as follows:

[0044] S101, according to the principle that people in large scenes appear larger when they are closer and smaller when they are farther away, the image block size is set in proportion to the size of the people in the vertical direction, so that the proportion of the people in the image block always remains approximately 1 / 2;

[0045] S102, unify the low-resolution image to (512, 512) by bilinear interpolation, and fill the insufficient part with 0.

[0046] S2. Detecting two-dimensional information based on the image blocks obtained after processing in S1, obtaining two-dimensional information of all characters in the large scene image by merging and deduplication operations, and selecting two-dimensional key points of some standing or walking characters to estimate camera parameters and ground equations, wherein the method for estimating camera parameters and ground equations is as follows:

[0047]

[0048] z' s ×p' s =K(z a ×K -1 p a +h×N)

[0049] Where K represents the camera parameter matrix; N represents the ground normal; D represents the constant term of the ground equation; L cos is the cosine distance; λ angle and λ mod is the weight of the corresponding loss term; p s and p a They are the center points of the two shoulders and the center points of the two ankles of the two-dimensional key points of the human body; p' s is the predicted shoulder center estimated using the camera parameters and the ground equation; ||·||2 is the second norm; z a is the depth of the ankle center; z' s is the predicted depth of the center point of the shoulder; h is the assumed height of the person, which is set to 1.7m here.

[0050] S3, based on the two-dimensional information and camera parameters of all characters obtained in S2, estimate the initialization parameterized model SMPL of each character in the large scene and the virtual interaction point of the three-dimensional human scene, and then obtain the motion trajectory of each character through the matching tracking method combining detection and prediction, and use the two-dimensional key points of the characters to preliminarily optimize the initialization SMPL model. The specific process is as follows:

[0051] S301, estimating the initialization parameter model SMPL model and the three-dimensional human-scene virtual interaction point in the local coordinate system according to the image blocks and the two-dimensional information of all characters;

[0052] S302, performing 3D reconstruction by combining bottom-up detection with top-down prediction, providing reconstructed 3D position information according to 3D human-scene virtual interaction points, matching the posture, position and appearance of adjacent people in space across time, and efficiently and quickly tracking the human body motion trajectory;

[0053] S303: Preliminarily optimize the posture and motion of the SMPL model using the human body motion trajectory information and the detected two-dimensional key points.

[0054] S4. Based on the preliminary optimization results of S3, the human posture prior optimizer is used to optimize the position and posture of the SMPL model of each frame, so as to obtain a more detailed and natural posture and improve the performance of this method in processing complex postures. The specific implementation process is as follows:

[0055] S401, using a human posture priori optimizer to perform root optimization, and adjusting the rotation matrix and translation matrix of the root node of the initially optimized SMPL model;

[0056] S402, using a human posture prior optimizer to perform SMPL optimization, further adjusting the rotation matrix of each joint point of the SMPL model, so that the character posture is more detailed and natural.

[0057] S5. Use variational autoencoders to train a human motion prior model, consider the posture changes of people in motion, and further optimize the SMPL model of each frame to effectively enhance the realism and smoothness of human motion sequences in dynamic scenes. The specific method of using the human motion prior model is as follows:

[0058] The AMASS dataset is used to train a human motion prior model with a variational autoencoder (VAE) structure of an encoder-decoder structure. During optimization, latent variable encoding is extracted from the motion, and the latent variable encoding is used as the optimization variable, and then decoded to obtain the optimized SMPL model, which not only enhances the motion smoothness and ground interaction harmony, but also can restore the correct motion from short-term occlusion through occlusion data enhancement.

[0059] S6. Design a crowd grouping optimization paradigm to divide the crowd in a large scene into several groups according to their trajectories, introduce asynchronous motion consistency loss for constraints, use the unoccluded sequence to guide the occluded sequence, and finally obtain a crowd dynamic reconstruction result that is coherent, realistic, occluded, and harmonious with the ground. The specific implementation process is as follows:

[0060] S601. Design a new crowd grouping paradigm, divide the motion sequence of the crowd in the global space into a segment every 64 frames, and then perform cluster analysis on the motion sequence to divide the crowd in the large scene into several groups with similar motion patterns;

[0061] S602, according to the two-dimensional key point detection confidence and joint importance, the unobstructed score of each character movement is calculated, and the occluded sequence and unobstructed sequence to be repaired are adaptively selected, as well as the corresponding optimization weight of the unobstructed sequence;

[0062] S603: Introduce asynchronous motion consistency loss for joint optimization, use time warping to perform optimal alignment between pose sequences, so that the unoccluded sequence guides the reconstruction of the occluded sequence.

[0063] Embodiment 2:

[0064] See also Figure 1-Figure 4 The specific implementation process of the dynamic 3D crowd reconstruction method for large scene videos described in Example 1 is as follows:

[0065] (I) Data preprocessing:

[0066] First, the large scene image is adaptively cropped and divided into blocks centered on people and unified to a resolution of 512×512. Then, the VitDet 2D detection method is used for each image block to obtain the bounding box and mask sequence of all visible subjects. The most advanced method DWPose is used to estimate the initial 2D key point information of each bounding box, and unrealistic postures are initially filtered out to obtain feasible 2D key points and their corresponding bounding boxes and masks. Some standing or walking people are selected as prior information of human posture to calibrate the ground plane equation and scene-level camera parameters.

[0067] (II) Tracking and crowd initialization process:

[0068] Since it is necessary to reconstruct multi-person motion videos in large scenes, cross-time correlation tracking of characters is essential. This embodiment introduces a three-dimensional virtual position representation - a three-dimensional human-scene virtual interaction point, that is, the projection point of the character's center of gravity on the ground, which provides more stable 3D position information and significantly improves matching accuracy; at the same time, in order to avoid the significant waste of computing resources caused by global matching, spatially adjacent people are matched based on a more stable position representation, which significantly improves the efficiency and practicality of the algorithm;

[0069] (III) Population grouping optimization paradigm:

[0070] A crowd grouping optimization paradigm is designed to use collective intra-group motion to allow unobstructed, high-quality motion to help restore the motion of occluded individuals: first, all human motion sequences are divided into segments of size 64 frames, and individuals with similar motion trajectories are clustered. The unobstructed score of each person's motion is calculated based on detection confidence and joint importance. It is adaptively determined which sequences need to be repaired, and high-quality unobstructed sequences and their corresponding optimization weights are identified. For sequences that need to be repaired, their own high-quality unobstructed sequence segments are preferentially used for optimization, and then the intra-group high-quality sequences with similar motion trajectories are used for guidance. If neither exists, a pre-set template is used for guidance. Then, an asynchronous motion consistency loss is introduced, and temporal warping is used to find the best alignment between pose sequences, thereby achieving an occluded sequence guided by the aligned high-quality pose sequence.

[0071] (IV) VirtualCrowd synthetic dataset:

[0072] The present invention uses ICity plug-in to create scenes, then obtains the ground of the scene, generates the motion trajectory of people based on the ground, generates motion sequences using the DIMOS method, and then renders through Blender to obtain a synthetic large-scene crowd motion video dataset. The dataset contains 4 scenes in total, and each scene produces motion sequence videos containing hundreds of people on average, with 200 frames; the specific results can be referred to Figure 2 .

[0073] Reference Figure 3 , character reconstruction is performed in real scenes. The reconstruction results not only match the input perspective with the character's movements, but also ensure the correct relative position in the world coordinate system; refer to Figure 4 ,By reconstructing characters in virtual scenes, it can be seen that a very smooth motion trajectory can be obtained even in the case of occlusion, and the obtained motion sequence has a high sense of reality and smoothness.

[0074] The quantitative results of the reconstruction method of Example 1 are compared with those of the current mainstream large-scene 3D reconstruction methods Crowd3D and GroupRec on the VirtualCrowd dataset. The specific comparison results can be found in Table 1:

[0075] Table 1 Comparison of the quantitative results of the reconstruction method of the present invention with those of Crowd3D and GroupRec

[0076] method PPDS PA-PPDS PCOD MPJPE PA-MPJPE WA-MPJPE W-MPJPE ACCEL Crowd3D 85.27 90.85 92.98 124.02 72.11 - - - GroupRec 74.76 75.97 87.28 95.14 55.74 63.58 76.12 124.09 The method of this embodiment 84.58 92.24 94.35 63.57 41.70 48.28 63.83 13.41

[0077] As can be seen from the table, there are eight evaluation indicators for quantitative results, namely PPDS, PA-PPDS, PCOD, MPJPE, PA-MPJPE, WA-MPJPE, W-MPJPE, and ACCEL. The higher the values ​​of the first three indicators, the better, and the lower the values ​​of the last five indicators, the better. Among them, PPDS is the pairwise percentage distance similarity, which is used to measure the relative position of the characters; PA-PPDS is the proportional percentage distance similarity using Procrustes alignment, excluding the influence of scale and rotation; PCOD is the percentage of correct ordinal depth, which is used to measure the ordinal depth relationship between all pairs of people in the image; MPJPE is the average joint position error, which is used to measure the accuracy of joint reconstruction; PA-MPJPE is the average joint position error after Procrustes alignment; WA-MPJPE is the MPJPE after the motion joint sequence is aligned according to the trajectory, which is used to measure the dynamic alignment of the motion sequence; W-MPJPE is the MPJPE after the first frame of the motion joint sequence is aligned; ACCEL is the joint acceleration, which is used to measure the smoothness of the motion. It can be seen that although the reconstruction result of the present invention is slightly lower than that of Crowd3D in terms of PPDS, it is much better than other methods in terms of other indicators. Therefore, the reconstruction result of the present invention achieves the best result on the VirtualCrowd dataset.

[0078] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and improved concepts of the present invention within the technical scope disclosed by the present invention, and they should be covered by the protection scope of the present invention.

Claims

1. A dynamic three-dimensional crowd reconstruction method for large scene videos, characterized in that: The steps include: S1, dividing the large scene image into several image blocks by an adaptive cropping method, and scaling the image blocks to a uniform size while maintaining the original aspect ratio of the image; S2, based on the image blocks obtained after the processing in S1, detect the two-dimensional information, obtain the two-dimensional information of all characters in the large scene image through merging and deduplication operations, and select several two-dimensional key points of the characters to estimate the camera parameters and ground equations; S3, based on the two-dimensional information and camera parameters of all characters obtained in S2, estimate the initialization parameter model SMPL of each character in the large scene and the virtual interaction point of the three-dimensional human scene, and then obtain the motion trajectory of each character through the matching tracking method combining detection and prediction, and use the two-dimensional key points of the characters to preliminarily optimize the initialization SMPL model; S4, based on the preliminary optimization results of S3, use the human posture prior optimizer to optimize the position and posture of the SMPL model of each frame; S5. Use variational autoencoders to train a human motion prior model, consider the posture changes of people during movement, and further optimize the SMPL model of each frame; S6. Design a crowd grouping optimization paradigm to divide the crowd in a large scene into several groups according to their trajectories, introduce asynchronous motion consistency loss for constraints, use the unoccluded sequence to guide the occluded sequence, and finally obtain the crowd dynamic reconstruction result.

2. According to claim 1, a method for dynamic three-dimensional crowd reconstruction for large scene videos is characterized in that: The S1 is specifically as follows: S101, according to the principle that characters in large scenes are larger when they are closer and smaller when they are farther away, the image block size is set in proportion to the size of the characters in the vertical direction; S102, unify the low-resolution image to (n,n) by bilinear interpolation, and fill the insufficient part with 0.

3. The method for dynamic three-dimensional crowd reconstruction for large scene videos according to claim 1, characterized in that: The steps of selecting several two-dimensional key points of characters in S2 to estimate camera parameters and ground equations are as follows: With' s ×p' s =K(z a ×K -1 p a +h×N) Where K represents the camera parameter matrix; N represents the ground normal; D represents the constant term of the ground equation; L cos is the cosine distance; λ angle and λ mod is the weight of the corresponding loss term; p s and p a They are the center points of the two shoulders and the center points of the two ankles of the two-dimensional key points of the human body; p' s is the predicted shoulder center estimated using the camera parameters and the ground equation; ||·||2 is the second norm; z a is the depth of the ankle center; z' s is the predicted depth of the center point of the shoulder; h is the preset character height.

4. The method for dynamic three-dimensional crowd reconstruction for large scene videos according to claim 1, characterized in that: The S3 is as follows: S301, estimating the initialization parameter model SMPL model and the three-dimensional human-scene virtual interaction point in the local coordinate system according to the image blocks and the two-dimensional information of all characters; S302, performing 3D reconstruction by combining bottom-up detection with top-down prediction, providing reconstructed 3D position information according to 3D human-scene virtual interaction points, and achieving matching of posture, position and appearance of adjacent persons in space across time; S303: Preliminarily optimize the posture and motion of the SMPL model using the human body motion trajectory information and the detected two-dimensional key points.

5. The method for dynamic 3D crowd reconstruction for large scene videos according to claim 4, characterized in that: The S4 is specifically as follows: S401, using a human posture priori optimizer to perform root optimization, and adjusting the rotation matrix and translation matrix of the root node of the initially optimized SMPL model; S402, using a human posture prior optimizer to perform SMPL optimization, and further adjust the rotation matrix of each joint point of the SMPL model.

6. The method for dynamic 3D crowd reconstruction for large scene videos according to claim 5, characterized in that: In S5, a human motion prior model is trained through a variational autoencoder structure of an encoder-decoder structure; during optimization, latent variable encoding is extracted from the motion, the latent variable encoding is used as the optimization variable, and then decoded to obtain the optimized SMPL model.

7. The method for dynamic three-dimensional crowd reconstruction for large scene videos according to claim 1, characterized in that: The S6 is specifically as follows: S601. Design a new paradigm for crowd grouping, divide the movement sequence of the crowd in the global space into several segments, and then perform cluster analysis on the movement sequence to divide the crowd in the large scene into several groups with similar movement patterns; S602, according to the two-dimensional key point detection confidence and joint importance, the unobstructed score of each character movement is calculated, and the occluded sequence and unobstructed sequence to be repaired are adaptively selected, as well as the corresponding optimization weight of the unobstructed sequence; S603, introducing asynchronous motion consistency loss for joint optimization, using time warping to align pose sequences, so that the unoccluded sequence guides the reconstruction of the occluded sequence.

Images