Multi-view multi-person three-dimensional human posture estimation and tracking method, system and device

Abstract

The present application relates to a kind of multi-view multi-person three-dimensional human posture estimation and tracking method, system and equipment, the method includes the following steps: using two-dimensional human posture estimation network estimates the two-dimensional human key point of each person in each view;Based on human symmetry hollow network and joint symmetry hollow network, using plane scanning stereo algorithm, the two-dimensional human key point estimated is carried out depth regression;Multiple three-dimensional skeletons formed by the plane scanning stereo of multiple views are gathered to form skeleton pool;Skeleton clustering algorithm is used to cluster skeleton pool and construct skeleton cluster;Skeleton cluster tracking algorithm is used, and each three-dimensional human posture is tracked using the skeleton cluster constructed.The application can realize three-dimensional human posture estimation and tracking, and improve the robustness of multi-view multi-person three-dimensional human posture estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a multi-view multi-person three-dimensional human pose estimation and tracking method, system and device. BACKGROUND

[0002] In recent years, with the continuous advancement of researchers on computer vision tasks, two-dimensional human pose estimation has been very excellent and available work, such as HRNet (Deep high-resolution representation learning for human pose estimation, Proceedings of the IEEE conference on computer vision and pattern recognition. 2019: 5693-5703) and Openpose (Realtime multi-person 2d pose estimation using part affinity fields, Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 7291-7299). Based on the excellent two-dimensional human pose estimation work, it is possible to estimate three-dimensional human pose.

[0003] In recent years, the research direction has appeared monocular single-person three-dimensional human pose estimation and monocular multi-person three-dimensional human pose estimation. The lack of depth information of monocular and the inaccurate 2D human pose caused by the self-occlusion and mutual occlusion of human body make the monocular three-dimensional human pose estimation research a challenging research. With the continuous in-depth research, researchers have proposed a multi-view multi-person three-dimensional human pose estimation task, but the multi-view multi-person three-dimensional human pose estimation task lacks stable tracking of human body, so it cannot be applied to the fields of human animation generation, motion analysis, behavior recognition, virtual human reconstruction and other frontier research fields as a markerless motion capture system.

[0004] Meanwhile, by analyzing the characteristics of 2D multi-person pose estimation, the following main problems are found: (1) the existing 2D human pose estimation algorithm has a lower bound of theoretical error due to the use of heat maps. (2) The existing 2D human pose estimation has occasional incorrect estimation on continuous video sequences. (3) When the limbs or people in the view angle are occluded, due to insufficient information, low-quality 2D human pose is generated. (4) Although the 2D human pose estimation produces inaccurate 2D human pose, the number of estimated human bodies is relatively stable, and can generally reflect the number of people in the space.

[0005] The existing representative works generally have the problems of low robustness and high computational cost. ZJUPose (Fast and robust multi-person 3d pose estimation and tracking from multiple views, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021) uses the multi-view geometry of 3DPS, the loop consistency constraint and the Re-ID module to perform robust multi-view multi-person three-dimensional human pose estimation. VoxelTrack (Voxeltrack: Multi-person 3d human pose estimation and tracking in the wild, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022) uses 3D CNN to solve the matching problem of people in 3D space, and uses ReID network, so that the overall computational complexity of the network is high. Iterative3DPose (Iterative greedy matching for 3d human pose tracking from multiple views, German Conference on Pattern Recognition. Springer, Cham, 2019: 537-550) uses Iterative greedy matching to perform multi-view three-dimensional human pose estimation. PlaneSweep (Multi-view multi-person 3d pose estimation with plane sweep stereo, Proceedings of the IEEE / CVF Conference On Computer Vision and Pattern Recognition. 2021: 11886-11895) is different from the previous work, which skips the matching step between people, and uses Plane Sweep to perform three-dimensional human pose estimation, which is very innovative, but because it skips the matching step between people, it will produce an incorrect three-dimensional skeleton.

[0006] In summary, the current study can effectively resist two-dimensional estimation error, and the multi-view multi-person three-dimensional human posture estimation and tracking method with strong robustness is particularly important. Therefore, a solution is needed to improve the robustness of the multi-view multi-person three-dimensional human posture estimation method. SUMMARY

[0007] To solve the above problems, the purpose of the present application is to provide a multi-view multi-person three-dimensional human posture estimation and tracking method, system and device, which can realize three-dimensional human posture estimation and tracking and improve the robustness of multi-view multi-person three-dimensional human posture estimation.

[0008] To achieve the above purpose, the present application adopts the following technical solution: a multi-view multi-person three-dimensional human posture estimation and tracking method, comprising: estimating two-dimensional human key points of each person in each view by using a two-dimensional human posture estimation network; based on a human symmetric hollow network and a joint symmetric hollow network, using a plane scanning stereo algorithm to regress the depth of the estimated two-dimensional human key points; aggregating a plurality of three-dimensional skeletons formed by plane scanning stereo of multiple views to form a skeleton pool; constructing a skeleton cluster after clustering the skeleton pool using a skeleton clustering algorithm; and tracking each three-dimensional human posture by using the constructed skeleton cluster using a skeleton cluster tracking algorithm.

[0009] Further, the two-dimensional human key points include a human center point and a joint point; the plane scanning stereo algorithm is used to regress the depth of the estimated two-dimensional human key points based on the human symmetric hollow network and the joint symmetric hollow network, specifically: adding the absolute depth value of the human center point obtained by the human symmetric hollow network to the relative depth value of each human joint point in the result obtained by the joint symmetric hollow network to obtain the spatial depth of the two-dimensional skeleton of each view.

[0010] Further, the multiple three-dimensional skeletons formed by plane scanning stereo of multiple views are aggregated to form a skeleton pool, which comprises:

[0011] The skeleton pool is obtained by combining the back projection of each two-dimensional skeleton and the depth value in all views; the i-th three-dimensional skeleton in the skeleton pool corresponds to a two-dimensional posture on the v-th view The two-dimensional posture h i，v The index of the two-dimensional skeleton of the i-th three-dimensional skeleton on the v-th view is represented as:

[0012]

[0013] Where q i，v，j represents the projection two-dimensional posture of the i-th three-dimensional skeleton on the v-th view obtained from the two-dimensional posture estimation, {p v，1 ..., p v，N} represents a two-dimensional skeleton on the vth view, n i as the index of the i th three-dimensional skeleton on the two-dimensional skeleton.

[0014] Further, the skeleton cluster algorithm is used to cluster the skeleton pool to construct a skeleton cluster, specifically including:

[0015] According to the skeleton pool The information of the two-dimensional skeleton that best matches each three-dimensional skeleton in each view is obtained from the I skeletons in the skeleton pool

[0016] After removing the repeated part, L classes are obtained, thereby obtaining a two-dimensional-three-dimensional matching class matrix and a matching two-dimensional skeleton confidence value matrix

[0017] By projecting the skeleton pool According to the two-dimensional-three-dimensional matching class matrix H, the skeleton cluster is divided into L classes, and the skeleton cluster clustering information vector b is obtained.

[0018] Further, the skeleton cluster tracking algorithm is used to track each three-dimensional human pose using the constructed skeleton cluster, including:

[0019] From the skeleton cluster information vector b, the two-dimensional-three-dimensional matching class matrix H, and the skeleton pool The skeleton cluster joint score matrix is obtained The score of the j th joint of the i th three-dimensional skeleton is :

[0020]

[0021] where q i，v represents the two-dimensional skeleton after projecting the i th skeleton in the skeleton pool P at the v th view, represents the two-dimensional skeleton matched at the v th view using the skeleton cluster information vector b i in the two-dimensional-three-dimensional matching class matrix H, and σ represents a hyperparameter that controls the width of the curve, represents the peak score of the two-dimensional human pose estimation network heatmap, i.e., the confidence of the two-dimensional joint, and ⊙ represents the Hadamard product.

[0022] For a given l th class in L classes, select the joint score of the i th three-dimensional skeleton in the l th class, and the index of the maximum score of each joint in the three-dimensional skeleton in the l th class as the high-quality skeleton cluster joint index vector Select the vector b of all indices i corresponding to all three-dimensional skeletons in the l th class in the skeleton pool lThus, a high-quality skeleton cluster matrix of class I is obtained is expressed as:

[0023]

[0024] Further, the skeleton cluster tracking algorithm is used to track each three-dimensional human posture by using the constructed skeleton cluster, and includes:

[0025] Input parameters of the skeleton cluster tracking algorithm are obtained:

[0026] The input parameters include a high-quality skeleton cluster C t at time t t-1 , a tracked skeleton T t-1 at time t-1 t , a tracking ID list Z t at time t-1 t-1 , a tracking number threshold value γ t at time t t-1 , a new person ID ξ t at time t-1 t-1 , and initialization of a tracking index list Θ, a new person tracking index list Θ', and a tracking distance list Δ.

[0027] Initialization of the skeleton cluster tracking algorithm is performed:

[0028] When the tracked skeleton result T t-1 at time t-1 t is empty, that is, there is no person in the scene in the first frame or the previous frame, indexes Ψ in the high-quality skeleton cluster C t are directly assigned to the tracking index list Θ and the new person tracking index list Θ' for initialization.

[0029] Tracking is performed by using the skeleton cluster tracking algorithm:

[0030] When there is a skeleton result detected in the previous frame, it is assumed that there is a pair of matched skeletons <τ t-1 , c t > and τ t-1 ∈ T t-1 and c t ∈ C t is the closest one among all the matched pairs, θ' is set as the index of c t in C t ; if the shortest index θ' is in the existing tracking index list Θ, it means that a person has left the space shot by multiple cameras, which will lead to the tracking ID of the skeleton being merged to another skeleton still in the scene; when there are multiple IDs on a skeleton, the ID corresponding to the smallest distance in the list Δ is selected in Z t-1 ; when the number of elements in Θ is less than γ t , it is assumed that a new person appears in the scene, and Θ' not in the current tracking index list Θ is added to Θ.

[0031] ID assignment for executing the skeleton cluster tracing algorithm:

[0032] When a new person appears in a space covered by multiple cameras, the new ID will be assigned to Z based on the number of elements in Θ′. t-1 In, and ξ t It will be updated to the new ID in the next frame;

[0033] Update the tracking ID list and new user list:

[0034] use Obtain the tracking result T at time t. t The list of tracking IDs Z at time t t By Z t -1 Updated, using ξ t-1 The sum of the number of newcomers n and the newcomer ID ξ at time t. t .

[0035] A multi-view, multi-person 3D human pose estimation and tracking system includes: a keypoint estimation module, which estimates the 2D human keypoints of each person in each view using a 2D human pose estimation network; a depth estimation module, which performs depth regression on the estimated 2D human keypoints using a planar scanning stereo algorithm based on a human symmetric hole network and an articulated symmetric hole network; a skeleton pooling module, which aggregates multiple 3D skeletons formed by planar scanning stereo from multiple views to form a skeleton pool; a clustering module, which constructs skeleton clusters by clustering the skeleton pool using a skeleton clustering algorithm; and a tracking module, which tracks the pose of each 3D human using a skeleton cluster tracking algorithm.

[0036] Furthermore, the two-dimensional human body key points include the human body center point and joint points;

[0037] The depth estimation module adds the absolute depth value of the human body center point obtained from the human body symmetric cavity network to the relative depth value of each human body joint point obtained from the joint symmetric cavity network, to obtain the spatial depth of the two-dimensional skeleton of each view.

[0038] A processing device, comprising at least a processor and a memory, wherein the memory stores a computer program, and the processor executes the steps of the above-described multi-view, multi-person, three-dimensional human pose estimation and tracking method when running the computer program.

[0039] A computer storage medium storing computer-readable instructions that can be executed by a processor to implement the steps of the above-described multi-view, multi-person, three-dimensional human pose estimation and tracking method.

[0040] The present application has the following advantages due to the above technical scheme:

[0041] (1) The technical scheme of the present application improves the detection robustness of multi-view multi-person three-dimensional human pose estimation. The present application obtains the depth values of the human body under different viewing angles by using the symmetric hollow network and the plane scanning stereo method, and obtains multiple 3D human skeleton results by back projection, thereby forming a skeleton pool, which greatly improves the robustness of the candidate results.

[0042] (2) The technical scheme of the present application obtains skeleton clusters by using skeleton clustering on the skeleton pool, and screens and tracks the human skeletons in the skeleton pool by using a skeleton cluster tracking algorithm, thereby obtaining stable and robust tracking results.

[0043] (3) The technical scheme of the present application greatly improves the detection rate of multi-view multi-person three-dimensional human pose estimation. In actual industrial applications, the present application can effectively overcome the error amplification problem caused by false 2D human poses, and can be widely applied in various applications such as visible animation generation, motion capture, virtual human reconstruction, etc. BRIEF DESCRIPTION OF DRAWINGS

[0044] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Throughout the drawings, the same reference numerals are used for the same components. In the drawings:

[0045] Figure 1 is the overall flowchart in the embodiment of the present application;

[0046] Figure 2 is the overall network structure diagram of the present application;

[0047] Figure 3 is the skeleton cluster tracking algorithm flowchart of the present application;

[0048] Figure 4 is the detection result schematic diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0049] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in conjunction with the drawings of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the described embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0050] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0051] The present application fully considers the characteristics of two-dimensional human pose estimation and the characteristics of multi-view geometry method, and realizes multi-view three-dimensional human pose estimation based on implicit skeleton cluster tracking, fully overcomes the error amplification and occasional errors in time sequence when the results of 2D pose estimation are used in 3D pose estimation, and still can produce stable and accurate three-dimensional human pose estimation and tracking results when more than half of the limbs or people in the view are seriously occluded. Can be applied to continuous video streams shot by multiple calibrated and time-synchronized cameras.

[0052] The present application proposes a multi-view three-dimensional human pose estimation and tracking method based on skeleton cluster tracking based on plane stereo scanning algorithm, which can well resist the problems of amplification of two-dimensional errors on three-dimensional models and low robustness of multi-view three-dimensional human pose estimation.

[0053] The multi-view three-dimensional human pose estimation and tracking method uses a plane stereo scanning method for depth regression on each two-dimensional human skeleton of each view after obtaining a two-dimensional human pose skeleton, regresses the depth information of each two-dimensional (2D) key point through a human symmetric hollow network and a joint symmetric hollow network, obtains multiple three-dimensional human poses based on the depth information under each view, and forms a skeleton pool. Skeleton clusters are obtained through a skeleton cluster clustering algorithm, and a skeleton cluster tracking algorithm is used on the obtained skeleton clusters, so that the final three-dimensional human pose and ID information are obtained.

[0054] Embodiment 1

[0055] As shown in Figure 1 The overall flowchart in the embodiment of the present application is given. The implementation process of multi-view multi-person three-dimensional human pose estimation and tracking includes three main parts, first constructing a skeleton pool P, then constructing a skeleton cluster through skeleton pool clustering, and finally tracking the skeleton cluster.

[0056] The multi-view multi-person three-dimensional human pose estimation and tracking method includes the following steps:

[0057] Step S1, estimating the two-dimensional human key points of each person in each view by using a two-dimensional human pose estimation network;

[0058] Step S2: Based on the human body symmetric cavity network and the joint symmetric cavity network, the estimated two-dimensional human body key points are subjected to depth regression using a planar scanning stereo algorithm.

[0059] Step S3: Gather multiple 3D skeletons formed by scanning the solid from multiple views to form a skeleton pool;

[0060] Step S4: Use a skeleton clustering algorithm to cluster the skeleton pool and construct skeleton clusters;

[0061] Step S5: Use the skeleton cluster tracking algorithm to track the pose of each three-dimensional human body using the constructed skeleton cluster.

[0062] In step S1, the key points of the human body include the center point of the human body and joint points, wherein the center point of the human body is located at the buttocks.

[0063] like Figure 2 As shown, in step S2, the absolute depth value of the human body center point obtained by the human body symmetric hole network is added to the relative depth value of each human body joint obtained by the joint symmetric hole network to obtain the spatial depth of the two-dimensional skeleton of each view.

[0064] In step S3, the skeleton pool is obtained by combining the back projection of each two-dimensional skeleton with the depth values ​​in all views.

[0065] In step S4, after obtaining the skeleton pool Next, it is necessary to supplement the matching relationship between the 3D skeleton and the 2D human pose on each view. This is to obtain the 3D skeleton h corresponding to the i-th view v. i，v Two-dimensional skeleton index n i The calculation is as follows:

[0066]

[0067] Where, q i，v Let {p} represent the 2D pose of the i-th 3D skeleton projected onto the v-th view from the 2D pose estimation network. v，1 ..., p v，N} represents the two-dimensional skeleton on the v-th view, n i It is the index of the i-th 3D skeleton on the 2D skeleton.

[0068] The i-th 3D skeleton in the skeleton pool corresponds to the 2D pose h on the v-th view. i，v The calculation is as follows:

[0069]

[0070] Thus, the skeleton library was obtained. The I skeletons correspond to the 2D human pose estimation results in V views. That is, the information of the best-matching 2D skeleton for each 3D skeleton in each view. After removing duplicates, we obtain L classes, thus obtaining the 3D-2D matching class matrix. and matching 2D skeleton confidence matrix It is the set of real numbers.

[0071] By skeleton pool Based on the two-dimensional and three-dimensional matching class matrix H, which is divided into L classes, the skeleton cluster clustering information vector is derived.

[0072] In step S5, after obtaining the skeleton cluster clustering information vector b and the skeleton pool... After assembling the skeleton clusters, a joint fraction matrix for the skeleton clusters needs to be constructed in order to perform high-quality tracking. and high-quality skeleton clusters From the skeleton cluster information vector b, the 2D-3D matching class matrix H, and the skeleton pool The joint fraction matrix of the skeleton cluster is obtained. The score of the j-th joint of the i-th 3D skeleton The calculation is as follows.

[0073]

[0074] Where, q i，v This represents the 2D skeleton projected onto the i-th skeleton in the skeleton pool in the v-th view. This indicates the use of b in the 3D-2D matching class matrix H. i The two-dimensional skeleton matched at the v-view, where σ represents the hyperparameter controlling the width of the curve. The peak score represents the peak score of the heatmap of the 2D human pose estimation network, which is the confidence score of the 2D joints. ⊙ represents the Hadamard product.

[0075] Given L classes, for the l-th class, choose The index of the maximum score for each joint in the 3D skeleton of class l will produce a high-quality skeleton group joint index vector. b l It is a skeleton pool The vector corresponding to all indices i of all 3D skeletons of class l. The final high-quality skeleton cluster matrix. According to b l j l and The calculation method for the 3D skeleton of class l is as follows: In obtaining a high-quality skeleton cluster matrix After that, the skeleton cluster tracking process is started.

[0076] As shown in the following figure, the skeleton cluster tracking process is given, including the following steps: Figure 3

[0077] ①Initialize the persistent tracking index list, new person tracking index list, and tracking distance list, specifically:

[0078] Input the high-quality skeleton group at time t, the skeleton tracked at time t-1, the tracking ID list at time t-1, the tracking number threshold at time t, and the new person ID at time t-1 into the algorithm, and initialize the persistent tracking index list, new person tracking index list, and tracking distance list.

[0079] ②Judge whether the tracking result of the previous frame is empty, specifically:

[0080] Judge whether the skeleton tracked at time t-1 is empty. If it is empty, add the high-quality tracking cluster index to the new person tracking index list and the persistent tracking index list. If the skeleton tracked at time t-1 is not empty, calculate the distance between the current high-quality skeleton cluster and the tracked result of the previous frame.

[0081] ③Judge whether the index with the shortest distance exists in the persistent tracking index list, specifically:

[0082] If the index with the shortest distance does not exist in the persistent tracking index list, add the high-quality skeleton cluster index with the shortest distance to the persistent tracking index list, and add the distance of the high-quality skeleton cluster with the shortest distance to the tracking distance list.

[0083] If the index with the shortest distance already exists in the persistent tracking index list, it means that the tracking ID has been merged, so replace the tracking ID with a closer distance with the tracking ID with a farther distance in the tracking distance list.

[0084] ④Judge whether the tracking result of the previous frame has been completely traversed. If it has not been completely traversed, return to step ③

[0085] ⑤If step ④ judges that it has been completely traversed, further judge whether the length of the persistent tracking list is less than the maximum 2D pose number in multiple perspectives, that is, the tracking number threshold at time t. If the length of the persistent tracking list is less than the tracking number threshold at time t, find the difference set between the high-quality tracking cluster index and the persistent tracking index, and add the difference set result to the new person tracking index list and the persistent tracking index list.

[0086] ⑥If the new person tracking list is not empty, add a new tracking ID, and finally extract the result from the high-quality skeleton cluster matrix c according to the persistent tracking index to obtain the tracking result and tracking ID.

[0087] ​Embodiment 2

[0088] The embodiment 1 above provides a multi-view multi-person three-dimensional human posture estimation and tracking method. Correspondingly, the embodiment provides a multi-view multi-person three-dimensional human posture estimation and tracking system. The system provided by the embodiment can implement the multi-view multi-person three-dimensional human posture estimation and tracking method of the embodiment 1. The system can be implemented by software, hardware or a combination of software and hardware. For example, the system can include integrated or separate functional modules or functional units to perform the corresponding steps in the methods of the embodiment 1. Since the system of the embodiment is basically similar to the method embodiment, the description process of the embodiment is relatively simple, and the related parts can be referred to the part of the description of the embodiment 1. The system of the embodiment is only illustrative.

[0089] The multi-view multi-person three-dimensional human posture estimation and tracking system provided by the embodiment includes:

[0090] A key point estimation module estimates two-dimensional human key points of each person in each view by using a two-dimensional human posture estimation network.

[0091] A depth estimation module uses a plane scanning stereo algorithm to perform depth regression on the estimated two-dimensional human key points based on a human symmetric hollow network and a joint symmetric hollow network.

[0092] A skeleton pool forming module aggregates a plurality of three-dimensional skeletons formed by plane scanning stereo of a plurality of views to form a skeleton pool.

[0093] A clustering module constructs a skeleton cluster by clustering the skeleton pool using a skeleton clustering algorithm.

[0094] A tracking module tracks each three-dimensional human posture by using a skeleton cluster tracking algorithm and using the constructed skeleton cluster.

[0095] The two-dimensional human key points include a human center point and a joint point. The depth estimation module adds an absolute depth value result of the human center point obtained by the human symmetric hollow network to a relative depth value result of each human joint point in the result obtained by the joint symmetric hollow network to obtain a spatial depth of a two-dimensional skeleton of each view.

[0096] Embodiment 3

[0097] In the embodiment, Shelf, Campus and Panoptic data sets are used.

[0098] The Shelf dataset captures the interaction of four people in an indoor environment using five cameras. Due to incomplete annotation of real 3D poses, the network was trained using synthetic 3D poses consistent with Tu, Wang, and Zeng (Tu H, Wang C, Zeng W. Voxelpose: Towards multi-camera 3d human pose estimation in wild environment [C] / / European Conference on Computer Vision. Springer, Cham, 2020:197-212.) and evaluated on the 300-600 frames of the test set for three of the four individuals that were mostly unoccluded for most of the time.

[0099] The Campus dataset (Belagiannis V, Amin S, Andriluka M, et al. 3D pictorial structures for multiple human pose estimation, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014:1669-1676.) captures the interaction of three people in an outdoor environment using three front-facing views of the camera. Due to incomplete annotation of real 3D poses, the network was trained using synthetic 3D poses consistent with Tu, Wang, and Zeng (Tu H, Wang C, Zeng W. Voxelpose: Towards multi-camera 3d human pose estimation in wild environment [C] / / European Conference on Computer Vision. Springer, Cham, 2020:197-212.) and evaluated on the 350-470 frames and 650-750 frames of the test set.

[0100] Panoptic dataset (Joo H, Liu H, Tan L, et al. Panoptic studio: A massively multiview system for social motion capture [C] / / Proceedings of the IEEE International Conference on Computer Vision. 2015: 3334-3342.) is a large-scale benchmark with 3D skeleton joint annotation. It captures the daily social activities of multiple individuals in an indoor environment. The same data sequence as Tu, Wang, and Zeng (Tu H, Wang C, Zeng W. Voxelpose: Towards multi-camera 3D human pose estimation in wild environment [C] / / European Conference on Computer Vision. Springer, Cham, 2020: 197-212.) is used, except for `160906_band3' in the training set, because it has broken images. A set of high-definition cameras (3, 6, 12, 13, 23) are used to shoot training and test sequences for evaluation.

[0101] 1) Train the symmetric dilated network using Adam with a learning rate of 0.0001 as the optimizer and using smooth L1 loss as the loss function. Train on the Shelf and Campus datasets using the synthetic dataset for 1,000 epochs. Train on the CMUPanoptic dataset for 100 epochs.

[0102] 2) Back-project the 2D human pose estimation and the depth results obtained by the symmetric dilated network under each view to obtain multiple 3D skeletons, then perform hierarchical clustering and weighted averaging on them to obtain the skeleton pool.

[0103] 3) After obtaining the skeleton pool, project the 3D human skeletons in the skeleton pool to each view in turn, and obtain the matching relationship between the 3D human skeleton and the 2D human skeleton according to the distance to the nearest 2D skeleton in the view. According to the matching relationship, classify the 3D human skeletons in the skeleton pool to obtain the skeleton cluster.

[0104] 4) After obtaining the skeleton cluster, use the skeleton cluster tracking algorithm to obtain the final 3D human pose estimation point.

[0105] In the present embodiment, the CPU of the computer executing the algorithm is Intel i9, the computer operating system is Ubuntu 18.04, the CUDA version is 11.2, the neural network framework used is Pytorch, and the version is 1.10.2. According to different training set sizes, the parameters can be adjusted appropriately, the input is the estimated 2D human pose estimation point, and the final output is obtained through the overall process of skeleton cluster tracking.

[0106] The present application can realize multi-person three-dimensional human pose estimation based on multiple views through the above steps. To verify the effectiveness and practicability of the method proposed in the present application, the following examples on the CMU Panoptic dataset are given. Table 1 is the detection result of the examples compared with Tu, Wang, and Zeng (Voxelpose: Towards multi-camera 3D human pose estimation in wild environment, European Conference on Computer Vision. Springer, Cham, 2020: 197-212.), Zhang et al. (Voxeltrack: Multi-person 3D human pose estimation and tracking in the wild, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.), Lin and Lee (Multi-view multi-person 3D pose estimation with plane sweep stereo, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 11886-11895.) on the test set, and the various measurement standards are AP (Average Precision) and MPJPE (Mean Per Joint Position Error).

[0107] Table 1 Results of different methods on the CMU Panoptic dataset using AP index and MPJPE error for evaluation

[0108] Method AP25 AP50 AP100 AP150 MPJPE Tu, Wang, and Zeng 83.6 98.3 99.8 99.9 17.68 mm Zhang et al. 85.9 98.3 99.5 - 16.97 mm Lin and Lee 92.1 99.0 99.8 99.8 16.75 mm Method of the invention 93.9 99.4 99.8 99.9 16.13 mm

[0109] As can be seen from Table 1, compared with other multi-view based three-dimensional human pose estimation methods, the improved algorithm provided by the present application has better performance for multi-three-dimensional human pose estimation. The use of the present embodiment has a greater improvement for most test categories, proving the effectiveness of the application.

[0110] In addition, Figure 4 The visual detection results given can also illustrate the superior performance of the present application.

[0111] Embodiment 4

[0112] The present embodiment provides a processing device corresponding to the multi-view multi-person three-dimensional human pose estimation and tracking method provided in Embodiment 1, which can be a processing device for a client, such as a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., to execute the method of Embodiment 1.

[0113] The processing device includes a processor, a memory, a communication interface and a bus, and the processor, the memory and the communication interface are connected through the bus to complete communication with each other. The memory stores a computer program executable on the processor, and the processor executes the computer program to perform the multi-view multi-person three-dimensional human pose estimation and tracking method provided in Embodiment 1.

[0114] In some embodiments, the memory can be a high-speed random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory.

[0115] In other embodiments, the processor can be a central processing unit (CPU), a digital signal processor (DSP) or various types of general-purpose processors, without limitation.

[0116] Embodiment 5

[0117] The multi-view multi-person three-dimensional human pose estimation and tracking method of Embodiment 1 can be specifically implemented as a computer program product, which can include a computer readable storage medium having loaded thereon computer readable program instructions for executing the multi-view multi-person three-dimensional human pose estimation and tracking method described in Embodiment 1.

[0118] The computer readable storage medium can be a tangible device that maintains and stores instructions for use by an instruction execution device. The computer readable storage medium can be, for example but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination of the above.

[0119] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for multi-view, multi-person 3D human pose estimation and tracking, characterized in that, include: Two-dimensional human pose estimation network is used to estimate the two-dimensional human key points of each person in each view; Based on the human body symmetric cavity network and the joint symmetric cavity network, a planar scanning stereo algorithm is used to perform depth regression on the estimated two-dimensional human body key points. Multiple 3D skeletons formed by scanning a solid from multiple views are aggregated to form a skeleton pool; The skeleton pool is clustered using a skeleton clustering algorithm to construct skeleton clusters; The skeleton cluster tracking algorithm is used to track each 3D human pose using the constructed skeleton cluster; The process of constructing skeleton clusters by clustering the skeleton pool using a skeleton clustering algorithm includes: According to the skeleton pool The I skeletons in the diagram correspond to the two-dimensional human pose estimation results in V views, and the information of the two-dimensional skeleton that best matches each three-dimensional skeleton in each view is obtained. After removing duplicates, we obtain L classes, thus generating a two-dimensional-three-dimensional matching class matrix. The matching two-dimensional skeleton confidence matrix It is the set of real numbers; By skeleton pool Based on the two-dimensional-three-dimensional matching class matrix H, which is divided into L classes, the skeleton cluster clustering information vector is derived.

2. The multi-view, multi-person, three-dimensional human pose estimation and tracking method according to claim 1, characterized in that, The two-dimensional human body key points include the human body center point and joint points; The method involves using a planar scanning stereo algorithm to perform depth regression on the estimated two-dimensional human body key points based on the human body symmetric cavity network and the joint symmetric cavity network. Specifically, the method involves adding the absolute depth value of the human body center point obtained from the human body symmetric cavity network to the relative depth value of each human body joint point obtained from the joint symmetric cavity network to obtain the spatial depth of the two-dimensional skeleton in each view.

3. The multi-view, multi-person, three-dimensional human pose estimation and tracking method according to claim 2, characterized in that, The process of aggregating multiple 3D skeletons formed by planar scanning of a solid from multiple views to form a skeleton pool includes: A skeleton pool is obtained by combining the back projection of each 2D skeleton with the depth value in all views; the i-th 3D skeleton in the skeleton pool corresponds to the 2D pose on the v-th view. The two-dimensional pose h i,v The index of the two-dimensional skeleton corresponds to the i-th three-dimensional skeleton on the view, represented as: Where, q i,v,j Let {p} represent the projected 2D pose of the i-th 3D skeleton in the v-th view, obtained from 2D pose estimation. v,1 ...,p v,N } represents the two-dimensional skeleton on the v-th view, n i As the index of the i-th 3D skeleton on the 2D skeleton.

4. The multi-view, multi-person, three-dimensional human pose estimation and tracking method according to claim 1, characterized in that, The method of using a skeleton cluster tracking algorithm to track each 3D human pose using the constructed skeleton cluster includes: From the skeleton cluster information vector b, the two-dimensional-three-dimensional matching class matrix H, and the skeleton pool The joint fraction matrix of the skeleton cluster is obtained. The score of the j-th joint of the i-th 3D skeleton for: Where, q i,v This represents the two-dimensional skeleton projected onto the i-th skeleton in the skeleton pool P at the v-th view. This indicates the use of the skeleton cluster information vector b in the 2D-3D matching class matrix H. i The two-dimensional skeleton matched at the v-view, where σ represents the hyperparameter. The peak score represents the peak score of the heatmap of the 2D human pose estimation network, which is the confidence score of the 2D joints. ⊙ represents the Hadamard product. For a given class l in L, select the joint score of the i-th 3D skeleton. The index of the maximum score for each joint in the 3D skeleton of class l is used as the joint index vector for high-quality skeleton groups. Select skeleton pool The vector b corresponding to all indices i of all 3D skeletons of class l. l This yields a high-quality skeleton cluster matrix for class l. Represented as:

5. The multi-view, multi-person, three-dimensional human pose estimation and tracking method according to claim 4, characterized in that, The method of using a skeleton cluster tracking algorithm to track each 3D human pose using the constructed skeleton cluster includes: Obtain the input parameters for the skeleton cluster tracing algorithm: The input parameters include the high-quality skeleton cluster C at time t. t Skeleton T tracked at time t-1 t-1 The list of tracking IDs Z at time t-1 t-1 The tracking number threshold γ at time t t The newcomer ID ξ at time t-1 t And initialize the tracking index list Θ, the new user tracking index list Θ′, and the tracking distance list Δ; Initialize the skeleton cluster tracing algorithm: The skeleton tracking result T at time t-1 t-1 When empty, meaning there is no one in the scene in the first or previous frame, then the high-quality skeleton cluster C will be... t The index Ψ in the index is directly assigned to the tracking index list Θ and the new user tracking index list Θ′ for initialization; Tracking is performed using a skeleton cluster tracing algorithm: When a skeleton result detected in the previous frame exists, assume there is a pair of paired skeletons <τ t-1 ,c t >,τ t-1 ∈T t-1 and c t ∈C t It is the closest of all these pairings, and θ′ is set to c. t In C t The index in Z; if the shortest distance index θ′ is in the existing list of tracking indices Θ, it means that a person has left the space captured by multiple cameras, which will cause the tracking ID of their skeleton to be merged into another skeleton that is still in the scene; when a skeleton has multiple IDs, in Z t-1 The ID corresponding to the minimum distance in list Δ is selected; when the number of elements in Θ is less than γ. t When a new person appears in the scene, Θ′, which is not in the current tracking index list, is added to Θ; ID assignment for executing the skeleton cluster tracing algorithm: When a new person appears in a space covered by multiple cameras, the new ID will be assigned to Z based on the number of elements in Θ′. t-1 In, and ξ t It will be updated to the new ID in the next frame; Update the tracking ID list and new user list: use Obtain the tracking result T at time t. t The list of tracking IDs Z at time t t By Z t-1 Updated, using ξ t-1 The sum of the number of newcomers n and the newcomer ID ξ at time t. t .

6. A multi-view, multi-person, three-dimensional human pose estimation and tracking system, used to implement the multi-view, multi-person, three-dimensional human pose estimation and tracking method as described in any one of claims 1 to 5, characterized in that, include: The key point estimation module uses a 2D human pose estimation network to estimate the 2D human key points of each person in each view; The depth estimation module, based on the human body symmetric cavity network and the joint symmetric cavity network, uses a planar scanning stereo algorithm to perform depth regression on the estimated two-dimensional human body key points; The skeleton pooling module aggregates multiple 3D skeletons formed by scanning a solid from multiple views to form a skeleton pool. The clustering module uses a skeleton clustering algorithm to cluster the skeleton pool and then construct skeleton clusters. The tracking module uses a skeleton cluster tracking algorithm to track the pose of each 3D human body using the constructed skeleton cluster.

7. The multi-view, multi-person, three-dimensional human pose estimation and tracking system according to claim 6, characterized in that, The two-dimensional human body key points include the human body center point and joint points; The depth estimation module adds the absolute depth value of the human body center point obtained from the human body symmetric cavity network to the relative depth value of each human body joint point obtained from the joint symmetric cavity network, to obtain the spatial depth of the two-dimensional skeleton of each view.

8. A processing apparatus, the processing apparatus comprising at least a processor and a memory, the memory storing a computer program, characterized in that, When the processor runs the computer program, it performs steps to implement the multi-view, multi-person, three-dimensional human pose estimation and tracking method according to any one of claims 1 to 5.

9. A computer storage medium, characterized in that, It stores computer-readable instructions that can be executed by a processor to implement the steps of the multi-view, multi-person, three-dimensional human pose estimation and tracking method according to any one of claims 1 to 5.

Images