Lightweight three-dimensional human body posture estimation method for uncalibrated multi-view camera

Abstract

The invention discloses a lightweight three-dimensional human body posture estimation method for an uncalibrated multi-view camera, and the method comprises the steps: firstly collecting a three-dimensional human body posture estimation data set of any view angle number, and defining a two-dimensional posture sequence from # imgabs0 dynamic view angles and a corresponding three-dimensional posture sequence; secondly, extracting features of the multi-view two-dimensional attitude sequence by using a bidirectional Mama model to obtain a prediction result, and pre-training and finely adjusting the bidirectional Mama model; and finally, training the bidirectional Mama model end to end by using a multi-view-angle data set shot by the uncalibrated movable camera, outputting a three-dimensional posture of each view angle, and aligning the postures of different view angles to one of the view angles by means of protoustes to complete human body posture estimation. According to the method, multi-view attitude estimation is adopted, so that the human body attitude of a person can still be estimated with high precision in an environment with more shelters, and a camera calibration process is avoided.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to a lightweight three-dimensional human body posture estimation method for an uncalibrated multi-view camera, mainly involving using a dual-stream space-time transformer to convert a two-dimensional posture sequence of multiple perspectives into three-dimensional postures of different perspectives, and then aligning the postures of different perspectives to the same coordinate system through the Procrustes method to calculate errors and optimize the neural network, thereby obtaining a method for obtaining a higher-precision three-dimensional human body posture. Background Art

[0002] Posture estimation refers to accurately inferring the posture information of the human body from an image, estimating the position coordinates of key points of the human body, including joint angles, positions, and motion trajectories. It has been applied to a wide range of fields, such as in the field of sports medicine, and in sports prone to injury, such as alpine skiing and gymnastics. Through posture estimation, these sports can be digitized, improving the movement techniques of athletes and reducing the probability of injury; at the same time, in the field of autonomous driving, the downstream tasks of posture estimation can be used to predict the possible behavior of pedestrians and improve the obstacle avoidance capability of autonomous driving.

[0003] 3D pose estimation can be divided into single-view pose estimation and multi-view pose estimation according to the number of views used in model training. Some single-view pose estimation methods directly obtain the 3D pose of the human body from the image through a neural network, while other methods use the 2D pose sequence obtained by 2D pose estimation as model input to obtain the 3D pose. Traditional multi-view pose estimation often requires obtaining a 2D pose sequence from an upstream task, or obtaining a 2D joint heat map from an image through a backbone convolutional neural network, and then jointly inferring the 3D pose through various types of neural networks.

[0004] However, in practical applications, existing methods often encounter various challenges:

[0005] (1) Single-view pose estimation often has large errors when the person is occluded by a large area.

[0006] (2) Traditional multi-view pose estimation requires camera calibration, which is very expensive, making it difficult to apply these methods to various fields. In addition, because there are few datasets containing camera calibration data, only a small amount of data can be used for model training, making it difficult for the model to achieve ideal results in specific fields.

[0007] (3) The preparation of pose estimation datasets is difficult, and professional personnel and equipment are required to collect the 3D position data of human joints;

[0008] (4) Network training requires too much time and hardware. The number of parameters that need to be adjusted often reaches millions. If there is a data set for a new target application scenario, the entire network can only be retrained, which is too costly.

[0009] (5) Neural networks have poor transfer generalization capabilities. They often perform well on a dataset, but perform poorly in an environment that is somewhat different from the dataset. Summary of the invention

[0010] The technical problem to be solved by the present invention is to propose a lightweight three-dimensional human body posture estimation method for an uncalibrated multi-view camera to solve the problem of the need to calibrate the camera when performing multi-view posture estimation. The present invention can obtain reliable estimation results in multi-view posture estimation, avoid camera calibration, and reliably estimate human body posture for videos shot by a mobile camera, effectively expanding the scope of use of the method and better serving downstream tasks of posture estimation.

[0011] In order to better improve the generalization ability of the model in different application environments, the present invention divides the training process of the model into two stages. The first stage is the pre-training stage, which aims to learn useful motion representations; the second stage can select similar data sets for training according to different application environments, thereby improving the generalization ability of the model in various occasions.

[0012] The objective of the present invention is achieved through the following technical solution: a lightweight three-dimensional human body posture estimation method for an uncalibrated multi-view camera, the method comprising the following steps:

[0013] In step S1, a 3D human pose estimation dataset with any number of viewpoints is collected. There is no special requirement for the camera used, and it is applicable to uncalibrated movable cameras. 2D pose sequence of dynamic viewpoints and the corresponding 3D pose sequence .in is the number of viewing angles, is the length of the pose sequence, is the number of human skeleton joints defined by the model, is the number of input channels, is the number of output channels.

[0014] In step S2, the bidirectional Mamba model is used to extract the features of the multi-view 2D posture sequence. The bidirectional Mamba model mainly includes a preprocessing module and multiple groups of bidirectional Mamba modules connected in series. The preprocessing module can process the input 2D posture sequence Mapping into a high-dimensional latent space ,in is the dimension of the latent space. Input multiple sets of bidirectional Mamba modules in series. Each bidirectional Mamba module contains two branches. The data input by one branch first passes through the time Mamba module and then the space Mamba module. The data input by the other branch first passes through the space Mamba module and then the time Mamba module. The two branches are fused by directly adding vectors. After passing through multiple sets of bidirectional Mamba modules with the same model parameter structure, the prediction results are obtained. With the data set Calculate the loss and optimize the bidirectional Mamba model parameters by back propagation.

[0015] In step S3, during the pre-training phase of the bidirectional Mamba model, the 3D posture in the data set is mapped to a complete 2D skeleton posture sequence using the orthogonal projection method, and then the partially defective 2D posture is obtained by random masking. , to achieve the purpose of data enhancement. At this stage, it is necessary to calculate the 3D key point loss, speed loss and 2D reprojection loss, as follows:

[0016] ;

[0017] in is the 3D keypoint loss, is the speed loss, is the 2D reprojection loss; and Represents the predicted and actual values ​​of the three-dimensional true pose; and Respectively represent the predicted value and actual value of the speed of the key point between video frames, namely: ; and Represent the predicted value and actual value of the two-dimensional reprojection posture, respectively. is the confidence of the 2D pose provided by the dataset. Represents each video frame, Represents each key point.

[0018] In step S4, during the fine-tuning phase of the bidirectional Mamba model, a simple task head is added to the pre-trained model to estimate the 3D pose. The 2D pose sequences of multiple viewpoints in the same scene are superimposed and input into the bidirectional Mamba model, and then some key points and viewpoints are masked by random masking to simulate the challenge of occlusion in the real environment and achieve the effect of data enhancement. Random masking is implemented by adding a control gate in the input channel of each viewpoint. , to mask the joint points or viewing angles, as follows:

[0019]

[0020]

[0021] in For sports characteristics, Perspective Output.

[0022] when When, perspective Valid, otherwise invalid. Yes The probability of a valid view is:

[0023]

[0024] In step S5, the entire bidirectional Mamba model is trained end-to-end using a multi-view dataset shot by an uncalibrated mobile camera. The model outputs the three-dimensional posture of each view. The postures of different views are aligned to one of the views by translation, rotation and scaling through procrustes, so as to achieve the result of posture alignment and unified coordinate system. In addition to calculating the three-dimensional key point loss, speed loss and two-dimensional reprojection loss, this stage also needs to calculate the multi-view loss, as follows:

[0025] First, the model predicts the 3D posture of each view output, which is recorded as: ,in: ;

[0026] Then use the first perspective as the reference view. If there are other perspectives, that is, , the 3D pose estimated from other perspectives can be aligned with the pose estimated from the first perspective through the procrustes method, and the Alignment of 3D joint points of two perspectives:

[0027]

[0028]

[0029] Procrustes alignment calculates the ratio between two sets of 3D point correspondences , Rotate and pan , the posture after procrustes alignment is recorded as:

[0030]

[0031] Finally, procrustes alignment is used to calculate the multi-view consistency loss:

[0032]

[0033] in is a binary indicator function used to decide whether to include multi-view consistency loss. represents the weighted multi-view consistency loss, , Represents the weight of the corresponding loss.

[0034] The advantages of the present invention compared with the prior art are:

[0035] To solve the problem that single-view pose estimation is incorrect in an environment with occlusion, the present invention adopts multi-view pose estimation, so that the human body pose can still be estimated with high precision in an environment with more occlusions.

[0036] In order to solve the problem that traditional multi-view pose estimation requires camera calibration, the present invention adopts the Procrustes method to unify the coordinate systems of poses from different perspectives through translation, rotation and scaling operations, thus avoiding the camera calibration process.

[0037] In order to solve the problem of high training cost and poor generalization ability of traditional posture estimation, this paper proposes a two-stage training framework, which learns the motion representation of the human body through pre-training and then adapts to various tasks through fine-tuning. When there is a new usage environment, it is only necessary to fine-tune the original pre-trained model with the data set of the usage environment, which greatly reduces the training time and hardware requirements and improves the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the overall flow chart of the present invention;

[0039] Figure 2 It is a schematic diagram of the posture lifting model of the present invention;

[0040] Figure 3 It is an architectural diagram of the bidirectional time / space Mamba module adopted by the present invention;

[0041] Figure 4 It is a visualization result diagram of the training model according to the present invention;

[0042] Figure 5 It is a comparison chart of indicators of the training model according to the present invention and other models. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the present disclosure is further elaborated in detail below with reference to the accompanying drawings and specific implementation methods.

[0044] like Figure 1 and Figure 2As shown, a multi-view 3D human posture estimation method based on an uncalibrated mobile camera includes the following steps:

[0045] S1. Collect 3D human pose estimation datasets with any number of viewpoints. There is no special requirement for the camera used. It is suitable for uncalibrated movable cameras. Definition from 2D pose sequence of dynamic viewpoints and the corresponding 3D pose sequence .in is the number of viewing angles, is the length of the pose sequence, is the number of human skeleton joints defined by the model, is the number of input channels, and is the number of output channels.

[0046] S2. Use the bidirectional Mamba model to extract the features of multi-view 2D posture sequences. The model mainly consists of a preprocessing module and multiple sets of bidirectional Mamba modules connected in series, such as Figure 3 In the preprocessing module, the 2D pose sequence Mapped to high-dimensional latent space through fully connected layers ,in is the dimension of the latent space. Then the time step Convert to an embedding vector , which encodes temporal information into a vector of the same dimension as the latent feature, and helps the network generate a feature representation that conforms to the temporal dynamics at each time point. The embedded vector is input into the MLP layer together with other features to further enhance the features and finally generate motion features. .

[0047] Then Input multiple sets of bidirectional Mamba modules in series. Each Mamba module contains two branches. The data input from one branch first passes through the time Mamba module and then the space Mamba module. The data input from the other branch first passes through the space Mamba module and then the time Mamba module. The two branches are fused by directly adding vectors. After passing through multiple sets of bidirectional Mamba modules with the same model parameter structure, the prediction results are obtained. With the data set Calculate the loss and optimize the model parameters by back propagation.

[0048] The trained model uses a 3D human pose estimation method based on a bidirectional Mamba module. The main contents include:

[0049] Temporal Mamba module: It is used to capture the temporal dependencies in motion sequences and adopts a bidirectional strategy to enhance temporal modeling capabilities. The input feature sequence is denoised through time step embedding guidance, and after normalization, linear projection, one-dimensional convolution and state space model processing, the final output sequence in the time dimension is generated. The details are as follows:

[0050] (1) Input feature sequence Applying normalization operations allows the model to avoid training instability caused by differences in feature ranges and speed up the training process.

[0051] (2) Normalize the feature sequence They are mapped to two new feature spaces through two fully connected layers, respectively, and ,in It is a higher feature dimension after mapping, which can help the model capture more feature information and better learn the laws of time series.

[0052] (3) For the mapped sequence Perform a 1D convolution and apply the SiLU activation function to get The 1D convolution can capture the local temporal dependencies in the sequence, and the SiLU activation function helps to avoid the gradient vanishing problem while providing nonlinear modeling capabilities.

[0053] (4) Features after convolution Perform two linear transformations respectively to obtain new features and Through further linear transformation, the model is able to map the convolutional features to a new representation space, enhancing the model’s understanding of features of different dimensions.

[0054] (5) After passing through the fully connected layer, the log-SoftPlus function is used to activate the output of the linear transformation to ensure that the intermediate parameters of the output time scale is a positive number. The calculation formula is as follows:

[0055]

[0056] in is a fully connected layer, is a learnable parameter, and the log-SoftPlus function is a smoother ReLU-like activation function.

[0057] (6) Yes and , and The two sets of parameters are respectively Sum and get the extended features and ,in is a learnable parameter. This transformation process enhances the representation ability of features, especially for the dependencies across time steps in time series.

[0058] (7) The transformed features , ,and As a Mamba module parameter, the feature As input to the Mamba module, get new features The role of the Mamba module is to further enhance the model's understanding of motion data by capturing the dependencies between space and time. and Add element by element, apply the fully connected layer and SiLU activation function, and get the output of the temporal Mamba module .

[0059] Spatial Mamba module: It is used to learn the spatial dependencies of human postures in a single frame, and adopts the same bidirectional strategy and structure as the temporal Mamba module. Its processing features are expanded along the latent space dimension, and the output in the spatial dimension is generated through linear projection, convolution and state space modeling, while ensuring the consistency of the output and input in shape. The specific steps are as follows:

[0060] Output of the Mamba module at a given time , first transpose it along the last two dimensions, and get .Will As the input of the spatial Mamba module, the subsequent calculation steps are similar to those of the temporal Mamba module, and finally we get .

[0061] In a bidirectional Mamba module, the final output comes from the output of the spatial Mamba module at the end of the two branches. Output of the and time Mamba modules , the output of the bidirectional Mamba module is obtained by adaptive fusion The specific process of adaptive fusion is as follows:

[0062] (1) First, Convert to , and then Splicing .

[0063] (2) Check again Use linear transformation to calculate attention and perform softmax operation to get normalized attention weights , and then Split by the last dimension and .

[0064] (3) Finally, use the attention weight to input and Perform weighted fusion to obtain :

[0065]

[0066] in Represents element-wise multiplication.

[0067] The output of N bidirectional Mamba modules After a fully connected layer, the prediction result is obtained , used for the calculation of subsequent loss loss.

[0068] S3. Use the orthogonal projection method to map the prediction results to a complete 2D skeleton pose sequence, and then obtain the partially defective 2D pose by random masking. , to achieve the purpose of data enhancement. At this stage, it is necessary to calculate the 3D key point loss, speed loss and 2D reprojection loss, as follows:

[0069] in is the 3D keypoint loss, is the speed loss, is the 2D reprojection loss; and Represents the predicted and actual values ​​of the three-dimensional true pose; and Respectively represent the predicted value and actual value of the speed of the key point between video frames, namely: ; and Respectively represent the predicted value and actual value of the two-dimensional reprojection posture. Represents each video frame, Indicates each key point; is the confidence of the 2D pose provided by the dataset.

[0070] S4. Add a simple task head to the pre-trained model obtained in S2 to estimate the 3D pose. Superimpose the 2D pose sequences of multiple viewpoints in the same scene and input them into the bidirectional Mamba model. Use random masking to mask some key points and viewpoints to simulate the challenge of occlusion in the real environment and achieve data enhancement. The random masking is implemented by adding a control gate in the input channel of each viewpoint. , to mask the joint points or viewing angles, as follows

[0071]

[0072]

[0073] in For sports characteristics, For perspective Output.

[0074] when When, perspective Valid, otherwise invalid. Yes The probability of a valid view is:

[0075]

[0076] S5. Use the multi-view dataset taken by an uncalibrated mobile camera to train the entire network end-to-end. The model outputs the 3D posture of each view. The postures of different views are aligned to one of them by translation, rotation and scaling through procrustes, so as to achieve the result of posture alignment and unified coordinate system. In addition to calculating the 3D key point loss, speed loss and 2D reprojection loss, this stage also needs to calculate the multi-view loss, as follows:

[0077] First, the model predicts the 3D posture of each view output, which is recorded as: ,in:

[0078] Then use the first perspective as the reference view. If there are other perspectives, that is, , the 3D pose estimated from other perspectives can be aligned with the pose estimated from the first perspective through the procrustes method, and the Alignment of 3D joint points between two perspectives.

[0079]

[0080]

[0081] Procrustes alignment calculates the ratio between two sets of 3D point correspondences , Rotate and pan , the posture after procrustes alignment is recorded as:

[0082]

[0083] Finally, procrustes alignment is used to calculate the multi-view consistency loss:

[0084]

[0085] The total loss at this stage is:

[0086] in is a binary indicator function used to decide whether to include multi-view consistency loss. represents the weighted multi-view consistency loss, , Represents the weight of the corresponding loss.

[0087] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0088] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

[0089] Example:

[0090] During the experiment, Human3.6M and AMASS human pose datasets were selected as pre-training datasets, and Human3.6M and SkiPose datasets were used as fine-tuning datasets. The following are the specific experimental steps and parameter settings:

[0091] Before model training, the data set needs to be standardized. The specific operation is to set the center point of the human pelvis as the coordinate origin to generate standardized posture data that can be directly used by the model.

[0092] During the pre-training phase, the following main parameters were set:

[0093] Number of batches: 128; learning rate: 0.001; learning rate decay: 0.99; number of frames: 243; number of key points: 17; the experiment uses two 3090 graphics cards, equipped with the proposed bidirectional Mamba network architecture. Compared with similar Transformer models, Mamba significantly reduces the training cost. Taking the training time per epoch as an example, the Mamba network only takes 40 minutes, while the similar Transformer model takes 30 hours.

[0094] The pre-training results of the Mamba model are as follows Figure 4As shown, through visualization, it can be seen that the model captures the posture sequence with high accuracy.

[0095] In the fine-tuning stage, the initial learning rate is adjusted to 0.0001, and the model is fine-tuned on the Human3.6M and SkiPose datasets. Through comparative experiments with the MotionBERT model, Figure 5 As shown in Table 1, it can be observed that on the SkiPose dataset, the overlap between the prediction results of the Mamba model and the true value is significantly better than that of MotionBERT. The final indicator comparison is shown in Table 1, which further verifies the superiority of the Mamba model.

[0096] Indicator explanation: In order to quantify the model performance, two indicators are used: the average joint position error MPJPE and the average joint position error P-MPJPE aligned by the Proctor analysis method:

[0097] MPJPE: represents the average Euclidean distance between the joint point position predicted by the model and the true position. The smaller the value, the higher the prediction accuracy of the model for the posture.

[0098] P-MPJPE: Based on MPJPE, Procrustes alignment is further performed to remove the scale, rotation and translation errors of the prediction results, so as to better measure the performance of the model in morphological matching.

[0099] Table 1 Index comparison

[0100]

[0101] Experimental results show that the Mamba model achieves the best performance in both MPJPE and P-MPJPE indicators on the SkiPose dataset.

Claims

1. A lightweight 3D human pose estimation method for uncalibrated multi-view cameras, characterized in that: The following steps are involved: Step S1, collect a 3D human pose estimation dataset of any number of viewpoints and define A 2D pose sequence of a dynamic perspective and a corresponding 3D pose sequence; Step S2, using the bidirectional Mamba model, extracting features of the multi-view two-dimensional posture sequence to obtain a prediction result; Step S3, pre-training and fine-tuning the bidirectional Mamba module; Step S4, using the multi-view dataset shot by an uncalibrated movable camera to train the bidirectional Mamba model end-to-end, output the three-dimensional posture of each view, align the postures of different views to one of the views through procrustes, and complete the human body posture estimation.

2. The lightweight 3D human pose estimation method for an uncalibrated multi-view camera according to claim 1, characterized in that: The step S1 specifically includes: collecting a 3D human posture estimation dataset of any number of viewing angles, using an uncalibrated movable camera; defining 2D pose sequence of dynamic viewpoints and the corresponding 3D pose sequence ,in is the number of viewing angles, is the length of the pose sequence, is the number of human skeleton joints defined by the model, is the number of input channels, is the number of output channels.

3. The lightweight 3D human pose estimation method for an uncalibrated multi-view camera according to claim 2, characterized in that: The bidirectional Mamba model comprises a preprocessing module and a plurality of groups of bidirectional Mamba modules connected in series; The preprocessing module takes the input 2D pose sequence Mapping into latent space ,in is the dimension of the latent space; Input multiple sets of bidirectional Mamba modules in series; each bidirectional Mamba module contains two branches, one of which first passes through the time Mamba module and then the space Mamba module, and the other first passes through the space Mamba module and then the time Mamba module. The two branches are fused by direct vector addition; after passing through multiple sets of bidirectional Mamba modules with the same model parameter structure, the prediction results are obtained. Compared with the data set Calculate the loss and optimize the bidirectional Mamba model parameters by back-propagation.

4. The lightweight 3D human pose estimation method for an uncalibrated multi-view camera according to claim 3, characterized in that: The specific implementation process of the time Mamba module is as follows: For the input feature sequence Apply the normalization operation to normalize the feature sequence Mapped to two new feature spaces through two fully connected layers, respectively, and ; After mapping, the sequence Perform a 1D convolution and apply the SiLU activation function to get ; For the convolutional features Perform two linear transformations respectively to obtain new features and ; then After passing through the fully connected layer, the log-SoftPlus function is used to activate the output of the linear transformation to ensure that the intermediate parameters of the output time scale is a positive number; right and , and The two sets of parameters are respectively Sum and get the extended features and ,in is a learnable parameter; The transformed features , ,and As a Mamba module parameter, the feature As input to the Mamba module, get new features ; The characteristics and Add element by element, apply the fully connected layer and SiLU activation function, and get the output of the temporal Mamba module .

5. The lightweight 3D human pose estimation method for an uncalibrated multi-view camera according to claim 4, characterized in that: The specific implementation process of the spatial Mamba module is as follows: Output of the Mamba module at a given time , first transpose it along the last two dimensions, and get ,Will As the input of the spatial Mamba module, the subsequent calculation steps are the same as those of the temporal Mamba module, and we get .

6. The lightweight 3D human pose estimation method for an uncalibrated multi-view camera according to claim 5, characterized in that: In the bidirectional Mamba module, the output comes from the spatial Mamba module and time Mamba module , the output of the bidirectional Mamba module is obtained through adaptive fusion ; The specific process of adaptive fusion is as follows: Convert to , and then Splicing ; right Use linear transformation to calculate attention and perform softmax operation to get attention weight ; then Split by the last dimension and ; Finally, the attention weights are used to and Perform weighted fusion to obtain : ； in Represents element-wise multiplication; The output of N bidirectional Mamba modules After a fully connected layer, the prediction result is obtained , used for the calculation of subsequent losses.

7. The lightweight 3D human pose estimation method for an uncalibrated multi-view camera according to claim 6, characterized in that: The specific implementation process of step S3 is as follows: Step S3.1: In the pre-training stage, orthogonal projection is used to project the prediction results into a complete 2D skeleton pose sequence, and then the partially defective 2D pose is obtained by random masking. , to achieve the purpose of data enhancement; in this stage, the 3D key point loss, speed loss and 2D reprojection loss are calculated as follows: ； in is the 3D keypoint loss, is the speed loss, is the 2D reprojection loss; and Represents the predicted and actual values ​​of the three-dimensional true pose; and Respectively represent the predicted value and actual value of the speed of the key point between video frames, namely: ; and Respectively represent the predicted value and actual value of the two-dimensional reprojection posture. Represents each video frame, Represents each key point, is the confidence of the 2D pose provided by the dataset; Step S3.2, in the fine-tuning phase, a task head is added to the pre-trained model to estimate the 3D pose; The 2D posture sequences of multiple perspectives in the same scene are superimposed and input into the bidirectional Mamba model. Then, some key points and perspectives are masked by random masking to simulate the occlusion challenges in the real environment. The random masking is implemented by adding a control gate in the input channel of each perspective. , masking joint points or viewing angles.

8. The lightweight 3D human pose estimation method for an uncalibrated multi-view camera according to claim 7, characterized in that: The specific implementation process of step S4 is as follows: a bidirectional Mamba model is trained end-to-end using a multi-view dataset shot by an uncalibrated mobile camera. The model outputs the three-dimensional posture of each view. The postures of different views are aligned to one of the views by translation, rotation and scaling through procrustes, so that the postures are aligned and the coordinate system is unified. In addition to calculating the three-dimensional key point loss, speed loss and two-dimensional reprojection loss, this stage also calculates the multi-view loss, which is as follows: First, the model predicts the 3D posture of each view output, which is recorded as: , ; Then use the first perspective as the reference view. If there are other perspectives, that is, , align the 3D pose estimated from other views with the pose estimated from the first view through the procrustes method; The posture after procrustes alignment is recorded as: ； Finally, procrustes alignment is used to calculate the multi-view consistency loss: ； The total loss at this stage is: ； in is a binary indicator function that determines whether to include multi-view consistency loss. represents the weighted multi-view consistency loss, , Represents the weight of the corresponding loss.

Images