Single-person 3D pose estimation method based on adaptive multi-view and temporal feature fusion

Abstract

The present invention discloses a single-person three-dimensional posture estimation method with adaptive multi-view and temporal feature fusion. Based on the video sequence pictures of multiple cameras as input, a robust feature extractor is designed to extract 2D posture features; based on the 2D posture features as input, an adaptive view self-attention transformation network is designed in the camera dimension, and the two-dimensional postures under any number of uncalibrated cameras are fused through relative camera position encoding and self-attention mechanism to obtain multi-view fusion posture features; based on the multi-view fusion posture features as input, a temporal self-attention transformation network is designed in the time dimension, and multi-frame features are adaptively fused through self-attention mechanism to obtain the final 3D posture. The present invention is reasonably designed, can be directly applied to scenes with any number of uncalibrated cameras without retraining, and the network calculation amount is very small.

Description

Technical Field

[0001] The invention relates to the field of artificial intelligence technology, and in particular to a single-person three-dimensional posture estimation method based on adaptive multi-view and temporal feature fusion. Background Art

[0002] The task of single-person 3D human pose estimation is a hot topic in the field of computer vision. It plays an important role in many applications, such as action recognition, human reconstruction, and robotic manipulation. According to the number of cameras, single-person 3D human pose systems can be mainly divided into monocular 3D human pose estimation and multi-camera 3D human pose estimation. For the monocular estimation model, only the visual information of the image under a single camera is used for feature encoding, which cannot effectively solve the problems of occlusion and depth ambiguity. The multi-camera model can use multi-view geometric constraints to compensate for the occluded joint information and the depth information lost after camera projection, so it can more accurately predict the 3D human pose. For multi-camera 3D human pose estimation, the existing models mainly use camera parameters to provide multi-view geometric constraints, and are heavily dependent on the configuration of multiple cameras, and cannot be directly applied to monocular or small camera scenes. In natural scenes, the position of the camera is often changing, and real-time camera calibration in dynamic scenes is unrealistic. In addition, the existing multi-camera models are computationally intensive and cannot use time series models to capture time series information to obtain smooth 3D poses. Summary of the invention

[0003] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a single-person 3D posture estimation method with adaptive multi-view and temporal feature fusion, which extracts robust 3D features through a carefully designed feature extractor, and fuses view geometry information and sequence timing information through a view self-attention transformation network and a temporal self-attention transformation network; thereby being better applied to the fields of action recognition, human body reconstruction and robot operation.

[0004] The present invention adopts the following technical solutions to solve the above technical problems:

[0005] A single-person 3D posture estimation method based on adaptive multi-view and temporal feature fusion proposed by the present invention comprises the following steps:

[0006] Step 1: Based on the video sequence images of multiple cameras as input, a robust feature extractor is used to extract 2D posture features;

[0007] Step 2: Based on the 2D posture feature as input, an adaptive view self-attention transformation network is designed in the camera dimension. The 2D postures of any number of uncalibrated cameras are fused through relative camera position encoding and self-attention mechanism to obtain multi-view fusion posture features.

[0008] Step 3: Based on the multi-view fusion posture features as input, a temporal self-attention transformation network is designed in the time dimension. The multi-frame features are adaptively fused through the self-attention mechanism to obtain the final 3D posture.

[0009] As a further optimization scheme of the single-person 3D posture estimation method of the present invention based on adaptive multi-view and temporal feature fusion, the feature extractor in step 1 is specifically as follows:

[0010] Step 1: Given N video sequences from cameras, each video sequence contains F frames, and N×F frames share the same feature extractor; each frame I has a width of W and a height of H, and each frame I contains three color channels: R, G, and B. It is a three-dimensional matrix space with dimensions W×H×3; each picture contains only one person;

[0011] The feature extractor includes a 2D posture detector and a 3D posture feature extractor. For each frame of image I, the 2D posture detector is first used Predicting 2D pose information

[0012]

[0013] Among them, the total number of joints is J, P 2D , C 2D Represent the 2D coordinates and confidence of J joints, p j is the 2D coordinate of the jth joint, c j is the confidence of the j-th joint;

[0014] Step 2: For P 2D and C 2D , J joints are divided into G groups according to the correlation of human joint movements:

[0015]

[0016]

[0017] Among them, g∈{1,2,...,G}, They are P 2D and C 2D The g-th subset of ; is a dimension of 2J g One-dimensional matrix space of; is a dimension J g One-dimensional matrix space of Represents the index of all joints in group g, Jg is the number of joints in group g, p i and c i Represent the 2D coordinates and confidence of the i-th joint respectively;

[0018] Step 3: The 3D pose feature extractor first uses the first fully connected layer The g-th group of 2D joint coordinates Mapping to Features It is a one-dimensional matrix space with a dimension of C / 2; C represents the characteristics of the combined G group of joints The channel dimension of the obtained global feature:

[0019]

[0020] Step 4. Second fully connected network enter Output the mapping matrix of the g-th group of joints is a dimension of (C / 2)×2J g Two-dimensional matrix space; Will Mapped to features with a channel number of C / 2 For modulation

[0021]

[0022]

[0023] Step 5: For each group in group G, and After addition, through the residual network of group g Further extract spatial information to obtain the g-th group of adjusted features

[0024]

[0025] Step 6: Group G features Spliced ​​together, through the third fully connected layer Mapped to a person's global characteristics is a one-dimensional matrix space with dimension C; Concat(f 1 ,f 2 ,…,f G ) represents splicing of G group joint features;

[0026] The global features of N×F frames are concatenated to obtain the features X of all images, where It is a three-dimensional matrix space with dimensions C×N×F.

[0027] As a further optimization scheme of the single-person 3D posture estimation method with adaptive multi-view and temporal feature fusion described in the present invention, the view self-attention transformation network in step 2 is composed of a relative camera position encoder and a view self-attention fusion module.

[0028] As a further optimization scheme of the single-person 3D pose estimation method with adaptive multi-view and temporal feature fusion described in the present invention, the view self-attention transformation network in step 2 is obtained by the following steps:

[0029] Step 201: In the camera dimension, From N camera features Splicing composition, among which, v∈{1,2,...,N}, It is a two-dimensional matrix space with dimension C×F; is the feature of the v-th camera, which is obtained by concatenating the global features of the F-frame images under the v-th camera. In the process of camera feature fusion, the temporal dimension F will be omitted, that is, simplified to v∈{1,2,...,N};

[0030] Step 202: The view self-attention transformation network first adaptively learns the relative position relationship between cameras through a neural network, and inputs the query variable of the ath camera and the key value variable of the bth camera Output the relative position relationship mapping matrix M between the a-th camera and the b-th camera ab And feature fusion weighting coefficient A ab ;in, Represent the a-th and b-th camera features respectively; It is a two-dimensional matrix space with dimension D×D; C = H × D;

[0031]

[0032]

[0033] in, and are two neural network layers that share the same residual network for obtaining and The relationship features between ; then use the fourth and fifth fully connected layers to output M ab and A ab ;

[0034] Step 203: Change the numerical characteristics of the bth camera The shape is divided into H D-dimensional local feature points in is the numerical feature after the shape is changed under the b-th camera; is a two-dimensional matrix space with dimension D×H; then through M ab right Perform linear mapping to achieve relative camera position encoding:

[0035]

[0036] in, It represents the feature after the a-th camera encodes the b-th camera feature relative to the camera position;

[0037] N×N camera feature combinations The encoded features will be obtained By N×N Splice to get, change After the shape is obtained, the feature V map ; in They are a four-dimensional matrix space with dimensions D×H×N×N and a three-dimensional matrix space with dimensions C×N×N;

[0038] Step 204, A={A ab |a∈{1,2,…,N},b∈{1,2,…,N}} represents N×N fusion coefficients A ab The fusion coefficient matrix composed of A and B is randomly set to 0 through the random masking strategy to change the number of cameras involved in the fusion.

[0039] Step 205: Use the randomly shielded A to map Perform weighted fusion to obtain the multi-view fusion feature V fuse , It is a two-dimensional matrix space with dimension C×N:

[0040] V fuse =sum((softmax(A)⊙V map )),

[0041] Among them, softmax(*) represents the normalized exponential function, which normalizes the third dimension of the fusion coefficient matrix A, ⊙ represents the dot product, and sum(*) represents the fusion of the features of N cameras in the third dimension.

[0042] As a further optimization scheme of the single-person 3D posture estimation method with adaptive multi-view and temporal feature fusion described in the present invention, the temporal self-attention transformation network in step three is composed of an encoding module and a two-layer feature fusion module.

[0043] As a further optimization scheme of the single-person 3D posture estimation method of the present invention with adaptive multi-view and temporal feature fusion, the temporal self-attention transformation network in step 3 is implemented by the following steps:

[0044] Step 301: In the process of temporal feature fusion, the camera dimension N is omitted, and the encoding module of the temporal self-attention transformation network first passes through the sixth fully connected layer V fuse Perform feature encoding:

[0045]

[0046] in, The features after feature encoding;

[0047] Step 302: The encoding module then uses cosine and sinine functions to construct sequence position encoding. The position encoding of Z is as follows:

[0048] Z 0 =Z+P emb ,

[0049] in, It is the feature after position encoding;

[0050] Step 303, the temporal self-attention transformation network includes two layers of feature fusion modules based on the self-attention mechanism; m∈{1,2} is the index of the number of layers of the feature fusion module, and the feature fusion process of the mth layer is:

[0051]

[0052]

[0053]

[0054]

[0055] in, and Through the seventh, eighth, and ninth fully connected layers respectively and For the time series features of the m-1 layer The query vector, key value vector and numerical vector obtained by mapping; FFN(*) is a multi-layer perceptron with residual connection; is the temporal fusion feature of the mth layer, where the input feature of the first layer fusion module

[0056] Step 304: Finally, pass through the tenth fully connected layer Regress the 3D pose P of each intermediate frame of the video sequence under N cameras 3D ,in It is a two-dimensional matrix space with dimension 3J×N:

[0057]

[0058] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0059] The feature extractor designed by the present invention can effectively use confidence to reduce the noise of the 2D pose detector, thereby extracting robust noise; the designed view self-attention transformation network can adaptively fuse any number of uncalibrated camera features, so that it can be directly applied to any camera configuration scene without retraining; the designed temporal self-attention transformation network can adaptively fuse single-frame and multi-frame temporal features, so that it can be directly applied to static scenes and dynamic video scenes without retraining;

[0060] (1) Modulating 2D pose features through a joint confidence feature extractor to reduce the impact of 2D pose noise on 3D pose estimation and extract robust pose features;

[0061] (2) The features of multiple views are adaptively fused through the view self-attention transformation network, which can be directly applied to any camera configuration (number of cameras and camera parameters) without camera calibration or retraining.

[0062] (3) The temporal features are adaptively fused through the temporal self-attention transformation network, which can be directly applied to single-frame and multi-frame scenarios without retraining. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is the overall flow chart of the present invention;

[0064] Figure 2 It is a framework diagram of a feature extractor of the present invention;

[0065] Figure 3 A block diagram of the attention mechanism in the view self-attention transformation network of the present invention; DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] A single-person 3D posture estimation method based on adaptive multi-view and temporal feature fusion includes the following steps:

[0068] Step 1: Based on the video sequence images of multiple cameras as input, a robust feature extractor is used to extract 2D posture features;

[0069] Step 2: Based on the 2D posture feature as input, an adaptive view self-attention transformation network is designed in the camera dimension. The 2D postures of any number of uncalibrated cameras are fused through relative camera position encoding and self-attention mechanism to obtain multi-view fusion posture features.

[0070] Step 3: Based on the multi-view fusion posture features as input, a temporal self-attention transformation network is designed in the time dimension. The multi-frame features are adaptively fused through the self-attention mechanism to obtain the final 3D posture.

[0071] Preferably, the feature extractor in step 1 uses a pre-trained 2D detector to obtain the posture information of the joint as input, and uses a fully connected network to extract robust features.

[0072] Preferably, the feature extractor in step 1 comprises the following steps:

[0073] like Figure 1 As shown in the figure, given a video sequence from N cameras, each video sequence contains F frames, and the N×F frames share the same feature extractor; each frame I has a width of W and a height of H, and each frame I contains three color channels: R, G, and B. It is a three-dimensional matrix space with dimensions W×H×3; each picture contains only one person;

[0074] The feature extractor includes a 2D posture detector and a 3D posture feature extractor. For each frame of image I, the 2D posture detector is first used Predicting 2D pose information

[0075]

[0076] Among them, the total number of joints is J, P 2D , C 2D Represent the 2D coordinates and confidence of J joints, p j is the 2D coordinate of the jth joint, c j is the confidence of the j-th joint;

[0077] Step 2: Figure 2 As shown, for P 2D and C 2D , J joints are divided into G groups according to the correlation of human joint movements:

[0078]

[0079]

[0080] Among them, g∈{1,2,...,G}, They are P 2D and C 2D The g-th subset of ; is a dimension of 2J g One-dimensional matrix space of; is a dimension J g One-dimensional matrix space of Represents the index of all joints in group g, J g is the number of joints in group g, p i and c i Respectively represent the 2D coordinates and confidence of the i-th joint; in the present invention, according to the movement correlation of each joint of the human body, it is divided into 5 groups, namely the head, left / right hand and left / right leg, that is, G=5;

[0081] Step 3: The 3D pose feature extractor first uses the first fully connected layer The g-th group of 2D joint coordinates Mapping to Features It is a one-dimensional matrix space with a dimension of C / 2; C represents the characteristics of the combined G group of joints The channel dimension of the obtained global feature:

[0082]

[0083] Step 4. Second fully connected network enter Output the mapping matrix of the g-th group of joints is a dimension of (C / 2)×2J g Two-dimensional matrix space; Will Mapped to features with a channel number of C / 2 For modulation

[0084]

[0085]

[0086] Step 5: For each group in group G, and After addition, through the residual network of group g Further extract spatial information to obtain the g-th group of adjusted features

[0087]

[0088] Step 6: Group G features Spliced ​​together, through the third fully connected layer Mapped to a person's global characteristics is a one-dimensional matrix space with dimension C; Concat(f 1 ,f 2 ,…,f G ) represents splicing of G group joint features;

[0089] The global features of N×F frames are concatenated to obtain the features X of all images, where It is a three-dimensional matrix space with dimensions C×N×F.

[0090] Preferably, the view self-attention transformation network in step 2 fuses view features based on a relative camera position encoding module and a self-attention fusion module.

[0091] Preferably, the view self-attention transformation network in step 2 is composed of the following steps:

[0092] The first step is to use the camera dimension. From N camera features Splicing composition, among which, v∈{1,2,...,N}, It is a two-dimensional matrix space with dimension C×F; is the feature of the v-th camera, which is obtained by concatenating the global features of the F-frame images under the v-th camera. In the process of camera feature fusion, the temporal dimension F will be omitted, that is, simplified to v∈{1,2,...,N};

[0093] Step 2: Figure 3 As shown in the figure, the view self-attention transformation network first adaptively learns the relative position relationship between cameras through a neural network, and inputs the query variable of the ath camera and the key value variable of the bth camera Output the relative position relationship mapping matrix M between the a-th camera and the b-th camera ab And feature fusion weighting coefficient A ab ;in, Represent the a-th and b-th camera features respectively; It is a two-dimensional matrix space with dimension D×D;

[0094] C = H × D;

[0095]

[0096]

[0097] in, and are two neural network layers that share the same residual network for obtaining and The relationship features between ; then use the fourth and fifth fully connected layers to output M ab and A ab ;

[0098] Step 3: Change the numerical characteristics of the bth camera The shape is divided into H D-dimensional local feature points in is the numerical feature after the shape is changed under the b-th camera; is a two-dimensional matrix space with dimension D×H; then through M ab right Perform linear mapping to achieve relative camera position encoding:

[0099]

[0100] in, It represents the feature after the a-th camera encodes the b-th camera feature relative to the camera position;

[0101] N×N camera feature combinations The encoded features will be obtained By N×N Splice to get, change The shape of the feature V map ; in They are a four-dimensional matrix space with dimensions D×H×N×N and a three-dimensional matrix space with dimensions C×N×N;

[0102] Step 4: A={A ab |a∈{1,2,…,N},b∈{1,2,…,N}} represents N×N fusion coefficients A ab The fusion coefficient matrix composed of A and B is randomly set to 0 through the random masking strategy to change the number of cameras involved in the fusion.

[0103] Step 5: Use the randomly shielded A to map Perform weighted fusion to obtain the multi-view fusion feature V fuse , It is a two-dimensional matrix space with dimension C×N:

[0104] V fuse =sum((softmax(A)⊙V map )),

[0105] Among them, softmax(*) represents the normalized exponential function, which normalizes the third dimension of the fusion coefficient matrix A, ⊙ represents the dot product, and sum(*) represents the fusion of the features of N cameras in the third dimension.

[0106] Preferably, the temporal self-attention transformation network in step three is composed of two temporal encoders, each of which is composed of a multi-head self-attention and forward propagation network in time.

[0107] Preferably, the temporal self-attention transformation network in step 3 is composed of the following steps:

[0108] The first step is to omit the camera dimension N in the process of temporal feature fusion. The encoding module of the temporal self-attention transformer network first passes through the sixth fully connected layer V fuse Perform feature encoding:

[0109]

[0110] in, The features after feature encoding;

[0111] The second step is to use the cosine and sinine functions to construct the sequence position code. The position encoding of Z is as follows:

[0112] Z 0 =Z+P emb ,

[0113] in, It is the feature after position encoding;

[0114] Step 3: The temporal self-attention transformation network contains two layers of feature fusion modules based on the self-attention mechanism; m∈{1,2} is the index of the number of feature fusion module layers, and the feature fusion process of the mth layer is:

[0115]

[0116]

[0117]

[0118]

[0119] in, and Through the seventh, eighth, and ninth fully connected layers respectively and For the time series features of the m-1 layer The query vector, key value vector and numerical vector obtained by mapping; FFN(*) is a multi-layer perceptron with residual connection; is the temporal fusion feature of the mth layer, where the input feature of the first layer fusion module

[0120] Step 304: Finally, pass through the tenth fully connected layer Regress the 3D pose P of each intermediate frame of the video sequence under N cameras 3D ,in It is a two-dimensional matrix space with dimension 3J×N:

[0121]

[0122] The feature extractor designed in the present invention can effectively use confidence to reduce the noise of the 2D pose detector, thereby extracting robust noise; the designed view self-attention transformation network can adaptively fuse any number of uncalibrated camera features, so that it can be directly applied to any camera configuration scene without retraining; the designed temporal self-attention transformation network can adaptively fuse single-frame and multi-frame temporal features, so that it can be directly applied to static scenes and dynamic video scenes without retraining.

[0123] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with the technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A single-person 3D pose estimation method based on adaptive multi-view and temporal feature fusion, characterized in that: The following steps are involved: Step 1: Based on the video sequence images of multiple cameras as input, a robust feature extractor is used to extract 2D posture features; Step 2: Based on the 2D posture feature as input, an adaptive view self-attention transformation network is designed in the camera dimension. The 2D postures of any number of uncalibrated cameras are fused through relative camera position encoding and self-attention mechanism to obtain multi-view fusion posture features. Step 3: Based on the multi-view fusion posture features as input, a temporal self-attention transformation network is designed in the time dimension. The multi-frame features are adaptively fused through the self-attention mechanism to obtain the final 3D posture.

2. According to the method for single-person 3D pose estimation based on adaptive multi-view and temporal feature fusion according to claim 1, the feature extractor in step 1 is specifically as follows: Step 1: Given N video sequences from cameras, each video sequence contains F frames, and N×F frames share the same feature extractor; Each frame of image I has a width of W and a height of H. Each frame of image I contains three color channels: R, G, and B. It is a three-dimensional matrix space with dimensions W×H×3; each picture contains only one person; The feature extractor includes a 2D posture detector and a 3D posture feature extractor. For each frame of image I, the 2D posture detector is first used Predicting 2D pose information in, The total number of joints is J, P 2D , C 2D Represent the 2D coordinates and confidence of J joints, p j is the 2D coordinate of the jth joint, c j is the confidence of the j-th joint; Step 2: For P 2D and C 2D , J joints are divided into G groups according to the correlation of human joint movements: where g∈{1, 2, ..., G}, They are P 2D and C 2D The g-th subset of ; is a dimension of 2J g One-dimensional matrix space of; is a dimension J g One-dimensional matrix space of Represents the index of all joints in group g, J g is the number of joints in group g, p i and c i Represent the 2D coordinates and confidence of the i-th joint respectively; Step 3: The 3D pose feature extractor first uses the first fully connected layer The g-th group of 2D joint coordinates Mapping to Features It is a one-dimensional matrix space with a dimension of C / 2; C represents the characteristics of the combined G group of joints The channel dimension of the obtained global feature: Step 4. Second fully connected network enter Output the mapping matrix of the g-th group of joints is a dimension of (C / 2)×2J g Two-dimensional matrix space; Will Mapped to features with a channel number of C / 2 For modulation Step 5: For each group in group G, and After addition, through the residual network of group g Further extract spatial information to obtain the g-th group of adjusted features Step 6: Group G features Spliced ​​together, through the third fully connected layer Mapped to a person's global characteristics is a one-dimensional matrix space with dimension C; Concat(f 1 , f 2 , …, f G ) represents splicing of G group joint features; The global features of N×F frames are concatenated to obtain the features X of all images, where It is a three-dimensional matrix space with dimensions C×N×F.

3. The method for single-person 3D posture estimation based on adaptive multi-view and temporal feature fusion according to claim 1, characterized in that: The view self-attention transformation network in step 2 consists of a relative camera position encoder and a view self-attention fusion module.

4. The method for single-person 3D posture estimation based on adaptive multi-view and temporal feature fusion according to claim 2, characterized in that: The view self-attention transformation network in step 2 is obtained by the following steps: Step 201: In the camera dimension, From N camera features Splicing composition, among which, v∈{1, 2, ..., N}, It is a two-dimensional matrix space with dimension C×F; is the feature of the v-th camera, which is obtained by concatenating the global features of the F-frame images under the v-th camera. In the process of camera feature fusion, the temporal dimension F will be omitted, that is, simplified to v∈{1, 2, ..., N}; Step 202: The view self-attention transformation network first adaptively learns the relative position relationship between cameras through a neural network, and inputs the query variable of the ath camera and the key value variable of the bth camera Output the relative position relationship mapping matrix M between the a-th camera and the b-th camera ab And feature fusion weighting coefficient A ab ;in, Represent the a-th and b-th camera features respectively; It is a two-dimensional matrix space with dimension D×D; C = H × D; in, and are two neural network layers that share the same residual network for obtaining and Then use the fourth and fifth fully connected layers to output M ab and A ab ; Step 203: Change the numerical characteristics of the bth camera The shape is divided into H D-dimensional local feature points in is the numerical feature after the shape is changed under the b-th camera; is a two-dimensional matrix space with dimension D×H; then through M ab right Perform linear mapping to achieve relative camera position encoding: in, It represents the feature after the a-th camera encodes the b-th camera feature relative to the camera position; N×N camera feature combinations The encoded features will be obtained By N×N Splice to get, change After the shape is obtained, the feature V map ; in They are a four-dimensional matrix space with dimensions D×H×N×N and a three-dimensional matrix space with dimensions C×N×N; Step 204, A={A ab |a∈{1,2,...,N},b∈{1,2,...,N}} represents N×N fusion coefficients A ab The fusion coefficient matrix composed of A and B is randomly set to 0 through the random masking strategy to change the number of cameras involved in the fusion. Step 205: Use the randomly shielded A to map Perform weighted fusion to obtain the multi-view fusion feature V fuse , It is a two-dimensional matrix space with dimension C×N: V fuse =sum((softmax(A)⊙V map )), Among them, softmax(*) represents the normalized exponential function, which normalizes the third dimension of the fusion coefficient matrix A, ⊙ represents the dot product, and sum(*) represents the fusion of the features of N cameras in the third dimension.

5. The method for single-person 3D posture estimation based on adaptive multi-view and temporal feature fusion according to claim 1, characterized in that: The temporal self-attention transformer network in step 3 consists of an encoding module and a two-layer feature fusion module.

6. The method for single-person 3D posture estimation based on adaptive multi-view and temporal feature fusion according to claim 4, characterized in that: The temporal self-attention transformer network in step 3 is implemented by the following steps: Step 301: In the process of temporal feature fusion, the camera dimension N is omitted, and the encoding module of the temporal self-attention transformation network first passes through the sixth fully connected layer V fuse Perform feature encoding: in, The features after feature encoding; Step 302: The encoding module then uses cosine and sinine functions to construct sequence position encoding. The position encoding of Z is as follows: WITH 0 =Z+P emb , in, It is the feature after position encoding; Step 303, the temporal self-attention transformation network includes two layers of feature fusion modules based on the self-attention mechanism; m∈{1,2} is the index of the number of layers of the feature fusion module, and the feature fusion process of the mth layer is: in, and Through the seventh, eighth, and ninth fully connected layers respectively and For the time series features of the m-1 layer The query vector, key value vector and numerical vector obtained by mapping; FFN(*) is a multi-layer perceptron with residual connection; is the temporal fusion feature of the mth layer, where the input feature of the first layer fusion module Step 304: Finally, pass through the tenth fully connected layer Regress the 3D pose P of each intermediate frame of the video sequence under N cameras 3D ,in It is a two-dimensional matrix space with dimension 3J×N:

Images