3D multi-person human pose estimation method and system in video based on spatio-temporal attention

Abstract

This invention discloses a method and system for 3D multi-person human pose estimation in videos based on spatiotemporal attention, relating to the field of pose estimation technology. This invention improves the human pose estimation scheme based on spatiotemporal feature fusion, effectively reconstructing the pose information of multiple people from monocular video sequences. The model can not only accurately estimate the 3D spatial position of each individual's joints, but also clearly reconstruct the relative positions and spatial hierarchy between people, effectively avoiding pose aliasing and computational complexity problems in multi-person scenes, and possessing excellent multi-person 3D human pose estimation capabilities. Without requiring multi-view or depth sensors, it achieves 3D multi-person human pose estimation results highly consistent with real-world scenes, verifying the effectiveness and practical value of the model in achieving high-precision 3D multi-person human pose estimation under monocular image conditions.

Description

Technical Field

[0001] This invention belongs to the field of pose estimation technology, and in particular relates to a method and system for 3D multi-person human pose estimation in videos based on spatiotemporal attention. Background Technology

[0002] 3D human pose estimation refers to locating the joints of a human body from input data obtained from motion capture devices or videos, thereby improving the understanding of human poses in complex scenes. With the rapid development of deep learning and the increasing availability of real data, research on 3D human pose estimation from monocular images or videos has made significant progress and has been widely applied in various fields such as action recognition, virtual reality, and human-computer interaction, becoming one of the most in-depth research areas in computer vision.

[0003] Research methods for 3D human pose estimation can be mainly divided into two types: direct estimation methods and 2D-to-3D boosting methods. Direct estimation methods do not require intermediate estimation of the 2D pose representation; they directly estimate the 3D human pose from the 2D image. Direct estimation methods offer fast inference speed, convenient end-to-end training, and avoid the accumulation of intermediate errors. However, they are sensitive to data distribution, easily affected by occlusion, and are difficult to train due to the lack of intermediate supervision, requiring a large amount of data. 2D-to-3D methods use existing 2D pose estimation models to predict 2D pose in the first stage, and then estimate the 3D pose through a 2D-to-3D boosting process in the second stage. 2D-to-3D methods can effectively utilize existing high-precision 2D pose estimation results, and their modular structure facilitates the combination and replacement of different components, making training easier.

[0004] Human pose estimation is categorized into single-person pose estimation and multi-person pose estimation based on the number of people in the estimated scene. Multi-person pose estimation is more challenging because it not only requires determining the number of people and camera depth based on single-person pose estimation but also needs to correctly group the keypoints of different individuals. Multi-person pose estimation methods are generally divided into top-down and bottom-up approaches. Top-down methods first estimate the region where each person is located, i.e., the bounding box, and then perform single-person pose estimation for each individual. Bottom-up methods first detect all human keypoints and then group these detected keypoints and assign them to each individual.

[0005] Due to issues such as occlusion and depth ambiguity in 3D human pose estimation, a 2D pose may have multiple reasonable 3D pose assumptions. In research on upscaling from 2D pose to 3D, emerging deep learning methods are mainly based on graph convolutional networks (GCNNs), Transformers, and Mamba. GCNN-based methods treat the skeleton representation of human joints as a graph structure, where joints are considered nodes and bones are considered edges, fusing features of related joints based on human kinematic constraints. Furthermore, since videos provide contextual temporal information, existing methods often fuse information from multiple frames for pose estimation. Temporal convolutional networks can estimate 3D pose based on continuous 2D sequences, but their processing capability for long sequences is limited due to a fixed receptive field size. Transformer technology has a stronger ability to fuse global information, but due to its quadratic complexity, it still suffers from performance bottlenecks in memory and time. Linear Transformers, based on linear attention technology, transform quadratic complexity into linear complexity, offering greater advantages in handling longer temporal contexts. The combination of temporal information and human kinematic constraints can more effectively predict human poses in occluded scenes, ensuring consistency of the human body across video frames.

[0006] In summary, most current 3D human pose estimation methods rely on complex deep learning architectures, fusing long-term temporal contextual information with human kinematic constraints to improve robustness in occluded scenes and alleviate depth ambiguity and inter-frame inconsistency. However, the quadratic computational complexity of the Transformer model leads to a surge in GPU resource requirements, making it difficult to deploy on mobile devices or real-time systems. Some methods employ improved temporal convolutional networks or semi-supervised strategies to reduce annotation dependence, but these suffer from lower accuracy in long sequences or highly occluded scenes due to limitations in receptive field fixedness and insufficient global modeling capabilities. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, this invention provides a method and system for estimating the pose of multiple people in 3D videos based on spatiotemporal attention, thereby solving the problems of high computational overhead of attention mechanisms and temporal coherence in long sequence modeling in existing methods.

[0008] The technical solution of this invention is as follows:

[0009] On one hand, this invention provides a method for estimating the pose of multiple people in a video based on spatiotemporal attention, comprising the following steps:

[0010] Obtain human body 3D key points and human body 2D key points datasets, and divide them into training set and test set according to a set ratio;

[0011] The human body 3D key points and human body 2D key points dataset includes several samples. Each sample is a series of human body images segmented from a video containing a human body, and the human body 3D key points and human body 2D key points corresponding to each human body image; the key points are joints in the human skeleton.

[0012] Construct a spatiotemporal linear attention model;

[0013] The spatiotemporal linear attention model takes multiple consecutive frames of human images and their corresponding 2D pose sequences as input, performs 3D human pose estimation to obtain 3D human relative pose, i.e., single-person 3D pose sequence, and then combines the estimated human depth to perform coordinate transformation, finally obtaining the single-person absolute pose in the scene; the 2D pose sequence includes the single-person 2D key points corresponding to each frame of human image.

[0014] The fusion spatiotemporal linear attention model is trained using the training set to obtain the trained fusion spatiotemporal linear attention model;

[0015] The test set is input into the trained spatiotemporal linear attention model to obtain the individual absolute pose of all human bodies in the same scene.

[0016] By integrating the absolute poses of all individuals, a multi-person 3D pose sequence is obtained.

[0017] Furthermore, the fusion spatiotemporal linear attention model includes a linear embedding module, a spatiotemporal encoder module, a regression head, and a pelvic depth estimator;

[0018] The linear embedding module is used to map the coordinates of 2D key points of a single person in a 2D pose sequence to a high-dimensional coordinate space through linear embedding, and obtain high-dimensional features as input to the spatiotemporal encoder module.

[0019] The spatiotemporal encoder module comprises L stacked spatiotemporal processing units. Each spatiotemporal processing unit includes a spatial attention module with a dual-branch structure and a temporally gated attention module. The spatial attention module with the dual-branch structure learns the spatial dependencies between the human skeleton structure and joints at both the local and global levels to obtain spatial augmented features. The temporally gated attention module models the joint motion changes and long-term temporal correlations of the spatial augmented features to obtain spatiotemporal augmented features. Finally, the spatiotemporal augmented features output by the Lth spatiotemporal processing unit are used as the input to the regression head.

[0020] The regression head is used to regress the spatiotemporal enhancement features output by the spatiotemporal encoder module through a linear layer, mapping the feature dimension to 3 to obtain a single person 3D pose sequence.

[0021] The pelvic depth estimator takes multiple consecutive frames of human images as input to estimate the depth of the human body, and combines them with a single person's 3D pose sequence to generate the single person's absolute pose in the scene.

[0022] Furthermore, the spatiotemporal encoder module logically first treats the input high-dimensional features as... That is, shared Let G = (V, E) be an independent undirected graph, where joints are used as nodes. Each undirected graph includes... There are nodes, and the feature dimension of each node is . V is the set of nodes; the human skeleton connections between joints are called edges, and E is the set of edges.

[0023] Introducing a learnable spatial location encoding into high-dimensional features The initial feature tensor is obtained and used as the input to the spatial attention module with a dual-branch structure in the first spatiotemporal processing unit;

[0024] (2);

[0025] in, This is the initial feature tensor; For learnable spatial location encoding;

[0026] The spatiotemporal encoder module treats nodes as independent samples and logically reassembles the spatial enhancement features output by the spatial attention module into... That is, shared 3 independent time series, each series having a length of 1. The feature dimension is Temporal location encoding is added through temporal location embedding; specifically, spatial enhancement features are added to the output of the spatial module. Perform dimensional transformation and add temporal position encoding to the time dimension. ;

[0027] (3);

[0028] in, This represents the feature vector after time-location encoding;

[0029] The time-location encoded feature vector is input into the temporal-gated attention module in the first spatiotemporal processing unit to obtain the spatiotemporal enhanced features. For the spatial attention modules in the second to Lth spatiotemporal processing units, the spatiotemporal enhanced features output by the temporal-gated attention module in the previous spatiotemporal processing unit are used as inputs. For the temporal-gated attention modules in the second to Lth spatiotemporal processing units, the enhanced features output by the spatial attention module in the current spatiotemporal processing unit are used as inputs.

[0030] Furthermore, regarding the first Input feature tensors of the spatial attention module in each spatiotemporal processing unit First, root mean square normalization is performed to stabilize training, unify feature scale, and accelerate convergence. Then, a two-branch parallel computation strategy is employed on the root-mean square normalized feature tensor, utilizing both local and global spatial feature branches to compute local spatial features. and global spatial features ;

[0031] In the local spatial feature branch, a high-order graph convolutional network is used to extract local spatial features. By utilizing the kinetic chain structure relationships in human kinematics, a high-order adjacency matrix that integrates human kinematic relationships is designed, namely, a kinematic adjacency matrix. This explicitly encodes the first-order and second-order connections of distal joints, while retaining only first-order connections for joints other than distal joints. The kinematic adjacency matrix... The adjacency weight in the diagram represents the connection relationship between joints. When there is a first-order connection between two joints, the corresponding adjacency weight value is 1. When there is a second-order connection between a distal joint and other joints, the corresponding adjacency weight value is 1. Otherwise, the adjacency weight value is 0. The kinetic chain structure relationship includes the lower limb chain: ankle-knee-hip; the upper limb chain: wrist-elbow-shoulder; and the axial structure: head-neck-chest. The distal joint is an end joint with only one connection.

[0032] The local spatial features The calculation formula is:

[0033] (4);

[0034] in, This indicates the root mean square normalization operation. This is a learnable channel transformation matrix; It is a learnable mask matrix; This indicates element-wise multiplication; It is a non-linear activation function.

[0035] In the global spatial feature branch, a global gated graph attention mechanism is adopted for the th The formula for calculating the output of each attention head is as follows:

[0036] (5);

[0037] in, For the first The output of each attention head, The number of the attention head; These are the query, key, and value matrices, respectively. It is the first The learnable global adjacency matrix of each attention head is initialized to a zero matrix; For the Softmax function; Let k be the feature dimension of the attention head; For transpose; For the Sigmoid function, For gating parameters, Let be the learnable gating weight matrix for the k-th attention head;

[0038] All The outputs of each attention point are concatenated and processed through the output matrix. The final global spatial features are obtained by fusion. The output shape remains unchanged. ;

[0039] (6);

[0040] in, Indicates splicing;

[0041] Local spatial features and global spatial features Add them together, and then use the residual connection with the input feature tensor. By combining these features, spatial aggregation characteristics are obtained. ;

[0042] (7);

[0043] Subsequently, spatial aggregation features After root mean square normalization, the input is fed into a multilayer perceptron module, which includes two fully connected layers and a nonlinear activation function. Then, a second residual connection is introduced to obtain the... The final output of the spatial attention module in each spatiotemporal processing unit, i.e., the spatial augmentation feature. ;

[0044] (8);

[0045] in, It is a multilayer perceptron module.

[0046] Furthermore, the temporal-gated attention module first performs layer normalization on the input feature vector; it then introduces a bidirectional scanning mechanism to arrange the layer-normalized feature vector in a forward and reverse order along the time axis, generating sequence streams in two directions; these two sequence streams are then input into a temporal-gated linear attention structure to capture past and future temporal dependencies, resulting in feature sequences in two directions; finally, the feature sequences in the two directions are bidirectionally fused, specifically by adding the forward feature sequence element-wise to the reversed feature sequence restored to forward order, resulting in a fused feature vector; a residual connection is used to introduce the layer-normalized feature vector into the fused feature vector, which is then input into a multilayer perceptron; finally, a residual connection is used to introduce the fused feature vector into the output of the multilayer perceptron, restoring it to its original state. Shape, as a spatiotemporal enhancement feature;

[0047] (9);

[0048] (10);

[0049] in, The input feature vector; LN denotes layer normalization; It is a bidirectional scan; This represents a temporally gated linear attention structure; Indicates two-way fusion; To fuse feature vectors; It is a spatiotemporal enhancement feature.

[0050] Furthermore, the processing procedure of the pelvic depth estimator is as follows:

[0051] First, the 2D coordinates of the pelvic nodes are extracted from the 2D key points. Then, the depth of the pelvic nodes is estimated based on the human body image using RootNet to obtain the three-dimensional coordinates of the pelvic nodes of each human body.

[0052] (11);

[0053] in, Here are the three-dimensional coordinates of the pelvic nodes. Numbering of the human body; and Let be the coordinates of the pelvic nodes of the human body in the human body image plane. It is the depth of the pelvic joints in the human body;

[0054] Secondly, the three-dimensional pixel coordinates of the pelvic nodes are back-projected into absolute coordinates in the camera coordinate system using the camera's intrinsic parameters. ;

[0055] (12);

[0056] in, absolute coordinates The components in; The coordinates of the principal point; Focal length;

[0057] Then, the single-person 3D pose sequence Perform unit transformation and translation to the root node position in the camera coordinate system:

[0058] (13);

[0059] in, The absolute 3D pose after translation;

[0060] The translated absolute 3D pose is reprojected back onto the 2D image plane to obtain the absolute 2D pose.

[0061] (14);

[0062] in, , , Let be the coordinates of a joint in the absolute 3D pose after translation. and The coordinates of a joint in an absolute 2D pose;

[0063] Finally, the average scale factor s is calculated based on the absolute 2D pose and then used to globally scale and correct the single-person 3D pose sequence.

[0064] Specifically, this involves calculating the ratio of the torso length in the original 2D human keypoints to the torso length in the absolute 2D pose:

[0065] (15);

[0066] in, A scale factor for the human body. The torso length in the original 2D human keypoints. The torso length in absolute 2D pose;

[0067] The average scaling factor is:

[0068] (16);

[0069] in, For the number of people;

[0070] The single-person 3D pose sequence is globally scaled and corrected as follows:

[0071] (17);

[0072] in, This is a single, absolute stance.

[0073] On the other hand, the present invention also provides a 3D multi-person human pose estimation system in video based on spatiotemporal attention, used to implement a method for 3D multi-person human pose estimation in video based on spatiotemporal attention, including:

[0074] The data acquisition and segmentation module is used to acquire human 3D key points and human 2D key points datasets, and divide them into training set and test set according to a set ratio;

[0075] The model building module is used to build a spatiotemporal linear attention model.

[0076] The training module is used to train the fusion spatiotemporal linear attention model using the training set, and obtain the trained fusion spatiotemporal linear attention model.

[0077] The single-person pose estimation module is used to input the test set into the trained fusion spatiotemporal linear attention model to obtain the single-person absolute pose of all human bodies in the same scene.

[0078] The multi-person pose estimation module is used to integrate the absolute poses of all individuals to obtain a multi-person 3D pose sequence.

[0079] Thirdly, this application proposes an electronic device comprising: one or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the aforementioned spatiotemporal attention-based 3D multi-person human pose estimation method in video.

[0080] Fourthly, this application proposes a computer-readable storage medium storing executable instructions that, when executed, cause a processor to perform the aforementioned method for estimating the pose of multiple people in a video based on spatiotemporal attention.

[0081] Fifthly, this application proposes a computer program product, including a computer program or instructions that, when executed by a processor, implement the aforementioned method for estimating 3D multi-person human pose in a video based on spatiotemporal attention.

[0082] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0083] This invention, based on a 2D to 3D human pose enhancement process, addresses the limited expressive power of simple spatial skeleton graph models caused by occlusion by introducing a graph structure attention modeling mechanism. By constructing attention modules on the graph structure of key human points, it adaptively models the kinematics and symmetry relationships of the human body in space, thereby enhancing the network's ability to extract human structural characteristics. Unlike existing graph attention mechanisms that directly aggregate global features, the gated spatial attention module in this invention uses an improved input-dependent sparse gating mechanism. After attention aggregation and before output, a non-linear gating is added, which not only acts as an activation function but also dynamically closes or filters out invalid joint connections caused by occlusion or detection errors based on the characteristics of the current node.

[0084] To address the depth ambiguity issue in 2D pose estimation for 3D, this invention introduces a linear gated attention mechanism based on GatedDeltaNet in the temporal dimension. This mechanism performs global temporal modeling of human poses in videos, ensuring the temporal coherence of long-sequence human pose modeling. Simultaneously, to address the high computational cost of existing attention mechanisms, this invention dynamically filters key spatial and temporal information through gating, combining fast attention erasure with precise memory modification of the Delta Rule to construct an explicit pose memory and correction mechanism. While maintaining linear computational complexity, it effectively captures long-term dependencies across frames. Using an embedded temporal bidirectional scanning and feature fusion architecture, the linear attention layer of GatedDeltaNet is wrapped within a bidirectional scanning path, enabling the model to utilize not only past information but also future information to constrain the current 3D pose estimation when processing video frames.

[0085] Finally, when aligning the human scale of multiple people in a scene, the relative human coordinates are converted into absolute human pose by combining the results of pelvic depth estimation, and the scaling factor is calculated to calibrate the scaling error, thereby solving the scale uncertainty problem of multi-person pose estimation.

[0086] This invention improves the human pose estimation scheme based on spatiotemporal feature fusion, effectively reconstructing the pose information of multiple people from monocular video sequences. The model not only accurately estimates the 3D spatial position of each individual's joints but also clearly reconstructs the relative positions and spatial hierarchy between people, effectively avoiding pose aliasing and computational complexity issues in multi-person scenes, demonstrating excellent multi-person 3D human pose estimation capabilities. Without requiring multi-view or depth sensors, it achieves 3D multi-person human pose estimation results highly consistent with real-world scenes, verifying the effectiveness and practical value of the model in achieving high-precision 3D multi-person human pose estimation under monocular image conditions. Attached Figure Description

[0087] Figure 1This is a flowchart of a method for estimating the pose of multiple people in a video based on spatiotemporal attention, as described in an embodiment of the present invention.

[0088] Figure 2 This is a structural diagram of the spatiotemporal linear attention model fused in an embodiment of the present invention;

[0089] Figure 3 This is a diagram illustrating the effect of 3D multi-person human pose estimation in an embodiment of the present invention. Detailed Implementation

[0090] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0091] Example 1:

[0092] Methods for 3D multi-person human pose estimation in videos based on spatiotemporal attention, such as Figure 1 As shown, it includes the following steps:

[0093] Step 1: Obtain the human body 3D key points and human body 2D key points dataset, and divide them into training set and test set according to the set ratio;

[0094] The human body 3D key points and human body 2D key points dataset includes several samples. Each sample is a series of human body images segmented from a video containing a human body, and the human body 3D key points and human body 2D key points corresponding to each human body image; the key points are joints in the human skeleton.

[0095] In this embodiment, the publicly available 3D human pose estimation dataset Human3.6M is first obtained. 3D human keypoints and their corresponding 2D keypoints are extracted from the original data source. Specifically, S1, S5, S6, S7, and S8 are downloaded as the training set, and S9 and S11 as the test set. All compressed packages named Poses_D3_Positions_S*.tgz are decompressed, the data is preprocessed, and parsed and converted using the cdflib library to finally generate standardized "data_3d_h36m.npz" and "data_2d_h36m_gt.npz" files. In addition, to enhance the generalization ability of the algorithm, human 2D key points from various sources were introduced, including human 2D key points obtained based on Mask R-CNN and CPN models. These 2D detection models (Mask R-CNN and CPN models) were pre-trained or fine-tuned on COCO or Human3.6M datasets respectively, and finally uniformly converted to npz format so as to obtain human 2D key points and human 3D key points for use.

[0096] Step 2: Construct a spatiotemporal linear attention model;

[0097] The spatiotemporal linear attention model takes multiple consecutive frames of human images and their corresponding 2D pose sequences as input, performs 3D human pose estimation to obtain 3D human relative pose, i.e., single-person 3D pose sequence, and then combines the estimated human depth to perform coordinate transformation, finally obtaining the single-person absolute pose in the scene; the 2D pose sequence includes the single-person 2D key points corresponding to each frame of human image.

[0098] The fusion spatiotemporal linear attention model, such as Figure 2 As shown, it includes a linear embedding module, a spatiotemporal encoder module, a regression head, and a pelvic depth estimator;

[0099] The linear embedding module is used to embed the coordinates of 2D key points of a single person in a 2D pose sequence. , The number of human joints is represented by a linear embedding and mapped to a high-dimensional coordinate space to obtain high-dimensional features, which are then used as input to the spatiotemporal encoder module.

[0100] The spatiotemporal encoder module comprises L stacked spatiotemporal processing units. Each spatiotemporal processing unit includes a spatial attention module with a dual-branch structure and a temporally gated attention module. The spatial attention module with a dual-branch structure learns the spatial dependencies between the human skeleton structure and joints at both the local and global levels to obtain spatially enhanced features. The temporally gated attention module models the joint motion changes and long-term temporal correlations of the spatially enhanced features to obtain spatiotemporally enhanced features. By alternating between the spatial attention module with a dual-branch structure and the temporally gated attention module, and keeping the feature dimension constant, the model can continuously fuse spatial structural information and temporal dynamic information at the same joint, thereby gradually enhancing the overall spatiotemporal consistency. Finally, the spatiotemporal enhanced features output by the Lth spatiotemporal processing unit are used as the input to the regression head.

[0101] The regression head is used to regress the spatiotemporal enhancement features output by the spatiotemporal encoder module through a linear layer, mapping the feature dimension to 3 to obtain a single person 3D pose sequence.

[0102] The pelvic depth estimator takes multiple consecutive frames of human images as input to estimate the depth of the human body, and combines them with a single person's 3D pose sequence to generate the single person's absolute pose in the scene.

[0103] To map low-dimensional coordinate information to a high-dimensional feature space, the linear embedding process of the linear embedding module is as follows:

[0104] (1);

[0105] in, This represents the high-dimensional features after linear mapping. For batch size, The number of frames representing human images in the video. The number of joints in the human body. For feature embedding dimension; It is a 2D pose sequence, and its tensor shape is , Number of coordinate channels; and These are the learnable weight matrix and bias vector of the linear layer, respectively. This linear transformation acts on the last dimension of the input tensor.

[0106] The spatial attention module with a dual-branch structure is used to model the relationships between different joints within each frame, in the time dimension ( In this step, the high-dimensional features are treated as independent samples. Therefore, to facilitate subsequent calculations of higher-order graph convolution and global gating attention, the spatiotemporal encoder module logically first treats the input high-dimensional features as independent samples. That is, shared Let G = (V, E) be an independent undirected graph, where joints are used as nodes. Each undirected graph includes... There are nodes, and the feature dimension of each node is . V is the set of nodes; the human skeleton connections between joints are called edges, and E is the set of edges.

[0107] To preserve the spatial identity information of joints in an undirected graph with invariant permutations, a learnable spatial location code is introduced into the high-dimensional features. The initial feature tensor is obtained and used as the input to the spatial attention module with a dual-branch structure in the first spatiotemporal processing unit;

[0108] (2);

[0109] in, This is the initial feature tensor; To encode learnable spatial locations, a broadcast mechanism is used to add them to high-dimensional features. On each frame and each sample, unique spatial location information is used to identify different joints in the human skeleton;

[0110] The temporal gating attention module models the joint trajectory in the time dimension. This module mainly focuses on different frames in the temporal trajectory of each node. The spatiotemporal encoder module models the relationships between spatial nodes ( ), therefore the spatiotemporal encoder module models the relationships between spatial nodes ( ). Treating these as independent samples, and in order to capture temporal dependencies, the spatial augmentation features output by the spatial attention module are logically reorganized into... That is, shared 3 independent time series, each series having a length of 1. The feature dimension is Temporal location encoding is added through temporal location embedding to identify the position of this human image frame in the 2D pose sequence. Specifically, this enhances the spatial features output by the spatial module. Perform dimensional transformation and add temporal position encoding to the time dimension. ;

[0111] (3);

[0112] in, This represents the feature vector after time-location encoding;

[0113] The time-position encoded feature vector is input into the temporal-gated attention module in the first spatiotemporal processing unit to obtain the spatiotemporal enhanced features. For the spatial attention modules in the second to Lth spatiotemporal processing units, the spatiotemporal enhanced features output by the temporal-gated attention module in the previous spatiotemporal processing unit are used as inputs. For the temporal-gated attention modules in the second to Lth spatiotemporal processing units, the enhanced features output by the spatial attention module in the current spatiotemporal processing unit are used as inputs.

[0114] Regarding the first Input feature tensors of the spatial attention module in each spatiotemporal processing unit First, root mean square normalization is performed to stabilize training, unify feature scale, and accelerate convergence. Then, a two-branch parallel computation strategy is employed on the root-mean square normalized feature tensor, utilizing both local and global spatial feature branches to compute local spatial features. and global spatial features ;

[0115] In the local spatial feature branch, a high-order graph convolutional network is used to extract local spatial features. By considering nodes that are not directly connected but are part of the human kinematics relationship, the accuracy of joint prediction is improved. Utilizing the kinetic chain structure in the human kinematics relationship, a high-order adjacency matrix integrating the human kinematics relationship is designed, namely the kinematic adjacency matrix. This explicitly encodes the first-order and second-order connections of distal joints. To save computation, only first-order connections are retained for joints other than distal joints. The kinematic adjacency matrix... The adjacency weight in the diagram represents the connection relationship between joints. When there is a first-order connection between two joints, the corresponding adjacency weight value is 1. When there is a second-order connection between a distal joint and other joints, the corresponding adjacency weight value is 1. Otherwise, the adjacency weight value is 0. The kinetic chain structure relationship includes the lower limb chain: ankle-knee-hip; the upper limb chain: wrist-elbow-shoulder; and the axial structure: head-neck-chest. The distal joint is an end joint with only one connection (such as the wrist and shoulder).

[0116] The local spatial features The calculation formula is:

[0117] (4);

[0118] in, This indicates the root mean square normalization operation. This is a learnable channel transformation matrix; It is a learnable mask matrix used to adaptively adjust the adjacency weights; This indicates element-wise multiplication; It is a non-linear activation function;

[0119] In the global spatial feature branch, a global gated graph attention mechanism is used to learn pose semantic representations in a data-driven manner, which adaptively and efficiently encodes global relations.

[0120] For the attention of the global gating graph, the first The formula for calculating the output of each attention head is as follows:

[0121] (5);

[0122] in, For the first The output of each attention head, The number of the attention head; These are the query, key, and value matrices, respectively. It is the first Each attention head has a learnable global adjacency matrix, initialized to zero. Its adjacency weights do not depend on the features of specific nodes but are freely updated during training. These weights represent the existence and strength of the connection between any two nodes. It can create new connections, offering greater flexibility; For the Softmax function; Let k be the feature dimension of the attention head; For transpose; For the Sigmoid function, For gating parameters, Let be the learnable gating weight matrix for the k-th attention head; this gating mechanism (i.e., in the formula) The item introduces sparsity of input dependencies, which can dynamically suppress irrelevant node connections (such as noise caused by occlusion) based on the current action.

[0123] All The outputs of each attention point are concatenated (Concat) and then processed through the output matrix. The final global spatial features are obtained by fusion. The output shape remains unchanged. ;

[0124] (6);

[0125] in, Indicates splicing;

[0126] Local spatial features and global spatial features Add them together, and then use the residual connection with the input feature tensor. By combining these features, spatial aggregation characteristics are obtained. ;

[0127] (7);

[0128] Subsequently, in order to further enhance the model's ability to express spatial features nonlinearly, spatial aggregate features were incorporated. After root mean square normalization, the input is fed into a multilayer perceptron module, which includes two fully connected layers and a nonlinear activation function. Then, a second residual connection is introduced to obtain the... The final output of the spatial attention module in each spatiotemporal processing unit, i.e., the spatial augmentation feature. ;

[0129] (8);

[0130] in, This is a multilayer perceptron module; the final spatial augmentation features are obtained. Maintain Dimension constant;

[0131] The temporally gated attention module first performs layer normalization on the input feature vector. To address the issue that causal linear attention cannot utilize future information, a bidirectional scanning mechanism is introduced, arranging the layer-normalized feature vector in both forward and reverse directions along the time axis to generate two-directional sequence streams. These two-directional sequence streams are then input into a temporally gated linear attention structure (Gated DeltaNet) to capture past and future temporal dependencies, resulting in two-directional feature sequences. The two-directional feature sequences are then bidirectionally fused: specifically, the forward feature sequence is added element-wise to the reversed feature sequence (restored to forward order) to obtain a fused feature vector. A residual connection is used to introduce the layer-normalized feature vector into the fused feature vector, which is then input into a multilayer perceptron. Finally, a residual connection is used to introduce the fused feature vector into the output of the multilayer perceptron, restoring it to its original state. Shape, as a spatiotemporal enhancement feature; the temporally gated linear attention structure (Gated DeltaNet) and the bidirectional scanning mechanism along... The time axis performs linear attention computations for both forward and backward directions. This process exchanges information over a global time scale while preserving the sequence length. constant;

[0132] (9);

[0133] (10);

[0134] in, The input feature vector; LN denotes layer normalization; It is a bidirectional scan; This represents a temporally gated linear attention structure (Gated DeltaNet). Indicates two-way fusion; To fuse feature vectors; For spatiotemporal enhancement features;

[0135] The processing procedure of the pelvic depth estimator is as follows:

[0136] First, the 2D coordinates of the pelvic nodes are extracted from the 2D key points. Then, the depth of the pelvic nodes is estimated based on the human body image using RootNet to obtain the three-dimensional coordinates of the pelvic nodes of each human body.

[0137] (11);

[0138] in, Here are the three-dimensional coordinates of the pelvic nodes. Numbering of the human body; and Let be the coordinates of the pelvic nodes of the human body in the human body image plane. It is the depth of the pelvic joints in the human body;

[0139] Secondly, the three-dimensional pixel coordinates of the pelvic nodes are back-projected into absolute coordinates in the camera coordinate system using the camera's intrinsic parameters. ;

[0140] (12);

[0141] in, absolute coordinates The components in; The coordinates of the principal point; Focal length;

[0142] Then, the single-person 3D pose sequence Perform unit conversion (to millimeters) and translate to the root node position in the camera coordinate system:

[0143] (13);

[0144] in, The absolute 3D pose after translation;

[0145] Then, the translated absolute 3D pose is reprojected back onto the 2D image plane to obtain the absolute 2D pose;

[0146] (14);

[0147] in, , , Let be the coordinates of a joint in the absolute 3D pose after translation. and The coordinates of a joint in an absolute 2D pose;

[0148] Finally, the average scale factor s is calculated based on the absolute 2D pose and then used to globally scale and correct the single-person 3D pose sequence.

[0149] Specifically, this involves calculating the ratio of the torso length in the original 2D human keypoints to the torso length in the absolute 2D pose:

[0150] (15);

[0151] in, A scale factor for the human body. The torso length in the original 2D human keypoints. The torso length in absolute 2D pose;

[0152] The average scaling factor is:

[0153] (16);

[0154] in, For the number of people;

[0155] The single-person 3D pose sequence is globally scaled and corrected as follows:

[0156] (17);

[0157] in, This is a single, absolute posture;

[0158] Step 3: Train the fusion spatiotemporal linear attention model using the training set to obtain the trained fusion spatiotemporal linear attention model;

[0159] Step 4: Input the test set into the trained spatiotemporal linear attention model to obtain the individual absolute poses of all human bodies in the same scene;

[0160] Step 5: Integrate the absolute poses of all individuals to obtain a multi-person 3D pose sequence, such as... Figure 3 As shown;

[0161] The method in this embodiment is applicable to the estimation of 3D human poses of multiple people from monocular video. It captures the relationship between human kinematics and structure through graph attention, models global time dependence through temporal attention, and combines pelvic depth and camera intrinsics to realize the conversion from relative pose to absolute pose, thus completing the 3D pose reconstruction of multiple people and achieving higher accuracy.

[0162] Table 1 Comparison of MPJPE of the present invention and other methods on the Human 3.6M dataset;

[0163]

[0164] Table 2 Comparison of P-MPJPE of the present invention and other methods on the Human 3.6M dataset;

[0165]

[0166] Table 3 Comparison of MPJVE between the present invention and other methods on the Human 3.6M dataset;

[0167]

[0168] Example 2:

[0169] A 3D multi-person human pose estimation system based on spatiotemporal attention in video is provided to implement a method for 3D multi-person human pose estimation in video based on spatiotemporal attention, comprising:

[0170] The data acquisition and segmentation module is used to acquire human 3D key points and human 2D key points datasets, and divide them into training set and test set according to a set ratio;

[0171] The model building module is used to build a spatiotemporal linear attention model.

[0172] The training module is used to train the fusion spatiotemporal linear attention model using the training set, and obtain the trained fusion spatiotemporal linear attention model.

[0173] The single-person pose estimation module is used to input the test set into the trained fusion spatiotemporal linear attention model to obtain the single-person absolute pose of all human bodies in the same scene.

[0174] The multi-person pose estimation module is used to integrate the absolute poses of all individuals to obtain a multi-person 3D pose sequence.

[0175] Example 4:

[0176] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the aforementioned 3D multi-person human pose estimation method in video based on spatiotemporal attention.

[0177] The electronic device may be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the 3D multi-person human pose estimation method in video based on spatiotemporal attention as described in the embodiments. It is understood that the electronic device may also include input / output (I / O) interfaces and communication components.

[0178] The processor is used to execute all or part of the steps in the spatiotemporal attention-based 3D multi-person human pose estimation method in video as described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0179] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the spatiotemporal attention-based 3D multi-person human pose estimation method in video described in the above embodiments.

[0180] Example 5:

[0181] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0182] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the spatiotemporal attention-based 3D multi-person human pose estimation method in video described in various embodiments of this application.

[0183] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disk, optical disk, server, APP (Application) application store, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the aforementioned spatiotemporal attention-based 3D multi-person human pose estimation method in video.

[0184] Example 6:

[0185] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned method for estimating 3D multi-person human pose in video based on spatiotemporal attention.

[0186] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0187] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0188] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of this disclosure and its equivalents, then the intent of this disclosure also includes these modifications and variations.

Claims

1. A method for estimating the pose of multiple people in 3D video based on spatiotemporal attention, characterized in that, Includes the following steps: Obtain human body 3D key points and human body 2D key points datasets, and divide them into training set and test set according to a set ratio; The human body 3D key points and human body 2D key points dataset includes several samples. Each sample is a series of human body images segmented from a video containing a human body, and the human body 3D key points and human body 2D key points corresponding to each human body image; the key points are joints in the human skeleton. Construct a spatiotemporal linear attention model; The spatiotemporal linear attention model takes multiple consecutive frames of human images and their corresponding 2D pose sequences as input, performs 3D human pose estimation to obtain 3D human relative pose, i.e., single-person 3D pose sequence, and then combines the estimated human depth to perform coordinate transformation, finally obtaining the single-person absolute pose in the scene; the 2D pose sequence includes the single-person 2D key points corresponding to each frame of human image. The fusion spatiotemporal linear attention model is trained using the training set to obtain the trained fusion spatiotemporal linear attention model; The test set is input into the trained spatiotemporal linear attention model to obtain the individual absolute pose of all human bodies in the same scene. By integrating the absolute poses of all individuals, a multi-person 3D pose sequence is obtained.

2. The method for estimating 3D multi-person human pose in video based on spatiotemporal attention according to claim 1, characterized in that, The fusion spatiotemporal linear attention model includes a linear embedding module, a spatiotemporal encoder module, a regression head, and a pelvic depth estimator. The linear embedding module is used to map the coordinates of 2D key points of a single person in a 2D pose sequence to a high-dimensional coordinate space through linear embedding, and obtain high-dimensional features as input to the spatiotemporal encoder module. The spatiotemporal encoder module includes L stacked spatiotemporal processing units. Each spatiotemporal processing unit includes a spatial attention module with a dual-branch structure and a temporal gating attention module. The spatial attention module with a dual-branch structure is used to learn the spatial dependencies between human skeleton structures and joints at the local and global levels on the high-dimensional input features to obtain spatially enhanced features. By using a temporally gated attention module, the spatial augmentation features are modeled by the motion changes of joints and their long-term temporal correlations to obtain the spatiotemporal augmentation features. Finally, the spatiotemporal enhancement features output by the Lth spatiotemporal processing unit are used as the input to the regression head; The regression head is used to regress the spatiotemporal enhancement features output by the spatiotemporal encoder module through a linear layer, mapping the feature dimension to 3 to obtain a single person 3D pose sequence. The pelvic depth estimator takes multiple consecutive frames of human images as input to estimate the depth of the human body, and combines them with a single person's 3D pose sequence to generate the single person's absolute pose in the scene.

3. The method for estimating 3D multi-person human pose in video based on spatiotemporal attention according to claim 2, characterized in that, The spatiotemporal encoder module logically first treats the input high-dimensional features as... That is, shared Let G = (V, E) be an independent undirected graph, where joints are used as nodes. Each undirected graph includes... There are nodes, and the feature dimension of each node is . V is the set of nodes; the human skeleton connections between joints are called edges, and E is the set of edges. Introducing a learnable spatial location encoding into high-dimensional features The initial feature tensor is obtained and used as the input to the spatial attention module with a dual-branch structure in the first spatiotemporal processing unit; （2）； in, This is the initial feature tensor; For learnable spatial location encoding; The spatiotemporal encoder module treats nodes as independent samples and logically reassembles the spatial enhancement features output by the spatial attention module into... That is, shared 3 independent time series, each series having a length of 1. The feature dimension is Temporal location encoding is added through temporal location embedding; specifically, spatial enhancement features are added to the output of the spatial module. Perform dimensional transformation and add temporal position encoding to the time dimension. ; （3）； in, This represents the feature vector after time-location encoding; The time-location encoded feature vector is input into the temporal-gated attention module in the first spatiotemporal processing unit to obtain the spatiotemporal enhanced features. For the spatial attention modules in the second to Lth spatiotemporal processing units, the spatiotemporal enhanced features output by the temporal-gated attention module in the previous spatiotemporal processing unit are used as inputs. For the temporal-gated attention modules in the second to Lth spatiotemporal processing units, the enhanced features output by the spatial attention module in the current spatiotemporal processing unit are used as inputs.

4. The method for estimating 3D multi-person human pose in video based on spatiotemporal attention according to claim 3, characterized in that, Regarding the first Input feature tensors of the spatial attention module in each spatiotemporal processing unit First, root mean square normalization is performed to stabilize training, unify feature scale, and accelerate convergence. Then, a two-branch parallel computation strategy is employed on the root-mean square normalized feature tensor, utilizing both local and global spatial feature branches to compute local spatial features. and global spatial features ; In the local spatial feature branch, a high-order graph convolutional network is used to extract local spatial features. By utilizing the kinetic chain structure relationships in human kinematics, a high-order adjacency matrix that integrates human kinematic relationships is designed, namely, a kinematic adjacency matrix. This explicitly encodes the first-order and second-order connections of distal joints, while retaining only the first-order connections for joints other than distal joints. The kinematic adjacency matrix... The adjacency weight in the diagram represents the connection relationship between joints. When there is a first-order connection between two joints, the corresponding adjacency weight value is 1. When there is a second-order connection between a distal joint and other joints, the corresponding adjacency weight value is 1. Otherwise, the adjacency weight value is 0. The kinetic chain structure relationship includes the lower limb chain: ankle-knee-hip; the upper limb chain: wrist-elbow-shoulder; and the axial structure: head-neck-chest. The distal joint is an end joint with only one connection. The local spatial features The calculation formula is: （4）； in, This indicates the root mean square normalization operation. This is a learnable channel transformation matrix; It is a learnable mask matrix; This indicates element-wise multiplication; It is a non-linear activation function; In the global spatial feature branch, a global gated graph attention mechanism is adopted for the th The formula for calculating the output of each attention head is as follows: （5）； in, For the first The output of each attention head, The number of the attention head; These are the query, key, and value matrices, respectively. It is the first The learnable global adjacency matrix of each attention head is initialized to a zero matrix; For the Softmax function; Let k be the feature dimension of the attention head; For transpose; For the Sigmoid function, For gating parameters, Let be the learnable gating weight matrix for the k-th attention head; All The outputs of each attention point are concatenated and processed through the output matrix. The final global spatial features are obtained by fusion. The output shape remains unchanged. ; （6）； in, Indicates splicing; Local spatial features and global spatial features Add them together, and then use the residual connection with the input feature tensor. By combining these features, spatial aggregation characteristics are obtained. ; （7）； Subsequently, spatial aggregation features After root mean square normalization, the input is fed into a multilayer perceptron module, which includes two fully connected layers and a nonlinear activation function. Then, a second residual connection is introduced to obtain the... The final output of the spatial attention module in each spatiotemporal processing unit, i.e., the spatial augmentation feature. ; （8）； in, It is a multilayer perceptron module.

5. The method for estimating 3D multi-person human pose in video based on spatiotemporal attention according to claim 2, characterized in that, The temporal-gated attention module first performs layer normalization on the input feature vector; then, it introduces a bidirectional scanning mechanism to arrange the layer-normalized feature vector in both forward and reverse order along the time axis, generating two-directional sequence streams; these two-directional sequence streams are then input into a temporal-gated linear attention structure to capture past and future temporal dependencies, resulting in two-directional feature sequences; finally, the two-directional feature sequences are bidirectionally fused, specifically by adding the forward feature sequence element-wise to the reversed feature sequence restored to forward order, obtaining a fused feature vector; a residual connection is used to introduce the layer-normalized feature vector into the fused feature vector, which is then input into a multilayer perceptron; finally, a residual connection is used to introduce the fused feature vector into the output of the multilayer perceptron, restoring it to its original state. Shape, as a spatiotemporal enhancement feature; （9）； （10）； in, The input feature vector; LN denotes layer normalization; It is a bidirectional scan; This represents a temporally gated linear attention structure; Indicates two-way fusion; To fuse feature vectors; It is a spatiotemporal enhancement feature.

6. The method for estimating 3D multi-person human pose in video based on spatiotemporal attention according to claim 2, characterized in that, The processing procedure of the pelvic depth estimator is as follows: First, the 2D coordinates of the pelvic nodes are extracted from the 2D key points. Then, the depth of the pelvic nodes is estimated based on the human body image using RootNet to obtain the three-dimensional coordinates of the pelvic nodes of each human body. （11）； in, Here are the three-dimensional coordinates of the pelvic nodes. Numbering of the human body; and Let be the coordinates of the pelvic nodes of the human body in the human body image plane. It is the depth of the pelvic joints in the human body; Secondly, the three-dimensional pixel coordinates of the pelvic nodes are back-projected into absolute coordinates in the camera coordinate system using the camera's intrinsic parameters. ; （12）； in, absolute coordinates The components in; The coordinates of the principal point; Focal length; Then, the single-person 3D pose sequence Perform unit transformation and translation to the root node position in the camera coordinate system: （13）； in, The absolute 3D pose after translation; The translated absolute 3D pose is reprojected back onto the 2D image plane to obtain the absolute 2D pose. （14）； in, , , Let be the coordinates of a joint in the absolute 3D pose after translation. and The coordinates of a joint in an absolute 2D pose; Finally, the average scale factor s is calculated based on the absolute 2D pose and then used to globally scale and correct the single-person 3D pose sequence. Specifically, this involves calculating the ratio of the torso length in the original 2D human keypoints to the torso length in the absolute 2D pose: （15）； in, A scale factor for the human body. The torso length in the original 2D human keypoints. The torso length in absolute 2D pose; The average scaling factor is: （16）； in, For the number of people; The single-person 3D pose sequence is globally scaled and corrected as follows: （17）； in, This is a single, absolute stance.

7. A 3D multi-person human pose estimation system in video based on spatiotemporal attention, used to implement the 3D multi-person human pose estimation method in video based on spatiotemporal attention as described in any one of claims 1-6, characterized in that, include: The data acquisition and segmentation module is used to acquire human 3D key points and human 2D key points datasets, and divide them into training set and test set according to a set ratio; The model building module is used to build a spatiotemporal linear attention model. The training module is used to train the fusion spatiotemporal linear attention model using the training set, and obtain the trained fusion spatiotemporal linear attention model. The single-person pose estimation module is used to input the test set into the trained fusion spatiotemporal linear attention model to obtain the single-person absolute pose of all human bodies in the same scene. The multi-person pose estimation module is used to integrate the absolute poses of all individuals to obtain a multi-person 3D pose sequence.

8. An electronic device, characterized in that, include: One or more processors, and a memory for storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the spatiotemporal attention-based 3D multi-person human pose estimation method in video according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed, cause the processor to perform the spatiotemporal attention-based 3D multi-person human pose estimation method in video as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program or instructions that, when executed by a processor, implement the spatiotemporal attention-based 3D multi-person human pose estimation method in video as described in any one of claims 1-6.

Images