Lightweight multi-view 3D human pose estimation method and system based on semantic graph

Abstract

This invention discloses a lightweight multi-view 3D human pose estimation method and system based on semantic graphs, relating to the field of computer vision. The method is as follows: S1, acquiring multi-view 2D human pose data; S2, constructing a 3D human pose estimation model; the 3D human pose estimation model includes a spatial location embedding block, a dynamic pruning block, a semantic graph transformer encoder, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block, and a regression layer; S3, training the 3D human pose estimation model based on the multi-view 2D human pose data to obtain a trained 3D human pose estimation model; S4, using the trained 3D human pose estimation model to perform 3D human pose estimation. This invention achieves lightweight model design based on semantic graphs, dynamic pruning, and multi-view 3D human pose estimation, while improving the accuracy and efficiency of 3D human pose estimation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a lightweight multi-view three-dimensional human body posture estimation method and system based on semantic graphs. Background Art

[0002] Traditional 3D human pose estimation methods usually rely on single-view image data, which leads to the problem of depth uncertainty in capturing key information, and the accuracy and integrity of the 3D structure of the human body are difficult to guarantee. This makes the pose estimation method based on single view unsatisfactory in complex scenes, especially in application scenarios such as public safety and motion analysis, which limits the effectiveness of the model in practical applications. With the development of deep learning and computer vision technology, the fusion method of multi-view information has gradually become a research hotspot. The introduction of deep learning provides new possibilities for the dynamic learning and fusion of pose features, which can effectively combine the position information of human joints, spatial structure and bone edge features, thereby significantly enhancing the representation ability of pose features. However, how to construct an effective multi-view spatiotemporal feature fusion framework that can mitigate the depth uncertainty under single-view conditions is still a difficult problem to be solved. In addition, the inherent differences in data collection between different edge devices, such as camera perspective, dynamic range and diversity of data modalities, also bring challenges to 3D pose estimation. These factors can lead to significant differences in the performance of the model between different devices, thus affecting the overall accuracy. At the same time, current high-precision 3D pose estimation models often require a lot of computing resources and storage space, which makes it difficult to achieve real-time pose estimation for many resource-constrained devices (such as mobile devices and embedded systems). Existing multi-view 3D pose estimation technologies generally rely on the accuracy of camera parameters and excessive model resource consumption. These technologies often face challenges such as low computational efficiency and difficulty in deployment in practical applications, which limits their application in real-time scenarios. Therefore, when designing and implementing multi-view 3D human pose estimation methods, how to reasonably and accurately fuse cross-view spatial features while taking into account the integrity of spatiotemporal information and the lightweight model is an important challenge facing current research. Summary of the invention

[0003] In response to the above problems, the present invention proposes a lightweight multi-view 3D human pose estimation method and system based on semantic graphs. By constructing a lightweight 3D human pose estimation model based on semantic graphs, the adaptability in different scenarios is increased. Moreover, the present invention does not depend on camera parameters, which significantly improves the accuracy and efficiency of multi-view 3D pose estimation.

[0004] On the one hand, a lightweight multi-view 3D human pose estimation method based on semantic graph, the specific steps are as follows:

[0005] S1, obtaining multi-view two-dimensional human body posture data;

[0006] S2, constructing a 3D human posture estimation model; the 3D human posture estimation model includes a plurality of spatial position embedding blocks, a plurality of dynamic pruning blocks, a plurality of semantic graph transformer encoders, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block and a regression layer;

[0007] A spatial position embedding block, a dynamic pruning block and a semantic graph transformer encoder are a group, and the input of each group comes from single-view 2D human pose data from different views of the same multi-view 2D human pose data;

[0008] The spatial position embedding block processes the single-view two-dimensional human posture data using a node position embedding function and a spatial graph embedding function to obtain a node position embedding and a spatial graph embedding, and splices the node position embedding and the spatial graph embedding into an input tag;

[0009] The dynamic pruning block performs pooling and pruning operations on the input tags to obtain representative tags;

[0010] The semantic graph transformer encoder extracts node features from the input representative tags as output;

[0011] The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph transformer encoders to obtain multi-view spatial fusion features;

[0012] The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features;

[0013] The regression layer converts the input multi-view spatiotemporal fusion features into three-dimensional posture coordinates as the output of the three-dimensional human posture estimation model;

[0014] S3, training a 3D human body posture estimation model based on the acquired multi-view 2D human body posture data to obtain a trained 3D human body estimation model;

[0015] S4, using the trained 3D human pose estimation model to perform 3D human pose estimation.

[0016] Preferably, the node position embedding function is expressed as:

[0017]

[0018] Where N represents the node position embedding; P ij Represents single-view 2D human posture data P 2D The position coordinates of the jth joint in the i-th frame; Represents the feature conversion function, which maps the input features to D n dimensional space; Indicates that all P ij Integrate into a representation; T represents the number of frames; J represents the number of joints.

[0019] Preferably, the spatial graph embedding function is expressed as:

[0020]

[0021] Where G represents spatial graph embedding; W s represents the weight matrix of the sth order; represents the adjacency matrix of the sth order after the introduction of self-connection and symmetric normalization; σ represents the nonlinear activation function; Represents the connection from S global to local graph features.

[0022] Preferably, the input tags are pooled and pruned to obtain representative tags, as follows:

[0023] The input tag M is pooled by spatial average pooling to generate the pooled tag M p ; for M p Apply density peak clustering algorithm to obtain representative markers And discard the rest of the pooling tokens.

[0024] Preferably, the density peak clustering algorithm is based on the k-nearest neighbor algorithm, according to the pooling mark M p The feature similarity is grouped into clusters, and the cluster center is used as a representative mark, as follows:

[0025] For the αth pooling tag m α ∈M p , m α The corresponding density ρ α It is expressed as:

[0026]

[0027] Among them, kNN(m α ) represents the pooling mark m α k nearest neighbors; k represents the number of k nearest neighbors used to calculate the local density; ||m α -m β || 2 Indicates m α and m β The Euclidean distance between Indicates m α and m β The square of the Euclidean distance between them;

[0028] Using Delta α Indicates the pooling mark m α The distance to other pooling markers; if there is a pooling marker m β The density ρ β Greater than the pooling mark m α The density ρ α , then δ α The pooling mark m α to all these denser pooling markers m β The minimum Euclidean distance between them; if there is no pooling marker m β The density ρ β Greater than ρ α , then δ α The pooling mark m α The maximum Euclidean distance to all other pooled tags; δ α The calculation formula is as follows:

[0029]

[0030] Among them, ρ β Indicates m β The corresponding density;

[0031] The pooling mark m α The cluster center score is denoted as ρ α ×δ α ;

[0032] Finally, the first f pooled markers with the highest scores are selected as cluster centers, and the remaining pooled markers are assigned to the nearest cluster centers with higher density; the cluster centers are used as representative markers Here, f represents a hyperparameter.

[0033] Preferably, the semantic graph transformer encoder deeply mines the semantic information hidden in the representative tags through semantic attention; the semantic attention is expressed as follows:

[0034]

[0035]

[0036] Among them, SA represents semantic attention; Q p Query parameter representing semantic attention; K p Key parameter representing semantic attention; V p The Value parameter representing semantic attention; ψ 1 , 2 and ψ 3 represents the linear layer; d represents the feature dimension; represents the pruned node position embeddings in the representative markup; Represents pruned spatial graph embeddings in representative tokens; As a bias term of position features, it represents the combination of joint space and position knowledge; K p The transpose of ; LN represents the normalization operation.

[0037] Preferably, multi-head semantic attention is used to further enhance the features of pruned node position embedding, expressed as:

[0038]

[0039] Among them, MHSA stands for multi-head semantic attention; represents the connection of H attention heads; Q represents the h-th attention head p ; K represents the h-th attention head p ; V represents the h-th attention head p .

[0040] Preferably, the node features output by all semantic graph transformer encoders are subjected to multi-view spatial fusion to obtain multi-view spatial fusion features, specifically as follows:

[0041] First, generate the final fusion feature output, the formula is as follows:

[0042]

[0043] Among them, η 1 , η 2 and η 3 represents a linear layer; Represents the node features output by perspective v1; represents the node features output by perspective v2; MHA represents the multi-head attention mechanism, As the query parameter of multi-head attention, As the key parameter of multi-head attention, As the Value parameter of multi-head attention; Represents fusion node features; LN represents normalization operation; MLP represents multi-layer perceptron; Represents the final fusion feature output;

[0044] Finally, and and Connect and generate multi-view fusion spatial features It is expressed as:

[0045]

[0046] Among them, Concat represents the concatenation operation of multiple feature vectors.

[0047] Preferably, the input multi-view spatial fusion features are subjected to multi-view cross-channel fusion, and then multi-view spatiotemporal fusion is performed to obtain multi-view spatiotemporal fusion features, specifically as follows:

[0048] The multi-view spatial fusion feature is input into the multi-view cross-channel fusion block to generate the multi-view cross-channel fusion feature Y N ; The obtained multi-view cross-channel fusion feature Y N Perform multi-view spatiotemporal fusion to obtain the multi-view spatiotemporal fusion feature Z; the calculation formula for the multi-view spatiotemporal fusion feature is as follows:

[0049] Z=M r +MCA(M r ,LN(Y N ),LN(Y N ))

[0050] Among them, MCA represents multi-head cross attention; LN represents normalization operation; M r represents a learnable temporal position embedding; M r As the query parameter of multi-head cross attention, LN(Y N ) as the Key and Value parameters of multi-head cross attention.

[0051] On the other hand, a lightweight multi-view 3D human pose estimation system based on semantic graph includes the following:

[0052] A data acquisition module, used to acquire multi-view two-dimensional human body posture data;

[0053] A model building module, for building a 3D human posture estimation model; the 3D human posture estimation model includes a number of spatial position embedding blocks, a number of dynamic pruning blocks, a number of semantic graph transformer encoders, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block and a regression layer;

[0054] A spatial position embedding block, a dynamic pruning block and a semantic graph transformer encoder are a group, and the input of each group comes from single-view 2D human pose data from different views of the same multi-view 2D human pose data;

[0055] The spatial position embedding block processes the single-view two-dimensional human posture data using a node position embedding function and a spatial graph embedding function to obtain a node position embedding and a spatial graph embedding, and splices the node position embedding and the spatial graph embedding into an input tag;

[0056] The dynamic pruning block performs pooling and pruning operations on the input tags to obtain representative tags;

[0057] The semantic graph transformer encoder extracts node features from the input representative tags as output;

[0058] The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph transformer encoders to obtain multi-view spatial fusion features;

[0059] The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features;

[0060] The regression layer converts the input multi-view spatiotemporal fusion features into three-dimensional posture coordinates as the output of the three-dimensional human posture estimation model;

[0061] A model training module is used to train a 3D human posture estimation model based on 2D human posture data from multiple perspectives to obtain a trained 3D human posture estimation model;

[0062] The 3D human pose estimation module is used to perform 3D human pose estimation using a trained 3D human pose estimation model.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] The present invention constructs a 3D human posture estimation model, inputs multi-view 2D human posture data into a spatial position embedding block to generate posture tags, and uses a dynamic pruning block to generate representative tags that retain high semantic diversity and remove redundant information; generates enhanced feature representations through a semantic graph transformer encoder in the model; and integrates temporal knowledge and spatial data, taking into account the integrity of spatiotemporal information; the lightweight model increases the applicability in different scenarios; in addition, the present invention does not rely on camera parameters, significantly improves the accuracy and efficiency of multi-view 3D posture estimation, and provides a more efficient and practical solution for related applications; BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The present invention is further described in detail below in conjunction with the accompanying drawings;

[0066] Figure 1 It is a flowchart of a lightweight multi-view 3D human body posture estimation method based on semantic graph according to an embodiment of the present invention;

[0067] Figure 2 A training flow chart of a lightweight multi-view 3D human pose estimation method based on a semantic graph according to an embodiment of the invention;

[0068] Figure 3 The structure block diagram of a lightweight multi-view 3D human posture estimation system based on a semantic graph according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present invention is further described below through specific implementation modes.

[0070] See also Figure 1 As shown, this embodiment discloses a lightweight multi-view 3D human posture estimation method based on semantic graph, which is as follows:

[0071] S1, obtain multi-view two-dimensional human posture data.

[0072] The acquired multi-view 2D posture data is expressed as Among them, T represents the number of frames and J represents the number of joints.

[0073] S2, build a 3D human pose estimation model; see Figure 2 As shown, the 3D human pose estimation model includes several spatial position embedding blocks, several dynamic pruning blocks, several semantic graph transformer encoders, multi-view spatial fusion blocks, multi-view spatiotemporal fusion blocks and regression layers. A spatial position embedding block, a dynamic pruning block and a semantic graph transformer encoder form a group, and the input of each group comes from single-view 2D human pose data of different views of the same multi-view 2D human pose data.

[0074] Spatial Position Embedding Block:

[0075] The spatial position embedding block uses the node position embedding function and the spatial graph embedding function to process the single-view two-dimensional human posture data to obtain the node position embedding and the spatial graph embedding, and concatenates the node position embedding and the spatial graph embedding into the input tag.

[0076] Specifically, the node position embedding function and the spatial graph embedding function are used to receive two-dimensional posture data from different perspectives to obtain the node position embedding and spatial graph embedding Among them, D n Denotes the feature dimension of node position embedding, D g Represents the feature dimension of the spatial graph embedding. The node position embedding N and the spatial graph embedding G are fused through a concatenation operation, that is, the two are concatenated on the feature dimension to form the final input tag The output input markup M combines node details and spatial structure information, allowing subsequent models to understand the input data more comprehensively.

[0077] The node position embedding function is as follows:

[0078]

[0079] Where N represents the node position embedding; P ij =(x ij ,y ij ), that is, the two-dimensional posture data P 2D The position coordinates of the jth joint in the i-th frame, x ij and ij Indicates the horizontal and vertical coordinates of the position; is a feature conversion function that maps input features to D n dimensional space. Indicates that all P ij Integrate into a representation and get position embedding.

[0080] The spatial graph embedding function constructs multi-order graph features from global to local based on the connection of human joints to describe the spatial structure of human joints. It specifically includes the following steps:

[0081] Define the adjacency matrix and weight matrix. Define the first-order graph adjacency matrix A to represent the relationship between the most directly adjacent nodes. If there is a connection between nodes b and c, then its element a bc =1, otherwise a bc = 0. For multi-order adjacency matrices, define A s The adjacency matrix of the sth order refers to the relationship between a node and its more distant neighbors, so as to obtain multi-order graph features from global to local. In the case of this multi-order graph feature, the corresponding weight matrix W is defined for each level s ={w ij}, these matrices are used to construct and adjust the spatial structural characteristics of human joints.

[0082] Introduce self-connection. In order to ensure that each node also considers its own characteristics when calculating, in the adjacency matrix A l Based on this, we introduce self-connection, i.e., the unit matrix I, and obtain

[0083] Symmetric normalization. Definition express The symmetric matrix of yes The symmetric normalization operation ensures that feature propagation will not be imbalanced due to differences in node degrees, thereby improving the generalization ability of the model.

[0084] Combined features. The final spatial graph embedding G is obtained by combining features of different orders. The specific formula is as follows:

[0085]

[0086] Here, σ represents a nonlinear activation function, which enables the model to capture complex feature patterns. Represents the connection from S global to local graph features.

[0087] The node position embedding of each perspective is fused with the spatial graph embedding to form the input markup of each perspective

[0088] Dynamic trim blocks:

[0089] The dynamic pruning block performs pooling and pruning operations on the input tokens to obtain representative tokens.

[0090] Specifically, for input mark Eliminate spatial redundancy through spatial average pooling and generate pooling marks To M p Apply efficient density peak clustering algorithm to obtain representative markers in represents the number of representative markers, while the rest are discarded, effectively reducing video redundancy.

[0091] The density peak clustering algorithm used in this embodiment is based on the k-nearest neighbor (DPC-kNN) algorithm. p The feature similarity of M is used to cluster M into several groups. The marker at the cluster center has a higher density and a larger distance relative to its neighbors. It contains more information than other markers. The cluster center is used as a representative marker for efficient estimation.

[0092] For the αth pooling tag m α ∈M p , whose density ρ α Calculated by the following formula:

[0093]

[0094] Among them, kNN(m α ) is the marker m α k nearest neighbors; k represents the number of k nearest neighbors used to calculate the local density; ||m α -m β || 2 Indicates m α and m β The Euclidean distance between Indicates m α and m β The square of the Euclidean distance between them.

[0095] Definition of δ α Indicates the pooling mark m α The distance to other pooled markers, δ of the pooled marker αThe calculation method is:

[0096]

[0097] That is, if there is a marker m β The density ρ β Greater than mark m α The density ρ α , then δ α Mark m α To all these higher density markers m β On the contrary, if there is no mark m β The density ρ β Greater than ρ α , then δ α Mark m α The maximum Euclidean distance to all other markers. This method is used to help identify high-density and relatively isolated markers in the feature space as potential cluster centers.

[0098] Mark m α The score of the cluster center is determined by the density (i.e., local degree) ρ α and the minimum distance δ α Multiply by ρ α ×δ α . A higher score indicates that the marker has a larger density and distance, showing a higher potential to become a cluster center. The top f input pose markers with the highest scores are selected as cluster centers, and the remaining markers are assigned to the nearest cluster centers with higher density. The cluster center has higher semantic diversity and contains more information than other markers. Therefore, the cluster center is taken as the representative marker and denoted as This enables efficient estimation, and the remaining tags are discarded to reduce video redundancy. Where f represents a hyperparameter.

[0099] Semantic Graph Transformer Encoder:

[0100] The semantic graph transformer encoder extracts node features from representative tokens of the input as output.

[0101] Specifically, the semantic graph transformer encoder deeply explores the hidden That is, the node position embedding after pruning and pruned spatial graph embedding The semantic attention is described as follows:

[0102]

[0103] Among them, SA represents semantic attention, Q pQuery parameter representing semantic attention, K p Key parameter representing semantic attention, V p The Value parameter representing semantic attention; ψ 1 , 2 and ψ 3 represents a linear layer, d is the feature dimension; As a bias term of position features, it represents the combination of joint space and position knowledge; K p The transpose of .

[0104] At the same time, multi-head semantic attention is used to further enhance the features of pruned node position embedding, expressed as:

[0105]

[0106] Among them, MHSA stands for multi-head semantic attention, H stands for the total number of attention heads, Q represents the h-th attention head p ; K represents the h-th attention head p ; V represents the h-th attention head p ; is the connection of H attention heads.

[0107] In addition, in order to make the features more expressive, by combining the node position embedding to form a residual structure, the model can better capture the relationship between the input features and the output features, and at the same time alleviate the gradient vanishing problem. The changes in node features from layer l-1 to layer l are as follows:

[0108]

[0109] in, represents the feature conversion function in the feedforward network, It is the feature representation after processing by the lth layer; Represents the feature representation after processing by the l-1th layer.

[0110] Multi-view spatial fusion block:

[0111] The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph transformer encoders to obtain multi-view spatial fusion features.

[0112] Specifically, spatial feature fusion between views is performed during the spatial semantic feature extraction process of each view. Assume that the output node features of the lth layer of views v1 and v2 are and They are first transformed using a linear layer and then fed into a general transform encoder for interaction and fusion. As the Query parameter and Key parameter of multi-head attention, the converted As the Value parameter of multi-head attention. The formula for generating the final fusion feature output is as follows:

[0113]

[0114] Among them, η 1 , η 2 and η 3 Represents a linear layer. MHA represents a multi-head attention mechanism, which is used to interact and fuse input features; Represents the output node features of the lth layer of view v1; Represents the output node features of the lth layer of view v2; Represents the fusion node features; After the layer LN (normalization) and MLP (multi-layer perceptron), the final fusion feature output is obtained

[0115] after, and and Concatenate to generate multi-view spatial fusion features And input into a deeper network.

[0116]

[0117] Among them, Concat represents the concatenation operation of multiple feature vectors.

[0118] Multi-view spatiotemporal fusion block:

[0119] The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features.

[0120] Specifically, for multi-view spatial fusion features First, a multi-view cross-channel fusion block (including batch normalization, 1x1 convolutional layer and layer normalization) is used to better preserve and refine the original spatial information. Converted to multi-view cross-channel fusion feature Y N Afterwards, the obtained multi-view cross-channel fusion feature Y N The input is sent to the spatiotemporal transform coding block for further spatiotemporal fusion to alleviate the depth ambiguity problem in 3D pose estimation and restore the complete time sequence. The time sequence recovery in the spatiotemporal transform coding block specifically includes the following steps:

[0121] Initialize the recovery attention module of the spatiotemporal transform encoding block. The recovery attention module is mainly composed of multi-head cross attention layers and restores the temporal information through the attention mechanism.

[0122] Using learnable temporal position embeddings As the query parameter of multi-head cross attention, denoted as Q; for multi-view cross-channel fusion feature Y N A normalization operation is performed as the Key parameter (denoted as K) and Value parameter (denoted as V) of multi-head cross attention to provide prompts for spatiotemporal information.

[0123] Through multi-head cross attention, a weighted sum is performed based on the input Q, K and V to generate a multi-view spatiotemporal fusion feature Z after restoring the timing information. The specific calculation formula of multi-head cross attention is as follows:

[0124] Z=M r +MCA(M r ,LN(Y N ),LN(Y N ))

[0125] Among them, MCA stands for Multi-Cross Attention.

[0126] Regression Layer:

[0127] The regression layer converts the input multi-view spatiotemporal fusion features into 3D pose coordinates as the output of the 3D human pose estimation model.

[0128] Specifically, the Z obtained by the time recovery module is integrated into a regression layer. The goal of this layer is to convert the processed features into the corresponding 3D posture coordinates Represents the 3D coordinates of each joint J in frame T.

[0129] S3, training a three-dimensional human body posture estimation model based on the acquired multi-view two-dimensional human body posture data to obtain a trained three-dimensional human body estimation model.

[0130] Specifically, in this embodiment, the loss function used to train the 3D human posture estimation model is constructed in the form of mean square error (MSE); the loss function measures the difference between the 3D posture predicted by the model and the actual posture.

[0131] The construction process of the 3D human body estimation model in this embodiment also includes a testing phase. First, some multi-view 2D posture data are selected from the data set as a test set; the trained multi-view 3D human body posture estimation model is used to extract features from the test set; the model generates a 3D posture feature representation by processing the input multi-view data; then, posture estimation is performed using these feature representations, and the prediction results of the 3D human body posture are output; finally, the estimation results are further verified and analyzed according to actual application requirements to ensure the accuracy and reliability of the 3D posture estimation.

[0132] S4, using the trained 3D human pose estimation model to perform 3D human pose estimation.

[0133] The lightweight multi-view 3D human pose estimation method based on semantic graph in this embodiment introduces deep semantic graph transformation in multi-view 3D human pose estimation, and uses dynamic pruning and fusion strategies to optimize the feature processing flow. This method achieves the lightweight of the model without relying on camera parameters, while greatly improving the accuracy and efficiency of multi-view 3D pose estimation.

[0134] like Figure 3 As shown, the present invention also discloses a lightweight multi-view 3D human posture estimation system based on semantic graph, comprising:

[0135] The data acquisition module 301 is used to acquire multi-view two-dimensional human body posture data.

[0136] A model building module 302 is used to construct a 3D human posture estimation model; the 3D human posture estimation model includes a plurality of spatial position embedding blocks, a plurality of dynamic pruning blocks, a plurality of semantic graph encoders, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block and a regression layer;

[0137] A spatial position embedding block, a dynamic pruning block and a semantic graph encoder form a group, and the input of each group comes from single-view 2D human pose data from different views of the same multi-view 2D human pose data;

[0138] The spatial position embedding block processes the single-view two-dimensional human posture data using a node position embedding function and a spatial graph embedding function to obtain a node position embedding and a spatial graph embedding, and splices the node position embedding and the spatial graph embedding into an input tag;

[0139] The dynamic pruning block performs pooling and pruning operations on the input tags to obtain representative tags;

[0140] The semantic graph encoder extracts node features from the input representative tags as output;

[0141] The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph converter encoders to obtain multi-view spatial fusion features;

[0142] The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features;

[0143] The regression layer converts the input multi-view spatiotemporal fusion features into three-dimensional posture coordinates as the output of the three-dimensional human posture estimation model.

[0144] The model training module 303 is used to train the 3D human body posture estimation model based on the acquired multi-view 2D human body posture data to obtain a trained 3D human body estimation model.

[0145] The 3D human body posture estimation module 304 is used to perform 3D human body posture estimation using a trained 3D human body posture estimation model.

[0146] The specific implementation of the lightweight multi-view 3D human pose estimation system based on semantic graph is the same as the lightweight multi-view 3D human pose estimation method based on semantic graph, and will not be repeated in this embodiment.

[0147] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A lightweight multi-view 3D human pose estimation method based on semantic graph, characterized in that: The following steps are involved: S1, obtaining multi-view two-dimensional human body posture data; S2, constructing a 3D human posture estimation model; the 3D human posture estimation model includes a plurality of spatial position embedding blocks, a plurality of dynamic pruning blocks, a plurality of semantic graph transformer encoders, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block and a regression layer; A spatial position embedding block, a dynamic pruning block and a semantic graph transformer encoder are a group, and the input of each group comes from single-view 2D human pose data from different views of the same multi-view 2D human pose data; The spatial position embedding block processes the single-view two-dimensional human posture data using a node position embedding function and a spatial graph embedding function to obtain a node position embedding and a spatial graph embedding, and splices the node position embedding and the spatial graph embedding into an input tag; The dynamic pruning block performs pooling and pruning operations on the input tags to obtain representative tags; The semantic graph transformer encoder extracts node features from the input representative tags as output; The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph transformer encoders to obtain multi-view spatial fusion features; The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features; The regression layer converts the input multi-view spatiotemporal fusion features into three-dimensional posture coordinates as the output of the three-dimensional human posture estimation model; S3, training a 3D human body posture estimation model based on the acquired multi-view 2D human body posture data to obtain a trained 3D human body estimation model; S4, using the trained 3D human pose estimation model to perform 3D human pose estimation.

2. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 1, characterized in that: The node position embedding function is expressed as: Where N represents the node position embedding; P ij Represents single-view 2D human posture data P 2D The position coordinates of the jth joint in the i-th frame; Represents the feature conversion function, which maps the input features to D n dimensional space; Indicates that all P ij Integrate into a representation; T represents the number of frames; J represents the number of joints.

3. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 2, characterized in that: The spatial graph embedding function is expressed as: Where G represents spatial graph embedding; W s represents the weight matrix of the sth order; represents the adjacency matrix of the sth order after the introduction of self-connection and symmetric normalization; σ represents the nonlinear activation function; Represents the connection from S global to local graph features.

4. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 1, characterized in that: The input tags are pooled and pruned to obtain representative tags, as follows: The input tag M is pooled by spatial average pooling to generate the pooled tag M p ; for M p Apply density peak clustering algorithm to obtain representative markers And discard the rest of the pooling tokens.

5. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 4, characterized in that: The density peak clustering algorithm is based on the k-nearest neighbor algorithm, according to the pooling mark M p The feature similarity is grouped into clusters, and the cluster center is used as a representative mark, as follows: For the αth pooling tag m α ∈M p , m α The corresponding density ρ α It is expressed as: Among them, kNN(m α ) represents the pooling mark m α k nearest neighbors; k represents the number of k nearest neighbors used to calculate the local density; ||m α -m β ||2 means m α and m β The Euclidean distance between Indicates m α and m β The square of the Euclidean distance between them; Using Delta α Indicates the pooling mark m α The distance to other pooling markers; if there is a pooling marker m β The density ρ β Greater than the pooling mark m α The density ρ α , then δ α The pooling mark m α to all these denser pooling markers m β The minimum Euclidean distance between them; if there is no pooling marker m β The density ρ β Greater than ρ α , then δ α The pooling mark m α The maximum Euclidean distance to all other pooled tags; δ α The calculation formula is as follows: Among them, ρ β Indicates m β The corresponding density; The pooling mark m α The cluster center score is denoted as ρ α ×δ α ; Finally, the first f pooled markers with the highest scores are selected as cluster centers, and the remaining pooled markers are assigned to the nearest cluster centers with higher density; the cluster centers are used as representative markers Here, f represents a hyperparameter.

6. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 1, characterized in that: The semantic graph transformer encoder deeply mines the semantic information hidden in the representative tokens through semantic attention; the semantic attention is expressed as follows: Among them, SA represents semantic attention; Q p Query parameter representing semantic attention; K p Key parameter representing semantic attention; V p The Value parameter represents the semantic attention; ψ1, ψ2 and ψ3 represent the linear layer; d represents the feature dimension; represents the pruned node position embeddings in the representative markup; Represents pruned spatial graph embeddings in representative tokens; As a bias term of position features, it represents the combination of joint space and position knowledge; K p The transpose of ; LN represents the normalization operation.

7. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 6, characterized in that: Multi-head semantic attention is used to further enhance the features of pruned node position embedding, expressed as: Among them, MHSA stands for multi-head semantic attention; represents the connection of H attention heads; represents the Q of the h-th attention head p ; K represents the h-th attention head p ; V represents the h-th attention head p .

8. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 1, characterized in that: The node features output by all semantic graph transformer encoders are subjected to multi-view spatial fusion to obtain multi-view spatial fusion features, as follows: First, generate the final fusion feature output, the formula is as follows: Among them, η1, η2 and η3 represent linear layers; Represents the node features output by perspective v1; represents the node features output by perspective v2; MHA represents the multi-head attention mechanism, As the query parameter of multi-head attention, As the key parameter of multi-head attention, As the Value parameter of multi-head attention; Represents fusion node features; LN represents normalization operation; MLP represents multi-layer perceptron; Represents the final fusion feature output; Finally, and and Connect and generate multi-view fusion spatial features It is expressed as: Among them, Concat represents the concatenation operation of multiple feature vectors.

9. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 1, characterized in that: The input multi-view spatial fusion features are subjected to multi-view cross-channel fusion, and then multi-view spatiotemporal fusion is performed to obtain multi-view spatiotemporal fusion features, specifically as follows: The multi-view spatial fusion feature is input into the multi-view cross-channel fusion block to generate the multi-view cross-channel fusion feature Y N ; The obtained multi-view cross-channel fusion feature Y N Perform multi-view spatiotemporal fusion to obtain the multi-view spatiotemporal fusion feature Z; The calculation formula of spatiotemporal fusion features is as follows: Z=M r +MCA(M r ,LN(Y N ),LN(Y N )); Among them, MCA represents multi-head cross attention; LN represents normalization operation; M r represents a learnable temporal position embedding; M r As the query parameter of multi-head cross attention, LN(Y N ) as the Key and Value parameters of multi-head cross attention.

10. A lightweight multi-view 3D human pose estimation system based on semantic graph, comprising: A data acquisition module, used to acquire multi-view two-dimensional human body posture data; A model building module, for building a 3D human posture estimation model; the 3D human posture estimation model includes a number of spatial position embedding blocks, a number of dynamic pruning blocks, a number of semantic graph transformer encoders, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block and a regression layer; A spatial position embedding block, a dynamic pruning block and a semantic graph transformer encoder are a group, and the input of each group comes from single-view 2D human pose data from different views of the same multi-view 2D human pose data; The spatial position embedding block processes the single-view two-dimensional human posture data using a node position embedding function and a spatial graph embedding function to obtain a node position embedding and a spatial graph embedding, and splices the node position embedding and the spatial graph embedding into an input tag; The dynamic pruning block performs pooling and pruning operations on the input tags to obtain representative tags; The semantic graph transformer encoder extracts node features from the input representative tags as output; The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph transformer encoders to obtain multi-view spatial fusion features; The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features; The regression layer converts the input multi-view spatiotemporal fusion features into three-dimensional posture coordinates as the output of the three-dimensional human posture estimation model; A model training module is used to train a 3D human posture estimation model based on 2D human posture data from multiple perspectives to obtain a trained 3D human posture estimation model; The 3D human pose estimation module is used to perform 3D human pose estimation using a trained 3D human pose estimation model.

Images