Lightweight multi-view three-dimensional human body posture estimation method and system based on semantic graph

Abstract

The invention discloses a lightweight multi-view three-dimensional human body posture estimation method and system based on a semantic graph, and relates to the field of computer vision, and the method comprises the following steps: S1, obtaining multi-view two-dimensional human body posture data; s2, constructing a three-dimensional human body posture estimation model; the three-dimensional human body posture estimation model comprises a space position embedding block, a dynamic trimming block, a semantic graph converter encoder, a multi-view space fusion block, a multi-view space-time fusion block and a regression layer; s3, training the three-dimensional human body posture estimation model based on the multi-view two-dimensional human body posture data to obtain a trained three-dimensional human body estimation model; and S4, performing three-dimensional human body posture estimation by using the trained three-dimensional human body posture estimation model. According to the method, the three-dimensional human body posture estimation model based on the semantic graph, the dynamic trimming and the multiple view angles is designed, so that the light weight of the model is realized, and meanwhile, the accuracy and the efficiency of three-dimensional human body posture estimation are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a lightweight multi-view three-dimensional human body posture estimation method and system based on semantic graphs. Background Art

[0002] Traditional 3D human pose estimation methods usually rely on single-view image data, which leads to the problem of depth uncertainty in capturing key information, and the accuracy and integrity of the 3D structure of the human body are difficult to guarantee. This makes the pose estimation method based on single view unsatisfactory in complex scenes, especially in application scenarios such as public safety and motion analysis, which limits the effectiveness of the model in practical applications. With the development of deep learning and computer vision technology, the fusion method of multi-view information has gradually become a research hotspot. The introduction of deep learning provides new possibilities for the dynamic learning and fusion of pose features, which can effectively combine the position information of human joints, spatial structure and bone edge features, thereby significantly enhancing the representation ability of pose features. However, how to construct an effective multi-view spatiotemporal feature fusion framework that can mitigate the depth uncertainty under single-view conditions is still a difficult problem to be solved. In addition, the inherent differences in data collection between different edge devices, such as camera perspective, dynamic range and diversity of data modalities, also bring challenges to 3D pose estimation. These factors can lead to significant differences in the performance of the model between different devices, thus affecting the overall accuracy. At the same time, current high-precision 3D pose estimation models often require a lot of computing resources and storage space, which makes it difficult to achieve real-time pose estimation for many resource-constrained devices (such as mobile devices and embedded systems). Existing multi-view 3D pose estimation technologies generally rely on the accuracy of camera parameters and excessive model resource consumption. These technologies often face challenges such as low computational efficiency and difficulty in deployment in practical applications, which limits their application in real-time scenarios. Therefore, when designing and implementing multi-view 3D human pose estimation methods, how to reasonably and accurately fuse cross-view spatial features while taking into account the integrity of spatiotemporal information and the lightweight model is an important challenge facing current research. Summary of the invention

[0003] In response to the above problems, the present invention proposes a lightweight multi-view 3D human pose estimation method and system based on semantic graphs. By constructing a lightweight 3D human pose estimation model based on semantic graphs, the adaptability in different scenarios is increased. Moreover, the present invention does not depend on camera parameters, which significantly improves the accuracy and efficiency of multi-view 3D pose estimation.

[0004] On the one hand, a lightweight multi-view 3D human pose estimation method based on semantic graph, the specific steps are as follows:

[0005] S1, obtaining multi-view two-dimensional human body posture data;

[0006] S2, constructing a 3D human posture estimation model; the 3D human posture estimation model includes a plurality of spatial position embedding blocks, a plurality of dynamic pruning blocks, a plurality of semantic graph transformer encoders, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block and a regression layer;

[0007] A spatial position embedding block, a dynamic pruning block and a semantic graph transformer encoder are a group, and the input of each group comes from single-view 2D human pose data from different views of the same multi-view 2D human pose data;

[0008] The spatial position embedding block processes the single-view two-dimensional human posture data using a node position embedding function and a spatial graph embedding function to obtain a node position embedding and a spatial graph embedding, and splices the node position embedding and the spatial graph embedding into an input tag;

[0009] The dynamic pruning block performs pooling and pruning operations on the input tags to obtain representative tags;

[0010] The semantic graph transformer encoder extracts node features from the input representative tags as output;

[0011] The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph transformer encoders to obtain multi-view spatial fusion features;

[0012] The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features;

[0013] The regression layer converts the input multi-view spatiotemporal fusion features into three-dimensional posture coordinates as the output of the three-dimensional human posture estimation model;

[0014] S3, training a 3D human body posture estimation model based on the acquired multi-view 2D human body posture data to obtain a trained 3D human body estimation model;

[0015] S4, using the trained 3D human pose estimation model to perform 3D human pose estimation.

[0016] Preferably, the node position embedding function is expressed as:

[0017]

[0018] in, Represents node position embedding; Represents single-view 2D human pose data Middle Frame No. The position coordinates of each joint; Represents the feature conversion function, which maps the input features to dimensional space; Indicates that all Integrate into a representation; T represents the number of frames; J represents the number of joints.

[0019] Preferably, the spatial graph embedding function is expressed as:

[0020]

[0021] in, Representing spatial graph embeddings; Indicates The weight matrix of order; represents the first The adjacency matrix of order; represents a nonlinear activation function; express A connection from global to local graph features.

[0022] Preferably, the input tags are pooled and pruned to obtain representative tags, as follows:

[0023] Marking the input Generate pooling marks through spatial average pooling ;right Apply density peak clustering algorithm to obtain representative markers , and discard the rest of the pooling tokens.

[0024] Preferably, the density peak clustering algorithm is based on the k-nearest neighbor algorithm, according to the pooling mark The feature similarity is grouped into clusters, and the cluster center is used as a representative mark, as follows:

[0025] For Pooling Tags , The corresponding density It is expressed as:

[0026]

[0027] in, Indicates pooling mark k nearest neighbors; Represents the local density used to calculate The number of nearest neighbors; express and The Euclidean distance between express and The square of the Euclidean distance between them;

[0028] use Indicates pooling mark The distance to other pooling markers; if there is a pooling marker Density Greater than pooling mark Density ,So Mark for pooling to all these denser pooling markers The minimum Euclidean distance between; if there is no pooling mark Density Greater than ,So Mark for pooling The maximum Euclidean distance to all other pooled tokens; The calculation formula is as follows:

[0029]

[0030] in, express The corresponding density;

[0031] Mark the pool The cluster center score of ;

[0032] Finally, select the top The pooled markers are used as cluster centers, and the remaining pooled markers are assigned to the nearest cluster centers with higher density; the cluster centers are used as representative markers. ;in, Represents a hyperparameter.

[0033] Preferably, the semantic graph transformer encoder deeply mines the semantic information hidden in the representative tags through semantic attention; the semantic attention is expressed as follows:

[0034]

[0035]

[0036]

[0037]

[0038] in, Represents semantic attention; Query parameter representing semantic attention; Key parameter representing semantic attention; The Value parameter representing semantic attention; , and represents a linear layer; Represents feature dimension; represents the pruned node position embeddings in the representative markup; Represents pruned spatial graph embeddings in representative tokens; As a bias term of position features, it represents the combination of joint space and position knowledge; express The transpose of Represents a normalization operation.

[0039] Preferably, multi-head semantic attention is used to further enhance the features of pruned node position embedding, expressed as:

[0040]

[0041] in, represents multi-head semantic attention; express The connection of the attention heads; represents the hth attention head ; represents the hth attention head ; represents the hth attention head .

[0042] Preferably, the node features output by all semantic graph transformer encoders are subjected to multi-view spatial fusion to obtain multi-view spatial fusion features, specifically as follows:

[0043] First, generate the final fusion feature output, the formula is as follows:

[0044]

[0045]

[0046]

[0047] in, , and represents a linear layer; Representation perspective Output node features; Representation perspective Output node features; represents the multi-head attention mechanism, As the query parameter of multi-head attention, As the key parameter of multi-head attention, As the Value parameter of multi-head attention; Represents the fusion node features; Represents a normalization operation; MLP represents a multi-layer perceptron; Represents the final fusion feature output;

[0048] Finally, and and Connect and generate multi-view fusion spatial features , expressed as:

[0049]

[0050] in, Indicates that multiple feature vectors are concatenated.

[0051] Preferably, the input multi-view spatial fusion features are subjected to multi-view cross-channel fusion, and then multi-view spatiotemporal fusion is performed to obtain multi-view spatiotemporal fusion features, specifically as follows:

[0052] Input the multi-view spatial fusion features into the multi-view cross-channel fusion block to generate multi-view cross-channel fusion features ; The obtained multi-view cross-channel fusion features Perform multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features ; The calculation formula of multi-view spatiotemporal fusion features is as follows:

[0053]

[0054] in, represents multi-head cross attention; LN represents normalization operation; Represents learnable temporal position embeddings; As the query parameter of multi-head cross attention, As the Key and Value parameters of multi-head cross attention.

[0055] On the other hand, a lightweight multi-view 3D human pose estimation system based on semantic graph includes the following:

[0056] A data acquisition module, used to acquire multi-view two-dimensional human body posture data;

[0057] A model building module, for building a 3D human posture estimation model; the 3D human posture estimation model includes a number of spatial position embedding blocks, a number of dynamic pruning blocks, a number of semantic graph transformer encoders, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block and a regression layer;

[0058] A spatial position embedding block, a dynamic pruning block and a semantic graph transformer encoder are a group, and the input of each group comes from single-view 2D human pose data from different views of the same multi-view 2D human pose data;

[0059] The spatial position embedding block processes the single-view two-dimensional human posture data using a node position embedding function and a spatial graph embedding function to obtain a node position embedding and a spatial graph embedding, and splices the node position embedding and the spatial graph embedding into an input tag;

[0060] The dynamic pruning block performs pooling and pruning operations on the input tags to obtain representative tags;

[0061] The semantic graph transformer encoder extracts node features from the input representative tags as output;

[0062] The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph transformer encoders to obtain multi-view spatial fusion features;

[0063] The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features;

[0064] The regression layer converts the input multi-view spatiotemporal fusion features into three-dimensional posture coordinates as the output of the three-dimensional human posture estimation model;

[0065] A model training module is used to train a 3D human posture estimation model based on 2D human posture data from multiple perspectives to obtain a trained 3D human posture estimation model;

[0066] The 3D human pose estimation module is used to perform 3D human pose estimation using a trained 3D human pose estimation model.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] The present invention constructs a 3D human posture estimation model, inputs multi-view 2D human posture data into a spatial position embedding block to generate posture tags, and uses a dynamic pruning block to generate representative tags that retain high semantic diversity and remove redundant information; generates enhanced feature representations through a semantic graph transformer encoder in the model; and integrates temporal knowledge and spatial data, taking into account the integrity of spatiotemporal information; the lightweight model increases the applicability in different scenarios; in addition, the present invention does not rely on camera parameters, significantly improves the accuracy and efficiency of multi-view 3D posture estimation, and provides a more efficient and practical solution for related applications; BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The present invention is further described in detail below in conjunction with the accompanying drawings;

[0070] Figure 1 It is a flowchart of a lightweight multi-view 3D human body posture estimation method based on semantic graph according to an embodiment of the present invention;

[0071] Figure 2 A training flow chart of a lightweight multi-view 3D human pose estimation method based on a semantic graph according to an embodiment of the invention;

[0072] Figure 3 The structure block diagram of a lightweight multi-view 3D human posture estimation system based on a semantic graph according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] The present invention is further described below through specific implementation modes.

[0074] See also Figure 1 As shown, this embodiment discloses a lightweight multi-view 3D human posture estimation method based on semantic graph, which is as follows:

[0075] S1, obtain multi-view two-dimensional human posture data.

[0076] The acquired multi-view 2D posture data is expressed as ,in, Indicates the frame number. Indicates the number of joints.

[0077] S2, build a 3D human pose estimation model; see Figure 2 As shown, the 3D human pose estimation model includes several spatial position embedding blocks, several dynamic pruning blocks, several semantic graph transformer encoders, multi-view spatial fusion blocks, multi-view spatiotemporal fusion blocks and regression layers. A spatial position embedding block, a dynamic pruning block and a semantic graph transformer encoder form a group, and the input of each group comes from single-view 2D human pose data of different views of the same multi-view 2D human pose data.

[0078] Spatial Position Embedding Block:

[0079] The spatial position embedding block uses the node position embedding function and the spatial graph embedding function to process the single-view two-dimensional human posture data to obtain the node position embedding and the spatial graph embedding, and concatenates the node position embedding and the spatial graph embedding into the input tag.

[0080] Specifically, the node position embedding function and the spatial graph embedding function are used to receive two-dimensional posture data from different perspectives to obtain the node position embedding and spatial graph embedding ,in, represents the feature dimension of the node position embedding, Represents the feature dimension of the spatial graph embedding. Embed the node position and spatial graph embedding The fusion is performed through the connection operation, that is, the two are spliced ​​in the feature dimension to form the final input tag . Output input tags The combination of node details and spatial structure information enables subsequent models to have a more comprehensive understanding of the input data.

[0081] The node position embedding function is as follows:

[0082]

[0083] in, Represents node position embedding; , that is, two-dimensional posture data Middle Frame No. The position coordinates of the joints, and Indicates the horizontal and vertical coordinates of the position; is a feature conversion function that maps input features to dimensional space. Indicates that all Integrate into a representation and get position embedding.

[0084] The spatial graph embedding function constructs multi-order graph features from global to local based on the connection of human joints to describe the spatial structure of human joints. It specifically includes the following steps:

[0085] Define the adjacency matrix and weight matrix. Define the 1st order graph adjacency matrix Indicates the relationship between the most directly adjacent nodes. If the node and If there is a connection between them, then their elements ,otherwise For multi-order adjacency matrices, define Indicates The adjacency matrix of the order refers to the relationship between a node and its more distant neighbors, in order to obtain multi-order graph features from global to local. In the case of this multi-order graph feature, the corresponding weight matrix is ​​defined for each level ,These matrices are used to construct and adjust the spatial structural characteristics of human joints.

[0086] Introduce self-connection. In order to ensure that each node also considers its own characteristics when calculating, in the adjacency matrix Based on this, we introduce self-connection, i.e., the identity matrix ,get .

[0087] Symmetric normalization. Definition express The symmetric matrix of yes The symmetric normalization operation ensures that feature propagation will not be imbalanced due to differences in node degrees, thereby improving the generalization ability of the model.

[0088] Combined features. Final spatial graph embedding It is obtained by combining features of different orders. The specific formula is as follows:

[0089]

[0090] in, Represents a nonlinear activation function, which enables the model to capture complex feature patterns. express A connection from global to local graph features.

[0091] The node position embedding of each perspective is fused with the spatial graph embedding to form the input markup of each perspective .

[0092] Dynamic trim blocks:

[0093] The dynamic pruning block performs pooling and pruning operations on the input tokens to obtain representative tokens.

[0094] Specifically, the input mark Eliminate spatial redundancy through spatial average pooling and generate pooling marks .right Apply efficient density peak clustering algorithm to obtain representative markers ,in represents the number of representative markers, while the rest are discarded, effectively reducing video redundancy.

[0095] The density peak clustering algorithm used in this embodiment is based on Nearest neighbor (DPC-kNN) algorithm, according to The feature similarity will be Clustered into several groups, the marker at the cluster center has a higher density, a larger distance relative to its neighbors, and contains more information than other markers. The cluster center is used as a representative marker for efficient estimation.

[0096] For Pooling Tags , its density Calculated by the following formula:

[0097]

[0098] in, Yes Mark of Near neighbors; Represents the local density used to calculate The number of nearest neighbors; express and The Euclidean distance between express and The square of the Euclidean distance between them.

[0099] definition Indicates pooling mark The distance to other pooled markers, the distance to the pooled markers The calculation method is:

[0100]

[0101] That is, if there is a tag Density Greater than mark Density ,So For marking to all these higher density markers On the contrary, if there is no mark Density Greater than ,So For marking The maximum Euclidean distance to all other markers. This method is used to help identify markers with high density and relatively isolated in feature space as potential cluster centers.

[0102] mark The score of the cluster center is determined by the density (i.e. locality) and minimum distance Multiply them together to calculate, expressed as A higher score indicates that the marker has a larger density and distance, showing a higher potential to become a cluster center. The input pose markers are selected as cluster centers, and the remaining markers are assigned to the nearest cluster centers with higher density. The cluster centers have higher semantic diversity and contain more information than other markers. Therefore, the cluster centers are taken as representative markers and are denoted as , thus achieving efficient estimation, the remaining markers are discarded to reduce video redundancy. Represents a hyperparameter.

[0103] Semantic Graph Transformer Encoder:

[0104] The semantic graph transformer encoder extracts node features from representative tokens of the input as output.

[0105] Specifically, the semantic graph transformer encoder deeply explores the hidden , that is, the node position embedding after pruning and pruned spatial graph embedding The semantic attention is described as follows:

[0106]

[0107]

[0108]

[0109]

[0110] in, represents semantic attention, Query parameter representing semantic attention, Key parameter representing semantic attention, The Value parameter representing semantic attention; , and represents a linear layer, is the characteristic dimension; As a bias term of position features, it represents the combination of joint space and position knowledge; express The transpose of .

[0111] At the same time, multi-head semantic attention is used to further enhance the features of pruned node position embedding, expressed as:

[0112]

[0113] in, represents multi-head semantic attention, represents the total number of attention heads, represents the hth attention head ; represents the hth attention head ; represents the hth attention head ; yes The connection of the attention heads.

[0114] In addition, in order to make the features more expressive, by combining the node position embedding to form a residual structure, the model can better capture the relationship between the input features and the output features, and at the same time alleviate the gradient vanishing problem. Layer to The layers change as follows:

[0115]

[0116]

[0117] in, represents the feature conversion function in the feedforward network, It is through Feature representation after layer processing; Indicates that after Feature representation after layer processing.

[0118] Multi-view spatial fusion block:

[0119] The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph transformer encoders to obtain multi-view spatial fusion features.

[0120] Specifically, spatial feature fusion between views is performed during the spatial semantic feature extraction process of each view. Assuming that the view and No. The output node features of the layer are and . They are first transformed using a linear layer and then fed into a general transform encoder for interaction and fusion. As the Query parameter and Key parameter of multi-head attention, the converted As the Value parameter of multi-head attention. The formula for generating the final fusion feature output is as follows:

[0121]

[0122]

[0123]

[0124] in, , and Represents a linear layer. Represents a multi-head attention mechanism, which is used to interact and fuse input features; Represents the output node features of the lth layer of view v1; Represents the output node features of the lth layer of view v2; Represents the fusion node features; Passing layer (i.e. normalized) and (i.e. multi-layer perceptron) to obtain the final fusion feature output .

[0125] after, and and Concatenate to generate multi-view spatial fusion features , and input into a deeper network.

[0126]

[0127] in, Indicates that multiple feature vectors are concatenated.

[0128] Multi-view spatiotemporal fusion block:

[0129] The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features.

[0130] Specifically, for multi-view spatial fusion features First, a multi-view cross-channel fusion block (including batch normalization, 1x1 convolutional layer and layer normalization) is used to better preserve and refine the original spatial information. Convert to multi-view cross-channel fusion features Afterwards, the obtained multi-view cross-channel fusion features The input is sent to the spatiotemporal transform coding block for further spatiotemporal fusion to alleviate the depth ambiguity problem in 3D pose estimation and restore the complete time sequence. The time sequence recovery in the spatiotemporal transform coding block specifically includes the following steps:

[0131] Initialize the recovery attention module of the spatiotemporal transform encoding block. The recovery attention module is mainly composed of multi-head cross attention layers, and restores the temporal information through the attention mechanism.

[0132] Using learnable temporal position embeddings As the query parameter of multi-head cross attention, it is recorded as ; Multi-view cross-channel fusion features Perform normalization operation as the key parameter of multi-head cross attention (denoted as ) and the Value parameter (denoted as ), to provide prompts of time and space information.

[0133] Through multi-head cross attention, according to the input , and Perform weighted summation to generate multi-view spatiotemporal fusion features after restoring temporal information The specific calculation formula for multi-head cross attention is as follows:

[0134]

[0135] in, Indicates multi-head cross attention.

[0136] Regression Layer:

[0137] The regression layer converts the input multi-view spatiotemporal fusion features into 3D pose coordinates as the output of the 3D human pose estimation model.

[0138] Specifically, the timing recovery module obtains Integrate into a regression layer. The goal of this layer is to transform the processed features into corresponding 3D pose coordinates ,express Each joint in the frame The three-dimensional coordinates of .

[0139] S3, training a three-dimensional human body posture estimation model based on the acquired multi-view two-dimensional human body posture data to obtain a trained three-dimensional human body estimation model.

[0140] Specifically, in this embodiment, the loss function used to train the 3D human posture estimation model is constructed in the form of mean square error (MSE); the loss function measures the difference between the 3D posture predicted by the model and the actual posture.

[0141] The construction process of the 3D human body estimation model in this embodiment also includes a testing phase. First, some multi-view 2D posture data are selected from the data set as a test set; the trained multi-view 3D human body posture estimation model is used to extract features from the test set; the model generates a 3D posture feature representation by processing the input multi-view data; then, posture estimation is performed using these feature representations, and the prediction results of the 3D human body posture are output; finally, the estimation results are further verified and analyzed according to actual application requirements to ensure the accuracy and reliability of the 3D posture estimation.

[0142] S4, using the trained 3D human pose estimation model to perform 3D human pose estimation.

[0143] The lightweight multi-view 3D human pose estimation method based on semantic graph in this embodiment introduces deep semantic graph transformation in multi-view 3D human pose estimation, and uses dynamic pruning and fusion strategies to optimize the feature processing flow. This method achieves the lightweight of the model without relying on camera parameters, while greatly improving the accuracy and efficiency of multi-view 3D pose estimation.

[0144] like Figure 3 As shown, the present invention also discloses a lightweight multi-view 3D human posture estimation system based on semantic graph, comprising:

[0145] The data acquisition module 301 is used to acquire multi-view two-dimensional human body posture data.

[0146] A model building module 302 is used to construct a 3D human posture estimation model; the 3D human posture estimation model includes a plurality of spatial position embedding blocks, a plurality of dynamic pruning blocks, a plurality of semantic graph encoders, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block and a regression layer;

[0147] A spatial position embedding block, a dynamic pruning block and a semantic graph encoder form a group, and the input of each group comes from single-view 2D human pose data from different views of the same multi-view 2D human pose data;

[0148] The spatial position embedding block processes the single-view two-dimensional human posture data using a node position embedding function and a spatial graph embedding function to obtain a node position embedding and a spatial graph embedding, and splices the node position embedding and the spatial graph embedding into an input tag;

[0149] The dynamic pruning block performs pooling and pruning operations on the input tags to obtain representative tags;

[0150] The semantic graph encoder extracts node features from the input representative tags as output;

[0151] The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph converter encoders to obtain multi-view spatial fusion features;

[0152] The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features;

[0153] The regression layer converts the input multi-view spatiotemporal fusion features into three-dimensional posture coordinates as the output of the three-dimensional human posture estimation model.

[0154] The model training module 303 is used to train the 3D human body posture estimation model based on the acquired multi-view 2D human body posture data to obtain a trained 3D human body estimation model.

[0155] The 3D human body posture estimation module 304 is used to perform 3D human body posture estimation using a trained 3D human body posture estimation model.

[0156] The specific implementation of the lightweight multi-view 3D human pose estimation system based on semantic graph is the same as the lightweight multi-view 3D human pose estimation method based on semantic graph, and will not be repeated in this embodiment.

[0157] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A lightweight multi-view 3D human pose estimation method based on semantic graph, characterized in that: The following steps are involved: S1, obtaining multi-view two-dimensional human body posture data; S2, constructing a 3D human posture estimation model; the 3D human posture estimation model includes a plurality of spatial position embedding blocks, a plurality of dynamic pruning blocks, a plurality of semantic graph transformer encoders, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block and a regression layer; A spatial position embedding block, a dynamic pruning block and a semantic graph transformer encoder are a group, and the input of each group comes from single-view 2D human pose data from different views of the same multi-view 2D human pose data; The spatial position embedding block processes the single-view two-dimensional human posture data using a node position embedding function and a spatial graph embedding function to obtain a node position embedding and a spatial graph embedding, and splices the node position embedding and the spatial graph embedding into an input tag; The dynamic pruning block performs pooling and pruning operations on the input tags to obtain representative tags; The semantic graph transformer encoder extracts node features from the input representative tags as output; The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph transformer encoders to obtain multi-view spatial fusion features; The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features; The regression layer converts the input multi-view spatiotemporal fusion features into three-dimensional posture coordinates as the output of the three-dimensional human posture estimation model; S3, training a 3D human body posture estimation model based on the acquired multi-view 2D human body posture data to obtain a trained 3D human body estimation model; S4, using the trained 3D human pose estimation model to perform 3D human pose estimation.

2. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 1, characterized in that: The node position embedding function is expressed as: ； in, Represents node position embedding; Represents single-view 2D human pose data Middle Frame No. The position coordinates of each joint; Represents the feature conversion function, which maps the input features to dimensional space; Indicates that all Integrate into a representation; T represents the number of frames; J represents the number of joints.

3. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 2, characterized in that: The spatial graph embedding function is expressed as: ； in, Representing spatial graph embeddings; Indicates The weight matrix of order; represents the first The adjacency matrix of order; represents a nonlinear activation function; express A connection from global to local graph features.

4. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 1, characterized in that: The input tags are pooled and pruned to obtain representative tags, as follows: Marking the input Generate pooling marks through spatial average pooling ;right Apply density peak clustering algorithm to obtain representative markers , and discard the rest of the pooling tokens.

5. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 4, characterized in that: The density peak clustering algorithm is based on the k-nearest neighbor algorithm, according to the pooling mark The feature similarity is grouped into clusters, and the cluster center is used as a representative mark, as follows: For Pooling Tags , The corresponding density It is expressed as: ； in, Indicates pooling mark k nearest neighbors; Represents the local density used to calculate The number of nearest neighbors; express and The Euclidean distance between express and The square of the Euclidean distance between them; use Indicates pooling mark The distance to other pooling markers; if there is a pooling marker Density Greater than pooling mark Density ,So Mark for pooling to all these denser pooling markers The minimum Euclidean distance between; if there is no pooling mark Density Greater than ,So Mark for pooling The maximum Euclidean distance to all other pooled tokens; The calculation formula is as follows: ； in, express The corresponding density; Mark the pool The cluster center score of ; Finally, select the top The pooled markers are used as cluster centers, and the remaining pooled markers are assigned to the nearest cluster centers with higher density; the cluster centers are used as representative markers. ;in, represents a hyperparameter.

6. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 1, characterized in that: The semantic graph transformer encoder deeply mines the semantic information hidden in the representative tokens through semantic attention; the semantic attention is expressed as follows: ； ； ； ； in, Represents semantic attention; Query parameter representing semantic attention; Key parameter representing semantic attention; The Value parameter representing semantic attention; , and represents a linear layer; Represents feature dimension; represents the pruned node position embeddings in the representative markup; Represents pruned spatial graph embeddings in representative tokens; As a bias term of position features, it represents the combination of joint space and position knowledge; express The transpose of Represents a normalization operation.

7. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 6, characterized in that: Multi-head semantic attention is used to further enhance the features of pruned node position embedding, expressed as: ； in, represents multi-head semantic attention; express The connection of the attention heads; represents the hth attention head ; represents the hth attention head ; represents the hth attention head .

8. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 1, characterized in that: The node features output by all semantic graph transformer encoders are subjected to multi-view spatial fusion to obtain multi-view spatial fusion features, as follows: First, generate the final fusion feature output, the formula is as follows: ； ； ； in, , and represents a linear layer; Representation perspective Output node features; Representation perspective Output node features; represents the multi-head attention mechanism, As the query parameter of multi-head attention, As the key parameter of multi-head attention, As the Value parameter of multi-head attention; Represents the fusion node features; Represents a normalization operation; MLP represents a multi-layer perceptron; Represents the final fusion feature output; Finally, and and Connect and generate multi-view fusion spatial features , expressed as: ； in, Indicates that multiple feature vectors are concatenated.

9. The lightweight multi-view 3D human pose estimation method based on semantic graph according to claim 1, characterized in that: The input multi-view spatial fusion features are subjected to multi-view cross-channel fusion, and then multi-view spatiotemporal fusion is performed to obtain multi-view spatiotemporal fusion features, specifically as follows: Input the multi-view spatial fusion features into the multi-view cross-channel fusion block to generate multi-view cross-channel fusion features ; The obtained multi-view cross-channel fusion features Perform multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features ; The calculation formula of multi-view spatiotemporal fusion features is as follows: ； in, represents multi-head cross attention; LN represents normalization operation; Represents learnable temporal position embeddings; As the query parameter of multi-head cross attention, As the Key and Value parameters of multi-head cross attention.

10. A lightweight multi-view 3D human pose estimation system based on semantic graph, comprising: A data acquisition module, used to acquire multi-view two-dimensional human body posture data; A model building module, for building a 3D human posture estimation model; the 3D human posture estimation model includes a number of spatial position embedding blocks, a number of dynamic pruning blocks, a number of semantic graph transformer encoders, a multi-view spatial fusion block, a multi-view spatiotemporal fusion block and a regression layer; A spatial position embedding block, a dynamic pruning block and a semantic graph transformer encoder are a group, and the input of each group comes from single-view 2D human pose data from different views of the same multi-view 2D human pose data; The spatial position embedding block processes the single-view two-dimensional human posture data using a node position embedding function and a spatial graph embedding function to obtain a node position embedding and a spatial graph embedding, and splices the node position embedding and the spatial graph embedding into an input tag; The dynamic pruning block performs pooling and pruning operations on the input tags to obtain representative tags; The semantic graph transformer encoder extracts node features from the input representative tags as output; The multi-view spatial fusion block performs multi-view spatial fusion on the node features output by all semantic graph transformer encoders to obtain multi-view spatial fusion features; The multi-view spatiotemporal fusion block performs multi-view cross-channel fusion on the input multi-view spatial fusion features, and then performs multi-view spatiotemporal fusion to obtain multi-view spatiotemporal fusion features; The regression layer converts the input multi-view spatiotemporal fusion features into three-dimensional posture coordinates as the output of the three-dimensional human posture estimation model; A model training module is used to train a 3D human posture estimation model based on 2D human posture data from multiple perspectives to obtain a trained 3D human posture estimation model; The 3D human pose estimation module is used to perform 3D human pose estimation using a trained 3D human pose estimation model.

Images