Layer chain constraint three-dimensional human body posture estimation method based on monocular video stream

Abstract

The invention discloses a layer chain constraint three-dimensional human body posture estimation method based on a monocular video stream, and relates to the field of three-dimensional human body posture estimation. According to the method, a layer chain constraint three-dimensional human body posture estimation model is adopted, the model comprises a space-time coding basic block, and the space-time coding basic block comprises an SFM module, an IFM module, an EFM module, a space-time attention module and a space-time diagram convolution module; the SFM module extracts inherent space information from the attitude features from a single frame angle; the IFM module carries out frame distribution limiting operation on the attitude features from an inter-frame angle and carries out modeling on inter-frame time and space dependency relationships. According to the method, the 2D attitude is modeled from three perspectives of single frame, inter-frame and whole, the physical topological structure of the human body can be explicitly expressed, and the task precision can be better improved through mutual fusion and conversion of the global level and the local level.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional human body posture estimation, and in particular relates to a layer-chain-constrained three-dimensional human body posture estimation method based on a monocular video stream. Background Art

[0002] 3D human pose estimation is an important research direction in the field of computer vision and has shown wide application value in many fields. At present, the most mainstream technology in this direction is to upgrade the 2D human joint points in human images or videos to 3D joint representation. This method can be decomposed into two consecutive steps. The first step is to use the existing two-dimensional human key point detector to obtain the 2D joint coordinates of the human body in the human image or video, and the second step is to output the obtained joint coordinates into three-dimensional space through the mapping of complex deep neural networks, and finally obtain the 3D coordinate representation of the human joints. Due to the serious depth ambiguity problem in this process, it is extremely challenging to estimate the human pose.

[0003] Previous studies have shown that graph convolutional networks (GCNs) have achieved remarkable results in 3D human pose estimation tasks. At the same time, the Transformer architecture has also shown extremely outstanding performance in recent years. The core of Transformer, the attention mechanism, has also been widely used in 3D human pose estimation tasks due to its ability to understand global context. The 3D human pose estimation architecture based on the attention mechanism and the 3D human pose estimation architecture based on the graph convolutional network have become the de facto benchmark architectures for the 3D human pose estimation task. Although both are benchmark architectures for this task, the two types of networks have different focuses on the extraction of pose information. The graph convolutional network focuses on capturing local fine features, while the attention mechanism focuses on capturing global feature information of the pose. They each learn temporal and spatial dependencies from both local and global levels.

[0004] In order to solve the problem of depth ambiguity in the task of 2D to 3D pose estimation, many studies use graph convolutional networks to learn the constraint information between local joints of the human body, or describe the overall spatiotemporal dependency of 2D pose sequences based on the attention mechanism of Transformer. These studies have indeed achieved remarkable results in the task of 3D human pose estimation. However, these studies still have the following problems:

[0005] (1) The process of regressing 2D human pose to 3D human pose is essentially an ill-posed problem. The core challenge of ill-posedness is that there are multiple reasonable and effective possible solutions to regressing 3D human pose from a single 2D skeleton. In addition, the inherent diversity and uncertainty of human pose, as well as the serious occlusion phenomenon in real field environments, make it difficult to accurately estimate the 3D pose of the human body.

[0006] (2) Most of the existing inventions of 3D human pose estimation based on video input directly learn 2D pose feature information through graph convolutional networks or attention mechanisms. This type of research can indeed capture partial temporal and spatial dependency information from the pose sequence input, but this method is too one-way: it only considers the temporal and spatial correlation of the enhanced features or the matrix operations within the mechanism, without explicitly summarizing the spatial feature information contained in the physical topological structure of the human body, which may lead to the loss of effective information.

[0007] (3) Considering the process of upgrading 2D pose to 3D pose from a macro perspective, we can also learn pose features from the local level and the global level, and graph convolutional networks and attention mechanisms focus on the local level and the global level respectively. However, most of the existing benchmark methods are separate attention architectures or graph convolution architectures, and do not combine the two well, nor do they make an exact division between the local and the global in terms of time and space.

[0008] In summary, most of these studies use a single learning method, usually by directly learning the spatiotemporal dependencies of 2D human poses and regressing them to 3D poses, without any dedicated techniques to explicitly express the temporal and spatial dependencies.

[0009] Based on this, this paper proposes a new method based on spatiotemporal attention and spatiotemporal graph convolution, called layer chain constraint network (LCC), for estimating 3D human posture from monocular video. This model integrates the spatiotemporal attention mechanism and spatiotemporal graph convolution network, and can explicitly express the physical topological structure of the human body. Through the mutual fusion conversion of the global and local levels, the task accuracy can be better improved.

[0010] In addition, three modules are innovatively introduced, aiming to deeply learn the long-term spatiotemporal dependency features of 2D posture from different dimensions to improve the estimation accuracy of the model. Specifically, the present invention designs an SFM module, which focuses on a single-frame perspective and is responsible for capturing fine topological spatial information of human body posture. A new frame domain joint constraint mechanism is constructed inside the module, and the spatial dependency relationship of the initial input features is learned by using methods such as node weight ratio setting and node classification preset. At the same time, the present invention designs an IFM module as an implementation of the inter-frame perspective. It receives the potential spatial features from the single-frame perspective, and then extracts and fuses the local spatiotemporal dependencies, and integrates these dependencies as intermediate features into the global perspective, and jointly participates in the capture of spatiotemporal feature vectors. Furthermore, the present invention also introduces an EFM module, which can promote the interaction and fusion of information at different levels, thereby further improving the accuracy of the estimation task. The present invention makes full use of the deep dependency between the long-term spatiotemporal feature information of 2D human body posture and optimizes the modeling ability of the model. Summary of the invention

[0011] The purpose of the present invention is to provide a layer-chain constrained three-dimensional human pose estimation method based on monocular video stream, so as to solve the problem that most of the 3D human pose estimation technologies in the prior art use a single learning method, which directly learns the spatiotemporal dependencies of 2D human poses and regresses them to 3D poses without explicitly summarizing and expressing the temporal and spatial dependencies.

[0012] To achieve the above purpose, the technical solution adopted by the present invention is:

[0013] In a first aspect, the present invention proposes a layer-chain-constrained 3D human posture estimation model based on a monocular video stream, comprising a spatiotemporal coding basic block, wherein M spatiotemporal coding basic blocks are serially connected to form a model body, where M=6; the spatiotemporal coding basic block comprises an SFM module, an IFM module, an EFM module, a temporal attention module, a spatial attention module, a temporal graph convolution module, and a spatial graph convolution module;

[0014] The temporal attention module and the spatial attention module use a multi-head attention mechanism to learn the global spatial features and the global temporal features of the posture sequence respectively;

[0015] The temporal graph convolution module and the spatial graph convolution module use graph convolution networks to process posture features from two aspects: local time and local space.

[0016] The SFM module extracts inherent spatial information from the posture features from a single frame perspective, physically constrains the human joint feature vectors through the frame domain joint constraint mechanism, and guides the joints with implicit associations in the single frame topology structure to interact;

[0017] The IFM module performs a frame distribution restriction operation on the posture features from an inter-frame perspective and models the temporal and spatial dependencies between frames;

[0018] The EFM module is used to extract the potential spatiotemporal correlation of the posture feature vector and perform feature fusion.

[0019] Preferably, the SFM module is as follows:

[0020] Input initial feature X l , use the frame domain joint constraint mechanism to set it a priori; after calculation, a frame domain intermediate feature vector X' is obtained l , represents the constraint feature information after the single frame posture is constrained;

[0021] The intermediate feature vector X' in the frame domain lThrough a fully connected layer and dimensional transformation, the number of human body joint points is determined, and then through a fully connected layer, it is regressed to the feature dimension vector X with the same input feature. l "; Use EFM module to X l "vector is learned; finally, the parallel time graph convolution module and spatial graph convolution module are used to complement and fuse the two feature information obtained; it is expressed as follows:

[0022] X l "=FFN(ρ(FFN(X' l )))

[0023] Γ l =S_GCN(α*X l “+β*EFM(X l "))+T_GCN(β*X l "+α*EFM(X l “))

[0024] Among them, α+β=1; ρ(*) refers to the dimensionality conversion of the feature; FFN represents the fully connected layer, S_GCN represents the spatial graph convolution module, and T_GCN represents the temporal graph convolution module.

[0025] Preferably, the frame domain joint constraint mechanism includes node classification preset and node weight ratio setting;

[0026] The node classification preset divides a number of fixed joint nodes with stable characteristics into the first layer, and divides two adjacent joints in the physical structure into the second layer and the third layer; divides all joints into five node types, namely, first layer overlapping points, first layer non-overlapping points, first and second layer overlapping points, second and third layer overlapping points, and third layer non-overlapping points; and sets node weight ratios for the five node types;

[0027] The frame domain joint constraint mechanism imposes guided constraints on feature inputs based on node type and level division: the node weight ratio setting coefficient is multiplied by five different node types; the weight ratio coefficient of the overlapping points is divided by the weight ratio of each of the two adjacent levels, that is,

[0028]

[0029] Among them, W i,j Represents the weight ratio coefficient of the joint overlap point, j = 0 means there is only one joint point, Indicates that a certain joint point is at the nth level among the three levels.

[0030] Preferably, the IFM module is as follows:

[0031] Input feature El , E l =X l +Γ l ; Input feature E l After the restricted frame distribution operation, the feature vector E′ is obtained l , it passes through the spatial graph convolution module and the temporal graph convolution module successively, and the posture feature vectors after the spatial graph convolution module and the temporal graph convolution module are connected with themselves through a fully connected module to form a residual connection, and then pass through the EFM module to complete the final feature learning; the output formula of the IFM module is:

[0032] E″ l =S_GCN(E′ l )

[0033]

[0034] Z l =EFM(E″) l +FFN(E″ l ))+EFM(E″′ l +FFN(E″′ l ))

[0035] in, Represents the posture feature vector E″ obtained by the spatial graph convolution module l By restricting the frame distribution for the second time; FFN represents the fully connected layer, S_GCN represents the spatial graph convolution module, and T_GCN represents the temporal graph convolution module.

[0036] Preferably, the limiting frame distribution operation includes setting the number of processing headers corresponding to the input frame and performing distribution processing on each limiting frame;

[0037] The input feature vector E l Divide into h equal length limit frames Then generate h distribution coefficient arrays that conform to the normal distribution Multiply it with each restricted frame to add constraints to the spatiotemporal information between frames and obtain the intermediate feature vector E′ distributed through the restricted frames l ; The process is expressed by the following formula:

[0038]

[0039] Preferably, the EFM module is as follows:

[0040] The EFM module consists of a fully connected layer and a 2D convolution; by taking the input feature X lIt is concatenated with the feature vector obtained through the fully connected layer to obtain the intermediate feature vector, and then the 2D convolution is used to downsample the features of the local field of view and regress to obtain the learned posture feature vector H. l ; The process is expressed by the following formula:

[0041] H l =Down_Samping(Concat(X l ,FFN(X l )))

[0042] Among them, Down_Samping is a regression operation; FFN represents a fully connected layer.

[0043] Preferably, the model is optimized using the MPJPE loss function to optimize the error between the estimate and the ground truth:

[0044]

[0045] Among them, g i and p i denote the ground truth 3D joint position and the estimated 3D joint position of joint i, respectively.

[0046] The second aspect of the present invention proposes a layer-chain-constrained 3D human posture estimation method based on a monocular video stream using the above model, comprising the following steps:

[0047] S1. Input initial feature vector X l To the SFM module, the SFM module extracts the inherent spatial information of the posture features from a single frame perspective and outputs a feature vector Γ containing the latent spatial features l ;

[0048] S2, the feature vector Γ containing the latent space features l and the initial eigenvector X l Input to the IFM module, IFM performs frame distribution restriction operation on the posture features from the perspective of inter-frame, and outputs the feature vector Z l ;

[0049] S3, eigenvector Z l and the eigenvector Γ l Input to the EFM module, which is used to promote the interaction and integration of information at different levels;

[0050] S4, eigenvector Z l Also as part of the input together with the initial eigenvector X l The input is sent to the spatial attention module and the spatial graph convolution module to fuse local spatiotemporal dependencies and integrate into the global perspective as an intermediate feature to jointly capture the spatiotemporal feature vector.

[0051] S5, the spatiotemporal feature vector captured in S4 is combined with the feature vector of different levels of information in S3, and then combined with the initial feature vector X l Input to the temporal attention module and the temporal graph convolution module respectively;

[0052] S6, using the temporal attention module and the temporal graph convolution module to fuse the global and local levels, the fused temporal feature vector together with the initial feature vector X l Interactive fusion to obtain a 3D pose sequence.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] (1) The present invention proposes a solution strategy to model 2D posture from three perspectives: single frame, inter-frame, and overall. At the same time, it implements a step-by-step interactive fusion of spatiotemporal features from both the global and local levels, helping to establish a stronger link relationship between long-term spatiotemporal dependencies and solving the limitations of previous studies.

[0055] (2) The present invention proposes a new architecture LCC; the architecture adopts a layer chain design, combines spatiotemporal attention and spatiotemporal graph convolution to communicate spatiotemporal information from different levels, and achieves better spatiotemporal dependency modeling of 2D postures.

[0056] (3) The present invention proposes a frame domain joint constraint mechanism to constrain the posture space characteristics from a single frame perspective, and designs a novel SFM module based on this. At the same time, a method for limiting frame distribution processing is proposed to design an IFM module.

[0057] (4) The present invention designs an EFM module with feature fusion function; the module can physically transform the initial input features to obtain better feature semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a block diagram of the architecture of the middle-layer chain-constrained spatiotemporal feature capturer of the present invention;

[0059] Figure 2 It is a structural block diagram of the spatiotemporal attention module and the spatiotemporal graph convolution module in the present invention;

[0060] Figure 3 It is a structural block diagram of the SFM module, IFM module and EFM module in the present invention;

[0061] Figure 4 Schematic diagram of the frame domain joint constraint mechanism and the restricted frame distribution operation in the present invention;

[0062] Figure 5This is a qualitative visual result diagram of the present invention on the Human3.6M test set. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Embodiment 1:

[0065] The present invention proposes a layer-chain constrained 3D human pose estimation method based on monocular video stream, which is a new 3D human pose estimation technology. The model adopts a layer-chain constrained spatiotemporal feature capturer (LCC) to regress 3D human pose from monocular video.

[0066] For the input two-dimensional posture sequence, LCC innovatively adopts a cross-learning strategy of three perspectives: single frame, inter-frame, and overall perspective. This method not only processes the posture details within the frame from a single frame perspective, but also captures the dynamic changes of spatiotemporal dependencies between frames and the global characteristics of the entire sequence. In addition, LCC uses a layer-chain design architecture to take the feature information learned from each perspective as input, and pass it together with the original posture sequence data to a higher-level encoder for further encoding and learning, thereby achieving a deep understanding and application of posture information. The layer-chain constrained feature capturer LCC proposed in the present invention is as follows: Figure 1 shown.

[0067] The capturer uses a total of 7 components: SFM module, IFM module, EFM module, two spatiotemporal attention modules and two spatiotemporal graph convolution modules. The 7 components together form a spatiotemporal encoding basic block, which is connected in series through M = 6 basic blocks to form the main part of the architecture.

[0068] See also Figure 2-3 , which is the component module of LCC architecture. Figure 2 As shown, the present invention uses the attention module and the graph convolution module to capture the corresponding potential spatiotemporal relationships from the two aspects of space and time respectively; Figure 2 (a) shows the spatial attention and temporal attention modules. The difference is that the input feature vector has undergone some changes in dimension. The left one in the dotted box represents the spatial feature, and the right one represents the temporal feature. This representation method is also applicable to Figure 2 (b); Figure 2 (b) shows the spatial graph convolution and temporal graph convolution modules, the difference lies in the input features and adjacency matrix. Figure 3 As shown, Figure 3 (c) is the SFM module, which extracts inherent spatial information from a single frame perspective; Figure 3 (d) is the IFM module, which performs a restricted vibration distribution operation on the input features from an inter-frame perspective; Figure 3 (e) is the EFM module, which is used for feature fusion. The following will introduce several components separately.

[0069] (i) Temporal and spatial attention module;

[0070] Transformer is described as a general representation learning architecture, whose attention module can convert arbitrary feature information between tokens, so as to process input information from different feature dimensions. Multi-head self-attention mechanism (MHSA) is the core mechanism of Transformer architecture, which mainly calculates the correlation matrix between tokens and propagates information across different tokens.

[0071] In representation, given N input tokens with embedding dimension C Multi-head self-attention MHSA is expressed as:

[0072] MHSA(Q,K,V)=Concat(H1,...,H i ,...,H h )W (O) (1)

[0073]

[0074] Among them, W (O) is the projection parameter matrix, Q (i) ,K (i) ,V (i) Denotes the i-th head attention H i The query Q, key K, and value V feature matrices are obtained by linearly mapping the input tokens.

[0075] Following the interactive fusion of global and local features, the attention mechanism is used to learn the global spatial features and global temporal features separately. In the spatial multi-head attention S-MHSA, the i-th head attention H i The query Q is obtained by linearly mapping the input token s , key K s Sum value V s Feature matrix. In order to calculate these three matrices, a spatial feature Multiply them with their respective projection parameter matrices, namely:

[0076]

[0077] Similarly, the temporal-attention module uses a temporal feature θ t Perform a similar multiplication operation with the three projection parameter matrices:

[0078]

[0079] However, the difference between the spatial attention module Spatial-Attention and the temporal attention module Temporal-Attention is that the feature vector θ used by both s and θ t There are some changes in the dimensions, namely the spatial characteristics The time characteristics The result of the calculation will be sent to the next feature extractor, such as Figure 2 (a) It should be noted that the spatial-attention module is used in the present invention to extract global spatial features and receive the posture feature vector of the implicit spatiotemporal information from the IFM module.

[0080] (ii) Temporal and spatial graph convolution modules;

[0081] Unlike the Transformer architecture that focuses on the global information of the pose sequence, the spatiotemporal graph convolutional structure focuses on local information. Input a 2D human pose sequence P and represent it as a spatiotemporal graph G = {ν t ,ε t |t=1,...,T}, where ν t = { t,i |i=1,...,N} represents the human body posture joint at time t, ε t Represents the posture skeleton corresponding to the moment. The general GCN layer used to aggregate adjacent node information can be expressed as follows:

[0082]

[0083] in, is the adjacency matrix with self-connection added, I N represents the identity matrix, is a diagonal matrix, the adjacency matrix Whether the two joint points are equal to 0 is used to distinguish whether the joints are directly connected. The graph convolutional network focuses on local feature learning and also processes posture features from two aspects: local time and local space. The GCN module used is expressed by the following formula:

[0084]

[0085] GCN(X (i) )=X(i-1) +X (i) +X (i) W1 (7)

[0086] Among them, W0 and W1 are the trainable weight matrices defined in each GCN layer.

[0087] The difference between Spatial-GCN and Temporal-GCN lies in the different input features and adjacency matrices. The difference in input features is similar to the representation in the spatiotemporal attention structure. For Spatial-GCN, the adjacency matrix Represents the initial human body physical topology, which is manifested as the corresponding positions of two adjacent nodes in the adjacency matrix are not equal to 0. For the temporal graph convolution Temporal-GCN, it is calculated by The similarity between individual nodes at different times is obtained, and then the k nearest nodes are selected as the connecting nodes in the adjacency matrix. The representation of spatiotemporal graph convolution is as follows: Figure 2 (b) as shown.

[0088] (iii) SFM module;

[0089] The SFM module extracts inherent spatial information from a single frame, physically constrains the human joint feature vectors by setting node weight ratios and node classification presets, and guides the implicitly associated joints in the single frame topology to interact. Figure 3 (c) as shown.

[0090] Input initial features In order to better express the potential constraints between human joints, the present invention uses the frame domain joint constraint mechanism to set them a priori. Figure 4 (a) is shown, wherein the node classification preset is based on a number of fixed joint nodes with relative invariance in the physical topological structure of the human body and two adjacent joints in the physical structure. Specifically, the present invention regards a number of fixed nodes with stable characteristics as the first level, specifically including 9 joint points of the head, nose, chest, left shoulder, right shoulder, spine, hip joint, left hip joint and right hip joint. The two adjacent joints in the physical structure are divided into the second and third levels, the second level includes 8 joint points of the left shoulder, left elbow, right shoulder, right elbow, left hip, left knee, right hip and right knee joints, and the third level includes 8 joint points of the left elbow, left wrist, right elbow, right wrist, left knee, left ankle, right knee and right ankle joints. The present invention defines L i To specifically indicate the characteristics of a certain level containing nodes:

[0091]

[0092] According to the prior knowledge of human topology, the present invention divides the first layer into 4 parts in total according to the physical distance and structure of the joints, and the head is divided into one part separately. Based on this, all joints are divided into 5 more detailed categories according to their positions in the three levels: first layer overlapping points (corresponding to joints 0 and 8), first layer non-overlapping points (corresponding to joints 7, 9, and 10), first and second layer overlapping points (1, 4, 11, and 14), second and third layer overlapping points (corresponding to joints 2, 5, 12, and 15), and third layer non-overlapping points (corresponding to joints 3, 6, 13, and 16), with C i To represent the classification of a node, and L i The expression is similar.

[0093] Considering that the joints farther from the relatively stable joints have a greater impact on the loss accumulation, the node weight ratio is set for each joint in advance, and the weight of all joints in the first layer is set to 1; the weight of the remaining nodes in the upper part of the body in the second layer is set to 1.5, and the weight of the nodes in the lower part of the body is set to 2; the weight of the remaining nodes in the upper part of the body in the third layer is set to 2.5, and the weight of the nodes in the lower part of the body is set to 3. So far, all joints of the 2D posture are in a specific layer, accompanied by corresponding node weights.

[0094] The present invention imposes guidance constraints on feature input by node type and level division: the node weight ratio setting coefficient is multiplied by 5 different node types. The weight ratio coefficient of the overlapping points is divided by the weight ratio of each of the two adjacent levels, that is,

[0095]

[0096] Among them, W i,j Represents the weight ratio coefficient of the joint overlap point, j = 0 means there is only one joint point, It means that a certain joint point is in the nth layer among the 3 layers. It should be noted that according to the setting of the present invention, there are no non-overlapping points in the second layer and overlapping points in the first three layers.

[0097] After calculation, we get an intermediate feature vector in the frame domain. It represents the constrained feature information after the single-frame posture is constrained. Then it is passed through a fully connected layer and dimensionally transformed to determine the number of human body posture joint points, and then regressed to the same feature dimension vector as the input feature through a fully connected layer. In order to obtain more spatial feature information contained in the vector, a feature learning module EFM is used to l "Vector learning. Finally, the parallel time graph convolution and spatial graph convolution are used to complement the two feature information obtained, which is expressed as follows:

[0098] X l "=FFN(ρ(FFN(X' l ))) (10)

[0099] Γ l =S_GCN(α*X l “+β*EFM(X l "))+T_GCN(β*X l "+α*EFM(X l “)) (11)

[0100] Wherein, α+β=1, and α=0.3 is preset. ρ(*) here refers to the dimension conversion of the feature.

[0101] The SFM module directly transmits the information extracted by the two graph convolution modules as input to the IFM module. The spatial information contained therein can be a huge influencing factor affecting the spatial information between frames.

[0102] See also Figure 4 , which is the frame domain joint constraint mechanism and restricted frame distribution method. Figure 4 (a) The frame domain joint constraint mechanism is completed from a single frame perspective using the node weight ratio setting and node classification preset method. Figure 4 (b) To restrict the frame distribution method, we design it from the perspective of frames. The core idea is to use a set of arrays that satisfy a certain distribution to exploit the hidden associations of their own values ​​to better learn the potential spatiotemporal information between frames.

[0103] (iv) IFM module;

[0104] The IFM module is designed to effectively model the temporal and spatial dependencies between frames. Since the input vector of this module contains the initial input feature vector X l and the feature vector Γ containing the latent space features transmitted from the SFM module l ,In order to better obtain the potential spatiotemporal dependencies, the input features are processed using two restricted frame distributions. Figure 3 (d) as shown.

[0105] The restricted frame distribution includes two steps: setting the number of processing headers corresponding to the input frame and distributing each restricted frame. The default connection time length of the LCC model is 1, that is, input a feature dimension vector E l =X l +Γ l , the next connected frame of a particular frame is its next frame rate in physical logic. Using this, the input feature vector E l Divide into h equal length limit frames Then generate h distribution coefficient arrays that conform to the normal distribution By multiplying it with each restricted frame, we can successfully add constraints to the spatiotemporal information between frames and obtain the intermediate feature vector E′ distributed through the restricted frames. l The process can be expressed by the following formula:

[0106]

[0107] like Figure 2 As shown in (b), after the initial input features are restricted by the frame distribution, they are successively passed through the spatial graph convolution Spatial-GCN and the temporal graph convolution Temporal-GCN. This is because the input features contain feature vectors containing latent spatial features transmitted from the SFM module. The spatial graph convolution that focuses on learning local features for the previous and next contexts with the same spatial features can maximize the module effect. It should be noted that the difference between the second restricted frame distribution and the first is that the distribution coefficient array γ i After a left-right scroll of length h / / 2, there is no change in value. The present invention believes that using a group of distribution coefficient arrays with the same value and satisfying normal distribution can better learn the potential spatiotemporal information between frames with the implicit association of its own value.

[0108] The posture feature vectors after spatial graph convolution Spatial-GCN and temporal graph convolution Temporal-GCN form a residual connection with themselves through a fully connected module, and then complete the final feature learning through the feature learning module EFM. The output formula of the IFM module is:

[0109] E″ l =S_GCN(E′ l )

[0110]

[0111] Z l =EFM(E″) l +FFN(E″ l ))+EFM(E″′ l +FFN(E″′ l ))

[0112] in, Represents the posture feature vector E″ obtained by spatial graph convolution Temporal-GCN l By limiting the frame distribution for the second time, the result Z l It also serves as a partial input into the overall learning module Spatial-Attention.

[0113] (V) EFM module;

[0114] The EFM module is a general feature learning module that can convert arbitrary feature information between tokens. This module is reused multiple times in the LCC architecture to extract the potential spatiotemporal correlation of a posture feature vector.

[0115] This module is mainly composed of a fully connected layer and a 2D convolution. l By concatenating the feature vector obtained through the fully connected layer, the intermediate feature vector can be obtained. Then, the features of the local field of view are downsampled using 2D convolution, and the learned posture feature vector can be regressed. The process is shown in the figure below:

[0116] H l =Down_Samping(Concat(X l ,FFN(X l ))) (14)

[0117] One advantage of this module is that the model size can be customized and it has universal applicability. Figure 3 (e) as shown.

[0118] The above are the 7 components that constitute the basic block of the capturer. In addition, in order to optimize the model of the present invention, the MPJPE loss function is used to optimize the error between the estimated value and the ground truth:

[0119]

[0120] where g i and p i denote the ground truth 3D joint position and the estimated 3D joint position of joint i, respectively.

[0121] Experimental verification:

[0122] The present invention uses two 3D human pose estimation benchmark datasets, Human3.6M and MPI-INF-3DHP, to conduct a comprehensive performance evaluation of the proposed LCC algorithm.

[0123] As a benchmark dataset in this field, Human3.6M is rich in that it contains more than 3.6 million carefully annotated video frames, recording the detailed postures of 11 professional actors performing 15 different actions including walking, talking, taking pictures, etc. The MPI-INF-3DHP dataset is known for its challenges and provides more than 1.3 million frames of 3D posture data, which show more diverse motion scenes than Human3.6M, including three different environmental settings: green screen (GS), non-green screen (noGS), and outdoor (Outdoor).

[0124] During the evaluation process, for the Human3.6M dataset, the present invention follows the convention and uses two core indicators: MPJPE (mean joint position error, Protocol #1) and PMPJPE (mean joint position error after rigid transformation alignment, Protocol #2). The former directly calculates the average Euclidean distance between the predicted and true joint positions, while the latter first aligns the predicted posture with the true posture through rigid transformation before calculating the error. For the MPI-INF-3DHP dataset, the focus is on the percentage of correct key points within the range of 150mm (PCK) and the area under the curve (AUC) within the PCK threshold range to comprehensively measure the performance of the model on this dataset.

[0125] Table 1 Quantitative comparison of HUMAN 3.6M with the current state-of-the-art multi-frame methods under Scheme #1 and Scheme #2

[0126]

[0127] As shown in Table 1, by comparing these evaluation results with the existing state-of-the-art methods on the Human3.6M dataset, the present invention found that, based on the use of CPN as a 2D key point detector to obtain an input 2D posture sequence, the model of the present invention achieved excellent scores of 38.9 mm and 32.7 mm on MPJPE and PMPJPE, respectively, significantly surpassing all previous methods, and the index data obtained by the model of the present invention is basically the best or second best. This achievement fully demonstrates the innovation and breakthrough of the present invention in the field of 3D human posture estimation.

[0128] The present invention also gives accurate 2D real coordinates as input, and the comparison results are shown in Table 2.

[0129] Table 2 Quantitative comparison with state-of-the-art methods on Human 3.6M under protocol #1

[0130]

[0131] As shown in Table 2, the method of the present invention ranked first in 13 of the 15 test actions, and achieved the best average error accumulation of 25.8 mm, which was better than all the compared methods. This verifies the effectiveness of the method of the present invention.

[0132] In the cross-dataset evaluation scenario, the present invention is trained on the Human3.6M training dataset and tested on the MPI-INF-3dHP test dataset, and the LCC method proposed in the present invention is compared with the previous state-of-the-art methods. The results are shown in Table 3.

[0133] Table 3 Performance comparison with the most advanced methods on the MPI-INF-3DHP dataset

[0134]

[0135]

[0136] As shown in Table 3, T represents the input frame rate, and PCK represents the percentage of correct key points within 150mm using MPJPE. The method in the present invention can achieve the best 98.9. At the same time, the area under the curve AUC can reach 86.1. The method in the present invention has achieved the best results in multiple scenes and indicators of this dataset, outperforming all other methods. This verifies the powerful generalization ability of the method in the present invention for unknown scenes.

[0137] The present invention uses the Walking and Sitting actions on the Human 3.6M dataset S9 and the Eating and Greeting actions on the S11 to evaluate the LCDF model. Figure 5 As shown, it can be seen that the model in the present invention can accurately predict the 3D human posture in the environment in different actions, which shows the effectiveness of the method proposed in the present invention.

[0138] The above description is only used to help understand the method of the present invention and its core essence, but the protection scope of the present invention is not limited thereto. For those skilled in the art in the art, equivalent replacement or change according to the technical solution and inventive concept of the present invention within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A layer-chain-constrained 3D human pose estimation model based on monocular video stream, It is characterized in that It includes a spatiotemporal coding basic block, and the model body is formed by connecting M spatiotemporal coding basic blocks in series, where M=6; the spatiotemporal coding basic block includes an SFM module, an IFM module, an EFM module, a temporal attention module, a spatial attention module, a temporal graph convolution module, and a spatial graph convolution module; The temporal attention module and the spatial attention module use a multi-head attention mechanism to learn the global spatial features and the global temporal features of the posture sequence respectively; The temporal graph convolution module and the spatial graph convolution module use graph convolution networks to process posture features from two aspects: local time and local space. The SFM module extracts inherent spatial information from the posture features from a single frame perspective, physically constrains the human joint feature vectors through the frame domain joint constraint mechanism, and guides the joints with implicit associations in the single frame topology structure to interact; The IFM module performs a frame distribution restriction operation on the posture features from an inter-frame perspective and models the temporal and spatial dependencies between frames; The EFM module is used to extract the potential spatiotemporal correlation of the posture feature vector and perform feature fusion.

2. The layer-chain-constrained 3D human posture estimation model based on monocular video stream according to claim 1, It is characterized in that The SFM module is as follows: Input initial features Use the frame domain joint constraint mechanism to set it a priori; after calculation, a frame domain intermediate feature vector is obtained Represents the constraint feature information after constraints are imposed on a single frame posture; The intermediate feature vector in the frame domain Pass through a fully connected layer and transform the dimension, and then return to the same feature dimension vector as the input feature through a fully connected layer Use EFM module to The vector is learned; finally, the parallel time graph convolution module and the spatial graph convolution module are used to complement and fuse the two feature information obtained; it is expressed as follows: Among them, α+β=1; ρ(*) refers to the dimensionality conversion of the feature; FFN represents the fully connected layer, S_GCN represents the spatial graph convolution module, and T_GCN represents the temporal graph convolution module.

3. The layer-chain-constrained 3D human posture estimation model based on monocular video stream according to claim 2, It is characterized in that The frame domain joint constraint mechanism includes node classification preset and node weight ratio setting; The node classification preset divides a number of fixed joint nodes with stable characteristics into the first layer, and divides two adjacent joints in the physical structure into the second layer and the third layer; divides all joints into five node types, namely, first layer overlapping points, first layer non-overlapping points, first and second layer overlapping points, second and third layer overlapping points, and third layer non-overlapping points; and sets node weight ratios for the five node types; The frame domain joint constraint mechanism imposes guided constraints on feature inputs based on node type and level division: the node weight ratio setting coefficient is multiplied by five different node types; the weight ratio coefficient of the overlapping points is divided by the weight ratio of each of the two adjacent levels, that is, Among them, W , Represents the weight ratio coefficient of the joint overlap point, j = 0 means there is only one joint point, Indicates that a certain joint point is at the nth level among the three levels.

4. The layer-chain-constrained 3D human posture estimation model based on monocular video stream according to claim 1, It is characterized in that The IFM modules are as follows: Input Features The input features After the restricted frame distribution operation, the feature vector is obtained It passes through the spatial graph convolution module and the temporal graph convolution module successively. The posture feature vectors passing through the spatial graph convolution module and the temporal graph convolution module are connected with themselves through a fully connected module to form a residual connection, and then pass through the EFM module to complete the final feature learning; the output formula of the IFM module is: in, Represents the posture feature vector obtained by the spatial graph convolution module By restricting the frame distribution for the second time; FFN represents the fully connected layer, S_GCN represents the spatial graph convolution module, and T_GCN represents the temporal graph convolution module.

5. The layer-chain-constrained 3D human posture estimation model based on monocular video stream according to claim 4, It is characterized in that The limiting frame distribution operation includes setting the number of processing headers corresponding to the input frame and performing distribution processing on each limiting frame; The input feature vector Divide into h equal length limit frames Then generate h distribution coefficient arrays that conform to the normal distribution Multiply it with each restricted frame to add constraints to the spatiotemporal information between frames and obtain the intermediate feature vector distributed through the restricted frames. The process is expressed by the following formula:

6. The layer-chain-constrained 3D human posture estimation model based on monocular video stream according to claim 1, It is characterized in that The EFM module is as follows: The EFM module consists of a fully connected layer and a 2D convolution; by It is concatenated with the feature vector obtained through the fully connected layer to obtain the intermediate feature vector, and then the 2D convolution is used to downsample the features of the local field of view and regress to obtain the learned posture feature vector. The process is expressed by the following formula: Among them, _Samping is a regression operation; FFN represents a fully connected layer.

7. The layer-chain-constrained 3D human posture estimation model based on monocular video stream according to claim 1, It is characterized in that The model is optimized using the MPJPE loss function to optimize the error between the estimate and the ground truth: Among them, g i and p i denote the ground truth 3D joint position and the estimated 3D joint position of joint i, respectively.

8. A method for estimating 3D human posture based on monocular video stream using the model described in claims 1-7, It is characterized in that The steps include: S1. Input initial feature vector To the SFM module, the SFM module extracts the inherent spatial information of the posture features from a single frame perspective and outputs a feature vector containing the latent spatial features S2, the feature vector containing the latent space features and the initial eigenvector Input to the IFM module, IFM performs frame distribution restriction operation on the posture features from the perspective of inter-frame, and outputs the feature vector S3, feature vector and the eigenvector Input to the EFM module, which is used to promote the interaction and integration of information at different levels; S4, feature vector Also as part of the input together with the initial eigenvector The input is sent to the spatial attention module and the spatial graph convolution module to fuse local spatiotemporal dependencies and integrate into the global perspective as an intermediate feature to jointly capture the spatiotemporal feature vector. S5, merge the spatiotemporal feature vector captured in S4 with the feature vector of different levels of information in S3, and then merge them with the initial feature vector Input to the temporal attention module and the temporal graph convolution module respectively; S6, using the temporal attention module and the temporal graph convolution module to fuse the global and local levels, the fused temporal feature vector together with the initial feature vector Interactive fusion to obtain a 3D pose sequence.

Images