Human body posture estimation device and method based on global pressure

Abstract

The invention provides a global pressure-based human body posture estimation device and a global pressure-based human body posture estimation method. The device comprises a spatial feature encoder used for extracting human body global pressure spatial features from a global pressure frame sequence of human body actions; the long and short term time sequence attention module is used for extracting human body global pressure time sequence features from the human body global pressure space features and fusing the human body global pressure time sequence features with the human body global pressure space features to obtain pressure space-time features; and the action regression device is used for performing nonlinear regression calculation on the pressure spatial-temporal characteristics to obtain posture and displacement parameters of the human body parameterized model. Visual information is replaced by pressure information which does not depend on illumination conditions, has no privacy invasion and is not affected by visual occlusion, and non-invasion expansion of human body posture estimation in special scenes is achieved.

Description

Technical Field

[0001] The present invention relates to the field of human body posture estimation in the field of deep learning technology, and in particular to a human body posture estimation device based on global pressure and a method thereof. Background Art

[0002] Human pose estimation technology is a method that converts human motion into quantifiable position and angle parameters. With the rapid advancement of deep learning technology, this technology has shown great potential in accurately generating human motion, especially in the field of virtual characters and humanoid robots, where its application value and research prospects have attracted much attention. At present, human pose estimation methods based on monocular images have become mainstream. This method uses a monocular human image as input and uses deep learning technology to accurately fit the parameters of three-dimensional joints or human skin models, thereby achieving a detailed representation of human motion.

[0003] However, the existing human posture estimation methods based on monocular images have high requirements on light: they cannot be shot under dark or weak light conditions; they are highly invasive to the privacy of the subject: they are difficult to accept in places with high privacy protection requirements such as hospitals and homes; they have high requirements on the subject's movements: there are large errors when the subject is blocked by objects or body parts; they only focus on the subject but not the environment in which the subject exists: unreasonable interactions with the ground often occur, which will limit the application of human posture estimation methods based on monocular images in scenes with uncertain lighting conditions, high privacy requirements, frequent visual occlusion or self-occlusion, and attention to the interaction between people and the ground. At the same time, these image-based motion estimation methods often do not take into account physical constraints or physical characteristics, resulting in abnormal phenomena that violate physical laws when integrated into real environments with physical engines, such as character models floating in the air, or unnatural excessive forward or backward leaning and other posture problems.

[0004] The current methods of using pressure to estimate human posture often only focus on the posture of the human body in a lying scene, or only focus on the pressure exerted by the feet on the ground, and cannot make a unified estimate from the whole body contacting the ground to the action of only the feet contacting the ground. Therefore, how to provide a full body human posture estimation method that does not rely on a monocular camera, satisfies the real physical laws, and has no restrictions on the type of human body movements is an urgent problem to be solved. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a human body posture estimation device and method based on global pressure.

[0006] The technical solution adopted by the present invention is:

[0007] A human posture estimation device based on global pressure, the device comprises: a spatial feature encoder, used to extract human global pressure spatial features from a global pressure frame sequence of human motion; a long-term and short-term temporal attention module, used to extract human global pressure temporal features from the human global pressure spatial features and fuse them with the human global pressure spatial features to obtain pressure spatiotemporal features; and a motion regressor, used to perform nonlinear regression calculation on the pressure spatiotemporal features to obtain posture and displacement parameters of a human parameterized model.

[0008] The present invention also provides a method for using the above-mentioned human body posture estimation device based on global pressure, the method comprising the following steps:

[0009] Step S1, constructing a network model of a human posture estimation device and training the network model;

[0010] Step S2, collecting a global pressure frame sequence of human body movements; inputting the global pressure frame sequence into a spatial feature encoder to obtain a global pressure spatial feature F of the human body s ;

[0011] Step S3: transform the global pressure spatial feature F s Input the long-term and short-term time series self-attention module to obtain the pressure spatiotemporal feature F st ;

[0012] Step S4: The pressure spatiotemporal feature F st Input the action regressor to obtain the final human posture and displacement parameters.

[0013] The beneficial effects of the present invention are as follows: first, attention is paid to the problem of incorrect estimation in vision-based human posture estimation methods in scenarios with uncertain lighting conditions, high privacy requirements, and frequent visual occlusion or self-occlusion. By replacing visual information with pressure information that is independent of lighting conditions, has no privacy infringement, and is not affected by visual occlusion, non-invasive expansion of human posture estimation in special scenarios is achieved; secondly, by extracting high-dimensional features of pressure information, in-depth extraction and utilization of human posture and real physical information are achieved, so that physical constraints and physical characteristics are introduced into the human posture estimation method; finally, the present invention breaks through the action type limitation based on pressure human posture estimation, and realizes human posture estimation of all types of actions from full body contact with the ground to non-restricted parts of the body contact with the ground, broadening the application prospects and space of the estimation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a structural diagram of the device of the present invention;

[0015] Figure 2 It is a structural diagram of a feature encoder in an embodiment of the present invention;

[0016] Figure 3 1 is a comparison diagram of the human body posture and displacement estimation effects in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] Embodiments of the present invention are described in detail below with reference to the accompanying drawings, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0018] like Figure 1 As shown, this embodiment provides a human posture estimation device based on global pressure, including: a spatial feature encoder, a long-term and short-term temporal attention module and an action regressor. The spatial feature encoder is used to extract the spatial features of the global pressure information of the human body from the global pressure frame sequence of the human body motion. The long-term and short-term temporal attention module is used to extract the human body global pressure temporal features from the continuous human body global pressure spatial features and fuse them with the human body global pressure spatial features to obtain the pressure spatiotemporal features. The action regressor is used to perform nonlinear regression calculation on the human body pressure spatiotemporal features to obtain the posture and displacement parameters of the human body parametric model, and finally drive the human body parametric model.

[0019] The specific implementation device includes a memory and one or more processors, wherein the memory stores codes and executable files of a spatial feature encoder, a long-term and short-term temporal attention module, and a motion regressor. When the processor executes the executable files of the visual spatial feature encoder, the long-term and short-term temporal attention module, and the motion regressor, it is used to implement a human body posture estimation method based on global pressure of the present invention.

[0020] Among them, the spatial feature encoder includes a spatial feature extraction network, the spatial feature extraction network is as follows Figure 2 As shown, it includes an initial convolution layer, a maximum pooling layer, a four-stage convolution layer, an average pooling layer and a fully connected layer, wherein each of the four-stage convolution layers consists of two residual blocks, and each residual block consists of two convolution layers.

[0021] The long-term and short-term temporal attention module includes a long-term and short-term temporal attention network, which includes a gated loop structure and a self-attention module. The gated loop structure is composed of two gating mechanisms, an update gate and a reset gate. The self-attention module includes a fully connected layer, a self-attention mechanism, a normalized exponential function, a random inactivation layer and a layer normalization function.

[0022] The action regressor includes an action regression network, which is composed of a fully connected layer and a random dropout layer. The fused features are subjected to nonlinear regression to output human posture parameters and displacement parameters.

[0023] This embodiment also provides a method for estimating human posture based on global pressure, comprising the following steps:

[0024] Step S1, constructing a network model of the above-mentioned human posture estimation device and training the network model;

[0025] Step S2, collecting the global pressure frame sequence of human body movements; inputting the global pressure frame sequence into the spatial feature encoder to obtain the global pressure spatial feature F of the human body s The global pressure frame sequence includes the pressure generated by the various parts of the human body that can contact the ground, such as the downward pressure exerted on the ground by the feet, knees, hips, torso, elbows, hands, head, etc. during the action, and also includes the relative position relationship between the parts of the human body that contact the ground.

[0026] The spatial features specifically include the pressure value generated by the human body on each pressure measurement unit of the pressure pad, the two-dimensional spatial position of each pressure measurement unit relative to the pressure pad coordinate system, and the distance, angle, area and other spatial information between each pressure measurement unit. For example: when the human body performs a plank, the wrist, elbow and foot will apply pressure to the ground. The pressure pad is composed of a plurality of small pressure measurement units arranged in an array, and these small units can record the pressure values ​​applied by the wrist, elbow and foot to the ground. The distribution of pressure values ​​of multiple pressure measurement units not only reflects the pressure and contact area of ​​each part in contact with the ground, but also characterizes the distance, offset angle and other position information of the pressure area between the wrist, elbow and foot.

[0027] The spatial feature encoder is Figure 2 As shown in the figure, it includes an initial convolution layer with a convolution kernel size of 3*3 and 64 channels, a 2*2 maximum pooling layer, a four-stage convolution layer, an average pooling layer, and a fully connected layer. Each of the four-stage convolution layers consists of two residual blocks, and each residual block consists of two 3*3 convolution layers. As the number of stages increases, the number of channels of the convolution kernel increases from 64 to 128, 256, and 512, thereby achieving high-dimensional feature extraction. Since the pressure frame sequence is a single-channel image, and the pressure information accounts for a small proportion in actions such as standing and squatting, changes are made according to the Resnet18 classic feature extractor, changing the input channel to 1, reducing the initial convolution kernel size, and reducing the pooling layer size to obtain more refined pressure space features.

[0028] Step S3: transform the global pressure spatial feature F s Input the long-term and short-term temporal self-attention module. Since the single-frame pressure information represents the action with great ambiguity, the global pressure spatial feature is obtained by paying attention to the local adjacent frames and the global long-term frames in the long-term and short-term self-attention module to obtain the pressure temporal and spatial feature F that can better represent the human body action. st .

[0029] The long-term and short-term temporal self-attention modules are as follows: Figure 1 As shown, it includes a gated loop structure and a self-attention module. From step S1, the pressure spatial feature F is obtained after the spatial feature encoder. s First, the pressure features between adjacent frames are integrated through the gated loop structure to obtain the pressure space feature F s The local time characteristics are introduced and combined with the initial pressure space characteristics F s Add together to get the local temporal spatial feature F slocal =GRU(F s )+F s , where GRU(F s ) represents the pressure space characteristic F s The gated loop structure outputs the result, which implicitly represents the change in the distance and angle of the pressure measurement unit represented by the pressure feature between the nearest time frames, and learns the local temporal spatial features of pressure from the change in the distance and angle of the pressure measurement unit represented by the pressure feature; then the local temporal spatial features F slocal After the self-attention module, the global temporal information in the pressure feature frame is integrated through the global cross-weighted mechanism of the self-attention module and then combined with the initial local temporal spatial feature F slocal Add together to get the pressure spatiotemporal characteristics F st =Attention(F slocal )+F slocal , where Attention(F slocal ) represents the output result of the local temporal spatial feature after the self-attention module, which contains the change representation of the pressure measurement distance and angle between the pressure features between time frames, and learns the pressure temporal and spatial features that can better represent the important information of human posture. The residual connection will make the output features have certain original features and better represent the pressure features at the temporal and spatial levels. Pressure temporal and spatial features F st It includes the pressure that represents the interactive relationship between the human body and the contact parts of the ground at the spatial level, as well as the spatial position relationship between the parts of the human body in contact with the ground. It also includes the mutual perception between local adjacent frames and global frames with longer intervals of human body movements, as well as the physical dynamic mechanism information related to contact and ground interaction.

[0030] Step S4: The pressure spatiotemporal feature F st Put it into the SMPL (A Skinned Multi-Person Linear Model) model action regressor to get the final human posture θ and displacement parameter T.

[0031] When the parameters of the spatial feature encoder, the long-term and short-term temporal attention module, and the action regressor are trained in step S1, the training sample data is input into the network model, and the training loss function is calculated as: L = L pose +L 3d +L trans +L contact , the network model is iteratively trained, the AdamW optimizer is used to optimize the network parameters, the learning rate is set to 5e-4, the total number of iterations is 1000, and the network model parameters are determined based on the training loss value.

[0032] The human posture loss function θ is the predicted human posture parameter, is the benchmark truth value of human body posture parameters;

[0033] Human body 3D joint loss function J(θ,T) is the 3D joint coordinates of the SMPL human body 3D parametric model controlled by the human body posture parameter θ and the human body displacement parameter T. is the true value of human body posture parameter benchmark and the true value of human displacement parameters The 3D joint coordinates of the SMPL human 3D parametric model under control;

[0034] The human body displacement parameter loss function is: T is the predicted human body displacement parameter, is the reference true value of human body displacement parameters;

[0035] The human body contact loss function is: J c (θ,T) is the coordinate of the three-dimensional joint point of the SMPL human body three-dimensional parametric model in contact with the ground under the control of the human body posture parameter θ and the human body displacement parameter T. is the true value of human body posture parameter and the true value of human displacement parameters The coordinates of the three-dimensional joint points of the SMPL human body three-dimensional parametric model under control in contact with the ground.

[0036] The three-dimensional joint point J that contacts the ground c (θ,T) refers to the projection point obtained by projecting the three-dimensional joint point onto the ground The sum of the pressure values ​​within a certain neighborhood Greater than the threshold τ 1 , and the Z-axis height of the three-dimensional joint point Less than the threshold τ 2 The formula is: Typically, the neighborhood size is 25 square centimeters, τ 1is 5, τ 2 Where J(θ,T) represents the 3D joint coordinates of the SMPL human body 3D parametric model under the control of the human body posture parameter θ and the human body displacement parameter T.

[0037] Compared with the prior art, firstly, the present invention uses pressure information to determine whether a three-dimensional joint is in contact with the ground. A three-dimensional joint is determined to be in contact with the ground only if there is a pressure value on the ground projection and the height is close to the ground height. Other methods often only have height determination in the Z-axis direction, which greatly improves the accuracy of determining whether a body part is in contact with the ground, and indirectly improves the accuracy of the human posture estimation method; secondly, the whole-body contact loss function of the present invention can not only handle the situation where the feet are in contact with the ground, but also realize the contact determination of the joints of the whole body that may be in contact with the ground through feature determination in two dimensions, which expands the method's posture estimation range for a wider range of actions, and can handle human posture estimation for actions with different types of contact with the ground, such as standing, handstand, plank support, sitting down, kneeling, etc.

[0038] Example:

[0039] A pressure mat is laid in the human activity area. The size of the pressure mat is about two meters long and one and a half meters wide or more. Participants perform a series of daily actions on the pressure mat, including standing, lying flat, sitting, and plank support, etc. No part of the body is in contact with the ground outside the pressure mat. In this way, the pressure frame sequence data generated by the whole body when the human body performs various exercises can be accurately collected. The overall pressure frame data is divided into a pressure frame sequence of T = 20 frames, and the global pressure frame sequence is input into the spatial feature encoder composed of a convolutional layer, a pooling layer, and a fully connected layer. The global ground pressure spatial feature F of the human body is obtained, which focuses on the parts and positions of the pressure generated by the human body and the ground, as well as the relative spatial position relationship between the whole body and the parts where the pressure is generated on the ground. s The pressure pad of this embodiment is composed of 120 rows and 160 columns, totaling 19,200 pressure measurement units, which are closely arranged in a matrix form. When a person moves on the pressure pad, pressure is applied to some of the pressure measurement units, causing some of the pressure measurement units to generate pressure values. These pressure measurement units with pressure values ​​have two-dimensional spatial positions relative to the pressure pad coordinate system due to the two-dimensional matrix arrangement structure. After passing through the spatial feature encoder, spatial information such as the relative distance, relative angle, and component area between the pressure measurement units with pressure can be obtained.

[0040] The pressure frame sequence first passes through a two-dimensional convolution layer with an input channel of 1 and an output channel of 64, a convolution kernel size of 3*3, and a step size of 2, and then passes through a two-dimensional maximum pooling layer with a kernel size of 2*2 and a step size of 1, and then passes through a four-stage convolution layer, each layer consists of two residual blocks, and each residual block consists of two 3*3 convolution layers. As the number of stages increases, the number of channels of the convolution kernel will increase from 64 to 128, 256, and 512, realizing high-dimensional feature extraction. Finally, the average pooling layer and the fully connected layer with an input channel of 512 and an output channel of 1024 are used to increase the dimension, thereby increasing the high-dimensional information extraction of the global pressure spatial characteristics of the human body.

[0041] The pressure spatial feature F that passes through the spatial feature encoder s The gated recurrent structure is put into the pressure features between adjacent frames to integrate. The gated recurrent structure consists of two layers of bidirectional gated recurrent units, with an input dimension of 1024 and a hidden layer dimension of 1024. s After introducing the gated loop structure, the pressure feature GRU1 (F s ), and the initial pressure space characteristic F s Perform residual connection to obtain the combined features of local time series features and original pressure space features Among them, GRU(F s ) represents the pressure space characteristic F s The output result of the gated loop structure. In order to fully integrate the local temporal features, the local temporal spatial features at this time It needs to go through the gated loop structure again and combine it with the original local temporal spatial features Perform residual connection to obtain the final local temporal spatial features in Representing pressure space characteristics The output result of the gated loop structure. Then the local temporal spatial feature F slocal The self-attention module is inserted, where the self-attention module is a multi-head self-attention module with 4 heads and an embedding dimension of 1024. Through the global cross-weighted mechanism of the self-attention module, the global temporal information in the pressure feature frame is integrated and then combined with the initial local temporal spatial feature F slocal Add together to get the pressure spatiotemporal characteristics F st =Attention(F slocal )+F slocal , where Attention(F slocal ) represents the local temporal spatial feature F slocal The output result of the self-attention module is calculated as follows: Among them, SoftMax is a normalized exponential function. is the video key vector F slocal The dimension size is usually 1024. is the local temporal spatial feature F slocal The transposed vector of .

[0042] The pressure spatiotemporal characteristics F st Put it into the SMPL model parameter regressor composed of multiple layers of fully connected layers to obtain the final human body posture parameter θ and displacement parameter T. The estimated human body posture and displacement parameters can drive the SMPL human body 3D parametric model.

[0043] In order to verify the superiority of the present invention in human posture and displacement estimation, the present invention is compared with other human posture estimation methods. By deploying these models on a unified data set, the performance differences between the present invention and other methods can be clearly revealed. Method one uses a convolutional neural network for human posture estimation; Method two is trained on more data sets based on Method one. The present invention measures the human posture and displacement estimation accuracy of each model by the average position error per joint of the whole body, the average position error per joint of the lower body, the average position error per joint of the whole body, the average position error per joint of the lower body and the global average position error per joint. The average position error per joint of the whole body, the average position error per joint of the lower body, the average position error per joint of the whole body and the average position error per joint of the lower body measure the root mean square error between the estimated human joints of the whole body and the lower body and the true value. The smaller the error of the two indicators, the smaller the difference between the estimated value and the true value, which indicates that the estimated human posture is more accurate. The global average per-joint position error measures the error between the global coordinate estimation value and the true value of the joint points of the whole body, and measures the accuracy of the global position estimation of each joint. The smaller the index, the more accurate the global position estimation of each joint point. Table 1 compares the present invention with other methods in terms of the five indicators of the average per-joint position error of the whole body, the average per-joint position error of the lower body, the average per-joint position error of the whole body, the average per-joint position error of the lower body, and the global average per-joint position error:

[0044] Table 1 Comparison of human body posture and displacement estimation indicators of existing common methods and the model of the present invention

[0045]

[0046] As can be seen from Table 1, compared with the existing human posture estimation method, the average position error of each joint of the whole body of the present invention is reduced by 42.4-208.7mm, the average position error of each joint of the lower body is reduced by 36.4-181.2mm, the average position error of each joint of the whole body is reduced by 30.6-134.9mm, and the global average position error of each joint is reduced by 15.7-295.2mm. Since the position error of the joint points of the lower body is lower than the whole body position error, this shows that pressure has a stronger ability to characterize the stability of the lower body of the human body, and has a stronger ability to estimate the posture and displacement of the body part in contact with the ground.

[0047] In order to demonstrate the human body posture and displacement estimation effect of the present invention, a visual comparison with other methods is performed on the same data, such as Figure 3 In the three stride and stand actions, the present invention outperforms the second method in both human posture and global displacement, which shows that the present invention has the ability to accurately estimate human posture and human global displacement when only sparse pressure information is used for human posture estimation.

Claims

1. A human posture estimation device based on global pressure, characterized in that: The device includes: A spatial feature encoder is used to extract the global pressure spatial features of the human body from the global pressure frame sequence of the human body motion; A long-term and short-term temporal attention module is used to extract the temporal characteristics of global human pressure from the spatial characteristics of global human pressure and fuse them with the spatial characteristics of global human pressure to obtain the temporal characteristics of pressure; The motion regressor is used to perform nonlinear regression calculation on the pressure spatiotemporal characteristics to obtain the posture and displacement parameters of the human body parameterized model.

2. The human body posture estimation device based on global pressure according to claim 1, characterized in that: The global pressure frame sequence includes the pressure exerted on the ground by the body parts that are in contact with the ground, and also includes the relative positional relationship between the body parts that are in contact with the ground.

3. The human body posture estimation device based on global pressure according to claim 1, characterized in that: The global pressure spatial characteristics of the human body include the pressure values ​​generated by the human body on each pressure measurement unit of the pressure pad, the two-dimensional spatial position of each pressure measurement unit relative to the pressure pad coordinate system, and the spatial information between each pressure measurement unit.

4. The human body posture estimation device based on global pressure according to claim 1, characterized in that: The spatial feature encoder includes an initial convolution layer, a maximum pooling layer, a four-stage convolution layer, an average pooling layer and a fully connected layer, wherein each of the four-stage convolution layers is composed of two residual blocks, and each residual block is composed of two convolution layers.

5. The human body posture estimation device based on global pressure according to claim 1, characterized in that: The long- and short-term temporal attention module includes a gated loop structure and a self-attention module, wherein the gated loop structure is composed of two gating mechanisms, an update gate and a reset gate, and the self-attention module includes a fully connected layer, a self-attention mechanism, a normalized exponential function, a random inactivation layer and a layer normalization function.

6. The human body posture estimation device based on global pressure according to claim 1, characterized in that: The action regressor includes a fully connected layer and a random dropout layer. The action regressor performs nonlinear regression on the fused features to output human body posture parameters and displacement parameters.

7. A method for using the human body posture estimation device based on global pressure as claimed in claim 1, characterized in that: The method comprises the following steps: Step S1, constructing a network model of a human posture estimation device and training the network model; Step S2, collecting a global pressure frame sequence of human body movements; inputting the global pressure frame sequence into a spatial feature encoder to obtain a global pressure spatial feature F of the human body s ; Step S3: Input the global pressure spatial feature Fs of the human body into the long-term and short-term temporal self-attention module to obtain the pressure spatiotemporal feature F st ; Step S4: The pressure spatiotemporal feature F st Input the action regressor to obtain the final human posture and displacement parameters.

8. The method according to claim 7, characterized in that In step S1, the training loss function is: L = L pose +L 3d +L trans +L contact , where L pose is the human posture loss function, L 3d is the loss function of the 3D joint points of the human body, L trans is the human body displacement parameter loss function, L contact is the human body contact loss function.

9. The method according to claim 8, characterized in that The human body contact loss function is: J c (θ, T) is the coordinate of the three-dimensional joint point of the SMPL human body three-dimensional parametric model in contact with the ground under the control of the human body posture parameter θ and the human body displacement parameter T, is the true value of human body posture parameter benchmark and the true value of human displacement parameters The coordinates of the three-dimensional joint points of the SMPL human body three-dimensional parametric model under control in contact with the ground.

10. The method according to claim 9, characterized in that 3D joint point coordinates J c (θ, T) = Where J(θ, T) represents the three-dimensional joint coordinates of the SMPL human body three-dimensional parametric model under the control of the human body posture parameter θ and the human body displacement parameter T. The projection point obtained by projecting the three-dimensional joint point onto the ground The sum of the pressure values ​​within a certain neighborhood range is is the Z-axis height of the 3D joint point, τ2 and τ1 represent the thresholds.

Images