Attitude estimation method and device and electronic equipment

Abstract

The embodiment of the invention discloses a posture estimation method and device and electronic equipment. According to one specific embodiment, the method comprises the steps that an observation information sequence corresponding to a human body target part is obtained, and the human body target part comprises at least one of the head part and the hand part; determining initial human body joint point features based on the observation information sequence; and performing feature interaction based on the initial human body joint point features, and estimating human body postures by using interaction features to obtain human body posture information. According to the embodiment, the estimated human body posture is more accurate and real.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular to a posture estimation method, device, and electronic device. Background Art

[0002] Human pose estimation is an important task in computer vision and an essential step for computers to understand human movements and behaviors. In XR (Extended Reality) scenarios, human pose estimation can further serve downstream tasks, such as games, and rendering the user's human pose can enhance the user's sense of immersion. Summary of the invention

[0003] This disclosure section is provided to introduce concepts in a brief form, which will be described in detail in the detailed description section below. This disclosure section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] In a first aspect, an embodiment of the present disclosure provides a posture estimation method, which includes: obtaining an observation information sequence corresponding to a target part of a human body, wherein the target part of a human body includes at least one of the following: a head and a hand; determining initial human joint point features based on the observation information sequence; performing feature interaction based on the initial human joint point features, and using the interaction features to estimate a human body posture to obtain human body posture information.

[0005] In a second aspect, an embodiment of the present disclosure provides a posture estimation device, which includes: an acquisition unit, used to acquire an observation information sequence corresponding to a target part of a human body, wherein the target part of a human body includes at least one of the following: a head and a hand; a determination unit, used to determine initial human joint point features based on the observation information sequence; and an interaction unit, used to perform feature interaction based on the initial human joint point features, and use the interaction features to estimate a human body posture to obtain human body posture information.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the posture estimation method as described in the first aspect.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the posture estimation method as described in the first aspect.

[0008] The posture estimation method, device and electronic device provided by the embodiments of the present disclosure obtain the observation information sequence corresponding to the head and / or hand; then, based on the observation information sequence, determine the initial human joint point features; then, based on the initial human joint point features, perform feature interaction, use the interaction features to estimate the human posture, and obtain human posture information. This method interacts with the features of the human joint points and uses the interaction features to estimate the human posture, so that the estimated human posture is more accurate and realistic. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0010] Figure 1 is a flow chart of an embodiment of a posture estimation method according to the present disclosure;

[0011] Figure 2 is a flow chart of another embodiment of a posture estimation method according to the present disclosure;

[0012] Figure 3 is a flow chart of another embodiment of a posture estimation method according to the present disclosure;

[0013] Figure 4 is a flow chart of another embodiment of a posture estimation method according to the present disclosure;

[0014] Figure 5 is a schematic diagram of an application scenario of the posture estimation method according to the present disclosure;

[0015] Figure 6 is a structural schematic diagram of an embodiment of a posture estimation device according to the present disclosure;

[0016] Figure 7 is an exemplary system architecture diagram in which various embodiments of the present disclosure may be applied;

[0017] Figure 8 It is a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0020] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0021] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0022] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0023] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0024] Please refer to Figure 1 , shows a process 100 of an embodiment of a posture estimation method according to the present disclosure. The posture estimation method comprises the following steps:

[0025] Step 101, obtaining an observation information sequence corresponding to a target part of a human body.

[0026] In this embodiment, the execution subject of the posture estimation method can obtain the observation information sequence corresponding to the target part of the human body. The observation information in the above observation information sequence is usually sorted in the order of the collection time point of the observation information. The above human body target part can include at least one of the following: head and hand.

[0027] Here, the above-mentioned execution subject can obtain a sequence of 6DoF (six degrees of freedom) observation information of the human hand joints and head joints. Specifically, the head display of the XR device can be used to collect the observation information of the head joints, and the handle of the XR device can be used to collect the observation information of the hand joints. 6DoF includes the three-dimensional coordinates and three rotation angles of the joints in the world coordinate system. These three rotation angles usually include roll angle, pitch angle and yaw angle. 6DoF XR devices can simulate almost all head and hand dynamics.

[0028] The above-mentioned execution entity can splice the three-dimensional coordinates and rotation angles of the joint points observed each time into a one-dimensional vector as an observation information, and add it to the observation information sequence.

[0029] Step 102, determining initial human body joint features based on the observation information sequence.

[0030] In this embodiment, the execution subject can determine the initial human joint point features based on the observation information sequence obtained in step 101. Here, the human joint points can be pre-set joint points, for example, a preset number (for example, 22) of joint points distributed in important parts of the human body (for example, head, shoulders, arms, hands, pelvis, legs, feet, etc.).

[0031] Here, the above-mentioned execution entity may store a correspondence table of the correspondence between the observation information sequence and the human joint point features, and the above-mentioned execution entity may search the human joint point features corresponding to the observation information sequence obtained in step 101 from the above-mentioned correspondence table as the initial human joint point features.

[0032] Step 103, performing feature interaction based on the initial human joint point features, and using the interaction features to estimate the human body posture to obtain human body posture information.

[0033] In this embodiment, the execution subject can perform feature interaction based on the initial human joint point features, and use the interaction features to estimate the human body posture to obtain human body posture information. Here, the interaction of the joint point features can also be understood as determining the correlation between the joint points. The correlation between the human body joint points is determined by the initial human body joint point features. If the correlation between the joint points is small, the current position of the joint point can be corrected to improve the correlation between the joint points to estimate the human body posture more accurately.

[0034] The method provided by the above embodiment of the present disclosure obtains the observation information sequence corresponding to the head and / or the hand; then, based on the above observation information sequence, the initial human joint point features are determined; then, based on the above initial human joint point features, feature interaction is performed, and the human body posture is estimated using the interaction features to obtain human body posture information. This method interacts with the features of the human body joint points and estimates the human body posture using the interaction features, thereby making the estimated human body posture more accurate and realistic.

[0035] In some optional implementations, the above observation information may include the movement speed and angular velocity of the joints. After collecting a sequence of three-dimensional coordinates and rotation angles of the human hand and / or head joints, the movement speed of the joints during the multiple collection processes can be determined using the three-dimensional coordinates collected multiple times, and the acceleration of the joints can be determined using the movement speed. After that, the three-dimensional coordinates, rotation angle, movement speed and acceleration of the joints observed each time can be spliced ​​into a one-dimensional vector as an observation information and added to the observation information sequence.

[0036] Please refer to Figure 2 , which shows a process 200 of another embodiment of a posture estimation method. The process 200 of the posture estimation method includes the following steps:

[0037] Step 201, obtaining an observation information sequence corresponding to a target part of a human body.

[0038] Step 202, determining initial human body joint features based on the observation information sequence.

[0039] In this embodiment, steps 201 - 202 may be performed in a manner similar to steps 101 - 102 , and will not be described in detail herein.

[0040] Step 203, based on the initial human joint point features, feature interaction is performed in the spatial dimension and / or the temporal dimension, and the human body posture is estimated using the interaction features to obtain human body posture information.

[0041] In this embodiment, the executor of the posture estimation method can perform feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features determined in step 202, and use the interaction features to estimate the human body posture to obtain human body posture information.

[0042] As an example, the execution subject can input the initial human joint features into a Transformer-based network structure to obtain interactive features in the spatial dimension and / or in the temporal dimension. Transformer is a neural network model structure that fully utilizes the attention mechanism for calculation, and has the characteristics of high parallelism, large receptive field, and the ability to process sequence data. The Transformer-based network structure can be used to perform feature interaction on the input initial human joint features.

[0043] from Figure 2 It can be seen that Figure 1 Compared with the corresponding embodiment, the process 200 of the posture estimation method in this embodiment embodies the steps of interacting the features of the joint points in the spatial dimension and / or in the temporal dimension. Therefore, the solution described in this embodiment can improve the accuracy of the interactive features.

[0044] In some optional implementations, the above-mentioned execution entity can perform feature interaction in the spatial dimension and / or in the temporal dimension based on the above-mentioned initial human joint point features in the following manner: for each collection point in the collection time period, the above-mentioned execution entity can determine the attention score between the initial human joint point features corresponding to the collection point and the target input features, and can use the above-mentioned attention score to interact with the initial human joint point features corresponding to the collection point and the above-mentioned target input features to obtain interaction features in the spatial dimension.

[0045] Here, the target input feature may be obtained by mapping the high-dimensional input feature to the same dimension as the initial whole-body joint feature information corresponding to the acquisition point. The high-dimensional input feature may be obtained by inputting the observation information sequence into the joint prediction sub-model, for example, by inputting the observation information sequence into the linear layer of the joint prediction sub-model. The acquisition time period may be the time period for acquiring the observation information sequence.

[0046] In this way, the correlation of the joint points in the spatial dimension can be captured, and the position of each joint point can be corrected through the correlation, making the posture estimation more accurate.

[0047] In some optional implementations, the execution subject may perform feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features in the following manner: for each of the human joint points, the execution subject may determine the attention score between multiple initial joint point features of the joint point within the acquisition time period, and then, the attention score may be used to interact with multiple initial joint point features corresponding to the joint point to obtain interaction features in the temporal dimension. The acquisition time period may be the time period for acquiring the observation information sequence.

[0048] In this way, the motion pattern of each joint point in the time dimension can be captured, thereby ensuring the rationality of the joint point during the motion process and further improving the accuracy of posture estimation.

[0049] In some optional implementations, the execution subject may perform feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint features in the following manner: the execution subject may input the initial human joint features and the target input features into a pre-trained feature interaction network to obtain interaction features. The feature interaction network may be used to characterize the correspondence between the joint features and the input features and the interaction features. The interaction features may include interaction features of human joints in the spatial dimension and interaction features of human joints in the temporal dimension, and the target input features may be determined based on the observation information sequence, for example, may be obtained by inputting the observation information sequence into the linear layer of a pre-trained joint prediction sub-model.

[0050] In some optional implementations, the feature interaction network may include at least two first encoding layers and at least two second encoding layers, the first encoding layer may be used to perform feature interaction in the spatial dimension, for example, it may be a Transformer-Encoder (encoding) layer of the spatial dimension, and the second encoding layer may be used to perform feature interaction in the time dimension, for example, it may be a Transformer-Encoder layer of the time dimension.

[0051] The execution subject may input the initial human joint point features and the target input features into a pre-trained feature interaction network to obtain the interaction features in the following manner: the execution subject may input the initial human joint point features and the target input features into the first coding layer and the second coding layer arranged alternately to obtain the interaction features. As an example, if the feature interaction network includes three first coding layers and three second coding layers, the input initial human joint point features and the target input features may pass through the first first coding layer, the first second coding layer, the second first coding layer, the second second coding layer, the third first coding layer and the third second coding layer.

[0052] This alternating network structure can iteratively correct the positional relationship in the spatial dimension and the motion relationship of the joints in the temporal dimension, thereby achieving better results.

[0053] It should be noted that the above-mentioned feature interaction network may also include a position encoding layer. In the spatial dimension, a learnable parameter with a dimension of 2×22×512 (where 2 represents the two dimensions of rotation angle and coordinate, and 22 represents the number of joint points) may be added to the initial human joint point features collected each time to represent the relative positions between all related nodes. In the temporal dimension, a learnable parameter with a dimension of t×512 (where t represents the number of collection points) may be added to the features of each joint point to represent the relative position of each joint point in the time series.

[0054] Further references Figure 3 , which shows a process 300 of another embodiment of a posture estimation method. The process 300 of the posture estimation method includes the following steps:

[0055] Step 301, obtaining an observation information sequence corresponding to a target part of a human body.

[0056] In this embodiment, step 301 may be performed in a manner similar to step 101 , and will not be described in detail herein.

[0057] Step 302, determining initial human body posture information based on the observation information sequence.

[0058] In this embodiment, the execution body of the posture estimation method can determine the initial human posture information based on the above observation information sequence. The above execution body can input the above observation information sequence into the first linear layer and the posture regression layer to obtain the initial human posture information.

[0059] Here, the first linear layer can be a fully connected layer, which is usually used to map the input observation information sequence into high-dimensional (e.g., 1024-dimensional) input features. The posture regression layer can be composed of a multilayer perceptron (MLP), which is usually used to map the input features output by the first linear layer into initial human posture information.

[0060] In this embodiment, the multilayer perceptron is a feedforward artificial neural network model that maps multiple input data sets to a single output data set. The multilayer perceptron here can be composed of two fully connected layers with a hidden layer dimension of 1024 and a layer of ReLU activation function layer. The output of the multilayer perceptron can be a rotation representation of 22 joint points with a dimension of 6, and the total output dimension is 6×22=132.

[0061] Step 303, correcting the initial human body posture information, and determining the initial human body joint point features based on the corrected human body posture information.

[0062] In this embodiment, the execution subject may correct the initial human body posture information, and may determine the initial human body joint point features based on the corrected human body posture information.

[0063] Here, the execution subject may determine whether the human body posture represented by the initial human body posture information is reasonable. If not, the initial human body posture information needs to be corrected to make the human body posture represented by it reasonable.

[0064] Afterwards, the corrected human posture information can be input into the second linear layer to obtain the initial human joint features. The second linear layer can be two fully connected layers, which are used to map the rotation angle and coordinate position in the corrected initial human posture information into high-dimensional (e.g., 512-dimensional) initial human joint features.

[0065] Step 304, performing feature interaction based on the initial human joint point features, and using the interaction features to estimate the human body posture to obtain human body posture information.

[0066] In this embodiment, step 304 may be performed in a manner similar to step 103 , which will not be described in detail herein.

[0067] from Figure 3 It can be seen that Figure 1 Compared with the corresponding embodiment, the process 300 of the posture estimation method in this embodiment embodies the steps of correcting the initial human posture information and determining the initial human joint point features based on the corrected human posture information. Therefore, the scheme described in this embodiment can make the initial human joint point features more accurate and reasonable, and make the human posture estimation more accurate and true.

[0068] In some optional implementations, the above-mentioned initial human body posture information may include the relative rotation angles of the human body joints under the human body parameterized mesh model and / or the joint point coordinates of the human body joints under the human body parameterized mesh model, and the above-mentioned observation information sequence may include the rotation angles and / or joint point coordinates of the joints on the target part of the human body.

[0069] The execution subject may modify the initial human body posture information in at least one of the following ways:

[0070] If the initial human posture information includes the relative rotation angle of the human joints under the human body parametric mesh model and the observation information sequence includes the rotation angle of the joints on the target part of the human body, the execution subject can replace the rotation angle of the joints on the target part of the human body in the initial human posture information with the rotation angle of the joints on the target part of the human body in the observation information sequence. It should be noted that since the rotation angle of the joints on the target part of the human body in the observation information sequence is a global rotation angle (i.e., an absolute rotation angle), it is necessary to first convert the relative rotation angle of the human joints under the human body parametric mesh model into a global rotation angle.

[0071] If the above initial human posture information includes the joint point coordinates of the human joint points under the human body parametric mesh model and the above observation information sequence includes the joint point coordinates of the joint points on the human body target part, then the above execution subject can replace the joint point coordinates of the joint points on the human body target part in the above initial human posture information with the joint point coordinates of the joint points on the human body target part in the above observation information sequence. It should be noted that, since the joint point coordinates of the joint points on the human body target part in the observation information sequence are coordinates in the world coordinate system, it is necessary to first convert the joint point coordinates of the above human joint points under the human body parametric mesh model into joint point coordinates in the world coordinate system.

[0072] In this way, the input rotation angles and joint point coordinates of the human body joints can be used to correct the rotation angles and joint point coordinates in the estimated human body posture information, making the initial human body joint point features more accurate.

[0073] Further references Figure 4 , which shows a process 400 of another embodiment of a posture estimation method. The process 400 of the posture estimation method includes the following steps:

[0074] Step 401, obtaining an observation information sequence corresponding to a target part of a human body.

[0075] In this embodiment, step 401 may be performed in a manner similar to step 101 , and will not be described in detail herein.

[0076] Step 402, inputting the observation information sequence into the pre-trained joint prediction sub-model to obtain initial human joint features.

[0077] In this embodiment, the execution subject of the posture estimation method can input the above observation information sequence into a pre-trained joint point prediction sub-model to obtain initial human joint point features. The above joint point prediction sub-model can be used to characterize the correspondence between the observation information sequence and the initial human joint point features.

[0078] Here, the joint prediction submodel may include a first linear layer, a posture regression layer, and a second linear layer. Specifically, the first linear layer may be a fully connected layer, which is usually used to map the input observation information sequence into high-dimensional (e.g., 1024-dimensional) input features. The posture regression layer may be composed of a multi-layer perceptron, which is usually used to map the input features output by the first linear layer into initial human body posture information. The second linear layer may be two fully connected layers, which are respectively used to map the rotation angle and coordinate position in the initial human body posture information into high-dimensional (e.g., 512-dimensional) initial human body joint features.

[0079] In this embodiment, the multilayer perceptron is a feedforward artificial neural network model that maps multiple input data sets to a single output data set. The multilayer perceptron here can be composed of two fully connected layers with a hidden layer dimension of 1024 and a layer of ReLU activation function layer. The output of the multilayer perceptron can be a rotation representation of 22 joint points with a dimension of 6, and the total output dimension is 6×22=132.

[0080] Here, the initial human posture information may include relative rotation angle parameters (often with dimensions of 6×22) and joint point coordinates (often with dimensions of 3×22) under the human body parameterized network model (Skinned Multi-Person Linear Model, SMPL). The relative rotation angle parameters may refer to rotation angle parameters of a joint point relative to other adjacent joint points. For example, the relative rotation angle of the ankle joint point may be the rotation angle relative to the knee joint point under the human body parameterized network model.

[0081] The joint point coordinates under the human body parametric network model can be obtained by decoding the relative rotation angle parameters under the human body parametric network model. The human body parametric network model is a parametric network model designed for the human body, including parameters for controlling shape and parameters for controlling posture.

[0082] Step 403: input the initial human joint point features into the pre-trained posture estimation sub-model to obtain human posture information.

[0083] In this embodiment, the execution subject may input the initial human joint point features obtained in step 402 into a pre-trained posture estimation sub-model to obtain human posture information. The posture estimation sub-model may be used to characterize the correspondence between human joint point features and human posture information. The posture estimation sub-model generally functions to capture the correlation between human joint points in spatial dimension and / or time dimension from the input initial human joint point features, thereby regressing an accurate and true human posture.

[0084] Here, the above-mentioned human body posture information may include relative rotation angle parameters of human body joints under the human body parameterized network model, may also include joint point coordinates of human body joints under the human body parameterized network model, and may also include joint point coordinates of human body joints in the world coordinate system. Among them, the joint point coordinates of the above-mentioned human body joints under the human body parameterized network model may be obtained by decoding the relative rotation angle parameters of human body joints under the human body parameterized network model, and the joint point coordinates of the above-mentioned human body joints in the world coordinate system may be obtained by aligning the joint point coordinates of the human body joints under the human body parameterized network model with the input head joint point coordinates and / or hand joint point coordinates.

[0085] Here, the above-mentioned posture estimation sub-model may include a Transformer-based network structure and a full-body posture regression layer.

[0086] Transformer is a neural network model structure that fully utilizes the attention mechanism for calculation. It has the characteristics of high parallelism, large receptive field, and the ability to process sequence data. The above Transformer-based network structure can be used to perform feature interaction on the input initial human joint point features.

[0087] The above-mentioned whole-body posture regression layer can be composed of a multilayer perceptron (MLP), which takes the features captured by the spatiotemporal relationship output by the Transformer-based network structure as input, and outputs the relative rotation angle parameters (the dimension is usually 6×22) corresponding to each acquisition point under the human body parameterized network model. The multilayer perceptron in the above-mentioned whole-body posture regression layer can be composed of two fully connected layers with a hidden layer dimension of 1024, a ReLU activation function layer, and a Group Normalization layer. Group normalization is a method that normalizes the input of the layer by re-centering and rescaling, thereby making the training of artificial neural networks faster and more stable.

[0088] Step 404, based on the human body posture information and the label posture information, a loss value is determined using a preset loss function.

[0089] In this embodiment, the execution subject may determine the loss value using a preset loss function (e.g., L1 norm) based on the human body posture information and the label posture information determined in step 403. The execution subject may compare the human body posture information with the label posture information and determine the difference as the loss value.

[0090] Step 405 , using the loss value, adjust the model parameters of the joint point prediction sub-model and the model parameters of the posture estimation sub-model to obtain the adjusted joint point prediction sub-model and posture estimation sub-model.

[0091] In this embodiment, the above-mentioned execution entity can use the above-mentioned loss value to adjust the model parameters of the above-mentioned joint point prediction sub-model and the model parameters of the above-mentioned posture estimation sub-model to obtain the adjusted joint point prediction sub-model and posture estimation sub-model.

[0092] Here, the training can be ended when the preset training end condition is met. For example, the preset training end condition here may include but is not limited to at least one of the following: the training time exceeds the preset duration; the number of training times exceeds the preset number of times; the calculated difference is less than the preset difference threshold.

[0093] Here, various implementation methods can be used to adjust the model parameters of the joint prediction sub-model and the model parameters of the posture estimation sub-model based on the loss value. For example, the BP (Back Propagation) algorithm or the SGD (Stochastic Gradient Descent) algorithm can be used to adjust the model parameters of the joint prediction sub-model and the model parameters of the posture estimation sub-model. It should be noted that any algorithm for adjusting model parameters can be used to adjust the model parameters, and there is no limitation here.

[0094] from Figure 4 It can be seen that Figure 1 Compared with the corresponding embodiment, the process 400 of the posture estimation method in this embodiment embodies the steps of determining the loss value using the human posture information and the label posture information when determining the human posture information through the joint point prediction submodel and the posture estimation submodel, and adjusting the model parameters of the joint point prediction submodel and the posture estimation submodel using the loss value to obtain the adjusted joint point prediction submodel and the posture estimation submodel. Therefore, the scheme described in this embodiment can adjust the model parameters of the joint point prediction submodel and the posture estimation submodel, so that the joint point prediction submodel and the posture estimation submodel can perform human posture estimation more accurately.

[0095] In some optional implementations, the above-mentioned human body posture information may include the relative rotation angle of the human body joints under the human body parametric mesh model, and the above-mentioned label posture information may include the label relative rotation angle, which generally refers to the label relative rotation angle of the human body joints under the human body parametric mesh model.

[0096] The execution subject can determine the loss value based on the human body posture information and the label posture information by using a preset loss function in the following manner: the execution subject can determine the difference between the relative rotation angle of the human body joint point under the human body parameterized mesh model and the relative rotation angle of the label as the loss value. This supervision method can be called SMPL rotation angle parameter supervision. In this way, the output result of the posture estimation sub-model can be made as close as possible to the label value, thereby improving the accuracy of the output result of the posture estimation sub-model.

[0097] In some optional implementations, the above-mentioned human body posture information may include the joint point coordinates of the human body joints under the human body parametric mesh model, and the above-mentioned label posture information may include the label joint point coordinates, which generally refer to the label joint point coordinates of the human body joints under the human body parametric mesh model.

[0098] The execution subject can determine the loss value based on the human body posture information and the label posture information by using a preset loss function in the following manner: the execution subject can determine the difference between the joint point coordinates of the human body joint points under the human body parameterized mesh model and the label joint point coordinates as the loss value. This supervision method can be called joint point position supervision in the SMPL coordinate system. Since the joint point coordinates of the human body joint points under the human body parameterized mesh model are obtained by the relative rotation angle parameters of the human body joint points under the human body parameterized mesh model, the accuracy of the output result of the posture estimation sub-model can be further improved in this way.

[0099] In some optional implementations, the above-mentioned human body posture information may include the joint point coordinates of the hand joints in the world coordinate system, and the above-mentioned label posture information may include the label human hand joint point coordinates, and the label human hand joint point coordinates generally refer to the label joint point coordinates of the hand joints in the world coordinate system.

[0100] The above-mentioned execution subject can determine the loss value based on the above-mentioned human posture information and label posture information by using a preset loss function in the following manner: the above-mentioned execution subject can determine the difference between the joint point coordinates of the above-mentioned hand joint points in the world coordinate system and the joint point coordinates of the above-mentioned label human hand as the loss value. This supervision method can be called hand alignment supervision. Since the joint point coordinates of the hand joint points in the world coordinate system are obtained from the joint point coordinates of the human body joint points in the human body parameterized mesh model, and the joint point coordinates of the human body joint points in the human body parameterized mesh model are obtained from the relative rotation angle parameters of the human body joint points in the human body parameterized mesh model, the accuracy of the output result of the posture estimation sub-model can be further improved in this way.

[0101] In some optional implementations, the human body posture information may include the movement speed of the human body joints within a preset time length, and the tag posture information may include the tag movement speed. As an example, the preset time length may also be understood as a preset collection period, and the preset collection period may be a collection period between N adjacent collection points, for example, the preset collection period may be a collection period between two adjacent collection points, or a collection period between three adjacent collection points, or a collection period between five adjacent collection points.

[0102] As an example, if the preset collection period is the collection period between two adjacent collection points, the corresponding label posture information includes the label movement speed of the human joints in the collection period between two adjacent collection points; if the preset collection period is the collection period between three adjacent collection points, the corresponding label posture information includes the label movement speed of the human joints in the collection period between three adjacent collection points; if the preset collection period is the collection period between five adjacent collection points, the corresponding label posture information includes the label movement speed of the human joints in the collection period between five adjacent collection points.

[0103] The above-mentioned execution subject can determine the loss value based on the above-mentioned human posture information and label posture information by using a preset loss function in the following manner: the difference between the movement speed of the above-mentioned human joint point within a preset time length and the above-mentioned label movement speed is determined as the loss value. Here, the above-mentioned execution subject can determine the difference between the movement speed of the above-mentioned human joint point in the collection period between two adjacent collection points and the corresponding label movement speed as the loss value; the difference between the movement speed of the above-mentioned human joint point in the collection period between three adjacent collection points and the corresponding label movement speed can also be determined as the loss value; the difference between the movement speed of the above-mentioned human joint point in the collection period between five adjacent collection points and the corresponding label movement speed can also be determined as the loss value. This supervision method can be called motion speed supervision. Through this supervision method, the determined human posture in motion can be made as smooth and realistic as possible.

[0104] In some optional implementations, the human body joints may include foot joints. The execution entity may determine the difference between the movement speed of the human body joints within a preset time period and the tag movement speed as a loss value in the following manner: if the foot is placed on the ground within the preset time period, the execution entity may determine the difference between the movement speed of the foot joints within the preset time period and the target speed as a loss value. The target speed is usually 0. This supervision method may be called foot contact supervision. Through this supervision method, the feet that have not left the ground can be made to have as little movement tendency as possible.

[0105] In some optional implementations, the execution subject can determine the loss value based on the human body posture information and the label posture information by using a preset loss function in the following manner: the execution subject can use the human body posture information to determine whether there are any joints in the human body that are lower than the ground; if so, the difference between the height of the lowest point in the human body joints that is lower than the ground and the height of the ground can be determined as the loss value. This supervision method can be called through-the-ground supervision. Through this supervision method, the human body posture can be made to conform to physical constraints as much as possible, and no joint of the body will be lower than the ground.

[0106] In some optional implementations, the execution subject may determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: if the human body's feet are placed on the ground, the execution subject may determine the difference between the height of the lowest point above the ground in the human body joints and the height of the ground as the loss value. This supervision method may be called foot height supervision. This supervision method allows the human body posture to conform to physical constraints as much as possible, so that the grounded feet will not float in the air.

[0107] Continue to see Figure 5 , Figure 5 FIG. 1 is a schematic diagram of an application scenario of the posture estimation method according to this embodiment. Figure 5 In the application scenario, the input time-series human hand and head observation information will pass through the initial whole-body joint-level feature construction module, the joint-correlation modeling module and the model supervision module.

[0108] In the initial whole-body joint point level feature construction module, each collected human hand and head observation information will pass through the first linear layer to obtain high-dimensional input features, and then the high-dimensional input features will be input into the whole-body posture regression module to obtain the initial whole-body human posture. The initial whole-body human posture is corrected by comparing the initial whole-body human posture with the human hand and head observation information. After that, the corrected initial whole-body human posture is input into the second linear layer to obtain the high-dimensional initial whole-body joint point level features.

[0109] In the joint correlation modeling module, the initial whole-body joint-level features are input into the converter network of space and time dimensions. The converter network of space and time dimensions includes N groups of sub-networks consisting of space converter modules and time converter modules. The space converter module is used to interact with features in the spatial dimension. The time converter module is used to interact with features in the time dimension. The interactive features after capturing the spatiotemporal relationship are input into the whole-body posture regression module to obtain the relative rotation angle parameters under SMPL.

[0110] In the model supervision module, a variety of supervision methods can be used to supervise the output results of the initial whole-body joint-level feature construction module and the joint correlation modeling module, such as SMPL rotation angle parameter supervision method, SMPL coordinate system whole-body joint coordinate supervision method, hand alignment supervision method, motion speed supervision method and physical constraint supervision method.

[0111] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a posture estimation device, which is similar to Figure 1 Corresponding to the method embodiment shown, the apparatus can be specifically applied to various devices.

[0112] like Figure 6 As shown, the posture estimation device 600 of this embodiment includes: an acquisition unit 601, a determination unit 602 and an interaction unit 603. The acquisition unit 601 is used to acquire an observation information sequence corresponding to a target part of a human body, wherein the target part of a human body includes at least one of the following: a head and a hand; the determination unit 602 is used to determine the initial human joint point features based on the observation information sequence; the interaction unit 603 is used to perform feature interaction based on the initial human joint point features, and use the interaction features to estimate the human body posture and obtain the human body posture information.

[0113] In this embodiment, the specific processing of the acquisition unit 601, the determination unit 602 and the interaction unit 603 of the posture estimation device 600 can refer to Figure 1 This corresponds to step 101, step 102 and step 103 in the embodiment.

[0114] In some optional implementations, the interaction unit 603 may be further configured to perform feature interaction based on the initial human joint point features in the following manner: based on the initial human joint point features, perform feature interaction in the spatial dimension and / or in the temporal dimension.

[0115] In some optional implementations, the interaction unit 603 can be further used to perform feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features in the following manner: for each collection point in the collection time period, determine the attention score between the initial human joint point features corresponding to the collection point and the target input features, and use the attention score to interact with the initial human joint point features corresponding to the collection point and the target input features to obtain interaction features in the spatial dimension, wherein the target input feature is obtained by mapping the high-dimensional input feature to the same dimension as the initial human joint point feature corresponding to the collection point, the high-dimensional input feature is determined based on the observation information sequence, and the collection time period is the time period for collecting the observation information sequence.

[0116] In some optional implementations, the interaction unit 603 can be further used to perform feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features in the following manner: for each joint point in the human body, determine the attention score between multiple initial joint point features of the joint point within a collection time period, and use the attention score to interact with multiple initial joint point features corresponding to the joint point to obtain interaction features in the time dimension, wherein the collection time period is the time period for collecting the observation information sequence.

[0117] In some optional implementations, the interaction unit 603 can be further used to perform feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features in the following manner: the initial human joint point features and the target input features are input into a pre-trained feature interaction network to obtain interaction features, wherein the interaction features include interaction features of human joint points in the spatial dimension and interaction features of human joint points in the temporal dimension, and the target input features are determined based on the observation information sequence.

[0118] In some optional implementations, the feature interaction network includes at least two first coding layers and at least two second coding layers, the first coding layer is used to perform feature interaction in the spatial dimension, and the second coding layer is used to perform feature interaction in the time dimension; and the interaction unit 603 can be further used to input the initial human joint point features and the target input features into a pre-trained feature interaction network to obtain interaction features in the following manner: input the initial human joint point features and the target input features into the alternating first coding layer and the second coding layer to obtain interaction features.

[0119] In some optional implementations, the determination unit 602 can be further used to determine the initial human body joint point features based on the observation information sequence in the following manner: determine the initial human body posture information based on the observation information sequence; correct the initial human body posture information, and determine the initial human body joint point features based on the corrected human body posture information.

[0120] In some optional implementations, the initial human body posture information includes the relative rotation angle of the human body joints under the human body parameterized mesh model and / or the joint point coordinates of the human body joints under the human body parameterized mesh model, and the observation information sequence includes the rotation angle and / or the joint point coordinates of the joint points on the target part of the human body; and the determination unit 602 can be further used to correct the initial human body posture information in at least one of the following ways: replacing the rotation angle of the joint points on the target part of the human body in the initial human body posture information with the rotation angle of the joint points on the target part of the human body in the observation information sequence; replacing the joint point coordinates of the joint points on the target part of the human body in the initial human body posture information with the joint point coordinates of the joint points on the target part of the human body in the observation information sequence.

[0121] In some optional implementations, the determination unit 602 can be further used to determine the initial human joint features based on the observation information sequence in the following manner: input the observation information sequence into a pre-trained joint prediction sub-model to obtain the initial human joint features; and the interaction unit 603 can be further used to perform feature interaction based on the initial human joint features in the following manner, and use the interaction features to estimate the human body posture to obtain human body posture information: input the initial human joint features into a pre-trained posture estimation sub-model to obtain human body posture information.

[0122] In some optional implementations, the posture estimation device 600 may further include: a loss value determination unit (not shown in the figure) and an adjustment unit (not shown in the figure). The loss value determination unit is used to determine the loss value based on the human body posture information and the label posture information using a preset loss function; the adjustment unit is used to use the loss value to adjust the model parameters of the joint point prediction sub-model and the model parameters of the posture estimation sub-model to obtain the adjusted joint point prediction sub-model and posture estimation sub-model.

[0123] In some optional implementations, the human body posture information includes the relative rotation angle of the human body joints under the human body parameterized mesh model, and the label posture information includes the label relative rotation angle; and the loss value determination unit can be further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: the difference between the relative rotation angle of the human body joints under the human body parameterized mesh model and the label relative rotation angle is determined as the loss value.

[0124] In some optional implementations, the human body posture information includes the joint point coordinates of the human body joints under the human body parameterized mesh model, and the label posture information includes the label joint point coordinates; and the loss value determination unit can be further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: the difference between the joint point coordinates of the human body joints under the human body parameterized mesh model and the label joint point coordinates is determined as the loss value.

[0125] In some optional implementations, the human body posture information includes the joint point coordinates of the hand joints in the world coordinate system, and the label posture information includes the joint point coordinates of the label human hand; and the loss value determination unit can be further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: the difference between the joint point coordinates of the hand joints in the world coordinate system and the joint point coordinates of the label human hand is determined as the loss value.

[0126] In some optional implementations, the human body posture information includes the movement speed of the human body joints within a preset time length, and the label posture information includes the label movement speed; and the loss value determination unit can be further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: the difference between the movement speed of the human body joints within a preset time length and the label movement speed is determined as the loss value.

[0127] In some optional implementations, the human body joints include foot joints; and the loss value determination unit can be further used to determine the difference between the movement speed of the human body joints within a preset time length and the tag movement speed as the loss value in the following manner: if the foot is placed on the ground within the preset time length, the difference between the movement speed of the foot joints within the preset time length and the target speed is determined as the loss value.

[0128] In some optional implementations, the loss value determination unit can be further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: using the human body posture information, determine whether there are any joints among the human body joints that are below the ground height; if so, determine the difference between the height of the lowest point among the human body joints below the ground and the height of the ground as the loss value.

[0129] In some optional implementations, the loss value determination unit can be further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: if the human body's feet are placed on the ground, the difference between the height of the lowest point above the ground among the human body joints and the height of the ground is determined as the loss value.

[0130] In some optional implementations, the observation information includes the movement speed and angular velocity of the joint point.

[0131] Figure 7 An exemplary system architecture 700 is shown to which embodiments of the pose estimation method of the present disclosure may be applied.

[0132] like Figure 7 As shown, the system architecture 700 may include terminal devices 7011, 7012, 7013, a network 702, and a server 703. The network 702 is used to provide a medium for communication links between the terminal devices 7011, 7012, 7013 and the server 703. The network 702 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0133] The user can use the terminal devices 7011, 7012, 7013 to interact with the server 703 through the network 702 to send or receive messages, etc. For example, the server 703 can receive the observation information sequence sent by the terminal devices 7011, 7012, 7013. Various communication client applications can be installed on the terminal devices 7011, 7012, 7013, such as image processing applications, image shooting applications, game applications, instant messaging software, etc.

[0134] Terminal devices 7011, 7012, and 7013 can be hardware or software. When terminal devices 7011, 7012, and 7013 are hardware, they can be various electronic devices with cameras, display screens, and support information interaction, including but not limited to VR headsets, AR glasses, smart cameras, smart phones, tablet computers, laptop portable computers, etc. When terminal devices 7011, 7012, and 7013 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0135] Terminal devices 7011, 7012, and 7013 can obtain observation information sequences corresponding to the head and / or hands; then, based on the above observation information sequences, initial human joint features can be determined; then, feature interaction can be performed based on the above initial human joint features, and the human body posture can be estimated using the interaction features to obtain human body posture information.

[0136] The server 703 may be a server that provides various services. For example, it may be a background server that processes the observation information sequence corresponding to the head and / or hand. The server 703 may first obtain the observation information sequence corresponding to the head and / or hand from the terminal devices 7011, 7012, 7013; then, it may determine the initial human joint point features based on the above observation information sequence; then, based on the above initial human joint point features, feature interaction is performed, and the human posture is estimated using the interaction features to obtain human posture information.

[0137] It should be noted that the server 703 can be hardware or software. When the server 703 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 703 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0138] It should also be noted that the posture estimation method provided in the embodiment of the present disclosure can be executed by the terminal devices 7011, 7012, and 7013, and the posture estimation device is usually set in the terminal devices 7011, 7012, and 7013; the posture estimation method provided in the embodiment of the present disclosure can also be executed by the server 703, and the posture estimation device is usually set in the server 703.

[0139] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0140] Reference below Figure 8 , which shows an electronic device (eg, Figure 7 A schematic diagram of the structure of the server in FIG. 800. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0141] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0142] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 8 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0143] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium described in the embodiment of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus, or a device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0144] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains hand images collected from at least two viewing angles; determines the initial hand 3D reconstruction results corresponding to each hand image based on a preset hand 3D reconstruction network, wherein the hand 3D reconstruction results include a hand 3D model and hand key points; fuses the initial hand 3D reconstruction results corresponding to the hand images collected from the at least two viewing angles to obtain a fused hand 3D reconstruction result.

[0145] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] According to one or more embodiments of the present disclosure, a method for three-dimensional reconstruction of a hand is provided.

[0148] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0149] According to one or more embodiments of the present disclosure, a posture estimation method is provided, which includes: obtaining an observation information sequence corresponding to a target part of a human body, wherein the target part of a human body includes at least one of the following: a head and a hand; determining initial human joint point features based on the observation information sequence; performing feature interaction based on the initial human joint point features, and estimating a human body posture using the interaction features to obtain human body posture information.

[0150] According to one or more embodiments of the present disclosure, feature interaction is performed based on initial human joint point features, including: based on the initial human joint point features, feature interaction is performed in a spatial dimension and / or in a temporal dimension.

[0151] According to one or more embodiments of the present disclosure, based on the initial human joint point features, feature interaction is performed in the spatial dimension and / or in the temporal dimension, including: for each acquisition point in the acquisition time period, determining the attention score between the initial human joint point features corresponding to the acquisition point and the target input features, using the attention score, interacting the initial human joint point features corresponding to the acquisition point with the target input features to obtain interaction features in the spatial dimension, wherein the target input feature is obtained by mapping a high-dimensional input feature to the same dimension as the initial human joint point features corresponding to the acquisition point, the high-dimensional input feature is determined based on an observation information sequence, and the acquisition time period is the time period for collecting the observation information sequence.

[0152] According to one or more embodiments of the present disclosure, based on the initial human joint features, feature interaction is performed in the spatial dimension and / or in the temporal dimension, including: for each human joint, determining the attention score between multiple initial joint features of the joint within a collection time period, and using the attention score to interact with multiple initial joint features corresponding to the joint to obtain interaction features in the temporal dimension, wherein the collection time period is the time period for collecting the observation information sequence.

[0153] According to one or more embodiments of the present disclosure, based on the initial human joint features, feature interaction is performed in the spatial dimension and / or in the temporal dimension, including: inputting the initial human joint features and the target input features into a pre-trained feature interaction network to obtain interaction features, wherein the interaction features include interaction features of the human joints in the spatial dimension and interaction features of the human joints in the temporal dimension, and the target input features are determined based on the observation information sequence.

[0154] According to one or more embodiments of the present disclosure, a feature interaction network includes at least two first coding layers and at least two second coding layers, the first coding layer is used to perform feature interaction in a spatial dimension, and the second coding layer is used to perform feature interaction in a temporal dimension; and initial human joint point features and target input features are input into a pre-trained feature interaction network to obtain interaction features, including: inputting the initial human joint point features and target input features into the alternatingly arranged first coding layer and second coding layer to obtain interaction features.

[0155] According to one or more embodiments of the present disclosure, initial human joint features are determined based on an observation information sequence, including: determining initial human body posture information based on the observation information sequence; correcting the initial human body posture information, and determining initial human body joint features based on the corrected human body posture information.

[0156] According to one or more embodiments of the present disclosure, initial human body posture information includes relative rotation angles of human body joints under a human body parameterized mesh model and / or joint point coordinates of human body joints under a human body parameterized mesh model, and an observation information sequence includes rotation angles and / or joint point coordinates of joint points on a target part of a human body; and the initial human body posture information is corrected, including at least one of the following: replacing the rotation angles of joint points on a target part of a human body in the initial human body posture information with the rotation angles of joint points on a target part of a human body in the observation information sequence; replacing the joint point coordinates of joint points on a target part of a human body in the initial human body posture information with the joint point coordinates of joint points on a target part of a human body in the observation information sequence.

[0157] According to one or more embodiments of the present disclosure, initial human joint features are determined based on an observation information sequence, including: inputting the observation information sequence into a pre-trained joint prediction sub-model to obtain initial human joint features; and performing feature interaction based on the initial human joint features, and estimating human body posture using the interaction features to obtain human body posture information, including: inputting the initial human joint features into a pre-trained posture estimation sub-model to obtain human body posture information.

[0158] According to one or more embodiments of the present disclosure, the method also includes: determining a loss value using a preset loss function based on human body posture information and label posture information; using the loss value, adjusting the model parameters of the joint point prediction sub-model and the model parameters of the posture estimation sub-model to obtain the adjusted joint point prediction sub-model and posture estimation sub-model.

[0159] According to one or more embodiments of the present disclosure, human body posture information includes the relative rotation angle of human body joints under the human body parameterized mesh model, and label posture information includes the label relative rotation angle; and based on the human body posture information and the label posture information, a loss value is determined using a preset loss function, including: determining the difference between the relative rotation angle of human body joints under the human body parameterized mesh model and the label relative rotation angle as the loss value.

[0160] According to one or more embodiments of the present disclosure, human body posture information includes joint point coordinates of human body joints under a human body parameterized mesh model, and label posture information includes label joint point coordinates; and based on the human body posture information and the label posture information, a loss value is determined using a preset loss function, including: determining the difference between the joint point coordinates of the human body joints under the human body parameterized mesh model and the label joint point coordinates as the loss value.

[0161] According to one or more embodiments of the present disclosure, the human body posture information includes the joint point coordinates of the hand joints in the world coordinate system, and the label posture information includes the joint point coordinates of the label human hand; and based on the human body posture information and the label posture information, a loss value is determined using a preset loss function, including: determining the difference between the joint point coordinates of the hand joints in the world coordinate system and the joint point coordinates of the label human hand as the loss value.

[0162] According to one or more embodiments of the present disclosure, human body posture information includes the movement speed of human body joints within a preset time length, and label posture information includes the label movement speed; and based on the human body posture information and the label posture information, a loss value is determined using a preset loss function, including: determining the difference between the movement speed of human body joints within a preset time length and the label movement speed as the loss value.

[0163] According to one or more embodiments of the present disclosure, human body joints include foot joints; and the difference between the movement speed of the human body joints within a preset time length and the tag movement speed is determined as a loss value, including: if the foot is placed on the ground within the preset time length, then the difference between the movement speed of the foot joints within the preset time length and the target speed is determined as the loss value.

[0164] According to one or more embodiments of the present disclosure, a loss value is determined based on human body posture information and label posture information using a preset loss function, including: using the human body posture information to determine whether there are joints in the human body that are below the ground height; if so, determining the difference between the height of the lowest point in the human body joints below the ground and the height of the ground as the loss value.

[0165] According to one or more embodiments of the present disclosure, a loss value is determined based on human body posture information and label posture information using a preset loss function, including: if the human body's feet are placed on the ground, the difference between the height of the lowest point above the ground among the human body's joints and the height of the ground is determined as the loss value.

[0166] According to one or more embodiments of the present disclosure, the observation information includes the movement speed and angular velocity of the joint point.

[0167] According to one or more embodiments of the present disclosure, a posture estimation device is provided, which includes: an acquisition unit, used to acquire an observation information sequence corresponding to a target part of a human body, wherein the target part of a human body includes at least one of the following: a head and a hand; a determination unit, used to determine initial human joint point features based on the observation information sequence; and an interaction unit, used to perform feature interaction based on the initial human joint point features, and estimate the human body posture using the interaction features to obtain human body posture information.

[0168] According to one or more embodiments of the present disclosure, the interaction unit is further used to perform feature interaction based on the initial human joint point features in the following manner: based on the initial human joint point features, perform feature interaction in the spatial dimension and / or in the temporal dimension.

[0169] According to one or more embodiments of the present disclosure, the interaction unit is further used to perform feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features in the following manner: for each acquisition point in the acquisition time period, determine the attention score between the initial human joint point features corresponding to the acquisition point and the target input features, and use the attention score to interact with the initial human joint point features corresponding to the acquisition point and the target input features to obtain interaction features in the spatial dimension, wherein the target input feature is obtained by mapping the high-dimensional input feature to the same dimension as the initial human joint point feature corresponding to the acquisition point, the high-dimensional input feature is determined based on the observation information sequence, and the acquisition time period is the time period for collecting the observation information sequence.

[0170] According to one or more embodiments of the present disclosure, the interaction unit is further used to perform feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features in the following manner: for each joint point in the human body, determine the attention score between multiple initial joint point features of the joint point within a collection time period, and use the attention score to interact with multiple initial joint point features corresponding to the joint point to obtain interaction features in the time dimension, wherein the collection time period is the time period for collecting the observation information sequence.

[0171] According to one or more embodiments of the present disclosure, the interaction unit is further used to perform feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features in the following manner: the initial human joint point features and the target input features are input into a pre-trained feature interaction network to obtain interaction features, wherein the interaction features include interaction features of human joint points in the spatial dimension and interaction features of human joint points in the temporal dimension, and the target input features are determined based on the observation information sequence.

[0172] According to one or more embodiments of the present disclosure, the feature interaction network includes at least two first coding layers and at least two second coding layers, the first coding layer is used to perform feature interaction in the spatial dimension, and the second coding layer is used to perform feature interaction in the time dimension; and the interaction unit is further used to input the initial human joint point features and the target input features into a pre-trained feature interaction network to obtain interaction features in the following manner: the initial human joint point features and the target input features are input into the alternatingly arranged first coding layer and second coding layer to obtain interaction features.

[0173] According to one or more embodiments of the present disclosure, the determination unit is further used to determine the initial human body joint point features based on the observation information sequence in the following manner: determine the initial human body posture information based on the observation information sequence; correct the initial human body posture information, and determine the initial human body joint point features based on the corrected human body posture information.

[0174] According to one or more embodiments of the present disclosure, the initial human body posture information includes the relative rotation angle of the human body joints under the human body parameterized mesh model and / or the joint point coordinates of the human body joints under the human body parameterized mesh model, and the observation information sequence includes the rotation angle and / or the joint point coordinates of the joint points on the target part of the human body; and the determination unit is further used to correct the initial human body posture information in at least one of the following ways: replacing the rotation angle of the joint points on the target part of the human body in the initial human body posture information with the rotation angle of the joint points on the target part of the human body in the observation information sequence; replacing the joint point coordinates of the joint points on the target part of the human body in the initial human body posture information with the joint point coordinates of the joint points on the target part of the human body in the observation information sequence.

[0175] According to one or more embodiments of the present disclosure, the determination unit is further used to determine the initial human joint features based on the observation information sequence in the following manner: the observation information sequence is input into a pre-trained joint prediction sub-model to obtain the initial human joint features; and the interaction unit is further used to perform feature interaction based on the initial human joint features in the following manner, and estimate the human body posture by using the interaction features to obtain the human body posture information: the initial human joint features are input into a pre-trained posture estimation sub-model to obtain the human body posture information.

[0176] According to one or more embodiments of the present disclosure, a posture estimation device includes a loss value determination unit and an adjustment unit. The loss value determination unit is used to determine the loss value based on human body posture information and label posture information using a preset loss function; the adjustment unit is used to use the loss value to adjust the model parameters of the joint point prediction sub-model and the model parameters of the posture estimation sub-model to obtain the adjusted joint point prediction sub-model and posture estimation sub-model.

[0177] According to one or more embodiments of the present disclosure, the human body posture information includes the relative rotation angle of the human body joints under the human body parameterized mesh model, and the label posture information includes the label relative rotation angle; and the loss value determination unit is further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: the difference between the relative rotation angle of the human body joints under the human body parameterized mesh model and the label relative rotation angle is determined as the loss value.

[0178] According to one or more embodiments of the present disclosure, the human body posture information includes the joint point coordinates of the human body joints under the human body parameterized mesh model, and the label posture information includes the label joint point coordinates; and the loss value determination unit is further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: the difference between the joint point coordinates of the human body joints under the human body parameterized mesh model and the label joint point coordinates is determined as the loss value.

[0179] According to one or more embodiments of the present disclosure, the human body posture information includes the joint point coordinates of the hand joints in the world coordinate system, and the label posture information includes the joint point coordinates of the label human hand; and the loss value determination unit is further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: the difference between the joint point coordinates of the hand joints in the world coordinate system and the joint point coordinates of the label human hand is determined as the loss value.

[0180] According to one or more embodiments of the present disclosure, the human body posture information includes the movement speed of the human body joints within a preset time length, and the label posture information includes the label movement speed; and the loss value determination unit is further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: the difference between the movement speed of the human body joints within a preset time length and the label movement speed is determined as the loss value.

[0181] According to one or more embodiments of the present disclosure, the human body joints include foot joints; and the loss value determination unit is further used to determine the difference between the movement speed of the human body joints within a preset time length and the tag movement speed as the loss value in the following manner: if the foot is placed on the ground within the preset time length, the difference between the movement speed of the foot joints within the preset time length and the target speed is determined as the loss value.

[0182] According to one or more embodiments of the present disclosure, the loss value determination unit is further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: using the human body posture information, determine whether there are any joint points in the human body joints that are below the ground height; if so, determine the difference between the height of the lowest point in the human body joints below the ground and the height of the ground as the loss value.

[0183] According to one or more embodiments of the present disclosure, the loss value determination unit is further used to determine the loss value based on the human body posture information and the label posture information using a preset loss function in the following manner: if the human body's feet are placed on the ground, the difference between the height of the lowest point above the ground among the human body joints and the height of the ground is determined as the loss value.

[0184] According to one or more embodiments of the present disclosure, the observation information includes the movement speed and angular velocity of the joint point.

[0185] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes an acquisition unit, a determination unit, and an interaction unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the acquisition unit may also be described as a "unit for acquiring an observation information sequence corresponding to a target part of the human body".

[0186] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A posture estimation method, characterized in that: include: Acquire an observation information sequence corresponding to a target part of a human body, wherein the target part of a human body includes at least one of the following: a head and a hand; Based on the observation information sequence, determining initial human joint point features; Based on the initial human joint point features, feature interaction is performed, and the human body posture is estimated using the interaction features to obtain human body posture information.

2. The method according to claim 1, characterized in that The performing feature interaction based on the initial human joint point features includes: Based on the initial human joint point features, feature interaction is performed in the spatial dimension and / or in the temporal dimension.

3. The method according to claim 2, characterized in that The performing feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features includes: For each collection point in the collection time period, an attention score between an initial human joint point feature corresponding to the collection point and a target input feature is determined, and the initial human joint point feature corresponding to the collection point is interacted with the target input feature using the attention score to obtain an interaction feature in a spatial dimension, wherein the target input feature is obtained by mapping a high-dimensional input feature to the same dimension as the initial human joint point feature corresponding to the collection point, the high-dimensional input feature is determined based on the observation information sequence, and the collection time period is a time period for collecting the observation information sequence.

4. The method according to claim 2, characterized in that: The performing feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features includes: For each joint point in the human body joints, determine the attention score between multiple initial joint point features of the joint point within a collection time period, and use the attention score to interact with the multiple initial joint point features corresponding to the joint point to obtain interaction features in the time dimension, wherein the collection time period is the time period for collecting the observation information sequence.

5. The method according to claim 2, characterized in that: The performing feature interaction in the spatial dimension and / or in the temporal dimension based on the initial human joint point features includes: The initial human joint point features and target input features are input into a pre-trained feature interaction network to obtain interaction features, wherein the interaction features include interaction features of human joint points in a spatial dimension and interaction features of human joint points in a temporal dimension, and the target input features are determined based on the observation information sequence.

6. The method according to claim 5, characterized in that The feature interaction network includes at least two first coding layers and at least two second coding layers, the first coding layers are used to perform feature interaction in a spatial dimension, and the second coding layers are used to perform feature interaction in a temporal dimension; as well as The step of inputting the initial human joint point features and the target input features into a pre-trained feature interaction network to obtain interaction features comprises: The initial human joint point features and the target input features are input into the alternately arranged first encoding layer and the second encoding layer to obtain interactive features.

7. The method according to claim 1, characterized in that The step of determining initial human joint features based on the observation information sequence includes: Based on the observation information sequence, determining initial human body posture information; The initial human body posture information is corrected, and initial human body joint point features are determined based on the corrected human body posture information.

8. The method according to claim 7, characterized in that The initial human body posture information includes relative rotation angles of human body joints in a human body parameterized mesh model and / or joint point coordinates of human body joints in a human body parameterized mesh model, and the observation information sequence includes rotation angles and / or joint point coordinates of joints on a target part of the human body; and The correcting of the initial human body posture information includes at least one of the following: Replacing the rotation angle of the joint point on the target part of the human body in the initial human posture information with the rotation angle of the joint point on the target part of the human body in the observation information sequence; The joint point coordinates of the joint points on the target part of the human body in the initial human posture information are replaced by the joint point coordinates of the joint points on the target part of the human body in the observation information sequence.

9. The method according to claim 1, characterized in that: The step of determining initial human joint features based on the observation information sequence includes: Inputting the observation information sequence into a pre-trained joint prediction sub-model to obtain initial human joint features; and The method of performing feature interaction based on the initial human joint point features and estimating the human body posture by using the interaction features to obtain human body posture information includes: The initial human body joint point features are input into a pre-trained posture estimation sub-model to obtain human body posture information.

10. The method according to claim 9, characterized in that The method further comprises: Based on the human body posture information and the label posture information, a loss value is determined using a preset loss function; The loss value is used to adjust the model parameters of the joint point prediction sub-model and the model parameters of the posture estimation sub-model to obtain the adjusted joint point prediction sub-model and posture estimation sub-model.

11. The method according to claim 10, characterized in that The human body posture information includes the relative rotation angle of the human body joints under the human body parameterized mesh model, and the tag posture information includes the tag relative rotation angle; and The method of determining the loss value based on the human body posture information and the label posture information by using a preset loss function includes: The difference between the relative rotation angle of the human joint point under the human body parameterized mesh model and the relative rotation angle of the label is determined as the loss value.

12. The method according to claim 10, characterized in that The human body posture information includes the joint point coordinates of the human body joints in the human body parameterized mesh model, and the label posture information includes the label joint point coordinates; and The method of determining the loss value based on the human body posture information and the label posture information by using a preset loss function includes: The difference between the joint point coordinates of the human body joint point under the human body parameterized mesh model and the label joint point coordinates is determined as the loss value.

13. The method according to claim 10, characterized in that The human body posture information includes the joint point coordinates of the hand joint points in the world coordinate system, and the label posture information includes the label human hand joint point coordinates; and The method of determining the loss value based on the human body posture information and the label posture information by using a preset loss function includes: The difference between the joint point coordinates of the hand joint point in the world coordinate system and the joint point coordinates of the labeled human hand is determined as the loss value.

14. The method according to claim 10, characterized in that The human body posture information includes the movement speed of the human body joints within a preset time period, and the tag posture information includes the tag movement speed; and The method of determining the loss value based on the human body posture information and the label posture information by using a preset loss function includes: The difference between the movement speed of the human joint point within a preset time period and the movement speed of the tag is determined as the loss value.

15. The method according to claim 14, characterized in that The human body joints include foot joints; as well as The step of determining the difference between the movement speed of the human joint point within a preset time period and the movement speed of the tag as a loss value includes: If the foot is placed on the ground within the preset time period, the difference between the movement speed of the foot joint point within the preset time period and the target speed is determined as the loss value.

16. The method according to claim 10, characterized in that The method of determining the loss value based on the human body posture information and the label posture information by using a preset loss function includes: Using the human body posture information, determining whether there are any joint points below the ground among the human body joint points; If so, the difference between the height of the lowest point below the ground among the human body joint points and the height of the ground is determined as the loss value.

17. The method according to claim 10, characterized in that The method of determining the loss value based on the human body posture information and the label posture information by using a preset loss function includes: If the feet of the human body are placed on the ground, the difference between the height of the lowest point above the ground among the joints of the human body and the height of the ground is determined as the loss value.

18. The method according to claim 1, characterized in that The observation information includes the movement speed and angular velocity of the joint point.

19. A posture estimation device, characterized in that: include: An acquisition unit, used to acquire an observation information sequence corresponding to a target part of a human body, wherein the target part of a human body includes at least one of the following: a head and a hand; A determination unit, used to determine initial human joint point features based on the observation information sequence; The interaction unit is used to perform feature interaction based on the initial human joint point features, and use the interaction features to estimate the human body posture to obtain human body posture information.

20. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 18.

21. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 18 is implemented.

Images