A method and system for estimating three-dimensional human posture with motion prompts

Abstract

The present invention provides a method and system for three-dimensional human pose estimation with action prompts, comprising: extracting pose position features, pose sequence features, and action features from a two-dimensional pose sequence using a pose encoder; obtaining text prompt features based on the pose position features; aligning the text prompt features with the action features to extract action category information; selecting pose prompt features corresponding to the action category information, combining them based on the correlation between the pose prompt features and the pose sequence features to obtain enhanced pose sequence features; and linearly mapping the enhanced pose sequence features to obtain a three-dimensional pose estimate. During the pose estimation process, the present invention mines prior information related to the action, introduces multimodal information of action-related text and pose features, and addresses depth ambiguity. Its plug-and-play module compactly extracts input data features, saving network model parameters. It significantly improves the accuracy of pose estimation for self-occlusion and complex actions, enhancing flexibility and scalability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, system, medium and terminal for estimating three-dimensional human body posture with action prompts. Background Art

[0002] 3D human pose estimation aims to predict the 3D spatial positions of human joints from a single monocular image or video. Specifically, it first estimates the 2D joints from the input image and then lifts the estimated 2D joints into a 3D pose. As a hot topic in computer vision, 3D human pose estimation holds broad research potential and is widely used in various fields, such as behavioral anomaly detection and human action recognition. However, due to the lack of depth information, this processing approach suffers from a severe depth ambiguity problem, which arises from the many-to-one mapping of multiple 3D poses to the same 2D projection.

[0003] In the paper “3D human pose estimation in video with temporal convolutions and semi-supervised training” published at the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), Pavllo et al. disclosed a method for estimating 3D pose using a simple and effective method called temporal dilated convolution. In the paper “3D human pose estimation with spatial and temporal transformers” published at the IEEE / CVF International Conference on Computer Vision (ICCV 2021), Zheng et al. disclosed a method for introducing Transformer into 3D pose estimation. In the paper “MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Videog” published at the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022), Zhang et al. disclosed a method for decoupling time and space, and introducing Transformer modules at the temporal and spatial levels to measure spatiotemporal correlation. However, all of the above attempts have used temporal consistency to alleviate these problems. However, these attempts only model motion-agnostic spatial and temporal correlations, ignoring the motion-related prior knowledge contained in pose sequences. It turns out that each action has unique characteristics that are beneficial for pose estimation, so it is necessary to find an effective way to embed motion cues into video-based 2D to 3D pose estimation.

[0004] A literature search of existing technologies revealed that existing methods embed motion cues by simultaneously considering pose estimation and action recognition, forming a multi-task learning framework. In their paper "2D / 3D poseestimation and action recognition using multitask deep learning," published at the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018), Luvizon et al. introduced a method that introduced a motion projection module and used a one-hot action label to classify pose sequences, but only achieved negligible performance gains in pose estimation. This is because using only one-hot action labels in the classification subtask cannot fully mine motion cues. One-hot action labels only contain overall motion category information, while ignoring the position and velocity information of the pose sequence. Therefore, compared to using only one-hot action labels, a more effective action label mining scheme needs to be carefully designed. Summary of the Invention

[0005] In response to the problems existing in the above-mentioned prior art, the present invention proposes a method, system, application, medium and terminal for estimating three-dimensional human posture with action prompts.

[0006] According to one aspect of the present invention, a method for estimating a 3D human body posture with motion prompts is provided, comprising:

[0007] S100, obtaining a two-dimensional posture sequence as input;

[0008] S200, extracting posture position features, posture sequence features and action features from the two-dimensional posture sequence;

[0009] S300, obtaining a text prompt feature from the action information text prompt according to the posture position feature;

[0010] S400, aligning the text prompt feature with the action feature to extract action category information;

[0011] S500, selecting a posture prompt feature corresponding to the action category information, and combining the posture prompt feature and the posture sequence feature based on the correlation between the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature;

[0012] S600, obtaining a three-dimensional pose estimate by linearly mapping the enhanced pose sequence features;

[0013] S700, updating the parameters and extracted features in S100 to S700 according to the classification error between the action category information and the real action category, and the posture position error between the 3D posture estimation and the real 3D posture position;

[0014] S800, repeat S100 to S700 until the motion information error and the calculated posture position error converge.

[0015] Preferably, the step S200 of extracting posture position features, posture sequence features, and motion features from a two-dimensional posture sequence includes:

[0016] For a two-dimensional posture sequence X containing F frames and J joint points, the posture sequence feature Z is obtained through the posture encoder d , wherein the posture encoder is an arbitrary multi-layer convolutional neural network;

[0017] Output the features of the first one-dimensional convolutional layer of the posture encoder as the posture position feature Z 0 ;

[0018] The posture position feature Z 0 , the action feature A is extracted through the action projection module composed of one-dimensional convolutional layers.

[0019] Preferably, the step S300 of obtaining a text prompt feature from the action information text prompt according to the posture position feature includes:

[0020] For an action information text prompt consisting of N self-learned text prompt templates and 1 action category text, obtain the text prompt feature T through a text encoder, where the text encoder is any multi-layer attention mechanism network;

[0021] The posture position feature Z 0 , extract the posture change feature Z through the difference module 0 ', where the difference module performs first-order difference on adjacent frames of posture position features to obtain the difference;

[0022] The posture position feature Z 0 and the posture change feature Z 0 'Concatenate to obtain the combined posture feature Z Q ;

[0023] The combined posture feature Z Q The correlation between the text prompt feature T and the text prompt feature T is calculated using a single-layer attention mechanism network using a cross-attention mechanism, where the cross-attention mechanism formula is:

[0024] Q=Z Q P Q ,K=TPK

[0025]

[0026] Where: P Q and P K are projection matrices, Q and K are combined feature query values ​​and text prompt feature key values, respectively, and A is the correlation coefficient matrix;

[0027] The correlation coefficient matrix A is used as a weighting coefficient, and the combined posture feature Z Q Weighted merging into the text prompt feature T to obtain enhanced text prompt feature

[0028] Preferably, the step S400 of aligning the text prompt feature with the action feature and extracting action category information includes:

[0029] Normalizing the action feature and the enhanced text prompt feature respectively so that their respective bi-norms are 1, and calculating the cosine similarity between the normalized action feature and the enhanced text prompt feature;

[0030] The cosine similarity is used as a weight to weight and combine the standardized action features and the enhanced text prompt features to obtain action category information of the posture sequence.

[0031] Preferably, the step S500 of selecting a posture prompt feature corresponding to the action category information, and combining the posture prompt feature with the posture sequence feature based on the correlation between the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature includes:

[0032] For the posture prompts consisting of M action categories, a posture prompt feature is obtained by a posture prompt encoder, wherein the posture prompt encoder is an arbitrary multi-layer convolutional neural network;

[0033] Selecting a gesture prompt feature corresponding to the action according to the action category information;

[0034] Calculate the correlation coefficient matrix between the posture prompt feature and the posture sequence feature using a cross attention mechanism;

[0035] The correlation coefficient matrix is ​​used as a weight to combine the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature.

[0036] Preferably, the linear regressor in S600 is any multi-layer perception network.

[0037] Preferably, the step S700 of updating the parameters and extracting features of steps S100 to S600 according to the classification error between the action category information and the actual action category, and the posture position error between the 3D posture estimation and the actual 3D posture position, includes:

[0038] Calculating the cross entropy loss function between the action category information and the true action category as the classification error;

[0039] Calculating a mean square error between the three-dimensional pose estimate and the true three-dimensional pose position as a pose position error;

[0040] Combining the classification error and the posture position error with a weighted coefficient to form a loss function;

[0041] Calculating the gradient of the model parameters based on the loss function, and updating the model parameters according to the gradient;

[0042] Extract features based on the updated model parameters.

[0043] According to a second aspect of the present invention, there is provided a system for estimating a three-dimensional human posture with motion prompts, comprising:

[0044] Posture data acquisition module: obtains a two-dimensional posture sequence as input;

[0045] Posture feature extraction module: extracts posture position features, posture sequence features and motion features from two-dimensional posture sequences;

[0046] Text feature extraction module: obtains text prompt features from the action information text prompt according to the posture position features;

[0047] Action category extraction module: aligning the text prompt feature with the action feature to extract action category information;

[0048] A posture feature enhancement module: selecting a posture prompt feature corresponding to the action category information, and combining the posture prompt feature with the posture sequence feature based on the correlation between the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature;

[0049] 3D pose estimation module: obtains 3D pose estimation by linear mapping the enhanced pose sequence features;

[0050] Posture model update module: updates the parameters and extracted features of the above steps according to the classification error between the action category information and the real action category, and the posture position error between the 3D posture estimation and the real 3D posture position.

[0051] The error convergence stop module repeatedly executes each module until the motion information error and the calculated posture position error converge.

[0052] According to a third aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the terminal can be used to execute the above-mentioned motion-prompted three-dimensional human body posture estimation method, or to execute the above-mentioned motion-prompted three-dimensional human body posture estimation system.

[0053] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above-mentioned motion-prompted three-dimensional human body posture estimation method, or to run the above-mentioned motion-prompted three-dimensional human body posture estimation system.

[0054] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0055] (1) The method and system for estimating 3D human posture with action prompts in the embodiments of the present invention mines action-related prior information, introduces multimodal information of action-related text features and posture features, and handles the depth ambiguity problem.

[0056] (2) The method and system for estimating three-dimensional human posture with motion prompts in the embodiment of the present invention can compactly extract input data features and save network model parameters by constructing a plug-and-play module including S200 to S500.

[0057] (3) The method and system for estimating three-dimensional human body posture with action prompts in the embodiment of the present invention significantly improves the accuracy of posture estimation containing self-occlusion and complex actions by introducing action text information in S400 and posture position information in S500, and has strong flexibility and scalability.

[0058] (4) The embodiments of the present invention were verified on collected virtual reality three-dimensional human motion videos. The results fully confirmed its ability to accurately estimate typical human postures in various real environments, and effectively alleviated problems such as joint occlusion, partial limb occlusion, and depth discontinuity. The present invention can be applied in application fields such as virtual reality and the metaverse to achieve real-time, high-precision, and high-robustness human posture estimation for moving human posture videos, thereby promoting the implementation of various mid- and downstream tasks (such as three-dimensional reconstruction and action recognition). BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0060] Figure 1 A flowchart of a method for estimating a 3D human body posture with motion prompts according to an embodiment of the present invention;

[0061] Figure 2 A schematic structural diagram of a 3D human posture estimation system with motion prompts according to an embodiment of the present invention;

[0062] Figure 3 Schematic diagram of 3D human posture estimation with action prompts in a specific example of the present invention.

[0063] In the picture:

[0064] 11- Posture data acquisition module, 12- Posture feature extraction module, 13- Text feature alignment module, 14- Action category extraction module, 17- Posture feature enhancement module, 16- Three-dimensional posture estimation module, 17- Posture model update module, 18- Error convergence stop module. DETAILED DESCRIPTION

[0065] The following describes an embodiment of the present invention in detail. This embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operating procedures. It should be noted that those skilled in the art may make various modifications and improvements without departing from the scope of the present invention, and these modifications and improvements fall within the scope of protection of the present invention. Portions not described in detail in the following embodiments may be implemented using existing technologies.

[0066] like Figure 1 FIG2 is a flowchart of a method for estimating a 3D human body posture with motion prompts according to an embodiment of the present invention. Figure 1 The method for estimating a 3D human body posture based on motion prompts of this embodiment includes the following steps:

[0067] S100: Obtain a two-dimensional posture sequence as input;

[0068] S200: extracting posture position features, posture sequence features and action features from the two-dimensional posture sequence input in S100;

[0069] S300: Obtaining text prompt features from the action information text prompt based on the posture position features obtained in S200;

[0070] S400: aligning the text prompt features obtained in S300 with the action features obtained in S200 to extract action category information;

[0071] S500: selecting a posture prompt feature corresponding to the action category information obtained in S400, and combining the posture prompt feature and the posture sequence feature obtained in S200 based on the correlation between the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature;

[0072] S600: Mapping the enhanced posture sequence features obtained in S500 through a linear regressor to obtain a three-dimensional posture estimate, wherein the linear regressor can be any multi-layer perception network;

[0073] S700: updating the parameters and extracted features of the above steps S100-S600 according to the classification error between the action category information obtained in S400 and the actual action category, and the posture position error between the 3D posture estimation obtained in S600 and the actual 3D posture position;

[0074] S800, repeat S100 to S700 until the motion information error and the calculated posture position error converge.

[0075] This embodiment saves the workload and time of manual parameter design, better captures posture-related features, and can significantly improve the accuracy of posture estimation containing self-occlusion and complex movements while saving the number of network parameters.

[0076] In a preferred embodiment of the present invention, the implementation of S200 further includes:

[0077] S201: For a two-dimensional posture sequence X containing F frames and J joint points, obtain the posture sequence feature Z through the posture encoder d , where the pose encoder can be any multi-layer convolutional neural network;

[0078] S202: Output the features of the first one-dimensional convolution layer of the posture encoder as the posture position feature Z 0 ;

[0079] S203: The posture position feature Z 0 , the action feature A is extracted through the action projection module composed of one-dimensional convolutional layers.

[0080] In a preferred embodiment of the present invention, the implementation of S300 further includes:

[0081] S301: For an action information text prompt consisting of N self-learned text prompt templates and 1 action category text, obtain a text prompt feature T through a text encoder, where the text encoder can be any multi-layer attention mechanism network;

[0082] S302: The posture position feature Z 0 , extract the posture change feature Z through the difference module 0' , where the difference module performs first-order difference on adjacent frames of posture and position features to obtain the difference;

[0083] S303: The posture position feature Z 0 and posture change feature Z 0' Connect in series to obtain the combined posture feature ZQ ;

[0084] S304: Combine the posture feature Z Q The correlation between the text prompt feature T and the cross attention mechanism is calculated using a single-layer attention mechanism network, where the cross attention mechanism formula is:

[0085] Q=Z Q P Q ,K=TP K

[0086]

[0087] Where: P Q and P K are projection matrices, Q and K are combined feature query values ​​and text prompt feature key values, respectively, and A is the correlation coefficient matrix;

[0088] S305: Use the correlation coefficient matrix A as the weighting coefficient and combine the posture feature Z Q Weighted merging into the text prompt feature T to obtain enhanced text prompt feature

[0089] In a preferred embodiment of the present invention, the implementation of S400 further includes:

[0090] S401: Normalize the action features and the enhanced text prompt features respectively so that their bi-norms are both 1, and calculate the cosine similarity between the normalized action features and the enhanced text prompt features;

[0091] S402: Using cosine similarity as a weight, weightedly combining the standardized action features and the enhanced text prompt features to obtain action category information of the posture sequence.

[0092] The above embodiment effectively embeds action information into posture features by aligning action features and enhancing text prompt features. By introducing action prior information in the text space, the posture sequences are preliminarily classified and aligned in the action space, which is particularly helpful for posture estimation involving self-occlusion and complex actions.

[0093] In a preferred embodiment of the present invention, the implementation of S500 further includes:

[0094] S501: For the posture prompts consisting of M action categories, obtain posture prompt features through a posture prompt encoder, where the posture prompt encoder can be any multi-layer convolutional neural network;

[0095] S502: Selecting a gesture prompt feature corresponding to the action according to the action category information;

[0096] S503: Calculate the correlation between posture hint features and posture sequence features using the cross attention mechanism;

[0097] S504: Using the correlation coefficient matrix as a weight to combine the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature.

[0098] This example mines the posture cue features of each action, calculates the correlation with the posture sequence features, and introduces the position information in the posture cue features. It uses the position information of each action to optimize the estimation of the posture sequence, further improving the accuracy of posture estimation with self-occlusion and complex actions. It has strong flexibility and scalability.

[0099] In a preferred embodiment of the present invention, the implementation of S700 further includes:

[0100] S701: Calculate the cross entropy loss function between the action category information and the true action category as the classification error;

[0101] S702: Calculate the mean square error between the 3D pose estimate and the true 3D pose position as a pose position error;

[0102] S703: Combining the classification error and the posture position error with weighted coefficients to form a loss function;

[0103] S704: Calculate the gradient of the model parameters based on the loss function, and update the model parameters according to the gradient.

[0104] See also Figure 2Based on the same inventive concept, the present invention provides a three-dimensional human posture estimation system with action prompts, including a posture data acquisition module 11, a posture feature extraction module 12, a text feature alignment module 13, an action category extraction module 14, a posture feature enhancement module 15, a three-dimensional posture estimation module 16, a posture model updating module 17 and an error convergence stopping module 18. The posture data acquisition module 11 obtains a two-dimensional posture sequence as input; the posture feature extraction module 12 extracts posture position features, posture sequence features and action features from the two-dimensional posture sequence; the text feature extraction module 13 obtains text prompt features from the action information text prompt based on the posture position features; the action category extraction module 14 aligns the text prompt features with the action features and extracts the action category information; the posture feature enhancement module 17 selects the posture prompt features corresponding to the action category information, and combines the posture prompt features with the posture sequence features based on the correlation between the posture prompt features and the posture sequence features to obtain enhanced posture sequence features; the three-dimensional posture estimation module 16 obtains a three-dimensional posture estimate by linear mapping the enhanced posture sequence features; the posture model update module 17 updates the parameters and extracted features of the above steps based on the classification error between the action category information and the true action category, as well as the posture position error between the three-dimensional posture estimate and the true three-dimensional posture position. The error convergence stop module 18 repeatedly executes each module until the action information error and the calculated posture position error converge.

[0105] The modules / units in the above examples of the present invention may specifically refer to the implementation techniques of the corresponding steps of the method for estimating three-dimensional human posture with action prompts in the above embodiments, which will not be described in detail here.

[0106] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it can be used to execute the above-mentioned motion-prompted three-dimensional human body posture estimation method, or, run the above-mentioned motion-prompted three-dimensional human body posture estimation system.

[0107] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0108] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories, and the aforementioned computer programs, computer instructions, data, etc. may be called by a processor.

[0109] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0110] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0111] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above-mentioned motion-prompted three-dimensional human body posture estimation method, or to run the above-mentioned motion-prompted three-dimensional human body posture estimation system.

[0112] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one location to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. Alternatively, the ASIC can be located in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0113] To better understand each of the above-mentioned parts, the following is an explanation using a detailed virtual reality application example. This embodiment provides a three-dimensional human posture estimation system with motion prompts. The system adopts the above-mentioned motion prompt network construction method. The embodiment is executed in parallel on the input two-dimensional human joint points. Each group of two-dimensional human joint points includes multiple joint points, and each joint point contains its two-dimensional coordinate information in the image coordinate system. The three-dimensional human posture estimation system with motion prompts obtains the three-dimensional coordinate information corresponding to each joint point through motion prompt network regression. To clearly describe the execution process, the following is combined with a virtual reality video, where the virtual reality video is about a moving human posture video with a series of complex movements. The number of nodes in the human skeleton topology structure is 16, and the number of video frames is 243 frames. The motion prompt network structure construction method for three-dimensional human posture estimation with motion prompts is described. It will be divided into a training part and a testing part. The testing part is similar to the training part process, so only the training part is introduced in detail below.

[0114] Training part:

[0115] 1. Posture data acquisition module: obtain the two-dimensional motion human posture sequence corresponding to the virtual reality video with a dimension of (243, 16, 2) as input X.

[0116] 2. Posture feature extraction module, which extracts posture position features, posture sequence features, and action features from the two-dimensional posture sequence. The specific process is as follows:

[0117] S21: For a two-dimensional pose sequence X containing 243 frames and 16 joint points, the pose sequence feature Z with a dimension of (1, 256) is obtained through the pose encoder. d , where the pose encoder can be any multi-layer convolutional neural network;

[0118] S22: Output the features of the first one-dimensional convolutional layer of the posture encoder as the posture position feature Z 0 , dimension is (81, 256);

[0119] S23: The posture position feature Z 0 , the action feature A with dimension (1, 256) is extracted through the action projection module composed of one-dimensional convolutional layers.

[0120] 3. The text feature extraction module obtains text prompt features from the action information text prompt based on the posture position features. The specific process is as follows:

[0121] S31: For the action information text prompt consisting of 10 self-learned text prompt templates and 17 action category texts, a text prompt feature T of dimension (17, 256) is obtained through a text encoder, where the text encoder can be any multi-layer attention mechanism network;

[0122] S32: The posture position feature Z 0 , extract the posture change feature Z with a dimension of (80, 256) through the difference module 0' , where the difference module performs first-order difference on adjacent frames of posture and position features to obtain the difference;

[0123] S33: The posture position feature Z 0 and the posture change feature Z 0' Concatenate and obtain the combined posture feature Z with a dimension of (81, 256) Q ;

[0124] S34: The combined posture feature Z Q The correlation between the text prompt feature T and the text prompt feature T is calculated using a single-layer attention mechanism network using a cross-attention mechanism, where the cross-attention mechanism formula is:

[0125] Q=Z Q P Q ,K=TP K

[0126]

[0127] Where: P Q and P K are projection matrices, Q and K are combined feature query values ​​and text prompt feature key values, respectively, and A is the correlation coefficient matrix;

[0128] S35: Using the correlation coefficient matrix as a weighting coefficient, the combined posture feature Z Q Weighted merging into the text prompt feature T to obtain an enhanced text prompt feature with a dimension of (17, 256)

[0129] 4. An action category extraction module aligns the text prompt feature with the action feature and extracts action category information.

[0130] S41: normalizing the action feature and the enhanced text prompt feature respectively so that their bi-norms are both 1, and calculating the cosine similarity between the normalized action feature and the enhanced text prompt feature, with a dimension of (1, 17);

[0131] S42: Using the cosine similarity as a weight, weightedly combining the standardized action features and the enhanced text prompt features to obtain action category information of the posture sequence.

[0132] 5. A posture feature enhancement module selects a posture prompt feature corresponding to the action category information, and combines the posture prompt feature with the posture sequence feature based on the correlation between the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature. The specific process is as follows:

[0133] S51: For the posture prompts consisting of 17 action categories, obtain a posture prompt feature with a dimension of (17, 256) through a posture prompt encoder, where the posture prompt encoder can be any multi-layer convolutional neural network;

[0134] S52: Selecting a posture prompt feature corresponding to the action according to the action category information, with a dimension of (1, 81, 256);

[0135] S53: Calculate the similarity between the posture prompt feature and the posture sequence feature, with a dimension of (1, 81, 81);

[0136] S54: The posture prompt feature and the posture sequence feature are combined using the similarity as a weight to obtain an enhanced posture sequence feature with a dimension of (1, 256).

[0137] 6. A three-dimensional pose estimation module maps the enhanced pose sequence features through a linear regressor to obtain a three-dimensional pose estimation with a dimension of (1, 16, 3), where the linear regressor can be any multi-layer perception network.

[0138] 7. The posture model update module updates the parameters and extracted features of the above steps based on the classification error between the action category information and the actual action category, as well as the posture position error between the 3D posture estimate and the actual 3D posture position. The specific process is as follows:

[0139] S71: Calculate the cross entropy loss function between the action category information and the true action category as the classification error;

[0140] S72: Calculate the mean square error between the three-dimensional pose estimation and the true three-dimensional pose position as a pose position error;

[0141] S73: Combining the classification error and the posture position error with a weighted coefficient to form a loss function;

[0142] S74: Calculate the gradient of the model parameters based on the loss function, and update the model parameters according to the gradient;

[0143] S75: Extract features based on the updated model parameters.

[0144] Implementation effect:

[0145] In this embodiment, the evaluation can be divided into objective and subjective evaluations. The objective evaluation involves statistically analyzing the results of 3D human pose estimation in VR videos of action-prompted 3D human poses, obtaining metrics such as mean joint position error (MPJPE). The subjective evaluation involves visualizing the results of 3D human pose estimation in VR videos of action-prompted 3D human poses. This embodiment compares the human pose estimation results of VR videos containing multiple actions with those of existing methods.

[0146] Experiments show that in terms of objective evaluation, this embodiment significantly improves MPJPE in virtual reality videos with self-occlusion and complex movements compared to existing methods, and also achieves an improvement in average MPJPE. Further data analysis proves that after adopting the method of this embodiment, additional action prior information is taken into account in the network construction method, and the rich action-related text information in the action label is adapted to the posture sequence. The position-aware posture pattern of each action is mined, and the posture features are refined by utilizing the correlation between the learnable pattern and the input posture sequence. This enables the model to well represent the structural information of the joint points when the input two-dimensional human posture has complex movements and self-occlusion, effectively improving the three-dimensional human posture estimation results of the action prompt.

[0147] After adopting the method of this embodiment, the 3D human posture estimation effect of the action prompts of complex actions and self-occluded actions in virtual reality sports human posture videos is improved. Figure 3 Visualization results: From top to bottom, the two rows represent two different input actions. The leftmost column shows the 3D human pose estimation results for action cues without the action cue network. The middle column shows the 3D human pose estimation results for action cues in this example. The rightmost column shows the ground truth results, which are the manually labeled semantic categories. It can be seen that by introducing context-related and position-aware action prior knowledge, the method in this example significantly improves the accuracy of 3D human pose estimation for action cues with complex and self-occluded actions.

[0148] The embodiments of the present invention were validated against collected virtual reality 3D human motion videos. The results fully demonstrated its ability to accurately estimate typical human poses in various real-world environments, effectively alleviating issues such as joint occlusion, partial limb occlusion, and depth discontinuity. These embodiments can be applied in fields such as virtual reality and the metaverse, enabling real-time, high-precision, and highly robust human pose estimation for moving human pose videos, thereby facilitating various mid- and downstream tasks (such as 3D reconstruction and motion recognition).

[0149] Those skilled in the art will understand that, unless at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification and all processes or units of any device disclosed so far may be combined in any combination.

[0150] Disclosed herein are only preferred embodiments of the present invention. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention and are not intended to limit the present invention. Any modifications and variations made by those skilled in the art within the scope of this specification are intended to fall within the scope of protection of the present invention.

Claims

1. A method for estimating 3D human posture with motion prompts, characterized in that: include: S100, obtaining a two-dimensional posture sequence as input; S200, extracting posture position features, posture sequence features and action features from the two-dimensional posture sequence; S300, obtaining a text prompt feature from the action information text prompt according to the posture position feature; S400, aligning the text prompt feature with the action feature to extract action category information; S500, selecting a posture prompt feature corresponding to the action category information, and combining the posture prompt feature and the posture sequence feature based on the correlation between the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature; S600, obtaining a three-dimensional pose estimate by linearly mapping the enhanced pose sequence features; S700, updating the parameters and extracted features in S100 to S700 according to the classification error between the action category information and the real action category, and the posture position error between the 3D posture estimation and the real 3D posture position; S800, repeat S100 to S700 until the motion information error and the calculated posture position error converge; The step S300 of obtaining a text prompt feature from the action information text prompt based on the posture position feature includes: For the inclusion self-learning text prompt templates and The action information text prompt is composed of action category texts, and the text prompt feature is obtained through the text encoder. , where the text encoder is an arbitrary multi-layer attention mechanism network; The posture position feature , extracting posture change features through the difference module , where the difference module performs first-order difference on adjacent frames of posture and position features to obtain the difference; The posture position feature and the posture change characteristics Connect in series to obtain combined posture features ; The combined posture feature With the text prompt feature Through a single-layer attention mechanism network, the correlation is calculated using the cross attention mechanism, where the cross attention mechanism formula is: ； ； in: and are the projection matrices, and They are the combined feature query value and the text prompt feature key value respectively. is the correlation coefficient matrix; The correlation coefficient matrix As a weighting coefficient, the combined posture feature Weighted integration into the text hint feature , get enhanced text prompt features ; The step S400 of aligning the text prompt feature with the action feature and extracting action category information includes: Normalizing the action feature and the enhanced text prompt feature respectively so that their respective bi-norms are 1, and calculating the cosine similarity between the normalized action feature and the enhanced text prompt feature; Using the cosine similarity as a weight, weightedly combining the standardized action features and the enhanced text prompt features to obtain action category information of the posture sequence; By aligning action features and enhancing text prompt features, action information is effectively embedded in posture features. By introducing action prior information in the text space, posture sequences are preliminarily classified and aligned in the action space, which has a guiding effect on posture estimation containing self-occlusion and complex actions.

2. The method for estimating 3D human posture with motion prompts according to claim 1, wherein: The step S200 of extracting posture position features, posture sequence features, and motion features from the two-dimensional posture sequence includes: For the inclusion frame J 2D pose sequence of joint points X , get the posture sequence features through the posture encoder , wherein the posture encoder is an arbitrary multi-layer convolutional neural network; Output the features of the first one-dimensional convolutional layer of the posture encoder as the posture position feature ; The posture position feature , extracting motion features through the motion projection module composed of one-dimensional convolutional layers .

3. The method for estimating 3D human posture with motion prompts according to claim 1, wherein: The step S500 of selecting a posture prompt feature corresponding to the action category information, and combining the posture prompt feature with the posture sequence feature based on a correlation between the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature, includes: for The posture prompts are composed of action categories, and the posture prompt features are obtained by the posture prompt encoder, wherein the posture prompt encoder is an arbitrary multi-layer convolutional neural network; Selecting a gesture prompt feature corresponding to the action according to the action category information; Calculate the correlation coefficient matrix between the posture prompt feature and the posture sequence feature using a cross attention mechanism; The correlation coefficient matrix is ​​used as a weight to combine the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature.

4. The method for estimating 3D human posture with motion prompts according to claim 1, wherein: The linear mapping in S600 is implemented using any multi-layer perceptron network.

5. The method for estimating 3D human posture with motion prompts according to claim 1, wherein: The step S700 updates the parameters and extracts features of steps S100 to S600 based on the classification error between the action category information and the actual action category, and the posture position error between the 3D posture estimate and the actual 3D posture position, including: Calculating the cross entropy loss function between the action category information and the true action category as the classification error; Calculating a mean square error between the three-dimensional pose estimate and the true three-dimensional pose position as a pose position error; Combining the classification error and the posture position error with a weighted coefficient to form a loss function; Calculating the gradient of the model parameters based on the loss function, and updating the model parameters according to the gradient; Extract features based on the updated model parameters.

6. A motion-cued 3D human posture estimation system, characterized in that: include: Posture data acquisition module: obtains a two-dimensional posture sequence as input; Posture feature extraction module: extracts posture position features, posture sequence features and motion features from two-dimensional posture sequences; Text feature extraction module: obtains text prompt features from the action information text prompt according to the posture position features; Action category extraction module: aligning the text prompt feature with the action feature to extract action category information; A posture feature enhancement module: selecting a posture prompt feature corresponding to the action category information, and combining the posture prompt feature with the posture sequence feature based on the correlation between the posture prompt feature and the posture sequence feature to obtain an enhanced posture sequence feature; 3D pose estimation module: obtains 3D pose estimation by linear mapping the enhanced pose sequence features; Posture model update module: updates the parameters of each module and extracts features based on the classification error between the action category information and the real action category, and the posture position error between the 3D posture estimate and the real 3D posture position; Error convergence stop module, repeatedly executing each module until the motion information error and the calculated posture position error converge; The text feature extraction module includes: Obtaining text prompt features from the action information text prompt based on the posture position features, including: For the inclusion self-learning text prompt templates and The action information text prompt is composed of action category texts, and the text prompt feature is obtained through the text encoder. , where the text encoder is an arbitrary multi-layer attention mechanism network; The posture position feature , extracting posture change features through the difference module , where the difference module performs first-order difference on adjacent frames of posture and position features to obtain the difference; The posture position feature and the posture change characteristics Connect in series to obtain combined posture features ; The combined posture feature With the text prompt feature Through a single-layer attention mechanism network, the correlation is calculated using the cross attention mechanism, where the cross attention mechanism formula is: ； ； in: and are the projection matrices, and They are the combined feature query value and the text prompt feature key value respectively. is the correlation coefficient matrix; The correlation coefficient matrix As a weighting coefficient, the combined posture feature Weighted integration into the text hint feature , get enhanced text prompt features ; The action category extraction module includes: Aligning the text prompt feature with the action feature and extracting action category information includes: Normalizing the action feature and the enhanced text prompt feature respectively so that their respective bi-norms are 1, and calculating the cosine similarity between the normalized action feature and the enhanced text prompt feature; Using the cosine similarity as a weight, weightedly combining the standardized action features and the enhanced text prompt features to obtain action category information of the posture sequence; By aligning action features and enhancing text prompt features, action information is effectively embedded in posture features. By introducing action prior information in the text space, posture sequences are preliminarily classified and aligned in the action space, which has a guiding effect on posture estimation containing self-occlusion and complex actions.

7. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it can be used to perform the method according to any one of claims 1 to 5, or run the system according to claim 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to perform the method according to any one of claims 1 to 5, or to run the system according to claim 6.

Images