A three-dimensional human pose estimation method, system, application, medium and terminal

Abstract

The application provides a three-dimensional human posture estimation method and system, comprising: obtaining different categories of joint embedding information in different ways, that is, obtaining absolute and relative graph topology distance information while inputting 2D posture information; constructing a posture-oriented self-attention mechanism and a Transformer network; summing up different categories of joint embedding information of each joint as a token and inputting into the Transformer network to extract posture-related global features; according to the extracted global features, performing uncertainty-guided sampling operation and posture improvement operation by estimating the uncertainty of the joint to realize improvement based on uncertainty guidance. The application introduces graph topology structure and uncertainty information in the self-attention mechanism to process deep fuzzy problems; the constructed Transformer network can compactly extract input data features and save network model parameter quantity; the posture estimation accuracy of the posture containing self-occlusion and complex action is significantly improved, and the application has strong flexibility and scalability.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a three-dimensional human pose estimation method, system, application, medium, and terminal. Background Technology

[0002] 3D human pose estimation aims to predict the 3D spatial position of human joints using a single monocular image. As a hot topic in the field of computer vision, 3D human pose estimation has broad research prospects and is widely used in various fields, such as behavior anomaly detection and behavior recognition.

[0003] In recent years, the performance of 2D human pose estimation has improved significantly, leading to an increasing number of studies that use 2D human pose estimation for 3D human pose estimation. Martinez et al., in their paper "A simple yet effective baseline for 3D human pose estimation" published at the IEEE International Conference on Computer Vision (ICCV 2017), disclosed a method to regress the corresponding 3D coordinates of estimated joints. Zhao et al., in their paper "Semantic graph convolutional networks for 3D human pose regression" published at the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), disclosed a method that treats the human skeleton as a graph topology and uses graph convolutional neural networks to model the relationships between its joints. Zhao et al., in their paper "GraFormer: Graph-oriented transformer for 3D pose estimation" published at the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022), disclosed a method that, due to the limited receptive field of graph convolutional neural networks, uses a Transformer to construct global joint connectivity relationships, achieving more accurate pose estimation.

[0004] A literature review of existing technologies revealed that the main difficulty in current Transformer-based 3D human pose estimation tasks lies in overcoming the non-uniqueness of the 2D-to-3D mapping, making it an ill-posed problem. Existing methods treat joints as equally important units input into the Transformer when modeling them, but this clearly ignores the prior information of the human body's topology. Therefore, for joints with high degrees of freedom, it is often difficult to estimate accurate 3D joint coordinates. Summary of the Invention

[0005] This invention addresses the problems existing in the prior art by proposing a three-dimensional human pose estimation method, system, application, medium, and terminal.

[0006] According to one aspect of the present invention, a three-dimensional human pose estimation method is provided, comprising:

[0007] By acquiring different types of keypoint embedding information through different methods, that is, while the input contains 2D pose information, absolute and relative graph topological distance information is obtained.

[0008] By constructing a pose-oriented self-attention mechanism, a pose-oriented Transformer network is constructed; the sum of the different categories of embedded information at each keypoint is used as a token and input into the Transformer network to extract pose-related global features.

[0009] Based on the output of the Transformer network, uncertainty-guided sampling and attitude improvement operations are performed by estimating the uncertainty of the joints, thereby achieving uncertainty-guided improvement.

[0010] Preferably, the step of obtaining different categories of joint embedding information through different methods, i.e., obtaining absolute and relative graph topological distance information while inputting 2D pose information, includes:

[0011] For a 2D pose input image X containing J joints, the feature embedding information Z of each joint is obtained by linear mapping through a fully connected layer.

[0012] For J key points, the key point position embedding information K corresponding to each key point is obtained through adaptive learning;

[0013] For J joints, they are grouped according to the shortest graph topological distance to the pelvic joints, and the group-level position embedding information G corresponding to each group is obtained through adaptive learning.

[0014] The feature embedding information Z, the key point location embedding information K, and the group-level location embedding information G are added together, and the final result is used as the input of the subsequent Transformer.

[0015] Preferably, the operations that the pose-oriented attention mechanism can perform include:

[0016] Based on the topological structure of the human body graph, calculate the shortest topological distance between each pair of joints, where the distance between two adjacent joints is 1.

[0017] The distance l is mapped and encoded using a multilayer perceptron as a mapping function, and the result is added to the attention matrix output by the standard multi-head attention mechanism MH-SA as an additional bias.

[0018] Preferably, the Transformer network has a total of L layers, each layer including a pose-oriented attention mechanism layer, a GELU nonlinear mapping layer and two multilayer perceptron layers, and performs corresponding short connection operations; the Transformer network is a pose-oriented network that extracts pose-related global features.

[0019] Preferably, the output of the root Transformer network is used to estimate the uncertainty of the joints, perform uncertainty-guided sampling and attitude improvement operations, and achieve uncertainty-guided improvement, including:

[0020] Based on the output of the pose-oriented Transformer network structure, the uncertainty σ of each joint is estimated by using a multilayer perceptron as a linear mapping layer;

[0021] Based on the output of the pose-oriented Transformer network structure, a rough 3D pose is obtained by using a multilayer perceptron as a linear mapping layer. This 3D pose is then used as the mean μ, and the joint uncertainty σ is used as the variance to obtain a Gaussian distribution. Finally, the distribution was sampled;

[0022] An uncertainty-guided attention mechanism is constructed using the estimated uncertainty σ of the joint point to adjust the attention weights of the corresponding joint points;

[0023] Finally, a multilayer perceptron is used as a linear mapping layer to obtain the improved and refined 3D pose.

[0024] Preferably, for the i-th and j-th tokens, the uncertainty-guided attention mechanism divides the attention weights obtained from the standard multi-head attention mechanism MH-SA with the uncertainty of the j-th keypoint. Based on the division result, the attention weights of keypoints with high uncertainty are reduced, and the attention weights of keypoints with low uncertainty are increased.

[0025] According to a second aspect of the present invention, a three-dimensional human pose estimation system is provided, comprising:

[0026] A multi-category embedding information acquisition module acquires keypoint embedding information of different categories through different methods. That is, while the input contains 2D pose information, it obtains absolute and relative graph topological distance information.

[0027] A pose-oriented Transformer network construction module is provided. This module constructs a pose-oriented Transformer network by building a pose-oriented self-attention mechanism. The sum of the different categories of embedded information at each key point is used as a token and input into the Transformer network to extract pose-related global features.

[0028] An uncertainty-guided attitude improvement module is constructed. Based on the output of the Transformer network, this module estimates the uncertainty of the joints, performs uncertainty-guided sampling operations and attitude improvement operations, and achieves uncertainty-guided improvement.

[0029] According to a third aspect of the present invention, a three-dimensional human pose estimation application system is provided, comprising:

[0030] The Transformer network construction module is based on the above-mentioned 3D human pose estimation method to construct a Transformer network, or based on the above-mentioned 3D human pose estimation system to construct a Transformer network.

[0031] The training module for the Transformer network, which is used to train the constructed Transformer network;

[0032] The 3D human pose estimation module is used to perform 3D human pose estimation using a trained Transformer network.

[0033] According to a fourth aspect of the present invention, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-described three-dimensional human pose estimation method, or to run the above-described three-dimensional human pose estimation system.

[0034] According to a fifth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the above-described three-dimensional human pose estimation method, or to run the above-described three-dimensional human pose estimation system.

[0035] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0036] (1) The three-dimensional human pose estimation method and system in the embodiments of the present invention can better handle the depth fuzzy problem by introducing graph topology and uncertain information into the self-attention mechanism.

[0037] (2) The three-dimensional human pose estimation method and system in the embodiments of the present invention can extract features from the input data more compactly by constructing a Transformer network, thus saving the number of network model parameters.

[0038] (3) The three-dimensional human pose estimation method and system in the embodiments of the present invention can significantly improve the pose estimation accuracy of self-occlusion and complex actions by improving uncertainty sampling and uncertainty attention mechanism, and has strong flexibility and scalability. Attached Figure Description

[0039] The embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0040] Figure 1 This is a flowchart of a three-dimensional human pose estimation method according to an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the structure of a three-dimensional human pose estimation system according to an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the structure of a three-dimensional human pose estimation application system according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of three-dimensional human pose estimation in a specific embodiment of the present invention.

[0044] Label Explanation:

[0045] 11-Multi-category embedding information acquisition module, 12-Pose-oriented Transformer network construction module, 13-Uncertainty-guided pose improvement module construction module.

[0046] 21 - Transformer network building module, 22 - Transformer network training module, 23 - 3D human pose estimation module. Detailed Implementation

[0047] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0048] like Figure 1 The diagram shown is a flowchart of a three-dimensional human pose estimation method according to an embodiment of the present invention. Please refer to... Figure 1 The three-dimensional human pose estimation method in this embodiment includes the following steps:

[0049] S100: By acquiring different types of keypoint embedding information through different methods, the network includes both absolute and relative graph topological distance information, while also containing 2D pose input information.

[0050] S200: Construct a pose-oriented Transformer network by building a pose-oriented self-attention mechanism; use the sum of different category embedding information of each keypoint obtained in S100 as a token and input it into the Transformer network to extract pose-related global features.

[0051] S300: Based on the attitude-oriented Transformer structure obtained from S200, uncertainty-guided sampling and attitude improvement operations are performed by estimating the uncertainty of the joints, thereby achieving uncertainty-guided improvement.

[0052] This embodiment saves the workload and time of manual parameter design, better captures pose-related features, and significantly improves the accuracy of pose estimation with self-occlusion and complex movements while saving the number of network parameters.

[0053] In a preferred embodiment of the present invention, S100 is further comprising:

[0054] S101: For a 2D pose input image X containing J joints, the feature embedding information Z of each joint is obtained by linear mapping;

[0055] S102: For J key points, the key point embedding location information K corresponding to each key point is obtained through adaptive learning;

[0056] S103: For J joints, group them according to the shortest graph topological distance to the pelvic joints, and obtain the group-level position embedding information G for each group through adaptive learning.

[0057] S104: Add the three types of embedding information Z, G and G obtained in S101, S102 and S103, and use the final result as the input of the subsequent Transformer network.

[0058] In this embodiment, these three types of embedding information are used as input to the Transformer. While preserving the basic 2D pose features, the group-level position embedding information can be used to describe the relatively difficult joints that are far from the root node (pelvic node) in a distance-related manner, thereby improving the prediction accuracy of difficult joints.

[0059] In a preferred embodiment of the present invention, S200 is further comprising:

[0060] S201: Based on the topological structure of the human body graph, calculate the shortest topological distance between each pair of key points;

[0061] S202: Use a multilayer perceptron as a mapping function to map and encode the topological shortest distance obtained in S201;

[0062] S203: The S202 encoding result is used as an additional bias and the output of the standard multi-head attention mechanism MH-SA is added to the attention matrix as the output of the pose-oriented attention mechanism;

[0063] S204: Connect the above pose-oriented attention mechanism with a multilayer perceptron and a GELU nonlinear mapping layer to construct a pose-oriented Transformer block. Stack multiple pose-oriented Transformer blocks to construct a pose-oriented Transformer network structure.

[0064] In this embodiment, S201-S203 is to construct a pose-oriented self-attention mechanism, and S204 is to construct a pose-oriented Transformer network. Thus, the Transformer network has a total of L layers. Each layer includes a pose-oriented attention mechanism layer constructed as in S201-S203, a GELU nonlinear mapping layer, and two multilayer perceptron layers, and performs corresponding short connection operations.

[0065] The Transformer network constructed in this embodiment can extract features from the input data more compactly, saving the number of network model parameters.

[0066] In a preferred embodiment of the present invention, S300 is further comprising:

[0067] S301: Based on the output of the pose-oriented Transformer network structure obtained in S200 (features about the pose, which integrate global and local information), the uncertainty of each joint is estimated by using a multilayer perceptron as a linear mapping layer (uncertainty represents the accuracy of the joint in 3D position estimation; the greater the uncertainty, the less accurate the estimation).

[0068] S302: Based on the output of the pose-oriented Transformer network structure obtained according to S200, a rough 3D pose is obtained by using a multilayer perceptron as a linear mapping layer. This 3D pose is then used as the mean μ, and the joint uncertainty σ is used as the variance to obtain a Gaussian distribution. Finally, the distribution was sampled;

[0069] A rough 3D pose is obtained by using a multilayer perceptron as a linear mapping layer. The 3D pose is then used as the mean and the joint uncertainty as the variance to model a Gaussian distribution. Finally, the distribution is sampled.

[0070] S303: Based on the uncertainty estimated in S301, an uncertainty-guided attention mechanism is constructed to reduce the weight of joints with high uncertainty in the attention mechanism. Specifically, for the i-th and j-th tokens, the uncertainty-guided attention mechanism module divides the attention weight obtained from the standard multi-head attention mechanism (MH-SA) module by the uncertainty of the j-th joint, thereby making the attention weight of joints with high uncertainty lower. Using S303, the coarse 3D pose in S302 is improved and refined, and the final 3D pose result is obtained.

[0071] This embodiment can significantly improve the accuracy of pose estimation with self-occlusion and complex movements, and has strong flexibility and scalability. By introducing uncertainty, this embodiment reduces or increases the weight of related joints. This measure can refine and improve pose features, and ultimately achieve the improvement and refinement of 3D pose.

[0072] See Figure 2Based on the same inventive concept, this invention provides a three-dimensional human pose estimation system, including a multi-category embedding information acquisition module 11, a pose-oriented Transformer network construction module 12, and an uncertainty-guided pose improvement construction module 13. The multi-category embedding information acquisition module 11 acquires keypoint embedding information of different categories through different methods, enabling the network to include absolute and relative graph topological distance information in addition to 2D pose input information. The pose-oriented Transformer network construction module 12 constructs a pose-oriented Transformer network by building a pose-oriented self-attention mechanism. The sum of the different categories of embedding information for each keypoint is used as a token and input into the Transformer network for pose-related global feature extraction. The uncertainty-guided pose improvement module construction module 13 estimates the uncertainty of keypoints based on the extracted global features, implementing uncertainty-guided sampling and pose improvement operations to construct the uncertainty-guided improvement module.

[0073] The specific implementation techniques of each module / unit in the above examples of the present invention can be referred to the steps of the three-dimensional human pose estimation method in the above embodiments, and will not be repeated here.

[0074] like Figure 3 As shown, based on the same inventive concept, in other embodiments of the present invention, a three-dimensional human pose estimation application system is provided, including: a Transformer network construction module 21, a Transformer network training module 22, and a three-dimensional human pose estimation module 23. The Transformer network construction module 21 constructs a Transformer network using the three-dimensional human pose estimation method of the above embodiments; or constructs a Transformer network using the three-dimensional human pose estimation system of the above embodiments. The Transformer network training module 22 is used to train the Transformer network constructed by the Transformer network construction module 21. The three-dimensional human pose estimation module 23 is used to perform three-dimensional human pose estimation using the Transformer network trained by the Transformer network training module 22.

[0075] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the above-described three-dimensional human pose estimation method, or to run the above-described three-dimensional human pose estimation system.

[0076] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0077] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0078] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0079] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0080] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described three-dimensional human pose estimation method, or to run the above-described three-dimensional human pose estimation system.

[0081] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0082] To better understand the above parts, a detailed application example is provided below. This embodiment provides a 3D human pose estimation system that uses the Transformer network construction method described above. This embodiment executes in parallel on the input 2D human joints. Each group of 2D human joints includes multiple joints, and each joint contains its 2D coordinate information in the image coordinate system. The 3D human pose estimation system obtains the 3D coordinate information corresponding to each joint through Transformer network regression. To clearly describe the execution process, a human skeleton topology with 16 nodes is used as an example to illustrate the Transformer network structure construction method for 3D human pose estimation.

[0083] 1. Multi-category embedded information acquisition module

[0084] The purpose of the multi-class embedding information acquisition module is to ensure that the input of the Transformer contains not only the original 2D pose input information, but also the absolute and relative distance information of the joints. Specifically, for a 2D pose input image X with dimensions (16, 2), feature embedding information Z with dimensions (16, 128) is obtained through linear mapping; and keypoint embedding location information K with dimensions (16, 128) is obtained through adaptive learning; secondly, based on the shortest graph topological distance to the pelvic joints, the 16 joints are divided into 5 groups, and group-level location embedding information G with dimensions (5, 128) is obtained through adaptive learning; for each joint, the above three embedding information are added together, and the final result with dimensions (16, 128) is used as the input of the subsequent Transformer.

[0085] 2. Pose-Oriented Transformer Network Building Module

[0086] The pose-oriented Transformer network building module, based on the different categories of joint position encodings obtained in S11, further constructs a pose-oriented Transformer structure by encoding the graph topology into a self-attention mechanism, and then extracts pose-related features. The specific steps are as follows:

[0087] Step 1: Based on the topological structure of the human body graph, calculate the shortest topological distance between each pair of joints.

[0088] Step 2: Use a multilayer perceptron as a mapping function to map and encode the distance described in Step 1 to the H dimension, where H is the number of heads in the multi-head attention mechanism, which is set to 6 here.

[0089] Step 3: The encoding result from Step 2 is used as an additional bias and the output of the standard multi-head attention mechanism MH-SA is added to the attention matrix as the output of the pose-oriented attention mechanism module.

[0090] Step 4: Connect the above pose-oriented attention mechanism module in series with the multilayer perceptron and GELU nonlinear unit to construct a pose-oriented Transformer block. Stack 8 pose-oriented Transformer blocks to construct a pose-oriented Transformer network structure.

[0091] 3. Uncertainty-guided attitude improvement module construction module

[0092] The purpose of the uncertainty-guided pose improvement module is to introduce uncertainty at the joints to improve the features and rough 3D pose, thereby obtaining a more accurate robust 3D pose output.

[0093] Step 1:

[0094] Based on the output of the pose-oriented Transformer network structure obtained from the pose-oriented Transformer network building block, the uncertainty of each joint is estimated by using a multilayer perceptron as a linear mapping layer, with dimensions (16, 3).

[0095] Step Two:

[0096] Based on the output of the pose-oriented Transformer network structure obtained from the pose-oriented Transformer network building module, a coarse 3D pose with dimension (16, 3) is obtained by using a multilayer perceptron as a linear mapping layer. This 3D pose is then used as the mean, and the joint uncertainty is used as the variance to model a Gaussian distribution. Finally, this distribution is sampled to obtain a sampling result with dimension (16, 3).

[0097] Step 3:

[0098] Based on the uncertainty obtained in step one, an uncertainty-guided attention mechanism module is constructed to refine the coarse 3D pose obtained in step two. In the Transformer network training module, the input for step three is the sampling result from step two. In the 3D human pose estimation module, the input for step three is the coarse 3D pose, thus obtaining the final 3D pose result.

[0099] Implementation results:

[0100] In this embodiment, the evaluation can be divided into objective evaluation and subjective evaluation. The former includes statistical analysis of the 3D human pose estimation results to obtain indicators such as MPJPE; the latter includes visualization of the 3D human pose estimation results. This embodiment compares the pose estimation results with those of existing methods on the Human3.6M dataset, which contains multiple actions.

[0101] Experiments show that, in terms of objective evaluation, this embodiment significantly improves MPJPE compared to existing methods in cases of self-occlusion and complex movements, while also achieving an improvement in average MPJPE. Further data analysis demonstrates that, by employing the method described in this embodiment, a denser graph topology is considered in the network construction method compared to the human skeleton. This means that the neural network can capture more local information from the input 2D human joint coordinates, effectively representing the structural information of the joints even in cases of complex movements and self-occlusion in the input 2D human pose, thus significantly improving the 3D human pose estimation results for that movement.

[0102] By employing the method of this embodiment, the effect of 3D human pose estimation for complex movements and self-occlusion movements is improved, as shown in the reference. Figure 4 The visualization results are as follows: the three rows from top to bottom represent three different input actions; the leftmost column shows the 3D human pose estimation results of the traditional Transformer network structure; the middle column shows the 3D human pose estimation results of this example; and the rightmost column shows the ground truth results, which are the semantic categories annotated by the user. It can be seen that by introducing graph topological priors into the self-attention mechanism and introducing uncertainty-guided improvements, the method in this embodiment helps to significantly improve the accuracy of 3D human pose estimation for complex actions and self-occluded actions.

[0103] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0106] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0107] Those skilled in the art will understand that, except that at least some of such features and / or processes or units are mutually exclusive, any combination may be used to combine all features disclosed in this specification and all processes or units of any device so disclosed.

[0108] The embodiments disclosed herein are merely preferred embodiments of the present invention. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the present invention, and are not intended to limit the present invention. Any modifications and variations made by those skilled in the art within the scope of this specification should fall within the protection scope of the present invention.

Claims

1. A method of three-dimensional human pose estimation, the method comprising: include: By acquiring different types of keypoint embedding information through different methods, that is, while the input contains 2D pose information, absolute and relative graph topological distance information is obtained. By constructing a pose-oriented self-attention mechanism, a pose-oriented Transformer network is constructed; the sum of the different categories of embedded information of each key point is used as a token and input into the Transformer network to extract pose-related global features. Based on the output of the Transformer network, uncertainty-guided sampling and attitude improvement operations are performed by estimating the uncertainty of the joints, thereby achieving uncertainty-guided improvement. The step involves estimating the uncertainty of the joints based on the output of the Transformer network, performing uncertainty-guided sampling and attitude improvement operations, and achieving uncertainty-guided improvement, including: According to the output of the pose-oriented Transformer network structure, an uncertainty of each joint node is estimated by a multi-layer perceptron as a linear mapping layer ; Based on the output of the pose-oriented Transformer network structure, a coarse 3D pose is obtained by using a multilayer perceptron as a linear mapping layer. an uncertainty of the joint node is estimated An uncertainty-guided attention mechanism is constructed, the attention weight of the corresponding joint node is adjusted, the adjusted attention weight is used to improve and refine the rough 3D pose, and a final 3D pose result is obtained. wherein the uncertainty of the estimated joint node is utilized The uncertainty guided attention mechanism is constructed, and the attention weight of the corresponding joint node is adjusted, comprising: For the i-th and j-th tokens, the uncertainty-guided attention mechanism divides the attention weights obtained from the standard multi-head attention mechanism MH-SA by the uncertainty of the j-th key. Based on the division result, the attention weights of key points with high uncertainty are reduced, and the attention weights of key points with low uncertainty are increased.

2. The three-dimensional human pose estimation method of claim 1, wherein, The acquisition of different types of joint embedding information through different methods, i.e., obtaining absolute and relative graph topological distance information while inputting 2D pose information, includes: For a 2D pose input image containing J key points, linear mapping is performed through a fully connected layer to obtain feature embedding information of each key point X Z ;​ for J Each keypoint has its own keypoint location embedding information obtained through adaptive learning. K ; for J The pelvic joints are grouped according to the shortest graph topological distance to the pelvic joints, and the group-level position embedding information for each group is obtained through adaptive learning. G ; the feature embedding information Z、 the keypoint position embedding information K and the group-level position embedding information G addition, the final result is taken as the input of the subsequent Transformer.

3. The method of claim 1, wherein, The operations that the pose-oriented attention mechanism can perform include: Based on the topological structure of the human body graph, calculate the shortest topological distance between each pair of joints, where the distance between two adjacent joints is 1. The distance l is mapped and encoded using a multilayer perceptron as a mapping function, and the result is added to the attention matrix output by the standard multi-head attention mechanism MH-SA as an additional bias.

4. The three-dimensional human pose estimation method of claim 3, wherein, The Transformer network has a total of L layers. Each layer includes a pose-oriented attention mechanism layer, a GELU nonlinear mapping layer, and two multilayer perceptron layers, and performs corresponding short connection operations. The Transformer network is a pose-oriented network that extracts pose-related global features.

5. A three-dimensional human pose estimation system for implementing the method of any one of claims 1-4, characterized in that, include: A multi-category embedding information acquisition module acquires keypoint embedding information of different categories through different methods. That is, while the input contains 2D pose information, it obtains absolute and relative graph topological distance information. A pose-oriented Transformer network construction module is provided. This module constructs a pose-oriented Transformer network by building a pose-oriented self-attention mechanism. The sum of the different categories of embedded information at each key point is used as a token and input into the Transformer network to extract pose-related global features. An uncertainty-guided attitude improvement module is constructed. The output of the Transformer network in this module estimates the uncertainty of the joints and performs uncertainty-guided sampling and attitude improvement operations to achieve uncertainty-guided improvement.

6. A three-dimensional human pose estimation application system characterized by comprising: include: The Transformer network construction module is based on the three-dimensional human pose estimation method according to any one of claims 1-4 to construct the Transformer network, or it is constructed using the three-dimensional human pose estimation system according to claim 5. The training module for the Transformer network, which is used to train the constructed Transformer network; The 3D human pose estimation module is used to perform 3D human pose estimation using a trained Transformer network.

7. A terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-4, or to run the system of any one of claims 5-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, When executed by a processor, the program can be used to perform the method of any one of claims 1-4, or to run the system of any one of claims 5-6.

Images