A 3D human pose estimation method and system based on multi-representation fusion

Abstract

The present invention discloses a 3D human posture estimation method and system with multi-representation fusion, which belongs to the field of artificial intelligence. The steps of the method are as follows: using a 2D human posture prediction network to obtain the two-dimensional coordinate position of the key points of the human body, and inputting it into the 3D human posture prediction network, first adding the initial features obtained by the dimensionality-raising network to the learnable spatial position embedding vector to obtain a human posture input sequence, passing the human posture input sequence through N feature extraction modules in sequence to obtain human posture features, and inputting the human posture features into the multi-representation fusion prediction head to obtain the final 3D human posture prediction result. The present invention adds multi-representation fusion to the 3D human posture prediction head, so that the 3D human posture prediction network can learn the prior knowledge of the human body more easily, thereby increasing the accuracy of the 3D human posture estimation of the 3D human posture prediction network, and making a certain contribution to the development of 3D human posture estimation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a multi-representation fusion 3D human posture estimation method and system. Background Art

[0002] 3D human pose estimation, which aims to predict the 3D coordinates of key human points from images or videos, is an important computer vision task with promising applications such as augmented reality, body language interpretation, and human interaction. It has garnered considerable academic attention this year. Generally speaking, there are two basic approaches to 3D human pose estimation: directly using neural networks to predict 3D human key point coordinates from images or videos, or first using a 2D human pose predictor to predict 2D coordinates, then using a neural network to predict 3D coordinates from the 2D coordinates. Because the latter approach has a smaller prediction error, current research in academia and industry favors the latter approach.

[0003] Although there are many 3D human pose estimation methods currently, they all focus on the feature extraction module and make complex network designs for the feature extraction module, while ignoring the analysis of the 3D human pose structure itself. Summary of the Invention

[0004] The present invention aims to address the problems existing in the prior art and provide a multi-representation fusion 3D human pose estimation method and system. In the prediction head, the method not only predicts the point coordinates of the 3D human pose, but also the edge angles and lengths of the 3D human pose. This method predicts both the points and edges in the 3D human pose graph structure and fuses the two 3D human pose representations to obtain the final result.

[0005] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a 3D human pose estimation method based on multi-representation fusion, comprising the following steps:

[0007] S1. Obtain an RGB image with a human body object and feed the RGB image into a pre-trained 2D human pose prediction network to obtain the 2D coordinate positions of key points of the human body;

[0008] S2. Input the 2D coordinate positions of the key points of the human body into the trained 3D human pose prediction network. The dimensionality-increasing network first processes the 2D coordinate positions of the key points of the human body to obtain initial features. The initial features are added to the learnable spatial position embedding vector to obtain a human pose input sequence. The human pose input sequence is sequentially passed through N feature extraction modules to obtain human pose features. The human pose features are input into the multi-representation fusion prediction head to obtain the final 3D human pose prediction result.

[0009] Each feature extraction module of the 3D human pose prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses spatial graph convolution and temporal graph convolution to extract features. Finally, an adaptive fusion module is used to perform weighted fusion of the features extracted by the two parallel branches.

[0010] The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules. The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, the two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. The first 3D human posture prediction result and the second 3D human posture prediction result are weighted averaged to obtain the final 3D human posture prediction result.

[0011] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0012] Preferably, the specific process of obtaining the two-dimensional coordinate position of the key points of the human body in step S1 is: sending the RGB image into the backbone network to extract image features, then sending the extracted image features into the target detection network to obtain a human target detection frame, and then inputting the image features in the human target detection frame into the 2D human posture prediction network, and calculating the 2D human posture as the two-dimensional coordinate position of the key points of the human body in the image coordinate system.

[0013] Preferably, in step S1, the 2D human posture prediction network adopts a CPN model.

[0014] Preferably, in step S1, there are a total of 17 key points of the human body, including: hip, right hip, right knee, right ankle, left hip, left knee, left ankle, spine, chest, neck, head, left shoulder, left elbow, left wrist, right shoulder, right elbow and right wrist.

[0015] Preferably, in step S2, the total loss of the 3D human posture prediction network during training includes three types of losses, namely, MPJPE loss, L m , 3D human body posture edge angle loss L d And 3D human posture edge length loss L l , the functional form of the total loss is:

[0016] L=L m +γ0L d +γ1L l

[0017]

[0018] Among them, γ0 and γ1 are hyperparameters used to adjust the loss ratio; is the 3D human posture prediction value of the i′th human key point, is the true 3D human body posture data of the i′th human body key point; i′ represents the human body key point index; is the predicted value of the 3D human body posture edge angle of the i-th edge; is the true 3D human body posture edge angle of the i-th edge; is the predicted value of the 3D human body posture edge length of the i-th edge; is the true value of the 3D human body posture edge length of the i-th edge.

[0019] Preferably, in step S2, during training of the 3D human posture prediction network, the AdamW optimizer is used to optimize the parameters of the 3D human posture prediction network.

[0020] Preferably, in step S2, the dimension-raising network is implemented by a linear layer.

[0021] Preferably, in step S2, the calculation process of weighted fusion using the adaptive fusion module is as follows:

[0022]

[0023] Where, F n Represents the output of the nth feature extraction module; and are the features extracted from each branch respectively; ω1 and ω2 represent the fusion weights of the features of each branch.

[0024] Preferably, the fusion weight of each branch feature is calculated as follows:

[0025]

[0026] Where W is the parameter matrix of the adaptive fusion module.

[0027] In a second aspect, the present invention provides a 3D human pose estimation system based on multi-representation fusion, comprising:

[0028] The data acquisition module is used to obtain RGB images with human objects, feed the RGB images into the pre-trained 2D human posture prediction network, and obtain the two-dimensional coordinate positions of key points of the human body;

[0029] The result acquisition module is used to input the two-dimensional coordinate positions of the key points of the human body into the trained 3D human pose prediction network. The two-dimensional coordinate positions of the key points of the human body are first processed by the dimensionality-increasing network to obtain the initial features. The initial features are added to the learnable spatial position embedding vector to obtain the human pose input sequence. The human pose input sequence is sequentially passed through N feature extraction modules to obtain human pose features. The human pose features are input into the multi-representation fusion prediction head to obtain the final 3D human pose prediction result;

[0030] Each feature extraction module of the 3D human pose prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses spatial graph convolution and temporal graph convolution to extract features. Finally, an adaptive fusion module is used to perform weighted fusion of the features extracted by the two parallel branches.

[0031] The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules. The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, the two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. The first 3D human posture prediction result and the second 3D human posture prediction result are weighted averaged to obtain the final 3D human posture prediction result.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention proposes a multi-representation fusion 3D human pose estimation method and system. The multi-representation fusion method makes the prediction process of 3D human pose estimation more reasonable and robust. During the prediction process, the method of the present invention not only predicts the 3D human pose prediction results (the coordinates of the key points of the human body), but also predicts the 3D human pose edge angle and 3D human pose edge length, and supervises the three predicted values ​​at the same time, so that the 3D human pose prediction network can learn different representations of 3D human pose, improve the robustness and accuracy of the prediction, and has good application value in the industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the steps of the present invention;

[0035] Figure 2 3D human body posture prediction network structure diagram of the present invention;

[0036] Figure 3 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0038] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0039] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned multi-representation fusion 3D human pose estimation method includes the following steps S1 to S2. The specific implementation process is described in detail below.

[0040] S1. Obtain an RGB image with a human object, feed the RGB image into a pre-trained 2D human pose prediction network, and obtain the two-dimensional coordinate positions of the key points of the human body.

[0041] It should be noted that in step S1 of the present invention, the 2D human posture prediction network can be selected according to actual needs, as long as the network model can obtain the two-dimensional coordinate positions of the key points of the human body. In addition, during the training process, the weights of the 2D human posture prediction network are frozen and no longer need to be updated.

[0042] In this embodiment, if Figure 2As shown, the 2D human pose prediction network is set as a CPN model. The implementation method of this model belongs to the existing technology. Its relevant technical principles and network structure refer to the existing literature: Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7103–7112, 2018. In combination with the above-mentioned CPN model, the specific process of obtaining the two-dimensional coordinate positions of the key points of the human body in step S1 of this embodiment is as follows: the RGB image is fed into the backbone network to extract image features, and then the extracted image features are fed into the target detection network to obtain a human target detection frame, and then the image features in the human target detection frame are fed into the 2D human pose prediction network. After calculation, the 2D human pose is obtained as the two-dimensional coordinate position (2D Pose) of the key points of the human body in the image coordinate system.

[0043] It should also be noted that in step S1, for each human target, the 17 commonly used human key points in the Human3.6M dataset include: hip, right hip, right knee, right ankle, left hip, left knee, left ankle, spine, chest, neck, head, left shoulder, left elbow, left wrist, right shoulder, right elbow and right wrist.

[0044] S2. Input the 2D coordinate positions of the key points of the human body into the trained 3D human pose prediction network. The dimensionality-increasing network first processes the 2D coordinate positions of the key points of the human body to obtain initial features. The initial features are added to the learnable spatial position embedding vector to obtain a human pose input sequence. The human pose input sequence is sequentially passed through N feature extraction modules to obtain human pose features. The human pose features are input into the multi-representation fusion prediction head to obtain the final 3D human pose prediction result.

[0045] Each feature extraction module of the 3D human pose prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses spatial graph convolution and temporal graph convolution to extract features. Finally, an adaptive fusion module is used to perform weighted fusion of the features extracted by the two parallel branches.

[0046] The prediction head of the 3D human posture prediction network contains three parallel multi-layer perceptron modules (Multi-Layer Perceptron, MLP). The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, the two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. The first 3D human posture prediction result and the second 3D human posture prediction result are weighted averaged to obtain the final 3D human posture prediction result.

[0047] It should be noted that in step S2, the total loss of the 3D human posture prediction network during training includes three types of losses, namely, MPJPE loss L m , 3D human body posture edge angle loss L d And 3D human posture edge length loss L l , the functional form of the total loss is:

[0048] L=L m +γ0L d +γ1L l

[0049]

[0050] Among them, γ0 and γ1 are hyperparameters used to adjust the loss ratio; is the 3D human posture prediction value of the i′th human key point, is the true 3D human body posture data of the i′th human body key point; i′ represents the human body key point index; is the predicted value of the 3D human body posture edge angle of the i-th edge; is the true 3D human body posture edge angle of the i-th edge; is the predicted value of the 3D human body posture edge length of the i-th edge; is the true value of the 3D human body posture edge length of the i-th edge.

[0051] It should be noted that in step S2, when the 3D human posture prediction network is trained, the AdamW optimizer is used to optimize the parameters of the 3D human posture prediction network.

[0052] It should be noted that in step S2, the dimension-raising network is implemented by a linear layer.

[0053] It should be noted that in step S2, the calculation process of weighted fusion using the adaptive fusion module is as follows:

[0054]

[0055] Where, F n Represents the output of the nth feature extraction module; and are the features extracted from each branch respectively; ω1 and ω2 represent the fusion weights of the features of each branch.

[0056] Furthermore, the calculation process of the fusion weight of each branch feature is as follows:

[0057]

[0058] Where W is the parameter matrix of the adaptive fusion module.

[0059] It should be noted that in the present invention, a branch in each feature extraction module uses a spatial multi-head self-attention mechanism and a temporal multi-head self-attention mechanism to extract features. The calculation process of the temporal multi-head self-attention mechanism is similar to that of the spatial multi-head self-attention mechanism, and both implementations belong to the prior art. In this embodiment, the calculation process of the spatial multi-head self-attention mechanism is taken as an example, which is as follows:

[0060] S-MHSA(Q s ,K s ,V s )=Concat(head1,…,head i ,…,head h )W s (O)

[0061]

[0062] V s (i) =F s W s (V,i)

[0063] In the formula, S-MHSA(Q s ,K s ,V s ) represents the spatial multi-head self-attention mechanism; Concat represents concatenation; head1,…,head i ,…,head h They represent the 1st,…,i,…,hth attention heads respectively; h represents the number of preset attention heads; W s (O)Represents the learnable linear layer parameter matrix; head i represents the i-th attention head; Softmax represents the Softmax function; V s (i) denote the query representation, key representation, and value representation of the i-th attention head respectively; d k represents the key representation of the i-th attention head The feature dimension of i represents the index value of the attention head; F s is the input feature of the current feature extraction module; W s (Q,i) , W s (K,i) , W s (V,i) are the parameter matrices corresponding to the query representation, key representation, and value representation of the i-th attention head respectively.

[0064] It should be noted that in the present invention, the other branch in each feature extraction module uses spatial graph convolution and temporal graph convolution to extract features. The calculation process of temporal graph convolution is similar to that of spatial graph convolution, and both implementations belong to the prior art. In this embodiment, the calculation process of spatial graph convolution is taken as an example, and the details are as follows:

[0065]

[0066] In the formula, GCN(F s ) represents the output of spatial graph convolution; σ represents the activation function, such as the ReLU function; Norm represents the batch normalization operation; represents the adjoint matrix; represents the adjoint matrix degree matrix; W1 and W2 represent weight matrices.

[0067] In order to better demonstrate the specific implementation and technical effects of the present invention, the 3D human posture estimation method based on multi-representation fusion shown in steps S1 to S2 in the above preferred implementation is applied to a specific example.

[0068] Example

[0069] The specific implementation process of the multi-representation fusion 3D human pose estimation method used in this embodiment is as described above and will not be repeated here. Only some key steps in the implementation of the method are shown below.

[0070] Step 1: Data preprocessing to obtain data for training and inference.

[0071] The data stream involved in this embodiment is a video, so it is necessary to sample a video and take an image every 10 frames as the input RGB image;

[0072] Step 2: Get the 2D coordinate position of the key points of the human body;

[0073] Step 3: Obtain human posture input sequence;

[0074] Step 4: Input the human posture input sequence into N consecutive feature extraction modules to obtain human posture features;

[0075] Step 5: Input the human posture features into the multi-representation fusion prediction head to obtain the final 3D human posture prediction result.

[0076] The method of the present invention is applied to the Human3.6M dataset and compared with multiple existing models. The test results are shown in Table 1:

[0077] Table 1: Experimental results on the Human3.6M dataset

[0078] method Input frame number Parameter quantity P1(mm) P2(mm) P1gt(mm) MotionAGFormer 81 11.7M 42.5 35.3 26.5 Method of the present invention 81 7.6M 41.9 34.4 22.7 MotionBERT 243 42.5M 39.2 32.9 17.8

[0079] In Table 1, MotionBERT is a 3D human pose estimation model that uses Transformer as a feature extraction module, which is derived from the existing technical literature: Zhu, W., Ma, X., Liu, Z., Liu, L., Wu, W., Wang, Y.: MotionBERT: Aunified perspective on learning human motion representations. In: Proceedings of the IEEE / CVF International Conference on Computer Vision. pp. 15085–15099 (2023). MotionAGFormer is a 3D human pose estimation model that combines GCN and Transformer as feature extraction modules, which is derived from the existing technical literature Mehraban, S., Adeli, V., Taati, B.: MotionAGFormer: Enhancing 3D human pose estimation with a transformer-gcnformer network. In: Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. pp. 6920–6930 (2024).

[0080] Experiments show that the method of the present invention surpasses multiple indicators while using fewer parameters and without introducing other data sets for training.

[0081] It should also be noted that the multi-representation fusion 3D human pose estimation method in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a multi-representation fusion 3D human pose estimation system corresponding to the multi-representation fusion 3D human pose estimation method provided in the above embodiment, such as Figure 3 As shown, it includes:

[0082] The data acquisition module is used to obtain RGB images with human objects, feed the RGB images into the pre-trained 2D human posture prediction network, and obtain the two-dimensional coordinate positions of key points of the human body;

[0083] The result acquisition module is used to input the two-dimensional coordinate positions of the key points of the human body into the trained 3D human pose prediction network. The two-dimensional coordinate positions of the key points of the human body are first processed by the dimensionality-increasing network to obtain the initial features. The initial features are added to the learnable spatial position embedding vector to obtain the human pose input sequence. The human pose input sequence is sequentially passed through N feature extraction modules to obtain human pose features. The human pose features are input into the multi-representation fusion prediction head to obtain the final 3D human pose prediction result;

[0084] Each feature extraction module of the 3D human pose prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses spatial graph convolution and temporal graph convolution to extract features. Finally, an adaptive fusion module is used to perform weighted fusion of the features extracted by the two parallel branches.

[0085] The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules. The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, the two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. The first 3D human posture prediction result and the second 3D human posture prediction result are weighted averaged to obtain the final 3D human posture prediction result.

[0086] It should also be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0087] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A 3D human pose estimation method based on multi-representation fusion, characterized in that: The following steps are involved: S1. Obtain an RGB image with a human body object and feed the RGB image into a pre-trained 2D human pose prediction network to obtain the 2D coordinate positions of key points of the human body; S2. Input the 2D coordinate positions of the key points of the human body into the trained 3D human pose prediction network. The dimensionality-increasing network first processes the 2D coordinate positions of the key points of the human body to obtain initial features. The initial features are added to the learnable spatial position embedding vector to obtain a human pose input sequence. The human pose input sequence is sequentially passed through N feature extraction modules to obtain human pose features. The human pose features are input into the multi-representation fusion prediction head to obtain the final 3D human pose prediction result. Each feature extraction module of the 3D human pose prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses spatial graph convolution and temporal graph convolution to extract features. Finally, an adaptive fusion module is used to perform weighted fusion of the features extracted by the two parallel branches. The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules. The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, the two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. The first 3D human posture prediction result and the second 3D human posture prediction result are weighted averaged to obtain the final 3D human posture prediction result.

2. The 3D human pose estimation method based on multi-representation fusion according to claim 1, wherein: The specific process of obtaining the two-dimensional coordinate positions of the key points of the human body in step S1 is as follows: the RGB image is sent to the backbone network to extract image features, and then the extracted image features are sent to the target detection network to obtain a human target detection frame, and then the image features in the human target detection frame are input into the 2D human posture prediction network. After calculation, the 2D human posture is obtained as the two-dimensional coordinate positions of the key points of the human body in the image coordinate system.

3. The 3D human pose estimation method based on multi-representation fusion according to claim 1, wherein: In step S1, the 2D human posture prediction network adopts a CPN model.

4. The 3D human pose estimation method based on multi-representation fusion according to claim 1, wherein: In step S1, there are a total of 17 key points of the human body, including: hip, right hip, right knee, right ankle, left hip, left knee, left ankle, spine, chest, neck, head, left shoulder, left elbow, left wrist, right shoulder, right elbow and right wrist.

5. The 3D human pose estimation method based on multi-representation fusion according to claim 4, wherein: In step S2, the total loss of the 3D human posture prediction network during training includes three types of losses, namely, MPJPE loss L m , 3D human body posture edge angle loss L d And 3D human posture edge length loss L l , the functional form of the total loss is: L=L m +γ0L d +γ1L l Among them, γ0 and γ1 are hyperparameters used to adjust the loss ratio; is the 3D human posture prediction value of the i′th human key point, is the true 3D human body posture data of the i′th human body key point; i′ represents the human body key point index; is the predicted value of the 3D human body posture edge angle of the i-th edge; is the true 3D human body posture edge angle of the i-th edge; is the predicted value of the 3D human body posture edge length of the i-th edge; is the true value of the 3D human body posture edge length of the i-th edge.

6. The 3D human pose estimation method based on multi-representation fusion according to claim 1, wherein: In step S2, during training, the 3D human posture prediction network is optimized using the AdamW optimizer.

7. The 3D human pose estimation method based on multi-representation fusion according to claim 1, wherein: In step S2, the dimension-raising network is implemented by a linear layer.

8. The 3D human pose estimation method based on multi-representation fusion according to claim 1, wherein: In step S2, the calculation process of weighted fusion using the adaptive fusion module is as follows: Where, F n Represents the output of the nth feature extraction module; and are the features extracted from each branch respectively; ω1 and ω2 represent the fusion weights of the features of each branch.

9. The 3D human pose estimation method based on multi-representation fusion according to claim 8, wherein: The calculation process of the fusion weight of each branch feature is as follows: Where W is the parameter matrix of the adaptive fusion module.

10. A 3D human pose estimation system based on multi-representation fusion, characterized in that: include: The data acquisition module is used to obtain RGB images with human objects, feed the RGB images into the pre-trained 2D human posture prediction network, and obtain the two-dimensional coordinate positions of key points of the human body; The result acquisition module is used to input the two-dimensional coordinate positions of the key points of the human body into the trained 3D human pose prediction network. The two-dimensional coordinate positions of the key points of the human body are first processed by the dimensionality-increasing network to obtain the initial features. The initial features are added to the learnable spatial position embedding vector to obtain the human pose input sequence. The human pose input sequence is sequentially passed through N feature extraction modules to obtain human pose features. The human pose features are input into the multi-representation fusion prediction head to obtain the final 3D human pose prediction result; Each feature extraction module of the 3D human pose prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses spatial graph convolution and temporal graph convolution to extract features. Finally, an adaptive fusion module is used to perform weighted fusion of the features extracted by the two parallel branches. The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules. The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, the two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. The first 3D human posture prediction result and the second 3D human posture prediction result are weighted averaged to obtain the final 3D human posture prediction result.

Images