Multi-representation fused 3D human body posture estimation method and system

Abstract

The invention discloses a multi-representation fusion 3D human body posture estimation method and system, and belongs to the field of artificial intelligence. The method comprises the following steps: acquiring two-dimensional coordinate positions of key points of a human body by using a 2D human body posture prediction network, inputting the two-dimensional coordinate positions into a 3D human body posture prediction network, firstly adding initial features acquired by a dimension raising network and learnable spatial position embedded vectors to obtain a human body posture input sequence, and finally, performing 3D human body posture prediction on the human body posture input sequence; enabling the human body posture input sequence to sequentially pass through N feature extraction modules to obtain human body posture features, and inputting the human body posture features into a multi-representation fusion prediction head to obtain a final 3D human body posture prediction result. According to the method, multi-representation fusion is added into the 3D human body posture prediction head, so that the 3D human body posture prediction network can learn human body priori knowledge more easily, the accuracy of 3D human body posture estimation of the 3D human body posture prediction network is further improved, and a certain contribution is made to the development of 3D human body posture estimation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a multi-representation fusion 3D human body posture estimation method and system. Background Art

[0002] 3D human pose estimation aims to predict the 3D coordinates of key points of the human body from images or videos. It is an important computer vision task with great application prospects, such as augmented reality, body language interpretation, human interaction, etc. It has received much attention from the academic community this year. In general, there are two basic routes for 3D human pose estimation: one is to directly use neural networks to predict the coordinates of 3D human key points from images or videos, and the other is to first use a 2D human pose predictor to predict 2D coordinates, and then use a neural network to predict 3D coordinates from 2D coordinates. Since the second method has a smaller prediction error, current research in academia and industry tends to favor the second method.

[0003] Although there are many 3D human pose estimation methods currently, they all focus on the feature extraction module and make complex network designs for the feature extraction module, while ignoring the analysis of the 3D human pose structure itself. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and provide a 3D human posture estimation method and system with multi-representation fusion. The method of the present invention not only predicts the point coordinates of the 3D human posture in the prediction head, but also predicts the side angles and side lengths of the 3D human posture, predicts both the points and the sides in the 3D human posture graph structure, and fuses the two 3D human posture representations to obtain the final result.

[0005] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a 3D human body posture estimation method based on multi-representation fusion, comprising the following steps:

[0007] S1. Obtain an RGB image with a human body object, and send the RGB image to a pre-trained 2D human posture prediction network to obtain the two-dimensional coordinate positions of key points of the human body;

[0008] S2. Input the two-dimensional coordinate positions of the key points of the human body into the trained 3D human posture prediction network. First, the two-dimensional coordinate positions of the key points of the human body are processed by the dimensionality-increasing network to obtain the initial features. The initial features are added to the learnable spatial position embedding vector to obtain the human posture input sequence. The human posture input sequence is sequentially passed through N feature extraction modules to obtain the human posture features. The human posture features are input into the multi-representation fusion prediction head to obtain the final 3D human posture prediction result.

[0009] Each feature extraction module of the 3D human posture prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses the spatial graph convolution and the temporal graph convolution to extract features. Finally, the adaptive fusion module is used to perform weighted fusion on the features extracted by the two parallel branches.

[0010] The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules. The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. After weighted averaging the first 3D human posture prediction result and the second 3D human posture prediction result, the final 3D human posture prediction result is obtained.

[0011] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0012] Preferably, the specific process of obtaining the two-dimensional coordinate position of the key points of the human body in step S1 is: sending the RGB image to the backbone network to extract image features, and then sending the extracted image features to the target detection network to obtain a human target detection frame, and then inputting the image features in the human target detection frame into the 2D human posture prediction network, and calculating the 2D human posture as the two-dimensional coordinate position of the key points of the human body in the image coordinate system.

[0013] Preferably, in step S1, the 2D human posture prediction network adopts a CPN model.

[0014] Preferably, in step S1, there are a total of 17 key points of the human body, including: hips, right hip, right knee, right ankle, left hip, left knee, left ankle, spine, chest, neck, head, left shoulder, left elbow, left wrist, right shoulder, right elbow and right wrist.

[0015] Preferably, in step S2, the total loss of the 3D human posture prediction network during training includes three types of losses, namely, MPJPE loss, L m , 3D human body posture edge angle loss L d And 3D human posture edge length loss L l , the functional form of the total loss is:

[0016] L=L m +γ0L d +γ1L l

[0017]

[0018] Among them, γ0 and γ1 are hyperparameters used to adjust the loss ratio; is the 3D human body posture prediction value of the i′th human body key point, is the real 3D human body posture data of the i′th human body key point; i′ represents the human body key point index; is the predicted value of the 3D human body posture edge angle of the i-th edge; is the true 3D human body posture edge angle of the i-th edge; is the predicted value of the 3D human body posture edge length of the i-th edge; is the true value of the 3D human body posture edge length of the i-th edge.

[0019] Preferably, in step S2, when the 3D human posture prediction network is trained, an AdamW optimizer is used to optimize the parameters of the 3D human posture prediction network.

[0020] Preferably, in step S2, the dimension-increasing network is implemented by a linear layer.

[0021] Preferably, in step S2, the calculation process of weighted fusion using the adaptive fusion module is as follows:

[0022]

[0023] In the formula, F n Represents the output of the nth feature extraction module; and are the features extracted from each branch respectively; ω1 and ω2 represent the fusion weights of the features of each branch.

[0024] Preferably, the calculation process of the fusion weight of the features of each branch is as follows:

[0025]

[0026] Where W is the parameter matrix of the adaptive fusion module.

[0027] In a second aspect, the present invention provides a multi-representation fusion 3D human posture estimation system, comprising:

[0028] The data acquisition module is used to acquire RGB images with human objects, and send the RGB images to the pre-trained 2D human posture prediction network to obtain the two-dimensional coordinate positions of key points of the human body;

[0029] The result acquisition module is used to input the two-dimensional coordinate positions of the key points of the human body into the trained 3D human posture prediction network. First, the two-dimensional coordinate positions of the key points of the human body are processed by the dimension-raising network to obtain the initial features, and the initial features are added to the learnable spatial position embedding vector to obtain the human posture input sequence. The human posture input sequence is sequentially passed through N feature extraction modules to obtain the human posture features, and the human posture features are input into the multi-representation fusion prediction head to obtain the final 3D human posture prediction result;

[0030] Each feature extraction module of the 3D human posture prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses the spatial graph convolution and the temporal graph convolution to extract features. Finally, the adaptive fusion module is used to perform weighted fusion on the features extracted by the two parallel branches.

[0031] The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules. The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. After weighted averaging the first 3D human posture prediction result and the second 3D human posture prediction result, the final 3D human posture prediction result is obtained.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention proposes a 3D human posture estimation method and system with multi-representation fusion, which uses the multi-representation fusion method to make the prediction process of 3D human posture estimation more reasonable and robust. In the prediction process, the method of the present invention not only predicts the 3D human posture prediction results (coordinates of key points of the human body), but also predicts the 3D human posture edge angle and 3D human posture edge length, and supervises the three predicted values ​​at the same time, so that the 3D human posture prediction network can learn different representations of 3D human posture, improve the robustness and accuracy of the prediction, and has good application value in the industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the steps of the present invention;

[0035] Figure 2 Schematic diagram of the 3D human body posture prediction network structure of the present invention;

[0036] Figure 3 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0038] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0039] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned multi-representation fusion 3D human posture estimation method includes the following steps S1 to S2. The specific implementation process is described in detail below.

[0040] S1. Obtain an RGB image with a human object, send the RGB image to a pre-trained 2D human posture prediction network, and obtain the two-dimensional coordinate positions of key points of the human body.

[0041] It should be noted that in step S1 of the present invention, the 2D human posture prediction network can be selected according to actual needs, as long as the network model that can obtain the two-dimensional coordinate positions of the key points of the human body can be used. In addition, during the training process, the weights of the 2D human posture prediction network are frozen and no longer need to be updated.

[0042] In this embodiment, if Figure 2As shown, the 2D human pose prediction network is set as a CPN model. The implementation method of the model belongs to the prior art. Its related technical principles and network structure refer to the existing literature: Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7103–7112, 2018. In combination with the above CPN model, the specific process of obtaining the two-dimensional coordinate position of the key points of the human body in step S1 in this embodiment is: sending the RGB image to the backbone network to extract image features, and then sending the extracted image features to the target detection network to obtain a human target detection frame, and then inputting the image features in the human target detection frame into the 2D human pose prediction network, and the 2D human pose is calculated as the two-dimensional coordinate position (2D Pose) of the key points of the human body in the image coordinate system.

[0043] It should also be noted that in step S1, for each human target, the 17 commonly used human key points in the Human3.6M dataset specifically include: hip, right hip, right knee, right ankle, left hip, left knee, left ankle, spine, chest, neck, head, left shoulder, left elbow, left wrist, right shoulder, right elbow and right wrist.

[0044] S2. Input the two-dimensional coordinate positions of the key points of the human body into the trained 3D human posture prediction network. First, the two-dimensional coordinate positions of the key points of the human body are processed by the dimensionality-increasing network to obtain the initial features. The initial features are added to the learnable spatial position embedding vector to obtain the human posture input sequence. The human posture input sequence is sequentially passed through N feature extraction modules to obtain the human posture features. The human posture features are input into the multi-representation fusion prediction head to obtain the final 3D human posture prediction result.

[0045] Each feature extraction module of the 3D human posture prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses the spatial graph convolution and the temporal graph convolution to extract features. Finally, the adaptive fusion module is used to perform weighted fusion on the features extracted by the two parallel branches.

[0046] The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules (Multi-Layer Perceptron, MLP). The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, the two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. The first 3D human posture prediction result and the second 3D human posture prediction result are weighted averaged to obtain the final 3D human posture prediction result.

[0047] It should be noted that in step S2, the total loss of the 3D human posture prediction network during training includes three types of losses, namely, MPJPE loss, L m , 3D human body posture edge angle loss L d And 3D human posture edge length loss L l , the functional form of the total loss is:

[0048] L=L m +γ0L d +γ1L l

[0049]

[0050] Among them, γ0 and γ1 are hyperparameters used to adjust the loss ratio; is the 3D human body posture prediction value of the i′th human body key point, is the real 3D human body posture data of the i′th human body key point; i′ represents the human body key point index; is the predicted value of the 3D human body posture edge angle of the i-th edge; is the true 3D human body posture edge angle of the i-th edge; is the predicted value of the 3D human body posture edge length of the i-th edge; is the true value of the 3D human body posture edge length of the i-th edge.

[0051] It should be noted that in step S2, when the 3D human posture prediction network is trained, the AdamW optimizer is used to optimize the parameters of the 3D human posture prediction network.

[0052] It should be noted that in step S2, the dimension-enhancing network is implemented by a linear layer.

[0053] It should be noted that in step S2, the calculation process of weighted fusion using the adaptive fusion module is as follows:

[0054]

[0055] In the formula, F n Represents the output of the nth feature extraction module; and are the features extracted from each branch respectively; ω1 and ω2 represent the fusion weights of the features of each branch.

[0056] Furthermore, the calculation process of the fusion weight of each branch feature is as follows:

[0057]

[0058] Where W is the parameter matrix of the adaptive fusion module.

[0059] It should be noted that, in the present invention, a branch in each feature extraction module uses a spatial multi-head self-attention mechanism and a temporal multi-head self-attention mechanism to extract features. The calculation process of the temporal multi-head self-attention mechanism is similar to that of the spatial multi-head self-attention mechanism, and both implementation methods belong to the prior art. In this embodiment, the calculation process of the spatial multi-head self-attention mechanism is taken as an example, which is as follows:

[0060] S-MHSA(Q s ,K s ,V s )=Concat(head1,…,head i ,…,head h )W s (O)

[0061]

[0062] V s (i) =F s W s (V,i)

[0063] In the formula, S-MHSA(Q s ,K s ,V s ) represents the spatial multi-head self-attention mechanism; Concat represents concatenation; head1,…,head i ,…,head h denote the 1st,…,i,…,hth attention heads respectively; h denotes the preset number of attention heads; W s (O)Represents the learnable linear layer parameter matrix; head i represents the i-th attention head; Softmax represents the Softmax function; V s (i) denote the query representation, key representation, and value representation of the i-th attention head respectively; d k represents the key representation of the i-th attention head The feature dimension of ; i represents the index value of the attention head; F s is the input feature of the current feature extraction module; W s (Q,i) , W s (K,i) , W s (V,i) are the parameter matrices corresponding to the query representation, key representation, and value representation of the i-th attention head respectively.

[0064] It should be noted that in the present invention, another branch in each feature extraction module uses spatial graph convolution and temporal graph convolution to extract features. The calculation process of temporal graph convolution is similar to that of spatial graph convolution, and both implementation methods belong to the prior art. In this embodiment, the calculation process of spatial graph convolution is taken as an example, which is as follows:

[0065]

[0066] In the formula, GCN(F s ) represents the output of spatial graph convolution; σ represents the activation function, such as the ReLU function; Norm represents the batch normalization operation; represents the adjoint matrix; Denotes the adjoint matrix The degree matrix of ; W1 and W2 represent the weight matrices.

[0067] In order to better demonstrate the specific implementation and technical effects of the present invention, a 3D human posture estimation method using multiple representation fusion as shown in steps S1 to S2 in the above preferred implementation is applied to a specific example.

[0068] Example

[0069] The specific implementation process of the multi-representation fusion 3D human posture estimation method used in this embodiment is as described above and will not be repeated here. Only some key steps in the implementation of the method are shown below.

[0070] Step 1: Data preprocessing to obtain data for training and inference.

[0071] The data stream involved in this embodiment is a video, so it is necessary to sample a section of the video, taking an image as the input RGB image every 10 frames;

[0072] Step 2: Get the 2D coordinate position of the key points of the human body;

[0073] Step 3: Obtain human body posture input sequence;

[0074] Step 4: Input the human posture input sequence into N consecutive feature extraction modules to obtain human posture features;

[0075] Step 5: Input the human posture features into the multi-representation fusion prediction head to obtain the final 3D human posture prediction result.

[0076] The method of the present invention is applied to the Human3.6M dataset and compared with multiple existing models. The test results are shown in Table 1:

[0077] Table 1: Experimental results on the Human3.6M dataset

[0078] method Input Frames Parameter quantity P1(mm) P2(mm) P1gt(mm) MotionAGFormer 81 11.7M 42.5 35.3 26.5 Method of the present invention 81 7.6M 41.9 34.4 22.7 MotionBERT 243 42.5M 39.2 32.9 17.8

[0079] In Table 1, MotionBERT is a 3D human pose estimation model that uses Transformer as a feature extraction module, which is derived from the prior art literature: Zhu, W., Ma, X., Liu, Z., Liu, L., Wu, W., Wang, Y.: Motionbert: Aunified perspective on learning human motion representations. In: Proceedings of the IEEE / CVF International Conference on Computer Vision. pp. 15085–15099 (2023). MotionAGFormer is a 3D human pose estimation model that combines GCN and Transformer as feature extraction modules, which is derived from the prior art literature Mehraban, S., Adeli, V., Taati, B.: Motionagformer: Enhancing 3D human pose estimation with a transformer-gcnformer network. In: Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision. pp. 6920–6930 (2024).

[0080] Experiments show that the method of the present invention surpasses multiple indicators while using fewer parameters and without introducing other data sets for training.

[0081] It should also be noted that the multi-representation fusion 3D human posture estimation method in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a multi-representation fusion 3D human posture estimation system corresponding to the multi-representation fusion 3D human posture estimation method provided in the above embodiment, such as Figure 3 As shown, it includes:

[0082] The data acquisition module is used to acquire RGB images with human objects, and send the RGB images to the pre-trained 2D human posture prediction network to obtain the two-dimensional coordinate positions of key points of the human body;

[0083] The result acquisition module is used to input the two-dimensional coordinate positions of the key points of the human body into the trained 3D human posture prediction network. First, the two-dimensional coordinate positions of the key points of the human body are processed by the dimension-raising network to obtain the initial features, and the initial features are added to the learnable spatial position embedding vector to obtain the human posture input sequence. The human posture input sequence is sequentially passed through N feature extraction modules to obtain the human posture features, and the human posture features are input into the multi-representation fusion prediction head to obtain the final 3D human posture prediction result;

[0084] Each feature extraction module of the 3D human posture prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses the spatial graph convolution and the temporal graph convolution to extract features. Finally, the adaptive fusion module is used to perform weighted fusion on the features extracted by the two parallel branches.

[0085] The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules. The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. After weighted averaging the first 3D human posture prediction result and the second 3D human posture prediction result, the final 3D human posture prediction result is obtained.

[0086] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0087] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A 3D human posture estimation method based on multi-representation fusion, characterized in that: The following steps are involved: S1. Obtain an RGB image with a human body object, and send the RGB image to a pre-trained 2D human posture prediction network to obtain the two-dimensional coordinate positions of key points of the human body; S2. Input the two-dimensional coordinate positions of the key points of the human body into the trained 3D human posture prediction network. First, the two-dimensional coordinate positions of the key points of the human body are processed by the dimensionality-increasing network to obtain the initial features. The initial features are added to the learnable spatial position embedding vector to obtain the human posture input sequence. The human posture input sequence is sequentially passed through N feature extraction modules to obtain the human posture features. The human posture features are input into the multi-representation fusion prediction head to obtain the final 3D human posture prediction result. Each feature extraction module of the 3D human posture prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses the spatial graph convolution and the temporal graph convolution to extract features. Finally, the adaptive fusion module is used to perform weighted fusion on the features extracted by the two parallel branches. The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules. The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. After weighted averaging the first 3D human posture prediction result and the second 3D human posture prediction result, the final 3D human posture prediction result is obtained.

2. The 3D human body posture estimation method based on multi-representation fusion as claimed in claim 1, characterized in that: The specific process of obtaining the two-dimensional coordinate position of the key points of the human body in step S1 is: sending the RGB image to the backbone network to extract image features, then sending the extracted image features to the target detection network to obtain a human target detection frame, and then inputting the image features in the human target detection frame into the 2D human posture prediction network, and calculating the 2D human posture as the two-dimensional coordinate position of the key points of the human body in the image coordinate system.

3. The 3D human body posture estimation method based on multi-representation fusion as claimed in claim 1, characterized in that: In step S1, the 2D human posture prediction network adopts a CPN model.

4. The 3D human body posture estimation method based on multi-representation fusion as claimed in claim 1, characterized in that: In step S1, there are 17 key points of the human body, including: hip, right hip, right knee, right ankle, left hip, left knee, left ankle, spine, chest, neck, head, left shoulder, left elbow, left wrist, right shoulder, right elbow and right wrist.

5. The 3D human body posture estimation method based on multi-representation fusion as claimed in claim 4, characterized in that: In step S2, the total loss of the 3D human posture prediction network during training includes three types of losses, namely, MPJPE loss, L m , 3D human body posture edge angle loss L d And 3D human posture edge length loss L l , the functional form of the total loss is: L=L m +γ0L d +γ1L l Among them, γ0 and γ1 are hyperparameters used to adjust the loss ratio; is the 3D human body posture prediction value of the i′th human body key point, is the real 3D human body posture data of the i′th human body key point; i′ represents the human body key point index; is the predicted value of the 3D human body posture edge angle of the i-th edge; is the true 3D human body posture edge angle of the i-th edge; is the predicted value of the 3D human body posture edge length of the i-th edge; is the true value of the 3D human body posture edge length of the i-th edge.

6. The 3D human body posture estimation method based on multi-representation fusion as claimed in claim 1, characterized in that: In step S2, when the 3D human posture prediction network is trained, the AdamW optimizer is used to optimize the parameters of the 3D human posture prediction network.

7. The 3D human body posture estimation method based on multi-representation fusion as claimed in claim 1, characterized in that: In step S2, the dimension-raising network is implemented by a linear layer.

8. The 3D human body posture estimation method based on multi-representation fusion as claimed in claim 1, characterized in that: In step S2, the calculation process of weighted fusion using the adaptive fusion module is as follows: In the formula, F n Represents the output of the nth feature extraction module; and are the features extracted from each branch respectively; ω1 and ω2 represent the fusion weights of the features of each branch.

9. The method for 3D human posture estimation based on multi-representation fusion as claimed in claim 8, characterized in that: The calculation process of the fusion weight of each branch feature is as follows: Where W is the parameter matrix of the adaptive fusion module.

10. A 3D human posture estimation system based on multi-representation fusion, characterized in that: include: The data acquisition module is used to acquire RGB images with human objects, and send the RGB images to the pre-trained 2D human posture prediction network to obtain the two-dimensional coordinate positions of key points of the human body; The result acquisition module is used to input the two-dimensional coordinate positions of the key points of the human body into the trained 3D human posture prediction network. First, the two-dimensional coordinate positions of the key points of the human body are processed by the dimension-raising network to obtain the initial features, and the initial features are added to the learnable spatial position embedding vector to obtain the human posture input sequence. The human posture input sequence is sequentially passed through N feature extraction modules to obtain the human posture features, and the human posture features are input into the multi-representation fusion prediction head to obtain the final 3D human posture prediction result; Each feature extraction module of the 3D human posture prediction network uses two parallel branches to extract features. One branch uses the spatial multi-head self-attention mechanism and the temporal multi-head self-attention mechanism to extract features, and the other branch uses the spatial graph convolution and the temporal graph convolution to extract features. Finally, the adaptive fusion module is used to perform weighted fusion on the features extracted by the two parallel branches. The prediction head of the 3D human posture prediction network includes three parallel multi-layer perceptron modules. The first multi-layer perceptron module is used to convert the input human posture features into the first 3D human posture prediction result, the second multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side length, and the third multi-layer perceptron module is used to convert the input human posture features into the 3D human posture side angle. After multiplying the 3D human posture side length and the 3D human posture side angle to obtain the 3D human posture side vector, two adjacent 3D human posture side vectors are connected in sequence according to the index of the 3D human posture side vector to obtain the second 3D human posture prediction result. After weighted averaging the first 3D human posture prediction result and the second 3D human posture prediction result, the final 3D human posture prediction result is obtained.

Images