Monocular vision three-dimensional human body posture estimation method and device fusing multi-joint features

Abstract

The invention provides a monocular vision three-dimensional human body posture estimation method and device fusing multi-joint features, and belongs to the technical field of digital image processing, and the method comprises the steps: carrying out the joint classification of an information enhancement code, and obtaining a joint group; performing multi-branch feature extraction on the joint group; two-dimensional coordinates of a middle frame in the two-dimensional coordinate sequence of the human body joint points are taken, and overall spatial position information of the two-dimensional coordinates of the middle frame is extracted; fusing the joint motion features of the multiple branches and the overall spatial position information to obtain fused features; and performing pose estimation on the fusion feature to obtain an estimation result. According to the method, a multi-component feature extraction framework is designed for time sequence redundancy, and then joints are grouped and recombined for the problem that local information and global information of the joints are not fully fused. And finally, aiming at the problem that the unreasonable attitude is predicted, the joint information of the middle frame is used as a constraint, so that the prediction accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image processing, and in particular relates to a method and device for estimating three-dimensional human body posture by monocular vision integrating multi-joint features. Background Art

[0002] In the field of computer vision, 3D human pose estimation is one of the important research topics in this field. It mainly uses the information provided in pictures or videos to reproduce the position and dynamic changes of the human body in 3D space. This technology has a wide range of applications, such as virtual reality, action recognition, motion analysis, etc. Traditional methods often rely on the assumption of joint mobility and estimate the 3D pose of the human body from its 2D projection by manually selecting parameters. Due to the lack of depth information from the monocular camera, traditional methods cannot accurately estimate the 3D pose of the human body. With the development of deep learning technology, convolutional neural networks can be used to estimate the 3D pose of the human body directly from RGB images without the need for intermediate supervision, and more accurate results are achieved compared with traditional methods. Since predicting the 3D pose of the human body directly from the image requires a high computational cost, this task is decomposed into two steps: 1) Using a 2D pose detector to detect the position of the 2D joint points of the human body in the image space; 2) Using the correspondence between the 2D and 3D human body structures, the 2D joint point coordinates are mapped to the 3D space. Benefiting from the improvement of the accuracy of existing 2D key point detection, most of the related methods from 2D to 3D in recent years are superior to end-to-end methods. Therefore, the application focuses on the second step of this task, with the aim of improving the accuracy of 3D human pose prediction.

[0003] In the second step, some methods choose to use a small number of frames or even use the information of a single frame to estimate the 3D pose of the human body. This method ignores the temporal information of the input sequence. Some methods use temporal information to solve the problem of depth ambiguity. Some scholars have proposed a sequence-to-sequence learning model LSTM (Long Short-Term Memory), which encodes the 2D sequence of human bodies in the video into a vector of fixed size and decodes it into a 3D pose sequence, but the output pose will shift on a longer sequence. Some scholars have also used matrix decomposition to deal with the offset problem in the LSTM model, but this method has high computational complexity. Some scholars have also proposed a temporal convolutional network TCN (Temporal Convolutional Network) based on the CNN architecture, which uses a stride design to extract the characteristics of the input in the temporal dimension, but this method does not consider the independence between different joints. Some scholars have applied the attention mechanism to TCN, but only weighted the intermediate layer tensors, only emphasized the temporal correlation of joints in different frames, lacked consideration of the integrity of joints in the same frame, and ignored the spatial information of the intermediate frames. Other scholars decomposed the posture estimation task into predicting bone length and bone direction to improve the performance of TCN, but lacked prior knowledge of human body structure. In order to enhance the robustness of the network in the time domain, some scholars masked part of the input sequence during the encoding stage, but adjacent postures are highly similar, so a high masking ratio is required, and too high a masking ratio will reduce the performance of the model. Other scholars proposed a cross-spatiotemporal transformer that can capture the spatiotemporal dependencies of joint movements at different time points to obtain better spatiotemporal feature representation of joints. Other scholars grouped joints with similar motion patterns and calculated the temporal correlation of joint movements within the group, but ignored the overall spatial information of the joints. Other scholars introduced two prior attention modules and used known human topological structures and motion trajectory information to promote multi-head attention to learn the characteristics of global motion changes in human posture.

[0004] Some methods use the spatial information of human joints to capture changes in human joints. Some scholars designed UGCN based on GCN to capture the short-term and long-term changes in human posture. Other scholars proposed a segmentation and recombination network to divide the human body into multiple joint groups and reassemble them in low-dimensional space, but the joint groups lack mutual connections. Some scholars have proposed a new bone directed graph representation method that can better represent the hierarchical characteristics of the human skeleton. Some scholars use Transformer encoders to model the mutual connections between human vertices-vertices and vertices-joints, and capture the non-local dependencies of human joints through self-attention mechanisms. Some scholars have proposed MHFormer, which uses spatial transformation encoders to predict multiple human postures and then aggregates multiple postures into the final 3D posture. Some scholars use graph structures to represent the relationship between human joint points, and then correspond the graph structure to the MLP model to fuse the local and global spatial features of the joints, but do not use the temporal information in the 2D sequence of the human body. Some scholars convert 2D joint points from pixel space to 3D standardized rays. At the same time, the camera extrinsic parameters are taken as input, and the 3D normalized rays and the camera extrinsic parameters are jointly modeled, but the performance of this model will degrade when the human body size varies greatly.

[0005] Some methods model the spatial and temporal information of human posture sequences simultaneously to maintain the consistency of human posture input and output. Other scholars have constructed a set of spatiotemporal graph convolutions on human posture sequences, performing convolutions between adjacent joints, but lacking consideration of the connection between non-adjacent joints. Other scholars have proposed a graph attention spatiotemporal convolutional network consisting of temporal convolutions and graph attention blocks. By using graph convolutions of different scales, the network can capture spatial and temporal features at different levels. In addition, some scholars have constructed a PoseFormer model that connects multiple spatial transform encoders and temporal transform encoders in series, which can sequentially capture the temporal and spatial dependencies between joints, but when the number of frames is increased to obtain high accuracy, it will generate a huge computational burden. Other scholars use a sequence-to-sequence method to estimate the 3D poses of all input frames at the same time to improve the consistency between input and output sequences. Other scholars have proposed a new architecture of ConvFormer, which introduces a new dynamic multi-head convolution self-attention mechanism that can dynamically aggregate queries, keys, and values ​​at different scales to form a richer feature representation. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present application proposes a monocular vision 3D human posture estimation method and device that integrates multi-joint features.

[0007] In the first aspect, the present application proposes a monocular vision 3D human posture estimation method integrating multi-joint features, comprising:

[0008] Obtain the two-dimensional coordinate sequence of human joint points;

[0009] Performing position information encoding and time information encoding on the two-dimensional coordinate sequence of the human body joint points;

[0010] The two-dimensional coordinate sequence, position information code and time information code of the human joint point are spliced ​​to obtain information enhancement code;

[0011] The information enhancement code is used to classify joints to obtain joint groups;

[0012] Perform multi-branch feature extraction on the joint group to obtain multi-branch joint motion features;

[0013] Taking the two-dimensional coordinates of the intermediate frame in the two-dimensional coordinate sequence of the human body joint points, and extracting the overall spatial position information of the two-dimensional coordinates of the intermediate frame;

[0014] The joint motion features of multiple branches and the overall spatial position information are fused to obtain fused features;

[0015] The pose estimation is performed on the fused features to obtain a monocular vision three-dimensional human pose estimation result.

[0016] The two-dimensional coordinate sequence of the human body joint points includes: the two-dimensional coordinates of the root joint, the two-dimensional coordinates of the intermediate frame and the two-dimensional coordinates of other frames.

[0017] The position information encoding is performed on the two-dimensional coordinate sequence of the human body joint points by using the difference between the two-dimensional coordinates of other frames and the two-dimensional coordinates of the root joint.

[0018] The time information encoding performs time information encoding on the two-dimensional coordinate sequence of the human joint points by using the difference between the two-dimensional coordinates of the intermediate frame and the two-dimensional coordinates of all the human joint points.

[0019] The multi-branch feature extraction is performed on the joint group to obtain multi-branch joint motion features, including:

[0020] Inputting the joint group into a first branch to obtain a first branch joint motion feature, wherein the first branch sequentially comprises: a first convolutional layer, a first normalization layer, a first average pooling layer, and a second normalization layer;

[0021] Inputting the joint group into the second branch to obtain the second branch joint motion feature, wherein the second branch sequentially includes: a second convolutional layer and a third normalization layer;

[0022] Inputting the joint group into the third branch to obtain the third branch joint motion feature, wherein the third branch sequentially includes: a third convolution layer, a fourth normalization layer, a fourth convolution layer, and a fifth normalization layer;

[0023] The first branch joint motion feature, the second branch joint motion feature, and the third branch joint motion feature are input into the fusion layer to obtain multi-branch joint motion features. The fusion layer includes: a fifth convolution layer, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer, a ninth convolution layer, and a first output layer.

[0024] The step of extracting the overall spatial position information from the two-dimensional coordinates of the intermediate frame includes:

[0025] Performing a first fully connected operation on the two-dimensional coordinates of the intermediate frame;

[0026] Activate the two-dimensional coordinates of the intermediate frame after the full connection operation;

[0027] Performing a second fully connected operation on the two-dimensional coordinates of the intermediate frame after the activation operation;

[0028] splicing the two-dimensional coordinates of the intermediate frame after the activation operation and the two-dimensional coordinates of the intermediate frame after the second full connection operation;

[0029] A third full connection operation is performed on the two-dimensional coordinates of the spliced ​​intermediate frames to obtain overall spatial position information, and the overall spatial position information is used as constraint information.

[0030] The multi-branch joint motion features and the overall spatial position information are fused to obtain fused features, including:

[0031] The multi-branch joint motion features are divided into features of the current group and features of non-current groups;

[0032] Concatenate the features of non-current groups in the channel dimension, and input the concatenated results into the first fully connected layer;

[0033] Input the output of the first fully connected layer and the features of the current group into the first feature fusion unit;

[0034] Input the output of the first feature fusion unit and the overall spatial position information into the second feature fusion unit;

[0035] The output of the second feature fusion unit is input into the second fully connected layer, and the output of the second fully connected layer is used as the fusion feature.

[0036] The first feature fusion unit or the second feature fusion unit includes:

[0037] The first input passes through the first multi-layer perceptron to obtain the output of the first multi-layer perceptron;

[0038] The second input passes through the second multi-layer perceptron to obtain the output of the second multi-layer perceptron;

[0039] According to the output of the first multi-layer perceptron, a spatial position matrix is ​​obtained;

[0040] According to the output of the second multi-layer perceptron, an information supplement matrix is ​​obtained;

[0041] The spatial position matrix and the information supplement matrix are multiplied to obtain the output of the first feature fusion unit or the output of the second feature fusion unit.

[0042] The spatial position matrix is ​​calculated as follows:

[0043]

[0044] Where l1 is the first input, y1 is the output of the first multi-layer perceptron, Lr(*) is the LeakyReLU activation function, Ln(*) is the fully connected layer, Bn(*) is the normalization operation, s1 is the spatial position matrix, For addition operation.

[0045] The information supplement matrix is ​​calculated as follows:

[0046]

[0047] Where l2 is the second input, y2 is the output of the first multi-layer perceptron, Lr(*) is the LeakyReLU activation function, Ln(*) is the fully connected layer, Bn(*) is the normalization operation, s2 is the spatial position matrix, For addition operation.

[0048] In the second aspect, the present application proposes a monocular vision 3D human posture estimation device integrating multi-joint features, comprising:

[0049] A coordinate acquisition module is used to obtain a two-dimensional coordinate sequence of human joint points;

[0050] A coding enhancement module is used to perform position information coding and time information coding on the two-dimensional coordinate sequence of the human joint points; the two-dimensional coordinate sequence of the human joint points, the position information coding and the time information coding are spliced ​​to obtain information enhancement coding;

[0051] The joint grouping module is used to classify the joints by encoding the information enhancement to obtain joint groups;

[0052] The joint motion feature extraction module is used to perform multi-branch feature extraction on the joint group to obtain multi-branch joint motion features;

[0053] The joint constraint module is used to obtain the two-dimensional coordinates of the intermediate frame in the two-dimensional coordinate sequence of the human joint points, and extract the overall spatial position information of the two-dimensional coordinates of the intermediate frame;

[0054] The joint feature fusion module is used to fuse the joint motion features of multiple branches and the overall spatial position information to obtain fused features;

[0055] The pose estimation module is used to perform pose estimation on the fused features to obtain a monocular vision three-dimensional human pose estimation result.

[0056] Beneficial effects:

[0057] This application proposes a monocular vision three-dimensional human posture estimation method and device that integrates multi-joint features. The overall network architecture uses video sequences to obtain initial estimates, which are combined with implicit constraints of 2D joint information in intermediate frames to jointly estimate accurate 3D poses. In the joint motion feature extraction module, multi-branch operations are used to extract the motion features of joints in the time dimension, and different features are fused to form a highly expressive feature representation. The joint feature fusion module integrates the global information of different joint groups and intermediate frames, and expresses the relative positions and mutual connections of different joint groups in high-dimensional space by means of matrix inner products. The joint constraint module introduces the spatial position relationship of the 2D joint points of the intermediate frames as implicit constraints to reduce unreasonable posture outputs and improve the accuracy of the final 3D pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flow chart of a method for estimating a 3D human body posture using monocular vision and integrating multi-joint features according to an embodiment of the present application;

[0059] Figure 2 Schematic diagram of the relative relationship of joints in an embodiment of the present application;

[0060] Figure 3 Schematic diagram of joint grouping results in an embodiment of the present application;

[0061] Figure 4 Schematic diagram of a monocular vision 3D human posture network integrating multi-joint features according to an embodiment of the present application;

[0062] Figure 5 Schematic diagram of a joint motion feature extraction module according to an embodiment of the present application;

[0063] Figure 6 Schematic diagram of fusion features in an embodiment of the present application;

[0064] Figure 7 Schematic diagram of a feature fusion unit in an embodiment of the present application;

[0065] Figure 8A schematic diagram of the visualization results of the embodiment of the present application;

[0066] Fig. 9 Schematic diagram of the results visualized on 3DHP of the embodiment of the present application;

[0067] Fig.10 Convergence comparison diagram of the embodiment of the present application;

[0068] Fig.11 A schematic diagram of a monocular 3D human posture estimation device integrating multi-joint features according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] The specific implementation of the present application is further described in detail below in conjunction with the drawings and examples.

[0070] Existing monocular vision-based 3D human posture estimation methods use video time domain information and joint spatial information to solve problems such as lack of depth information. At present, such methods fail to effectively deal with time series redundancy problems and are difficult to capture small changes in human joints. To this end, the present application proposes a monocular vision 3D human posture estimation method and device that integrates multi-joint features, that is, a 3D human posture estimation network that integrates multi-joint features is proposed. In view of the problem that time series redundancy is difficult to capture small joint motion changes, a multi-branch architecture is designed, each branch captures joint motion features in different dimensions, and enhances the network's sensitivity to small joint changes. Secondly, in view of the problem that the fusion of local joint information and global joint information is insufficient, the present application groups joints according to the topological structure of the human body, and recombine the grouped joints at the end. Finally, in view of the problem of predicting unreasonable postures, the present application uses intermediate frame joint information as constraints, and integrates the constraint information with the initial estimated value to further improve the accuracy of the prediction.

[0071] The present application proposes a monocular vision 3D human posture estimation method that integrates multi-joint features, which mainly includes: 1) A network that integrates multi-joint features is proposed to effectively model the spatiotemporal motion relationship of joints and reduce the generation of unreasonable postures. 2) The present application proposes a joint motion feature extraction module, which adopts a multi-branch parallel feature extraction method. This design enhances the network's ability to capture comprehensive and local motion changes of joints. By combining the features of different branches, the temporal representation of joint motion changes is enriched. 3) The present application proposes a joint feature fusion module. After the joints are grouped, each group of features is relatively independent, lacking information from other joint groups, which is not conducive to maintaining the consistency of body posture. Therefore, the present application proposes a feature fusion module, which obtains the relative position of the joint group in high-dimensional space and the spatial position in the overall posture through feature matrix operations. 4) The present application proposes a joint constraint module. The 2D posture of the human body and the corresponding 3D position have an implicit connection in space. The spatial relationship of the joint 2D is used to reduce the generation of unreasonable postures, thereby improving the accuracy of 3D posture estimation.

[0072] Embodiment 1:

[0073] This embodiment proposes a monocular vision 3D human posture estimation method integrating multi-joint features, such as Figure 1 As shown, including:

[0074] Step S1: Obtain a two-dimensional coordinate sequence of human joint points;

[0075] In this embodiment, the input of the network is a two-dimensional coordinate sequence of human joint points, which can be expressed as Where T represents the length of the input sequence, and J represents the serial number of the human joint point. The two-dimensional coordinate sequence of the human joint point includes: the two-dimensional coordinate of the root joint, the two-dimensional coordinate of the intermediate frame, and the two-dimensional coordinate of other frames. The existing method of estimating the positions of other three-dimensional joints relative to the root joint only allows the network to focus on the global motion trajectory of the human posture in three-dimensional space, without considering the consistency of the input and output distribution. The inconsistent distribution of input and output will reduce the robustness of the global motion. In order to solve this problem, the input is encoded with information.

[0076] Step S2: encoding the position information and the time information of the two-dimensional coordinate sequence of the human body joint points;

[0077] In this embodiment, the position information of the input is encoded by subtracting the coordinates of the root joint from the coordinates of other joint points, that is:

[0078]

[0079] Among them, k 0represents the 2D coordinate of the root joint, k j represents the 2D coordinates of all joints, K p is the position information encoding, and j is the jth joint. Position information encoding enables the network to capture only meaningful spatial information related to human posture, which is beneficial to enhancing the network's robustness to global motion.

[0080] In this embodiment, the 3D pose of the intermediate frame is estimated using a 2D joint point sequence, so other poses play an auxiliary role in the continuity of the time domain for the pose of the intermediate frame, but are not as important as the pose of the intermediate frame. In order to emphasize the relative position relationship between the pose of the intermediate frame and other poses, the input is encoded with time information, which can be expressed as:

[0081]

[0082] in, represents the 2D joint coordinates of the intermediate frame, k t represents the 2D joint coordinates of all frames, K T For temporal information encoding, t is the tth moment, and the temporal information encoding uses the difference between the two-dimensional coordinates of the intermediate frame and the two-dimensional coordinates of all human joints. Temporal information encoding of the two-dimensional coordinate sequence of the human joints allows the network to focus on the changes in the positions of the surrounding joints in the intermediate frame, rather than the absolute position of each posture. When local movements with a small range of motion occur, these changes will be amplified, which helps to model the 3D pose with high precision.

[0083] Step S3: splicing the two-dimensional coordinate sequence of the human body joint points, the position information code and the time information code to obtain information enhancement code;

[0084] In this embodiment, the relative relationship diagram of the joints is as follows: Figure 2 As shown in the figure, the long dashed line represents the movement change of the intermediate frame joint in the time dimension, and the short dashed line represents the spatial position relationship of the root joint relative to other joints. The original input, position information encoding, and time information encoding are spliced ​​so that the model can learn different types of features at the same time. This process can be expressed as

[0085] K e =Concat(K,K p ,K T ) (3)

[0086] Among them, K e Represents the final information enhancement code.

[0087] Step S4: classify the information enhancement code into joints to obtain joint groups;

[0088] In this embodiment, human joints are grouped, and the results are as follows: Figure 3 shown.

[0089]

[0090] In this embodiment, according to the dynamic chain structure of the human body, the body is divided into five parts, namely, the trunk, the left leg, the right leg, the left arm and the right arm, which are represented as Where i represents the number of joint groups, J i Represents the number of joint points in the i-th group. The grouped joint groups are input into the joint motion feature extraction module in parallel. In this module, the temporal characteristics of the joint motion trajectory in each group are modeled.

[0091] Step S5: performing multi-branch feature extraction on the joint group to obtain multi-branch joint motion features;

[0092] The traditional temporal convolution model, which is composed of multiple temporal convolution blocks connected in series, plays an important role in extracting joint motion features. However, its single structure has a weak ability to extract joint temporal changes and insufficient feature expression capabilities. In order to enable the network to better learn the representation of human posture motion trajectories in the time dimension, more abundant motion features of human joints in the time dimension are extracted from a limited input sequence. The module of this embodiment adopts a multi-branch architecture, and each branch extracts different motion features of the joints. The input of this module is a joint group that has been grouped after information enhancement. The advantage of using a convolution operation is that the motion change features of the joints at different times can be extracted based on the size of the convolution kernel to maintain the consistency of joint motion changes. Because the input sequence is relatively redundant and the motion changes between adjacent frames are small, the average pooling operation is used. This operation can enhance the module's ability to capture subtle changes in joint motion. Finally, the features obtained from the three branches are fused to enhance the final feature's ability to characterize joint motion changes. The overall architecture of the joint motion feature extraction module is shown in the figure. Figure 5 shown.

[0093] The multi-branch feature extraction is performed on the joint group to obtain multi-branch joint motion features, including:

[0094] Step S5.1: inputting the joint group into the first branch to obtain the first branch joint motion feature, wherein the first branch sequentially includes: a first convolutional layer, a first normalization layer, a first average pooling layer and a second normalization layer;

[0095] In this embodiment, in the joint motion feature extraction module, the first branch joint motion feature is input through a 1×1 convolution to transform the spatial relationship of the joint points into a geometric relationship in the vector space, so as to better express the similarities and differences of the motions between different joints. The process is expressed as:

[0096] f1=Bn(Conv(Ki ,W 1×1 )) (5)

[0097] Then, the average pooling layer is used to obtain the spatial aggregation information of joint changes between adjacent times, so as to better capture the subtle changes of joints over time. The process is expressed as:

[0098] b1=Bn(Avg(f1,S 1×3 )) (6)

[0099] Where K i represents the input joint group, f1 represents the input feature after the first convolution layer and the first normalization layer, and b1 represents the feature output result of the first branch. Avg(*) represents the average pooling operation, S 1×3 It represents a pooling window size of 1×3 and a step size of 3.

[0100] Step S5.2: inputting the joint group into the second branch to obtain the second branch joint motion feature, wherein the second branch sequentially comprises: a second convolutional layer and a third normalization layer;

[0101] The second branch is the traditional time convolution operation, expressed as:

[0102] b2=Bn(Conv(K i ,W 1×3 )) (7)

[0103] Among them, b2 represents the feature output result of the second branch, which has undergone the second convolutional layer and the third normalization layer in sequence.

[0104] Step S5.3: inputting the joint group into the third branch to obtain the third branch joint motion feature, wherein the third branch sequentially includes: a third convolution layer, a fourth normalization layer, a fourth convolution layer, and a fifth normalization layer;

[0105] In the third branch, the role of 1×3 convolution is to gradually reduce the sequence length and aggregate the change information of the same joint at different times. The process is expressed as:

[0106] f2=Bn(Conv(K i ,W 1×1 )) (8)

[0107] b3=Bn(Conv(f2,W 1×3 )) (9)

[0108] Among them, f2 represents the feature of the input after the third convolution and the fourth normalization operation, b3 represents the feature output result of the third branch, which passes through the fourth convolution layer and the fifth normalization layer in sequence. Conv(*) represents the convolution operation, Bn(*) represents the normalization operation, W1×1 Represents the size of the convolution kernel is 1×1, W 1×3 Indicates that the size of the convolution kernel is 1×3.

[0109] Step S5.4: Input the first branch joint motion features, the second branch joint motion features, and the third branch joint motion features into the fusion layer to obtain multi-branch joint motion features, and the fusion layer includes: the fifth convolution layer, the sixth convolution layer, the seventh convolution layer, the eighth convolution layer, the ninth convolution layer, and the first output layer.

[0110] The features obtained from the three branches are fused, and the process can be expressed as

[0111]

[0112] Among them, b4 represents the fused feature, Represents the addition operation. The fused features are subjected to multiple strided convolutions. The process can be expressed as:

[0113] F B =Conv(Conv(b4,W 1×3 ),W 1×1 ) (11)

[0114] Among them, F B The final feature obtained is represented by the fifth convolutional layer, the sixth convolutional layer, the seventh convolutional layer, the eighth convolutional layer, and the ninth convolutional layer in sequence. The first output layer obtains the final feature. Hierarchical feature extraction enables the deeper layers of the model to learn the features of more complex human posture changes and longer time ranges based on the previous features, so that the model can better learn the changing rules of posture sequences in the time dimension.

[0115] Step S6: taking the two-dimensional coordinates of the intermediate frame in the two-dimensional coordinate sequence of the human body joint points, and extracting the overall spatial position information of the two-dimensional coordinates of the intermediate frame;

[0116] The step of extracting the overall spatial position information from the two-dimensional coordinates of the intermediate frame includes:

[0117] Performing a first fully connected operation on the two-dimensional coordinates of the intermediate frame;

[0118] Activate the two-dimensional coordinates of the intermediate frame after the full connection operation;

[0119] Performing a second fully connected operation on the two-dimensional coordinates of the intermediate frame after the activation operation;

[0120] splicing the two-dimensional coordinates of the intermediate frame after the activation operation and the two-dimensional coordinates of the intermediate frame after the second full connection operation;

[0121] A third full connection operation is performed on the two-dimensional coordinates of the spliced ​​intermediate frames to obtain overall spatial position information, and the overall spatial position information is used as constraint information.

[0122] Most methods that estimate the 3D pose of the intermediate frame by inputting a 2D joint point sequence mainly focus on the generation of 3D pose hypotheses, ignoring the correspondence between the 2D joint point information of the intermediate frame and the 3D pose. Although 2D joint points cannot provide depth information of the 3D pose, they not only indicate the possible positions of human joints in 3D space, but also contain the overall spatial relationship of the intermediate frame. In order to make full use of the geometric prior of the input 2D joint points, this embodiment proposes a joint constraint module to reduce the generation of unreasonable joint positions.

[0123] The input of this module is the 2D joint point information at the intermediate moment, expressed as The process of the constraint module is expressed as:

[0124] r1=δ(Bn(Ln(K c ))) (12)

[0125]

[0126] Among them, δ(*) is the Relu activation function, r1 is the feature after the first full connection operation and activation function, and f r For constraint information.

[0127] Step S7: Fuse the joint motion features of multiple branches and the overall spatial position information to obtain fused features, such as Figure 6 As shown, including:

[0128] Step S7.1: Divide the multi-branch joint motion features into features of the current group and features of the non-current group;

[0129] Step S7.2: concatenate the features of non-current groups in the channel dimension, and input the concatenated results into the first fully connected layer;

[0130] Step S7.3: Input the output of the first fully connected layer and the features of the current group into the first feature fusion unit;

[0131] Step S7.4: inputting the output of the first feature fusion unit and the overall spatial position information into the second feature fusion unit;

[0132] Step S7.5: Input the output of the second feature fusion unit to the second fully connected layer, and use the output of the second fully connected layer as the fusion feature.

[0133] In this embodiment, in the joint motion feature extraction module, the temporal features of the joint motion trajectory in each group are modeled. The obtained temporal features are then input in parallel to the joint feature fusion module to fuse the current group features, non-current group features and global information. In this module, the relative spatial position relationship between different joint groups is explicitly learned to connect all groups, and the fused features are added to the constraint information to obtain the final output, which is expressed as

[0134] The constraint information is obtained in another branch of the network. The 2D joint points of the intermediate frame are input into the joint constraint module to obtain the spatial position information of the entire joint in the intermediate frame. The information is then used as constraint information and global information and input into the main network. Overall network architecture Figure 4 shown.

[0135] After the joints are grouped, although each group extracts joint features with temporal significance, the relationship between each group is relatively independent. When inferring the 3D posture, the joints of the current group are completely unaware of the spatial positions of the joints of other groups, which is not conducive to maintaining the consistency of the overall body posture. Ideally, the network should ensure the continuity between each group of joints. Therefore, this embodiment proposes a joint feature fusion module that can integrate non-current group information and global information into the current group. Its architecture is as follows: Figure 4 shown.

[0136] First, the features of the non-current group are concatenated in the channel dimension, and then the joint features of the four groups are fused through the fully connected layer to establish the spatial position relationship between the joints of the non-current group. The process is expressed as:

[0137]

[0138] Among them, f m1 Indicates the characteristics of the non-current group of joints after fusion, Represents the features of four joint groups, Concat(*) represents concatenation, and Ln(*) represents the fully connected layer.

[0139] The features of the current joint group and the non-current joint group are input into the feature fusion module for fusion to obtain the spatial position relationship between the current joint group and the non-current joint group:

[0140]

[0141] Then the fused features and global information are simultaneously input into the feature fusion module to obtain the high-dimensional spatial position of the current group of joints in the overall posture, and then pass through the fully connected layer. The process is expressed as:

[0142] F J =Ln(FFM(f g ,fm2 )) (16)

[0143] in, Indicates the joint features of the current group, f m2 represents the fusion features of the current group and the non-current group, f g represents global information, F J It represents the final output feature of the joint feature fusion module, and FFM(*) represents the feature fusion operation.

[0144] Specifically, the first feature fusion unit or the second feature fusion unit, such as Figure 7 As shown, including:

[0145] The first input passes through the first multi-layer perceptron to obtain the output of the first multi-layer perceptron;

[0146] The second input passes through the second multi-layer perceptron to obtain the output of the second multi-layer perceptron;

[0147] According to the output of the first multi-layer perceptron, a spatial position matrix is ​​obtained;

[0148] According to the output of the second multi-layer perceptron, an information supplement matrix is ​​obtained;

[0149] The spatial position matrix and the information supplement matrix are multiplied to obtain the output of the first feature fusion unit or the output of the second feature fusion unit.

[0150] Among them, in the first feature fusion unit, the first input is specifically the output of the first fully connected layer, and the second input is the feature of the current group. In the second feature fusion unit, the first input is specifically the output of the first feature fusion unit, and the second input is the overall spatial position information;

[0151] The input of the feature fusion module is the feature containing the joint position information. The calculation process of the spatial position matrix can be expressed as:

[0152]

[0153] Where l1 is the first input, y1 is the output of the first multi-layer perceptron, Lr(*) is the LeakyReLU activation function, Ln(*) is the fully connected layer, Bn(*) is the normalization operation, s1 is the spatial position matrix, For addition operation.

[0154] The calculation process of the information supplement matrix can be expressed as:

[0155]

[0156] Where l2 is the second input, y2 is the output of the first multi-layer perceptron, Lr(*) is the LeakyReLU activation function, Ln(*) is the fully connected layer, Bn(*) is the normalization operation, s2 is the spatial position matrix, For addition operation.

[0157] Through the matrix multiplication operation, the two pieces of information can complement each other and generate better conditional weights to adjust the position of the input joint group in the high-dimensional space and explicitly express the relative spatial position relationship between the features. The process can be expressed as

[0158] S=Ln(s1⊙s2) (19)

[0159] Among them, S represents the characteristic matrix and ⊙ represents matrix multiplication.

[0160] Step S8: performing pose estimation on the fused features to obtain a monocular vision 3D human pose estimation result.

[0161] In this embodiment, fusion features are used to perform pose estimation to obtain an initial pose estimation value of the intermediate frame, and then pose estimation values ​​of other frames are obtained based on the initial pose estimation value of the intermediate frame, and finally a monocular vision three-dimensional human pose estimation result is obtained.

[0162] In this embodiment, the loss function is specifically expressed as the mean square error of the coordinates of the predicted joint point and the true value joint point, which is specifically expressed as:

[0163]

[0164] Among them, L MPJPE is the loss function value, T represents the total number of input frames, J represents the number of joint points, represents the 3D true value of the pose, 3D pose coordinates predicted by the network.

[0165] In order to prove the effectiveness of the method described in this embodiment, a comparative test was carried out, which is described in detail as follows:

[0166] (1) Dataset

[0167] This embodiment uses three publicly available benchmark datasets for training and evaluation, namely the Human3.6 dataset (Ionescu C, Papava D, Olaru V and Sminchisescu C. 2014. Human3.6m: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7): 1325-1339), the HumanEva-I (Leonid Sigal, Alexandru O, BalanMichael and J. Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International journal of computer vision, 87(1-2): 4) dataset and the MPI-INF-3DHP (3DHP) dataset (Mehta D, Rhodin H, Casas D, Fua P, Sotnychenko O and Xu W. 2017. Monocular 3D Human Pose Estimation in the Wild Using Improved CNN) Supervision / / Proceedings of 2017InternationalConference on 3d Vision.Qingdao,China:IEEE:506-516). Humman3.6M is one of the largest indoor datasets, divided into 15 scenes. This embodiment selects the data of 5 subjects (S1, S5, S6, S7, S8) as training data, and the data of 2 subjects (S9, S11) as evaluation data. The division of training data and evaluation data is consistent with the comparison method. HumanEva-I is a smaller dataset, which consists of three subjects and six actions. This embodiment selects the object (S1, S2, S3) action (Walk, Jog). MPI-INF-3DH is a large-scale dataset, consisting of three scenes: green screen, non-green screen and outdoor. By using 14 cameras, the dataset records 8 participants performing 8 training activities and 7 evaluation activities.

[0168] (2) Experimental environment

[0169] The operating system used in this experiment is Ubuntu 18.04, the deep learning framework is Pytorch 1.10.1, cuda 10.2, and the Lion optimizer is used. The RTX 1080Ti GPU is used for training and reasoning. In the setting of hyperparameters, the initial learning rate is set to 1×10 -4 , the learning decay rate is 0.95 and the batch size is 256.

[0170] (3) Evaluation indicators

[0171] This experiment uses two common protocols as evaluation indicators on the Humman3.6M and HumanEva-I datasets: Protocol 1 is MPJPE (Mean Per Joint Position Error, MPJPE), which represents the Euclidean distance between the coordinates of the predicted joint point and the true joint point. For video pose estimation, the error is averaged over all time series; Protocol 2 is the pose-processing mean per joint position error (P-MPJPE), which represents the average error of each joint position after rigid transformation and root joint alignment. Both indicators are in millimeters (mm). The lower the value of the indicator, the smaller the error of the model estimation. MPJPE, PCK and AUC are used as evaluation indicators on the 3DHP dataset. PCK (percentage of correct keypoints) represents the percentage of correct key points within 150mm, and AUC (area under curve) represents the area under the ROC curve. It can be expressed as

[0172]

[0173] Among them, Tr represents the threshold, d i represents the Euclidean distance between the predicted value and the true value of the i-th joint point, d represents the scale factor of the human body, is the Sigmod function.

[0174] (4) Experimental results and analysis

[0175] (4.1) Quantitative comparative analysis

[0176] On the Human3.6M dataset, the proposed method of this embodiment is quantitatively compared with the current advanced methods. In order to ensure the fairness and accuracy of the comparison, the experimental results in the original paper are used in the following table. Table 1 and Table 2 show the errors of the subjects in 15 movements of S9 and S11, and the last column represents the average error. The results of the true value represent the upper limit of this method. As shown in Table 1, among the 15 movements, the method of this embodiment achieves the best in 8 movements and the suboptimal in 2 movements compared with the previous method. Especially in dealing with some challenging postures with complex movements and rapid changes (such as Sitting and WalkDog), compared with the suboptimal method, this embodiment reduces the error by 2.3% in Sitting and 3.9% in WalkDog, achieving a significant advantage, which proves that the joint points predicted by the model of this embodiment have better accuracy.

[0177] Table 1 Results on the Human3.6M dataset using the true value of 2D pose as input

[0178]

[0179] In Table 1, the boldface indicates the best result, and the underline indicates the second best result.

[0180] Table 2 shows the results of the method of this embodiment using the 2D pose value detected by CPN as input, and the results obtained in MPJPE and P-MPJPE. Because the 2D pose value detected by CPN is noisy and inaccurate compared to the 2D true value, the result represents the lower limit predicted by the method of this embodiment. It can be seen that compared with similar methods, this method has achieved the best results in MPJPE and P-MPJPE, which are 45.0mm and 35.4mm respectively. In terms of the MPJPE index, 7 actions have reached the optimal value, and 4 actions have reached the suboptimal value. In terms of the P-MPJPE index, 5 actions have reached the optimal value, and 5 actions have reached the suboptimal value. This proves that the method of this embodiment can still achieve good results when facing inaccurate and noisy 2D detection points, and has strong robustness.

[0181] Table 2 Results on the Human3.6M dataset using 2D pose values ​​detected by CPN as input

[0182]

[0183]

[0184] In Table 2, the boldface indicates the best result, and the underline indicates the second best result.

[0185] This embodiment uses the HumanEva dataset to evaluate the generalization ability of the method of this embodiment. The input is the joints detected by MRCPN, the output is 15 joints, and the evaluation index is P-MPJPE. The results are shown in Table 3.

[0186] Table 3 Results on the HumanEva dataset

[0187]

[0188] Among them, the bold in Table 3 represents the best result, and the underline represents the second best result.

[0189] The high error in S3's Walk action is caused by the damage of the motion capture data. It can be seen that the method of this embodiment reaches the best in two actions and the second best in one action. It verifies that this method still has no significant impact on the overall performance of the network under the influence of inaccurate 2D joint points.

[0190] In order to verify the accuracy of the method of this embodiment in estimating human posture on the 3DHP dataset, the three modules are used simultaneously for training on the dataset, and the results are shown in Table 4.

[0191] Table 4 Results on the 3DHP dataset

[0192]

[0193] Among them, the bold in Table 4 represents the best result, and the underline represents the second best result.

[0194] The PCK, AUC and MPJPE of the method in this embodiment are 96.7%, 72.7% and 37.0, which are significantly lower than the second best result by 12.9% and 4.1% respectively. The results show that the method in this embodiment has strong generalization ability.

[0195] Visual qualitative comparison:

[0196] In order to further study the superiority of the method of this embodiment in expressing complex human postures, Figure 8As shown, the method of this embodiment and Li et al. (Li W, Liu H, Tang H, Wang P and Van G L. 2022. MHFormer: multi-hypothesis transformer for 3D human pose estimation / / Proceedings of 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition. New Orleans), Li et al. (Li W, Liu H, Ding R, Liu M, Wang P and Yang W. 2023. Exploiting Temporal Contexts With Strided Transformer for 3D Human Pose Estimation. IEEE Transactions on Multimedia, 25: 1282-1293) methods are visualized on 17 joint points on the H36M data set. It can be seen that the method of this embodiment has the highest degree of overlap between the solid line and the dotted line, indicating that the accuracy of human posture estimation is the highest.

[0197] This embodiment selects representative images of common problems in human posture tasks as input samples. In the first and second rows of actions, human self-occlusion occurs, especially in the second row of actions, the right elbow of the human body is severely occluded. From the visualization results, the elbow angle estimated by the method of this embodiment is closest to the true value, and other methods have large deviations. In the fourth row of actions, some human joints are relatively dense and the spatial position relationship is complex, but the method of this embodiment achieves more accurate overall modeling compared with other methods.

[0198] like Fig. 9 As shown, it is the result of visualization on 3DHP. It can be seen that the left arm position predicted by this method is closer to the true value, and the overall posture is more accurate.

[0199] After rigorous quantitative evaluation and qualitative comparison of multiple human datasets, the results show that this method can be adapted to a variety of human datasets and shows significant advantages over existing methods on each dataset. This series of experimental evidence strongly confirms that this method has excellent generalization ability and can stably and effectively improve the accuracy of pose prediction in different scenarios and conditions.

[0200] In order to verify the convergence of the method of this embodiment, the method of this embodiment is compared with other methods on the Huaman3.6 dataset. The results are as follows: Fig.10As shown, the vertical axis is the mean square error of the coordinates of the predicted joint point and the true value joint point of formula (20). The horizontal axis is the training time in hours. Compared with others, the method of this embodiment (dark curve) has a faster decrease in error and a lower error at the same time.

[0201] (4.2) Ablation experiment

[0202] In order to verify the effectiveness of the module proposed in this embodiment, this embodiment conducts an ablation experiment on the Human3.6M dataset, and verifies whether the joint motion feature extraction module, the joint feature fusion module and the joint constraint module are effective according to the changes in the two indicators of MPJPE and P-MPJPE. Table 5 shows the ablation experiment results of the method of this embodiment. Analyzing the experimental results, M4 adds a joint feature ablation module compared to M1, which reduces MPJPE by 2.3% and P-MPJPE by 1.2%. From the results, it can be seen that the joint feature fusion module can effectively establish the overall spatial relationship of the joint and reduce the error. M4 adds a joint motion feature module compared to M2, which reduces MPJPE by 1% and P-MPJPE by 1.2%. From the results, it can be seen that the joint motion feature extraction module can extract richer features, enhance the expressiveness of features, and thus reduce errors. Compared with M4, M5 adds a joint constraint module, which reduces MPJPE by 0.6% and P-MPJPE by 1.7%. The constraint effect of the joint constraint module on the joint in high-dimensional space is verified. Compared with M1, M5 adds two modules, which reduces MPJPE by 3% and P-MPJPE by 2.9%. After analysis, it can be seen that using each module alone can reduce the error of posture estimation to varying degrees, but using a module alone has little effect on the overall network. When the number of modules used increases, the error of posture estimation also decreases, which proves the effectiveness of the method of this embodiment in the joint use of modules.

[0203] Table 5 Ablation experiment

[0204]

[0205] Among them, the bold font in Table 5 is the lowest error in each column.

[0206] The experimental results show that compared with the MHFormer method, the MPJPE result of this method on the Human3.6M dataset is 29.0mm, and the error is reduced by 4.9%. On the MPI-INF-3DHP dataset, the MPJPE index is reduced by 36.2%, the AUC index is improved by 12.9%, and the PCK index is improved by 3%. The experimental results on the HumanEva dataset verify that this method has high generalization, and the ablation experiment further verifies the effectiveness of each module. Conclusion The research results show that the network proposed in this embodiment can better improve the accuracy of monocular vision 3D human posture estimation and has high generalization because it effectively integrates the multi-joint features of the human body.

[0207] The three key modules proposed in this embodiment are used to solve the problems of missing depth information and difficulty in capturing subtle changes in joints in three-dimensional human posture estimation. The first is the joint motion feature extraction module, which helps the model extract the motion features of human joints in the time dimension. By using a multi-branch architecture, the feature information is enriched and the impact of occlusion on the model is reduced. The second is the joint feature fusion module, which enables the model to establish the joint position relationship of different joints in the same space, strengthen the connection between different joints, and thus establish a global relationship. Finally, the joint constraint module makes full use of the joint information of the intermediate frame to provide implicit constraints, reduces ambiguous postures, and enables the model to generate reasonable three-dimensional postures, thereby enhancing the robustness and accuracy of the model. The comprehensive evaluation results show that the method proposed in this study can achieve smaller errors than similar methods. The ablation experiment further proves the effectiveness of the module proposed in this study. However, the method of this study is still limited by inaccurate and noisy 2D detection results. Future research will focus on reducing the length of the input sequence while maintaining the accuracy of model prediction, reducing network parameters, and simplifying the model.

[0208] Embodiment 2:

[0209] This embodiment proposes a monocular vision 3D human posture estimation device integrating multi-joint features, such as Fig.11 The module includes: coordinate acquisition module, coding enhancement module, joint grouping module, joint motion feature extraction module, joint constraint module, joint feature fusion module, and posture estimation module;

[0210] The coordinate acquisition module is connected to the coding enhancement module, the coding enhancement module is connected to the joint grouping module, the joint grouping module is connected to the joint motion feature extraction module, the coordinate acquisition module is connected to the joint constraint module, the joint motion feature extraction module and the joint constraint module are respectively connected to the joint feature fusion module, and the joint feature fusion module is connected to the posture estimation module;

[0211] A coordinate acquisition module is used to obtain a two-dimensional coordinate sequence of human joint points;

[0212] A coding enhancement module is used to perform position information coding and time information coding on the two-dimensional coordinate sequence of the human joint points; the two-dimensional coordinate sequence of the human joint points, the position information coding and the time information coding are spliced ​​to obtain information enhancement coding;

[0213] The joint grouping module is used to classify the joints by encoding the information enhancement to obtain joint groups;

[0214] The joint motion feature extraction module is used to perform multi-branch feature extraction on the joint group to obtain multi-branch joint motion features;

[0215] The joint constraint module is used to obtain the two-dimensional coordinates of the intermediate frame in the two-dimensional coordinate sequence of the human joint points, and extract the overall spatial position information of the two-dimensional coordinates of the intermediate frame;

[0216] The joint feature fusion module is used to fuse the joint motion features of multiple branches and the overall spatial position information to obtain fused features;

[0217] The pose estimation module is used to perform pose estimation on the fused features to obtain a monocular vision three-dimensional human pose estimation result.

[0218] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0219] The protection scope of the present application is not limited to the above-mentioned embodiments. Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the scope and spirit of the present disclosure. If these changes and modifications fall within the scope of the claims of the present disclosure and their equivalents, the intention of the present disclosure also includes these changes and modifications.

Claims

1. A monocular vision 3D human posture estimation method integrating multi-joint features, characterized in that: include: Obtain the two-dimensional coordinate sequence of human joint points; Performing position information encoding and time information encoding on the two-dimensional coordinate sequence of the human body joint points; The two-dimensional coordinate sequence, position information code and time information code of the human joint point are spliced ​​to obtain information enhancement code; The information enhancement code is used to classify joints to obtain joint groups; Perform multi-branch feature extraction on the joint group to obtain multi-branch joint motion features; Taking the two-dimensional coordinates of the intermediate frame in the two-dimensional coordinate sequence of the human body joint points, and extracting the overall spatial position information of the two-dimensional coordinates of the intermediate frame; The joint motion features of multiple branches and the overall spatial position information are fused to obtain fused features; The pose estimation is performed on the fused features to obtain a monocular vision three-dimensional human pose estimation result.

2. The monocular vision 3D human posture estimation method integrating multi-joint features according to claim 1 is characterized in that: The two-dimensional coordinate sequence of the human body joint points includes: the two-dimensional coordinates of the root joint, the two-dimensional coordinates of the intermediate frame and the two-dimensional coordinates of other frames.

3. The monocular vision 3D human posture estimation method integrating multi-joint features according to claim 1 is characterized in that: The position information encoding is performed on the two-dimensional coordinate sequence of the human body joint points by using the difference between the two-dimensional coordinates of other frames and the two-dimensional coordinates of the root joint.

4. The monocular vision 3D human posture estimation method integrating multi-joint features according to claim 1, characterized in that: The time information encoding performs time information encoding on the two-dimensional coordinate sequence of the human joint points by using the difference between the two-dimensional coordinates of the intermediate frame and the two-dimensional coordinates of all the human joint points.

5. The monocular vision 3D human posture estimation method integrating multi-joint features according to claim 1, characterized in that: The multi-branch feature extraction is performed on the joint group to obtain multi-branch joint motion features, including: Inputting the joint group into a first branch to obtain a first branch joint motion feature, wherein the first branch sequentially comprises: a first convolutional layer, a first normalization layer, a first average pooling layer, and a second normalization layer; Inputting the joint group into the second branch to obtain the second branch joint motion feature, wherein the second branch sequentially includes: a second convolutional layer and a third normalization layer; Inputting the joint group into the third branch to obtain the third branch joint motion feature, wherein the third branch sequentially includes: a third convolution layer, a fourth normalization layer, a fourth convolution layer, and a fifth normalization layer; The first branch joint motion feature, the second branch joint motion feature, and the third branch joint motion feature are input into the fusion layer to obtain multi-branch joint motion features. The fusion layer includes: a fifth convolution layer, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer, a ninth convolution layer, and a first output layer.

6. The monocular vision 3D human posture estimation method integrating multi-joint features according to claim 1, characterized in that: The step of extracting the overall spatial position information from the two-dimensional coordinates of the intermediate frame includes: Performing a first fully connected operation on the two-dimensional coordinates of the intermediate frame; Activate the two-dimensional coordinates of the intermediate frame after the full connection operation; Performing a second fully connected operation on the two-dimensional coordinates of the intermediate frame after the activation operation; splicing the two-dimensional coordinates of the intermediate frame after the activation operation and the two-dimensional coordinates of the intermediate frame after the second full connection operation; A third full connection operation is performed on the two-dimensional coordinates of the spliced ​​intermediate frames to obtain overall spatial position information, and the overall spatial position information is used as constraint information.

7. The monocular vision 3D human posture estimation method integrating multi-joint features according to claim 1, characterized in that: The multi-branch joint motion features and the overall spatial position information are fused to obtain fused features, including: The multi-branch joint motion features are divided into features of the current group and features of non-current groups; Concatenate the features of non-current groups in the channel dimension, and input the concatenated results into the first fully connected layer; Input the output of the first fully connected layer and the features of the current group into the first feature fusion unit; Input the output of the first feature fusion unit and the overall spatial position information into the second feature fusion unit; The output of the second feature fusion unit is input into the second fully connected layer, and the output of the second fully connected layer is used as the fusion feature.

8. The monocular vision 3D human posture estimation method integrating multi-joint features according to claim 7 is characterized in that: The first feature fusion unit or the second feature fusion unit includes: The first input passes through the first multi-layer perceptron to obtain the output of the first multi-layer perceptron; The second input passes through the second multi-layer perceptron to obtain the output of the second multi-layer perceptron; According to the output of the first multi-layer perceptron, a spatial position matrix is ​​obtained; According to the output of the second multi-layer perceptron, an information supplement matrix is ​​obtained; The spatial position matrix and the information supplement matrix are multiplied to obtain the output of the first feature fusion unit or the output of the second feature fusion unit.

9. The monocular vision 3D human posture estimation method integrating multi-joint features according to claim 7, characterized in that: The spatial position matrix is ​​calculated as follows: Where l1 is the first input, y1 is the output of the first multi-layer perceptron, Lr(*) is the LeakyReLU activation function, Ln(*) is the fully connected layer, Bn(*) is the normalization operation, s1 is the spatial position matrix, is the addition operation; The information supplement matrix is ​​calculated as follows: Where l2 is the second input, y2 is the output of the first multi-layer perceptron, Lr(*) is the LeakyReLU activation function, Ln(*) is the fully connected layer, Bn(*) is the normalization operation, s2 is the spatial position matrix, For addition operation.

10. A monocular vision 3D human posture estimation device integrating multi-joint features, characterized in that: include: A coordinate acquisition module is used to obtain a two-dimensional coordinate sequence of human joint points; A coding enhancement module, used for encoding the position information and time information of the two-dimensional coordinate sequence of the human body joint points; The two-dimensional coordinate sequence, position information code and time information code of the human joint point are spliced ​​to obtain information enhancement code; The joint grouping module is used to classify the joints by encoding the information enhancement to obtain joint groups; The joint motion feature extraction module is used to perform multi-branch feature extraction on the joint group to obtain multi-branch joint motion features; The joint constraint module is used to obtain the two-dimensional coordinates of the intermediate frame in the two-dimensional coordinate sequence of the human joint points, and extract the overall spatial position information of the two-dimensional coordinates of the intermediate frame; The joint feature fusion module is used to fuse the joint motion features of multiple branches and the overall spatial position information to obtain fused features; The pose estimation module is used to perform pose estimation on the fused features to obtain a monocular vision three-dimensional human pose estimation result.

Images