Three-dimensional human pose estimation method and system based on multi-path spatial fusion

Abstract

The application discloses a three-dimensional human posture estimation method and system based on multi-path space fusion, skillfully fuses the advantages of a graph convolution network, a Transformer structure and an SE module, and constructs an efficient posture estimation framework.The method comprises the following steps: performing high-dimensional vector representation on two-dimensional coordinates extracted from a three-dimensional human posture data set; cross-fusing local spatial relationships and global spatial relationships between human joints by using a graph convolution network and a Transformer structure; inputting obtained spatial vectors into an SE module for deep channel fusion; and finally, regressing to obtain three-dimensional human posture coordinates.Compared with a traditional method, the application realizes significant performance improvement under the condition of a relatively low parameter quantity, and provides more reliable technical support for actual application scenarios such as motion capture, virtual reality, augmented reality and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional human pose estimation in computer vision, more specifically, relates to a three-dimensional human pose estimation method and system based on graph convolution network and Transformer structure. BACKGROUND

[0002] Three-dimensional human pose estimation is one of the core technologies of computer vision and computer graphics. This technology aims to estimate the coordinates of human joints in three-dimensional space through image or video analysis. This technology has important practical application value, such as motion capture, virtual reality, augmented reality, human-computer interaction, motion evaluation, and medical rehabilitation, etc.

[0003] Thanks to the rapid development of deep learning technology, machine learning-based methods have made significant progress in three-dimensional human pose estimation. These methods extract key human pose features from input images by training a large number of labeled data samples. For example, methods based on the combination of graph neural networks and convolutional neural networks capture the spatial relationship between human joints; methods using long short-term memory networks (LSTM) or Transformer structure improve the ability of time series modeling; methods using attention mechanisms and multi-scale feature fusion enhance the expressive power of the model.

[0004] When performing three-dimensional human pose estimation on input images or video data, it is of practical importance to effectively extract spatial features and achieve this goal without significantly increasing the number of parameters. Lightweight parameter settings help improve the generalization ability and inference speed of the model, which is crucial for three-dimensional human pose estimation tasks. Therefore, in the research and application of this field, it is crucial to focus on how to extract spatial features as much as possible while keeping the number of parameters from increasing too much. SUMMARY

[0005] To solve the above problems, the purpose of the present application is to design a three-dimensional human pose estimation method and system based on multi-path spatial fusion, which can reduce the training and testing inference time based on the extraction of local spatial features and global features of human joints. Further, this method can effectively improve the accuracy and robustness of three-dimensional human pose estimation on various datasets.

[0006] In order to achieve the above-mentioned goal, the present application provides a three-dimensional human pose estimation method based on multi-path spatial fusion, comprising the following steps:

[0007] Obtain a three-dimensional human pose dataset;

[0008] Data processing:

[0009] extracting two-dimensional key point coordinates of a human body in the three-dimensional human pose data set;

[0010] high-dimensionally extending the two-dimensional key point coordinates into high-dimensional vectors, and positionally encoding the extended high-dimensional vectors, and inputting the high-dimensional vectors with the positionally encoded information into a multi-path fusion network;

[0011] multi-path network fusion:

[0012] splitting the high-dimensional vectors after positionally encoding;

[0013] alternately processing the split high-dimensional vectors using a graph convolution network and a Transformer module: inputting a first layer of uniformly split vectors into the graph convolution network, retaining the output thereof, inputting the output vectors of the first layer into a second layer of the Transformer module, retaining the output of the second layer, inputting the output vectors of the second layer into a third layer of the graph convolution network, retaining the output of the third layer, repeating several times until all split vectors are processed, so as to fully fuse local information and global information of the input vectors;

[0014] splicing the output vectors of each layer;

[0015] channel integration:

[0016] further extracting information from the spliced vectors using an SE module;

[0017] regression output:

[0018] inputting the information extracted by the SE module into a multi-layer perceptron model;

[0019] after the above steps are repeated and iterated for a preset number of times, converting the extracted information into three-dimensional human pose key point coordinates by the multi-layer perceptron regression model.

[0020] Further, the high-dimensional vectors after positionally encoding are uniformly split into six layers along the channel direction, and the split high-dimensional vectors are alternately processed using a graph convolution network and a Transformer module: inputting a first layer of uniformly split vectors into the graph convolution network, retaining the output thereof, inputting the output vectors of the first layer into a second layer of the Transformer module, retaining the output of the second layer, inputting the output vectors of the second layer into a third layer of the graph convolution network, retaining the output thereof, inputting the output vectors of the fourth layer into a fifth layer of the graph convolution network, retaining the output thereof, inputting the output vectors of the fifth layer into a sixth layer of the Transformer module, and retaining the output of the sixth layer.

[0021] Further, the output vectors of each layer are spliced along the channel direction, and the splicing length is equal to the channel length before splitting.

[0022] Further, while splicing each layer output vector, the multi-layer processed information is retained.

[0023] Further, after extracting the two-dimensional key point coordinates, the two-dimensional key point coordinates are normalized.

[0024] Further, the preset number of times is 3, so as to accurately realize three-dimensional human body posture estimation.

[0025] Further, the application also provides a three-dimensional human body posture estimation system based on multi-path space fusion, comprising:

[0026] The input data unit is used for processing the acquired three-dimensional human body posture data set, expanding the two-dimensional key point coordinates extracted from the three-dimensional human body posture data set to a high-dimensional space and performing position coding.

[0027] The multi-path fusion network unit is used for cutting the high-dimensional vector along the channel direction, inputting into the graph convolution network and the Transformer module in an alternating input mode, respectively, splicing the obtained output vectors, and further processing the spliced high-dimensional vectors.

[0028] The channel integration unit is used for inputting the spliced high-dimensional vectors into the SE module and further processing information.

[0029] The regression output unit is used for integrating and making regression prediction on the features extracted by the SE template in the MLP, so as to obtain the three-dimensional space coordinates of the human body key points and realize three-dimensional human body posture estimation.

[0030] Further, the application also provides a computer program product for executing the three-dimensional human body posture estimation method based on multi-path space fusion, which relates to a computer readable storage medium. The computer readable storage medium stores a set of computer executable instructions, which can make the computer device complete the three-dimensional human body posture estimation method when the instructions are executed on the computer device.

[0031] Further, the application also provides a computer device for realizing the three-dimensional human body posture estimation method, which comprises a processing unit, a storage unit and a computer program. The computer program is stored in the storage unit and is configured to be executed by the processing unit. The computer program comprises instructions for executing the three-dimensional human body posture estimation method.

[0032] Further, the processing unit is used for executing each step in the three-dimensional human body posture estimation method and outputting the three-dimensional human body posture key point coordinates.

[0033] Compared with the prior art, the application has the following beneficial effects:

[0034] (1) The application provides a novel three-dimensional human posture estimation method and system, which extracts the spatial information of the human skeleton structure in a six-layer structure, cross-extracts and fuses spatial features through a graph convolution network and a Transformer module, and can more accurately represent the human skeleton structure as a vector. This design reduces the number of parameters to a certain extent, and also improves the generalization ability and reasoning speed of the model.

[0035] (2) The method and system suitable for three-dimensional human posture estimation provided by the application input the extracted spatial feature vector into the SE (Squeeze Excitation) module, thereby enhancing the important features in the channel, and improving the accuracy of human posture estimation, which has good practicability and wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly describe the drawings of the embodiments. Obviously, the drawings mentioned below only contain some embodiments of the application, and are not limited to the application.

[0037] Figure 1 The method flowchart of the application;

[0038] Figure 2 The graph convolution network module schematic diagram of the application;

[0039] Figure 3 The Transformer module schematic diagram based on the self-attention mechanism of the application;

[0040] Figure 4 The SE module schematic diagram of the application. DETAILED DESCRIPTION

[0041] In order to make the target, technical scheme and advantages of the embodiments of the application more clear, the following contents will elaborate the technical strategy of the embodiments of the application with the help of the drawings. Obviously, the described embodiments are part of the embodiments of the application, not all. According to the description of the embodiments of the application, all other embodiments that can be obtained by those skilled in the art with certain technical level should be considered as covered by the protection scope of the application. Specific embodiment 1:

[0043] Input image: the embodiments of the application use single frame image as input, and the image comes from three-dimensional human posture data set (Human3.6M).

[0044] Image preprocessing: In order to adapt to different sizes and formats of input images, pre-processing operations such as cropping and scaling are performed on the images to ensure uniform size and format of the input images.

[0045] Two-dimensional human pose estimation: Use existing two-dimensional human pose estimator (such as CPN) to process the input image and extract two-dimensional key point information. The extracted two-dimensional key point coordinates are 17 joint points.

[0046] Normalization: First, translate the two-dimensional joint point coordinates so that the center joint point (such as the pelvis or torso center) is at the origin; then, scale the joint point coordinates; finally, rotate the joint point coordinates to ensure specific joint alignment. Such normalization helps to eliminate scale, translation and rotation differences, thereby improving the accuracy and robustness of subsequent three-dimensional pose estimation.

[0047] High-dimensional expansion: The input two-dimensional key point coordinates are expanded to a high dimension, making the shape a 17*240 vector. The expansion method changes the channel number to 240 dimensions through a linear layer.

[0048] Position encoding: In order to preserve the spatial relationship between key points, the high-dimensional vector is position encoded. The position information is integrated through a learnable position parameter encoding matrix.

[0049] Uniformly split along the channel: The high-dimensional vector is uniformly split into six layers along the channel. The splitting method uses the following formula:

[0050] split_vector = np.split(high_dim_vector, 6, axis = -1)

[0051] Alternately use graph convolutional network and Transformer module: The embodiment of the present invention alternately processes the split vector using graph convolutional network and Transformer module. GCN is used to capture local structural information between key points, while Transformer is used to establish long-distance dependency.

[0052] Graph convolutional network module: GCN module can perform convolution operation on input features according to graph structure. In the embodiment of the present invention, the operation of GCN module can be described using the following formula:

[0053] H' = ReLU(D^(-1 / 2)AD^(-1 / 2)H θ ), where A is the adjacency matrix, D is the degree matrix, H is the input feature matrix, θ is the weight matrix, and ReLU is the nonlinear activation function.

[0054] Transformer module: The Transformer module mainly includes self-attention mechanism, layer normalization, and regularization layer. In the embodiments of the present application, the self-attention mechanism can be described using the following formula:

[0055] Attention(Q, K, V) = softmax(Q*K^T / sqrt(d_k))V, where Q, K, and V represent query, key, and value matrices respectively, and d_k is the dimension.

[0056] Alternating processing: In the embodiments of the present application, the first layer of uniformly divided vectors is first input into the GCN module, and the output result is a 17*40 vector. Then, the output result of the GCN module is added to the second layer of vectors and input into the Transformer module, and then a 17*40 vector is output. In this way, the GCN module and the Transformer module are alternately used to process the divided vectors until the last layer of vectors is processed.

[0057] Concatenating the output vectors: The output vectors after six layers of processing are concatenated by channels to form a high-dimensional vector of 17*240.

[0058] SE (Squeeze Excitation) module: The concatenated high-dimensional vector is input into the SE module to further fuse the information of each channel. The functions of the SE module include performing global average pooling on the input feature vector to generate a channel descriptor; then, using two fully connected layers and a ReLU activation function, learning the nonlinear relationship between channels; passing the output of the fully connected layer through a Sigmoid activation function to obtain the weight of each channel; multiplying these weights with the original input feature map to realize the channel attention mechanism.

[0059] Multi-layer perceptron (MLP): The output vector of the SE module is input into the MLP for final feature extraction and mapping.

[0060] Repeated blending: In order to improve the performance and robustness of the model, the multi-layer fusion spatial feature extraction part is repeated three times. In each repetition, the parameters can be updated to adapt to different input features.

[0061] Regression output three-dimensional key point coordinates: After three times of repeated operation, the three-dimensional key point coordinates output by the model should be 17*3. These key point coordinates can be used for three-dimensional human pose estimation in actual application scenarios.

[0062] Training and verification: The training and verification of the model are performed using the Human3.6M dataset. The cross-validation method is used to optimize the model parameters and improve the generalization ability of the model.

[0063] Evaluation index: The Mean Per Joint Position Error (MPJPE) and Procrustes MPJPE (P-MPJPE) are used as evaluation indexes to measure the performance of the model. The experimental results show that the embodiments of the present application have achieved significant performance improvement in various indicators.

[0064] In summary, the embodiments of the present application provide a lightweight and efficient three-dimensional human pose estimation method. Through the design of the multi-path fusion network and the SE module, high-precision and robust three-dimensional human pose estimation is achieved. The multi-path fusion network designed in the present application includes a graph convolution network and a Transformer module to capture local structural information between key points and establish long-distance dependency relationships.

[0065] The embodiments of the present application not only achieve significant performance improvement in existing evaluation indicators, but also have strong generalization ability and application prospects. In future research, the design of the multi-path fusion network can be further optimized, and the robustness of the model in complex scenarios can be improved.

[0066] In summary, the embodiments of the present application provide a high-efficiency lightweight three-dimensional human pose estimation method based on multi-path spatial fusion. The key technologies and implementation steps are in the design of the multi-path fusion network, the application of the SE module, etc. The experimental results show that the embodiments of the present application have achieved significant performance improvement in various indicators, and have strong generalization ability and application prospects. Specific embodiment 2:

[0068] In this embodiment, the computer device is mainly composed of a storage component, a processing component, and a computer program located in the storage component and executable on the processing component. When the processing component executes the program, the related steps of the three-dimensional human pose estimation method in embodiment 1 will be implemented.

[0069] This embodiment provides a variety of processing module options, such as central processing units, general processing units, digital signal processing units, custom integrated circuits, field programmable gate arrays, other programmable logic components, discrete logic gates or transistors, and independent hardware components. The general processing unit can be a micro processing unit or other commonly used processing unit. The storage module can include read-only memory and random access memory to provide the processing module with the required instructions and data. The storage module can also include non-volatile random access memory. In addition, the storage module can store device type-related information.

[0070] Those skilled in the art should understand that the method disclosed in the embodiments can be realized by hardware, software or a combination of both. In addition, the present solution can also be implemented in the form of one or more computer program products containing computer interpretable code, such as magnetic storage and optical storage.

[0071] The present solution is further analyzed by means of flowcharts and / or block diagrams in the embodiments. Those skilled in the art should know that in the flowcharts and / or block diagrams, each action and / or block, and their combinations, can be completed by computer program scripts. Instructions executed by the processor of a computer or other programmable data processing device can realize the functions determined by one or more steps and / or blocks presented in the flowcharts and / or block diagrams.

[0072] These computer program instructions are saved in a specific computer readable storage medium, so as to guide the computer or other programmable data processing device to operate efficiently according to the preset strategy. In this way, the instructions in the computer readable storage medium can generate a product including instruction components. These instruction components are used to realize various functions in the flowcharts and / or block diagrams, including the functions illustrated by one or more stages and / or blocks. In this case, these components improve the efficiency and effectiveness of the entire implementation process by optimizing different stages and functions.

[0073] In addition, these computer programs can be transmitted to a computer or other programmable data processing device, and then a series of operation steps are carried out on the computer or other programmable device to establish a computer-driven processing architecture. Therefore, these instructions executed on the computer or other programmable device will provide operation steps to realize the functions illustrated by one or more actions in the flowcharts and / or one or more blocks in the block diagrams. In the entire implementation process, efficiency and practicability are ensured.

[0074] Finally, it should be particularly pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and do not limit them. Various adjustments and improvements to the technical solutions of the present application made by those skilled in the art on the basis of not deviating from the design concept of the present application should be included in the protection scope of the present application.

Claims

1. A three-dimensional human posture estimation method based on multi-path spatial fusion, characterized in that: The following steps are involved: Obtain a 3D human posture dataset; Data processing: Extracting the coordinates of two-dimensional key points of a human body from the three-dimensional human posture dataset; Expanding the two-dimensional key point coordinates into a high-dimensional vector, performing position encoding on the expanded high-dimensional vector, and inputting the position encoding information into a multi-channel fusion network after adding the high-dimensional vector; Multi-channel network fusion: Split the high-dimensional vector after position encoding; The split high-dimensional vectors are processed alternately using graph convolutional networks and Transformer modules: the evenly split first-layer vectors are input into the graph convolutional network, and its output is retained. The output vectors of the first layer are input into the Transformer module of the second layer, and the output of the second layer is retained. The output vectors of the second layer are input into the graph convolutional network of the third layer, and the output of the third layer is retained. This process is repeated several times until all the split vectors are processed, so as to fully integrate the local information and global information of the input vectors. Splice the output vectors of each layer; while splicing the output vectors of each layer, retain the information after multi-layer processing; Channel integration: Use the SE module to further extract information from the spliced ​​vector; Regression output: Inputting the information extracted by the SE module into a multi-layer perceptron model; When the above steps are repeated for a preset number of times, the multi-layer perceptron regression model converts the extracted information into the coordinates of the key points of the three-dimensional human body posture; Among them, the high-dimensional vector after position encoding is evenly split into six layers along the channel direction, and the split high-dimensional vector is processed alternately using the graph convolution network and the Transformer module: the uniformly split first-layer vector is input into the graph convolution network, and its output is retained, the output vector of the first layer is input into the Transformer module of the second layer, and the output of the second layer is retained, the output vector of the second layer is input into the graph convolution network of the third layer, and its output is retained, the output vector of the fourth layer is input into the graph convolution network of the fifth layer, and its output is retained, the output vector of the fifth layer is input into the Transformer module of the sixth layer, and the output of the sixth layer is retained.

2. The 3D human body pose estimation method according to claim 1, wherein: The output vectors of each layer are concatenated along the channel direction, and the concatenated length is equal to the channel length before segmentation.

3. The 3D human body pose estimation method according to claim 1, wherein: After extracting the two-dimensional key point coordinates, the two-dimensional key point coordinates are normalized.

4. The 3D human body pose estimation method according to claim 1, wherein: The preset number of times is 3 times to accurately achieve three-dimensional human posture estimation.

5. A 3D human pose estimation system based on multi-path spatial fusion, characterized in that: The method for estimating a three-dimensional human body posture according to any one of claims 1 to 4 is applied, comprising: Input data unit: used to process the acquired 3D human posture data set, expand the 2D key point coordinates extracted from the 3D human posture data set into a high-dimensional space and perform position encoding; Multi-channel fusion network unit: splits the high-dimensional vector along the channel direction, inputs it into the graph convolutional network and Transformer module in an alternating manner, and concatenates the obtained output vectors; Channel integration unit: inputs the concatenated high-level vector into the SE module and further processes the information; Regression output unit: In the MLP, the features extracted from the SE template are integrated and regression prediction is performed to obtain the three-dimensional spatial coordinates of the key points of the human body to achieve three-dimensional human posture estimation.

6. A computer program product for executing the three-dimensional human pose estimation method according to claim 1, characterized in that: The invention comprises a computer-readable storage medium, wherein the computer-readable storage medium stores a computer-executable instruction set, and when the instruction set is executed on a computer device, the computer device can perform the three-dimensional human body posture estimation method described in claim 1.

7. A computer device for implementing the three-dimensional human body posture estimation method described in claim 1, characterized in that: The invention comprises a processing unit, a storage unit and a computer program, wherein the computer program is stored in the storage unit and is configured to be executed by the processing unit, and the computer program contains instructions for executing the three-dimensional human body posture estimation method according to claim 1.

8. The computer device according to claim 7, wherein: The processing unit is used to execute each step of the three-dimensional human body posture estimation method according to claim 1 and output the coordinates of the three-dimensional human body posture key points.

Images