3D human body posture spatial feature modeling method

Abstract

The invention provides a 3D human body posture spatial feature modeling method, which comprises the following steps: detecting and generating a two-dimensional coordinate of each human body joint, and generating a 2D skeleton; spatial position codes are added to the 2D skeleton where a plurality of Origin-based Parts are formed, then processed data are input into STE, and global spatial features of the whole 2D skeleton are captured; the method comprises the following steps: carrying out local spatial feature modeling on each Origin-based Part; fusing the global spatial features and the local spatial features to generate spatial features; time position codes are added to the spatial features, and the dependency relationship of each articulation point in the time dimension is captured through a self-attention mechanism; and through a regression module, the spatial-temporal characteristics are converted into three-dimensional coordinates of the joint points, and a 3D human body posture estimation result is generated. According to the method, 3D human body posture spatial feature modeling based on Transform is realized.

Description

Technical Field

[0001] The present invention belongs to the field of 3D human body posture estimation in computer vision, and in particular, relates to a 3D human body posture spatial feature modeling method. Background Art

[0002] Human pose estimation (HPE) is an important and challenging task in the field of computer vision, and is of great significance to many application fields such as action recognition, virtual reality, and human-robot interaction. The purpose of HPE is to predict the position of each human joint from the input image or video. According to whether the predicted joints contain depth information, human pose estimation can be divided into 2D human pose estimation (2D HPE) and 3D human pose estimation (3D HPE). With the development of deep learning technology, the field of 2D HPE has now developed to a relatively mature level. The accuracy and generalization of 2D detectors have reached advanced levels, but the model output only contains limited 2D information. In contrast, although 3DHPE faces more challenges, the addition of depth information enables it to provide richer 3D spatial information for human poses and a better understanding of human actions and interactions. Therefore, the 2D-to-3D method that applies mature 2D detectors to 3D HPE tasks has become a typical monocular solution.

[0003] Due to the depth ambiguity problem in monocular data, multiple potential 3D poses may be mapped from the same 2D pose. Therefore, it is difficult to recover the accurate 3D pose based only on the 2D joint position information of a single frame. Recently, driven by the ability of Transformer to capture long-distance dependencies, significant progress has been made in 2D-to-3D solutions using video frame sequences containing motion timing features. Among them, MixSTE further divides the entire human pose into multiple joints to model more refined temporal features. However, most of the existing solutions roughly use the entire 2D skeleton as the source of model spatial features when designing the Spatial Block, or simply divide it into several independent parts.

[0004] How to overcome the above-mentioned defects and shortcomings of 3D human posture estimation is a technical problem that needs to be solved at present. Summary of the invention

[0005] We observed the performance of the Spatial Block in MixSTE in modeling the spatial correlation between joints, and studied the average degree of dependence between 17 joints in the Human3.6M dataset. We applied an interpretable scheme to average all the attention weights calculated in the middle layer of the model and then plotted them into a confusion matrix. We can observe that not all human joints have high dependence in spatial position. The spatial correlation between joints that are far away is generally low, such as elbow joints and knee joints. Such joint pairs are not directly related in human geometry and are independent of each other during movement. However, joints belonging to the same part are closer and generally have higher spatial correlation, such as hip joints, knee joints, and ankle joints. Such joint pairs are closely connected in human geometry and move synchronously during movement. In addition, we observed that all joints have a high degree of correlation with the hip joint, which is called the origin of the human skeleton. The position of the origin directly determines the spatial position of the human body, which in turn affects the spatial position of all other joints. Therefore, the origin is used as a bridge to connect different parts to protect the integrity of the human pose. Specifically, the origin is combined with each human body part to form multiple origin-based parts. The spatial features of each origin-based part are modeled separately, and finally concat together to form a complete local spatial feature.

[0006] In addition, spatial relationships outside of the body parts are not ignored, such as the left ankle and right ankle joints, and the left knee and right knee joints. Although these joint pairs do not belong to the same body part, they still have certain dependencies. The relationship between all joints is modeled to generate global spatial features that complement the local spatial features. Therefore, the proposed invention has a parallel structural design, and two parallel channels are responsible for capturing global and local spatial features respectively. Finally, the outputs of the two channels are fused and then passed through the Temporal Block to obtain the temporal features of each joint at multiple time steps to generate more accurate 3D posture results.

[0007] The contents of this invention can be summarized into three aspects: 1. A new Origin-based PartTransformer (OPFormer) block is designed. While modeling the spatial relationship of the internal joints of different parts, the Origin of the skeleton is used as a bridge to connect different parts to protect the integrity of the human pose.

[0008] 2. A new alternating network structure is designed, with a dual-channel parallel structure to capture spatial features of different ranges, and a temporal block to capture the temporal features of different joints to improve the results of 3D pose estimation.

[0009] 3. The overall network structure mainly consists of five components. The Origin-based Part TransformerBlock and the Spatial Transformer Block form a two-channel parallel structure, which is used to capture the spatial features of different sizes of a single-frame 2D pose. Among them, the proposed OPFormer Block is responsible for capturing the correlation between the joints within the part and generating local spatial features. In addition, the Spatial Transformer Block is used to capture the dependencies of all joints of the human body and generate global spatial features. The results of the two channels are fused in an adaptive manner, and the two are combined to enhance the network's modeling capabilities for spatial features of different ranges. The fused spatial features are then input into the Temporal TransformerBlock to capture the temporal features of each joint. The modeling process of spatial and temporal features is executed alternately. After several rounds of iterative structures, the final 3D pose sequence is obtained through Regression Head processing.

[0010] A 3D human body posture spatial feature modeling method, the method comprising the following steps: Step 1: Using a pre-trained 2D human body posture detector to process an input image or video frame, detect and generate the two-dimensional coordinates of each human body joint, and generate a 2D skeleton. This process ensures that accurate input data is provided for subsequent 3D posture estimation.

[0011] Step 2: Divide the 2D skeleton into several major parts, including the right leg, left leg, head, right arm, and left arm. This division is based on the spatial and motion correlation between joints. For example, joints in the same part, such as the knee and ankle, shoulder and elbow, usually have a high correlation during movement. By taking these parts as objects to capture local spatial features, the motion characteristics of local joints can be better understood while reducing the complexity of global processing.

[0012] Step 3: Combine the joint points of each part with the skeleton origin to form multiple Origin-based Parts. This design not only captures the joint motion relationship within the part, but also maintains the continuity and integrity of the overall posture of the human body by connecting the local joints to a global reference point (Origin). The connection between Origin and each part ensures that the global consistency of the human body posture is not lost when estimating the local posture.

[0013] Step 4: Add spatial position encoding (SpatialPositionalEncoding) to the 2D skeleton that forms multiple origin-based parts. This encoding can represent the relative position of each joint point in space. Then input the processed data into SpatialTransformerEncoder (STE) to capture the global spatial characteristics of the entire skeleton. STE can effectively capture the long-distance dependencies between the joints of the human body through the self-attention mechanism. This process helps the model understand the interactions between non-locally related joints, such as hands and feet, shoulders and knees, etc., thereby improving the prediction accuracy of the overall posture.

[0014] Step 5: In addition to capturing global spatial features, local spatial features are also modeled for each origin-based part. Using STE, each origin-based part is used as a unit to extract the spatial dependency features within the joint through the self-attention mechanism to achieve local spatial feature modeling for each origin-based part; these local features can well capture the motion correlation within the part; next, the global spatial features are fused with the local spatial features to generate a spatial feature containing comprehensive information, thereby improving the accuracy of 3D posture prediction.

[0015] The present invention adopts a dual-channel parallel design in the model structure, where one channel is used to capture local spatial features and the other channel is used to capture global spatial features, and the two features are fused to generate more accurate spatial features.

[0016] Step 6: After completing the modeling of spatial features, temporal positional encoding is added to the fused spatial features to represent the motion trajectory of each joint point between multiple time frames; the dependency of each joint point in the time dimension is captured through the self-attention mechanism; finally, the spatiotemporal features are converted into three-dimensional coordinates of the joint points through the regression module to generate 3D human posture estimation results.

[0017] Furthermore, the generated 2D joint points constitute the skeleton structure of the human body, including the head, shoulders, hips, knees, and ankles; the generated 2D joint point data usually contains the x, y coordinates of each joint point and the confidence of the detection; the Hip joint is defined as the origin of the human body because it has a high spatial correlation in the human body structure and is a reference point for the spatial positions of other joints.

[0018] Furthermore, the fusion process uses an adaptive fusion method, including element-by-element multiplication of local spatial features and global spatial features, and weighted averaging the two features through weights generated by linear transformation, thereby generating a spatial feature containing comprehensive information, ensuring that the information of the two can be fully integrated.

[0019] Furthermore, the fused spatial features are further passed through the Temporal Transformer Encoder (TTE) to capture the temporal features of each joint in multiple time frames, that is, to capture the dependencies of each joint point in the time dimension to generate more accurate 3D pose estimation results; TTE can effectively process video sequences and analyze the movement patterns of joints over time, thereby improving the model's understanding of dynamic poses.

[0020] Furthermore, the image or video frame is based on a video captured by a monocular camera.

[0021] Furthermore, the optimization objectives of the generated 3D human pose estimation results include the position loss function And the speed loss function ; Among them, the position loss function Used to measure the accuracy of the estimated position, speed loss function Used to measure the smoothness of joint movement; . BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a flow chart of a 3D human body posture spatial feature modeling method disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following is a description of the implementation of the present application by specific specific embodiments. People familiar with the technology can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0025] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0026] The overall network structure mainly consists of five components. The Origin-based Part TransformerBlock and the Spatial Transformer Block form a two-channel parallel structure to capture the spatial features of different size ranges of a single-frame 2D pose. Among them, the proposed OPFormer Block is responsible for capturing the correlation between the joints within the part and generating local spatial features. In addition, the Spatial Transformer Block is used to capture the dependencies of all joints of the human body and generate global spatial features. The results of the two channels are fused in an adaptive manner, and the two are combined to enhance the network's modeling capabilities for spatial features of different ranges. The fused spatial features are then input into the Temporal TransformerBlock to capture the temporal features of each joint. The modeling process of spatial and temporal features is executed alternately. After several rounds of iterative structures, the final 3D pose sequence is obtained through Regression Head processing.

[0027] like Figure 1 As shown, a 3D human body posture spatial feature modeling method is provided, including the following steps: Step 1: The model inputs a 2D skeleton sequence ,in is the number of frames, is the number of joints, is the number of input channels. First, the Linear Embedding layer is used to map the input into a high-dimensional feature. , and then add a learnable spatial position encoding After the prepared features are input into the parallel channel, the Origin-based Part Transformer Block is used to calculate the local spatial features. ,in is the number of channels of embedded features, , is the network depth.

[0028] Step 2: Origin-based Part Transformer Block is responsible for capturing the local spatial features of a single-frame 2D posture. First, the complete human skeleton is divided into five parts: right leg, left leg, head, right arm, left arm. Specifically, the complete human features Split into 5 parts ,in For the The number of joints in each part, The above process is defined as: Based on the fact that all joints are closely related to the hip joint, the hip joint is defined as the origin of the human skeleton. For each part, the origin and its internal joints are combined into an origin-based part. Specifically, the feature corresponding to the origin and Get new features through Concat , The above process is defined as: .

[0029] Step 3: Features corresponding to each Origin-based Part , first use the Linear layer to map it to the corresponding query ,key Sum , passed to This generates the spatial features of a specific part , . It not only contains the spatial dependencies of the joints inside the part, but also contains the spatial position information of the part relative to the origin. The above process is defined as: ;in, The matrix representing the linear map, Represents the calculation process of the Multi-Head Self-Attention Layer for spatial modeling.

[0030] Step 4: Then, the obtained features Reclassify as Origin feature and part characteristics , Finally, the calculated values ​​from different Origin-based Parts are Reconcat into a feature , and then recombines the calculated different part features together with the origin features into a complete human body spatial feature The above process is defined as: ;in, is a learnable linear transformation.

[0031] Step 5: Use Spatial Transformer Block to calculate global spatial features ,in is the number of channels of embedded features, , is the network depth. After the two parallel channels model the spatial features of different ranges respectively, the outputs of the two channels are fused by adaptive fusion to generate complete spatial features. , . This process is defined as: ;in, is element-wise production, adaptive fusion weight and The calculation process is defined as: ;in, is a learnable linear transformation.

[0032] Step 6: Finally, the fused features Add time position coding , passed into the Temporal Transformer Block to calculate the temporal characteristics of joint motion , Finally, Apply a linear transformation Regression Head to estimate the final 3D pose ,in is the number of channels of the output feature.

[0033] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0034] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A 3D human body posture spatial feature modeling method, characterized in that: The method comprises the following steps: step 1: using a pre-trained 2D human posture detector to process an input image or video frame, detecting and generating the two-dimensional coordinates of each human joint, and generating a 2D skeleton; step 2: dividing the 2D skeleton into several main parts, including the right leg, the left leg, the head, the right arm and the left arm; step 3: combining the joint points of each part with the skeleton origin to form a plurality of origin-based parts; step 4: adding a spatial position encoding to the 2D skeleton formed with the plurality of origin-based parts, and the encoding can represent the relative position of each joint point in space; inputting the processed data into a Spatial Transformer Encoder (STE) to capture the global spatial features of the entire 2D skeleton; Step 5: Using STE, taking each Origin-basedPart as a unit, extract the spatial dependency features inside the joint through the self-attention mechanism to achieve local spatial feature modeling of each Origin-basedPart; fuse the global spatial features with the local spatial features to generate a spatial feature containing comprehensive information; Step 6: Add temporal positional encoding to the spatial features to represent the motion trajectory of each joint point between multiple time frames; capture the dependency relationship of each joint point in the time dimension through the self-attention mechanism; After the regression module, the spatiotemporal features are converted into three-dimensional coordinates of the joint points to generate 3D human posture estimation results.

2. A 3D human body posture spatial feature modeling method according to claim 1, characterized in that: Step 1: The 2D joint point data contained in the 2D skeleton includes the x, y coordinates of each joint point and the confidence of the detection; and the Hip joint is defined as the origin of the human body.

3. A 3D human body posture spatial feature modeling method according to claim 2, characterized in that: The method of fusing the global spatial features with the local spatial features to generate a spatial feature containing comprehensive information includes: fusing in an adaptive fusion manner, specifically: performing element-by-element multiplication of the local spatial features and the global spatial features, and performing weighted averaging of the two features using weights generated by linear transformation to generate a spatial feature containing comprehensive information.

4. A 3D human body posture spatial feature modeling method according to claim 3, characterized in that: The self-attention mechanism is used to capture the dependencies of each joint point in the time dimension, including: further passing the spatial features through the Temporal Transformer Encoder (TTE) to capture the timing features of each joint in multiple time frames, that is, capturing the dependencies of each joint point in the time dimension.

5. A 3D human body posture spatial feature modeling method according to claim 4, characterized in that: The image or video frame is based on a video captured by a monocular camera.

6. A 3D human body posture spatial feature modeling method according to claim 5, characterized in that: The optimization objectives of the generated 3D human pose estimation results include the position loss function And the speed loss function ; Among them, the position loss function Used to measure the accuracy of the estimated position, speed loss function Used to measure the smoothness of joint movement; .

Images