Adaptive temporal modeling-driven methods and media for human pose estimation in motion scenes

Abstract

This invention discloses an adaptive temporal modeling-driven method and medium for human pose estimation in motion scenes. It constructs a 3D human pose estimation network model for motion scenes based on the speed of motion, utilizing physical constraint-based motion priors and the uncertainty of predicting 3D human joints to improve the model's robustness to motion scenes. Using human motion videos at different speeds as input, it first obtains the corresponding 2D human joint sequence based on an existing high-quality 2D human pose estimation model. This 2D human joint sequence is then input into the network model to obtain a trained model. This invention can adaptively adjust the receptive field according to the motion speed, using a small receptive field to capture instantaneous changes in fast motion and a large receptive field to capture long-term dependencies in slow motion, thus adapting to different motion speeds and improving the accuracy of human pose estimation in motion scenes at different speeds.

Description

Technical Field

[0001] This invention belongs to the field of human pose estimation technology, specifically relating to an adaptive time modeling-driven method and medium for human pose estimation in motion scenes. Background Technology

[0002] Human pose estimation is a core task in computer vision. Its goal is to reconstruct the complete human pose structure by accurately detecting and locating key skeletal nodes (such as the head, shoulders, elbows, and knees). It is one of the classic challenges in computer vision research, and this technology can be used for motion analysis, health detection, human-computer interaction, and virtual reality. Currently, various 3D human pose estimation methods have been proposed and achieved good results under different motion scene data conditions. However, in real-world motion scenes, due to the temporal complexity brought about by rapid movement, drastic changes in pose, and the diversity of different motion patterns, existing models often employ fixed temporal modeling strategies. When faced with motion scenes with large speed changes and drastic movement amplitudes, this fixed strategy leads to insufficient utilization of temporal information, a significant decrease in model performance, and an inability to effectively assess the uncertainty of prediction results, making it difficult to meet the accuracy and reliability requirements of practical applications such as sports analysis and action recognition.

[0003] Most existing methods capture temporal dependencies in video sequences using fixed temporal modeling strategies. The PoseFormer approach, which employs two modules—a spatial Transformer and a temporal Transformer—models joint relationships within a single frame and temporal dependencies across frames, respectively, achieving excellent performance on standard datasets such as Human3.6M. However, this method uses a fixed-size time window (e.g., 243 frames) and a uniform attention mechanism to handle all motion modes. This not only fails to adaptively adjust the temporal receptive field in the face of rapid motion or drastic pose changes, leading to insufficient utilization of keyframe information, but also, due to the lack of uncertainty quantification mechanisms, the model makes overconfident and erroneous predictions under challenging conditions such as occlusion and motion blur, making it impossible to assess the reliability of the predictions. These two issues limit the robustness and reliability of 3D human pose estimation in moving scenes. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an adaptive time modeling-driven method for estimating human pose in motion scenes. This method can adaptively adjust the temporal receptive field for rapid motion or motion with drastic pose changes and provide confidence scores for joints when predicting their coordinates, thereby improving the robustness and reliability of human pose estimation in motion scenes.

[0005] The present invention discloses an adaptive time modeling-driven method for human pose estimation in motion scenes, comprising the following steps.

[0006] S1. Construct a training dataset of human motion videos containing high-speed and low-speed motion, and extract and preprocess it to obtain 2D human joint sequence.

[0007] S2. Construct a 3D human pose estimation network model that includes a spatiotemporal coding network, an adaptive time modeling module, a physical prior constraint module, an uncertainty perception module, and a pose regression layer. The time Transformer branch of the adaptive time modeling module integrates the adaptive time modeling module and can dynamically adjust the temporal receptive field according to the motion speed.

[0008] S3. Input the preprocessed 2D human joint sequence into the 3D human pose estimation network model, and combine the physical prior constraint loss and uncertainty heteroscedasticity loss to complete the model training and obtain the trained network model.

[0009] S4. Extract 2D human joint sequence from the human motion video to be detected and preprocess it. Input it into the trained network model. After being processed by each module in sequence, the pose regression layer outputs the 3D human pose estimation result and joint confidence, realizing human pose estimation under different motion speed scenarios.

[0010] Furthermore, a spatiotemporal coding network is used to extract spatiotemporal features from 2D keypoint sequences, and a separate spatiotemporal attention mechanism is used to model the spatial and temporal dimensions separately.

[0011] The adaptive temporal modeling module is used to understand the skeleton structure and motion patterns, learn the spatial relationships between joints and the changes of joints over time, and fuse spatial and temporal information to perform deep feature extraction.

[0012] The physical prior constraint module is used to learn the physical constraints of human motion, including bone length prior prediction, joint angle range prediction, and adaptive constraint weight prediction, thereby providing reasonable guidance for posture estimation.

[0013] The uncertainty perception module is used to predict the uncertainty of the joints. When the uncertainty is high, it automatically reduces the uncertainty loss weight, and when the uncertainty is low, it automatically increases the uncertainty loss weight to more robustly predict the 3D joint coordinates, while enhancing the interpretability of the model.

[0014] The pose regression layer fuses the features after spatiotemporal encoding, adaptive temporal modeling, and physical constraint enhancement, and outputs 3D joint coordinates and corresponding uncertainty estimates to obtain the final human pose estimation result.

[0015] Furthermore, the spatiotemporal coding network adopts a split spatiotemporal Transformer architecture, which includes two parallel branches. The processing order is spatial first and temporal first and spatial first, respectively. The outputs of the two branches are dynamically fused through a learnable attention fusion layer.

[0016] Furthermore, the adaptive temporal modeling module is a Transformer-based network that learns the correlation between different joints in the same frame and the temporal correlation of the same joint between different frames by modeling the joints in time and space based on the input 2D human joint sequence.

[0017] Furthermore, the adaptive temporal modeling module consists of a motion velocity estimator and a multi-scale temporal attention unit. The motion velocity estimator learns motion velocity features by calculating inter-frame feature differences and using an MLP network. It also assigns weights to different time scales through a dynamic router. The multi-scale temporal attention unit computes temporal attention in parallel at multiple time scales. For high-speed motion, a time downsampling rate less than a set value is used to capture local motion details. For low-speed motion, a time downsampling rate greater than a set value is used to preserve global motion dependencies. Through a dynamic fusion mechanism, the multi-scale temporal attention output is adaptively weighted and fused according to the routing weights output by the motion velocity estimator to obtain the temporal modeling features for motion perception.

[0018] Furthermore, the uncertainty perception module is divided into two prediction heads: a mean prediction head and an uncertainty prediction head. The coordinates of each joint point follow a Gaussian distribution. The mean prediction head outputs the 3D joint coordinate prediction mean, and the uncertainty prediction head predicts the variance to represent the uncertainty of the joint point. By using heteroscedasticity loss during training, the model output is made more robust.

[0019] Furthermore, during the training phase of S3, the adaptive temporal modeling module performs the following steps:

[0020] The 2D key sequence with pre-encoded features is used as input, and the global features are input into the motion velocity estimation module to calculate the inter-frame differences and estimate the weights at different time scales.

[0021] Another branch uses three different scales of temporal dimension downsampling to simulate receptive fields of different scales to adapt to different speeds of motion. Then it is input into temporal attention to learn the features of key points in the temporal dimension. Finally, it obtains features of the same size in the temporal dimension through temporal dimension upsampling.

[0022] The weights of the three time scales obtained from motion speed perception and the features obtained from the three-scale time attention are dynamically weighted and fused to finally obtain adaptive time modeling features.

[0023] Furthermore, during the training phase of S3, the uncertainty-aware module performs the following steps:

[0024] The feature vectors obtained after the adaptive time modeling module are input into the physical prior network, and the model output is made more accurate by using bone length loss, joint angle loss and symmetry constraint loss that the left and right postures of the human body should be symmetrical.

[0025] The obtained feature vector is input into the uncertainty perception module, which is divided into two branches: the mean prediction head and the variance prediction head. The mean prediction head is responsible for outputting the mean. The 3D coordinates of the key points are represented by the variance prediction head, which is responsible for outputting the variance. , representing the confidence level of each joint, utilizes uncertainty-aware loss to reduce the loss weight when the uncertainty is higher than a set value, and increases the loss weight when the uncertainty is lower than a set value to improve the robustness of the model.

[0026] Further, step S1 includes the following steps:

[0027] S1.1: Collect videos containing different motion speeds from existing human motion datasets (such as the Human3.6M dataset), including high-speed motion scenes (such as running and jumping) and low-speed motion scenes (such as walking and slow turning).

[0028] S1.2: Preprocess the collected videos, normalize the resolution, and segment the sequences according to a fixed number of frames (243 frames) to ensure that each training sample contains a complete motion cycle.

[0029] S1.3: Use a pre-trained high-quality 2D human pose detector (such as OpenPose, AlphaPose) to perform joint detection on each frame of the video and obtain a 2D joint coordinate sequence;

[0030] S1.4: Post-process the detected 2D joint sequence, including coordinate normalization, normalizing the image coordinates to the range of [-1,1], and interpolating missing joints to ensure the quality of the input data.

[0031] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0032] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0033] In summary, this invention discloses an adaptive temporal modeling-driven method for human pose estimation in motion scenes, belonging to the field of human pose estimation technology. It uses human motion videos containing different motion speeds to construct a training dataset; it constructs an adaptive temporal modeling 3D human pose estimation network model for motion scenes based on the speed of motion, and also uses physical constraint-based motion priors and the uncertainty of predicting 3D human joints to improve the model's robustness to motion scenes. During the training phase, using human motion videos at different speeds as input, the corresponding 2D human joint sequence is obtained based on an existing high-quality 2D human pose estimation model. This 2D human joint sequence is then input into the network model to obtain a trained model. During the testing phase, a different 2D human joint sequence than the training set is used as input into the trained network model to obtain the 3D human pose estimation result. Compared with traditional methods, the network model proposed in this invention can adaptively adjust the receptive field according to the motion speed, using a small receptive field to capture instantaneous changes in fast motion and a large receptive field to capture long-term dependencies in slow motion, thus adapting to different motion speeds and improving the accuracy of human pose estimation in motion scenes of different speeds.

[0034] The beneficial effects of this invention are as follows:

[0035] This invention explores and constructs temporal dependencies under different speed modes in motion scenes, proposing a method for 3D human pose estimation in motion scenes that does not rely on a fixed time window but utilizes adaptive temporal modeling and uncertainty perception. This method can directly construct training samples using videos of different motion speeds in existing datasets. Compared to human pose estimation methods based on fixed temporal modeling, the method described in this invention is more flexible and efficient. This invention introduces an adaptive temporal attention module, which predicts motion speed and dynamically adjusts the temporal receptive field. Compared to traditional fixed-window temporal modeling methods, this module can provide a wider temporal range for fast-moving scenes. The invention significantly reduces redundant computational overhead in slow-motion scenes, achieving a balance between computational efficiency and prediction accuracy. It also introduces a physical prior constraint module, which effectively learns the physical constraints of human motion (including bone length consistency, joint angle limitations, left-right symmetry, etc.), thereby significantly improving the physical plausibility of the prediction results. Furthermore, the invention introduces an uncertainty perception mechanism, which quantifies the reliability of each joint prediction by simultaneously predicting 3D joint coordinates and logarithmic variance. Compared to traditional deterministic regression methods, this mechanism provides credibility assessment in challenging scenarios such as occlusion and motion blur, avoiding overconfident erroneous predictions and improving the model's interpretability. Attached Figure Description

[0036] Figure 1 This is a flowchart illustrating the method described in this invention;

[0037] Figure 2 This is a flowchart of the training dataset construction process described in this invention;

[0038] Figure 3 This is a schematic diagram of the network model described in this invention;

[0039] Figure 4 This is a schematic diagram of the workflow of the adaptive time modeling module of the present invention;

[0040] Figure 5 This is a schematic diagram of the time attention sampling module of the present invention;

[0041] Figure 6 This is a schematic diagram of the spatial attention module workflow of the present invention;

[0042] Figure 7 This is a schematic diagram of the uncertainty perception workflow of the present invention;

[0043] Figure 8 This is a diagram showing the experimental results of an embodiment of the present invention. Detailed Implementation

[0044] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific implementation examples and accompanying drawings.

[0045] like Figure 1 As shown, the adaptive time modeling-driven motion scene human pose estimation method of this invention improves upon the basic 3D human pose estimation process. It adds an adaptive time modeling module, an uncertainty perception module, and a physical prior constraint module to the original spatiotemporal Transformer backbone network and pose regression layer, thereby making it easier to perform human pose estimation in complex motion scenes and improving the accuracy and robustness of human pose estimation at different motion speeds. Specifically, it includes the following steps:

[0046] S1. Construct a training dataset of human motion videos containing high-speed and low-speed motion, and extract and preprocess it to obtain 2D human joint sequence.

[0047] S2. Construct a 3D human pose estimation network model that includes a spatiotemporal coding network, an adaptive time modeling module, a physical prior constraint module, an uncertainty perception module, and a pose regression layer. The time Transformer branch of the adaptive time modeling module integrates the adaptive time modeling module and can dynamically adjust the temporal receptive field according to the motion speed.

[0048] S3. Input the preprocessed 2D human joint sequence into the 3D human pose estimation network model, and combine the physical prior constraint loss and uncertainty heteroscedasticity loss to complete the model training and obtain the trained network model.

[0049] S4. Extract 2D human joint sequence from the human motion video to be detected and preprocess it. Input it into the trained network model. After being processed by each module in sequence, the pose regression layer outputs the 3D human pose estimation result and joint confidence, realizing human pose estimation under different motion speed scenarios.

[0050] like Figure 2 As shown, step S1 specifically includes the following steps:

[0051] High-speed and low-speed human motion videos were collected from existing human motion video datasets (such as Human3.6M and MPI-INF-3DHP). OpenPose was used to extract the 2D joint coordinate sequence of the human body. The obtained 2D joint sequence was further augmented by data augmentation, horizontal flipping, and Gaussian noise was added to simulate 2D test error. Finally, the continuous sequence was cut into fixed-length time segments. The high-speed and low-speed 2D joint features [B,T,J,2] obtained at this time were used for training, where B represents the batch size, T represents the number of frames in the video sequence, J represents the number of human joints, and 2 represents the horizontal and vertical coordinates of the joints.

[0052] like Figure 3 As shown, S2 specifically includes a constructed adaptive time modeling-driven uncertainty-aware motion scene human pose estimation network model, including a spatiotemporal coding network, an adaptive time modeling module, a physical prior constraint module, an uncertainty perception module, and a pose regression layer.

[0053] The spatiotemporal coding network is used to extract the spatiotemporal features of 2D joint sequences. It adopts a separate spatiotemporal attention mechanism to model the spatial and temporal dimensions separately. The spatiotemporal coding network is used to map 2D coordinates to a high-dimensional feature space to enhance expressive power. It injects spatial and temporal position information into the joints through temporal position encoding and spatial position encoding.

[0054] The adaptive temporal modeling module is used to dynamically adjust the temporal receptive field according to the motion speed. Through a multi-scale temporal attention mechanism and a motion speed estimator, it provides adaptive temporal modeling capabilities for high-speed and low-speed motion scenarios. The adaptive temporal modeling module is used to understand the skeleton structure and motion patterns, learn the spatial relationships between joints and the changes of joints over time, and fuse spatial and temporal information to perform deep feature extraction.

[0055] The physical prior constraint module is used to learn the physical constraints of human motion, including bone length prior prediction, joint angle range prediction, and adaptive constraint weight prediction, thereby providing reasonable guidance for posture estimation. The physical prior constraint module consists of multiple MLP modules, which use physical prior information such as bone length, joint angle, and skeletal symmetry as constraints to allow the model to learn joint features that are more consistent with human structure.

[0056] The uncertainty perception module is used to predict the uncertainty of the joints. When the uncertainty is high, it automatically reduces the uncertainty loss weight, and when the uncertainty is low, it automatically increases the uncertainty loss weight to more robustly predict the 3D joint coordinates and enhance the interpretability of the model. In other words, the uncertainty perception module is used to predict the uncertainty of each joint, dynamically adjust the feature weights through uncertainty, and use uncertainty weighted loss to improve the robustness of the model to difficult samples during training.

[0057] The pose regression layer fuses the features after spatiotemporal encoding, adaptive temporal modeling and physical constraint enhancement, and outputs 3D joint coordinates and corresponding uncertainty estimates to obtain the final human pose estimation result.

[0058] The spatiotemporal coding network adopts a split spatiotemporal Transformer architecture, which includes two parallel branches (spatial-temporal module and temporal-spatial module), and adopts a processing order of spatial first and temporal first and spatial first, respectively. The outputs of the two branches are dynamically fused through a learnable attention fusion layer.

[0059] The adaptive temporal modeling module is a Transformer-based network that learns the relationships between different joints in the same frame and the temporal relationships between the same joints in different frames by modeling the joints in time and space based on the input 2D human joint sequence.

[0060] The adaptive temporal modeling module consists of a motion velocity estimator and a multi-scale temporal attention unit. The motion velocity estimator learns motion velocity features by calculating inter-frame feature differences and using an MLP network. It also assigns weights to different time scales through a dynamic router. The multi-scale temporal attention unit computes temporal attention in parallel at multiple time scales. For high-speed motion, it uses a downsampling rate greater than the set value to capture long-term dependencies, while for low-speed motion, it uses a downsampling rate less than the set value to preserve detailed information. Through a dynamic fusion mechanism, the multi-scale temporal attention output is adaptively weighted and fused according to the routing weights output by the motion velocity estimator to obtain the temporal modeling features for motion perception.

[0061] The physical prior constraint module consists of a bone length prior prediction network and a joint angle prior prediction network. The bone length prior prediction network predicts the reasonable bone length distribution of 16 limbs from global features, and the joint angle prior prediction network predicts the mean and standard deviation of 18 joint angles to construct a prior Gaussian distribution of joint angles, which is used to constrain joint angles within a reasonable range of human physiology.

[0062] The uncertainty perception module includes an uncertainty prediction head and an uncertainty gating unit. The uncertainty prediction head is a linear layer that maps the pose representation features to the logarithmic variance of the 3D coordinates of each joint, which is used to quantify the confidence of the model's prediction for each joint. The uncertainty gating unit uses the uncertainty of the prediction to dynamically weight the features, and suppresses the feature response in high uncertainty areas through the gating mechanism, thereby enhancing the feature expression in low uncertainty areas.

[0063] When training the network model, the adaptive time modeling module performs the following steps:

[0064] The 2D key sequence with pre-encoded features is used as input, and the global features are input into the motion velocity estimation module to calculate the inter-frame differences and estimate the weights at different time scales.

[0065] Another branch uses three different scales of temporal dimension downsampling to simulate receptive fields of different scales to adapt to different speeds of motion. Then it is input into temporal attention to learn the features of key points in the temporal dimension. Finally, it obtains features of the same size in the temporal dimension through temporal dimension upsampling.

[0066] The weights of the three time scales obtained from motion speed perception and the features obtained from the three-scale time attention are dynamically weighted and fused to finally obtain adaptive time modeling features.

[0067] For example, the following steps are taken: First, obtain the features of the input 2D keypoint sequence. ,exist Velocity characteristics were calculated at these three different time scales. Represents batch size. Represents the length of the time series. Represents the number of key points. Represents the number of feature channels. Representing the Frame; Features Input motion speed estimator, and Representing the first Frame and the The features of the frames are used to calculate the differences in features between frames as a motion proxy, as shown in the following expression:

[0068]

[0069] The global motion agent is obtained by averaging over the time dimension:

[0070]

[0071] Extracting velocity features using a velocity estimator:

[0072]

[0073] in and Represents the learnable weight parameters in a fully connected layer network. This represents the activation function.

[0074] Finally, weights are assigned to different time scales using a dynamic router:

[0075]

[0076] in The number representing the time scale. Representing the Routing weights at various scales Represents the learnable linear layer weight parameters. Represents the activation function. This represents the normalization operation, and the final route weight is... ,like Figure 4 As shown.

[0077] like Figure 5 As shown, for each time scale , will feature The input to the adaptive time modeling module first performs time downsampling. After further reshaping the features, temporal attention is calculated, as shown in the following expression:

[0078]

[0079] in , , , , , Represents the learnable projection matrix. , , This represents the feature tensor after projection. represent transpose, As a scaling factor, the data is finally upsampled back to the original time dimension. This is an upsampling operation.

[0080]

[0081] Finally, features from multiple different time scales were combined. Perform weighted fusion:

[0082]

[0083] in Represents broadcast multiplication. Representing the The weights for each time scale are used to obtain the final adaptive time modeling features. .

[0084] Traditional temporal modeling methods use a fixed temporal receptive field to handle all motion scenarios, which cannot effectively cope with drastic changes in motion speed. For high-speed motion, a fixed, small receptive field is insufficient to capture long-term temporal dependencies; for low-speed motion, a fixed, large receptive field introduces redundant information and loses detailed features. In contrast, this embodiment proposes a motion-aware adaptive temporal modeling method. This method can dynamically adjust the scale of the temporal receptive field according to the motion speed features of the input sequence. Through a multi-scale parallel temporal attention mechanism and a learnable dynamic router, a smaller temporal downsampling rate is allocated for high-speed motion scenarios to capture local motion details, while a larger temporal downsampling rate is allocated for low-speed motion scenarios to preserve global motion dependencies, thereby achieving adaptive temporal modeling for different motion speed scenarios. This method improves the accuracy of pose estimation in complex motion scenarios while significantly enhancing the model's robustness to changes in motion speed.

[0085] When training the human pose estimation network model for the aforementioned motion scene, the spatial attention module performs the following steps; such as... Figure 6 As shown, for the input 2D pose sequence, each frame contains There are 1 key points, and the coordinates of each key point are represented as follows: Input all joint coordinates into the spatial modeling module. First, use MLP to linearly project the original coordinates of each joint into a high-dimensional feature space.

[0086]

[0087] in , Representing the High-dimensional spatial features of each joint For batch size, The number of key points. For embedded dimensions, Represents a linear layer.

[0088] Then, to preserve the spatial location information of each joint, spatial location encoding is superimposed on the embedded features of each joint:

[0089] ,

[0090] in , representing the learnable spatial location encoding matrix. Finally, Input TransformerEncoder to calculate spatial attention between joints:

[0091]

[0092] in , , , , , Represents the learnable projection matrix. , , This represents the feature tensor after projection. represent transpose, As a scaling factor, the final TransformerEncoder outputs spatial encoded features. .

[0093] The obtained temporal-space modeling features are input into the physical prior constraint module. The physical prior constraint module further predicts bone length prior, joint angle prior, and adaptive constraint weights after extracting global features. The expression for bone length prediction is as follows:

[0094]

[0095] in and Representing the The two ends of the bone length and Representing the 3D coordinates of the joints at the first and second endpoints of a bone. The prior loss expression for bone length, representing the number of frames, is as follows:

[0096]

[0097] in Representing the Predicted bone length, Representing the First, the length of the bone is the primary factor. This represents the batch size.

[0098] The expression for joint angle prediction is as follows:

[0099]

[0100] in and For the first The angle prior loss expression for the two limb vectors corresponding to each joint angle is as follows:

[0101]

[0102] in and Representing the first The prior mean and standard deviation of each joint angle. Represents batch size. Represents the number of frames.

[0103] The prior mean and standard deviation are calculated by iterating through all samples in the dataset:

[0104]

[0105]

[0106] in Representing the The nth sample, the nth Each joint angle value This represents the number of samples in the dataset.

[0107] When training the human pose estimation network model for the motion scene, the uncertainty perception module performs the following steps: inputting the feature vector obtained after the adaptive time modeling module into the physical prior network, and using bone length loss, joint angle loss and symmetry constraint loss that the left and right postures of the human body should be symmetrical to make the model output more accurate.

[0108] The obtained feature vector is input into the uncertainty perception module, which is divided into two branches: the mean prediction head and the variance prediction head. The mean prediction head is responsible for outputting the mean. The 3D coordinates of the key points are represented by the variance prediction head, which is responsible for outputting the variance. , representing the confidence level of each joint, utilizes uncertainty-aware loss to reduce the loss weight when the uncertainty is higher than a set value, and increases the loss weight when the uncertainty is lower than a set value to improve the robustness of the model;

[0109] By obtaining the confidence information of key points, we can clearly identify which predictions are unreliable, which can enhance the interpretability of the model and be used to evaluate the predictive quality of the model.

[0110] For example:

[0111] First, the pose representation features after spatiotemporal encoding and adaptive temporal modeling are obtained from the last layer of the network, TransformerBlock. The resulting features are: ,in Represents batch size. Represents the length of the time series. Represents the number of key points. Channel dimensions representing pose characteristics. For example... Figure 7 As shown, the reshaped pose representation features are simultaneously input into the 3D coordinate prediction branch and the uncertainty prediction branch, and the two branches are processed in parallel using independent linear layers; then a joint prediction operation is performed to obtain the 3D joint coordinates and the corresponding uncertainty estimates, as shown in the following expression:

[0112]

[0113]

[0114] in Represents the 3D coordinate prediction weight matrix. Represents the uncertainty prediction weight matrix. This represents the deviation in 3D coordinate prediction. Representing human joints coordinate, Represents the logarithm of uncertainty. This represents the bias in predictions due to uncertainty. To adaptively adjust feature weights using uncertainty information, a gating mechanism is designed, such as... Figure 7 As shown, the expression is as follows:

[0115]

[0116] in For learnable parameters, when uncertainty When it is large, A smaller value indicates lower confidence in the joint prediction. Then, the expanded gating weights are multiplied element-wise (Hadamard product) with the human pose features obtained from the 3D coordinate prediction branch to obtain the gated features:

[0117]

[0118]

[0119] in Represents the Hadamard product. This represents the features after uncertainty gating. Representing a linear layer, the feature responses of joints with uncertainties higher than a set value are suppressed, while the feature responses of joints with uncertainties lower than a set value are preserved. This is for the final 3D joint coordinate prediction.

[0120] The uncertainty loss function is:

[0121]

[0122] in Indicates the predicted 3D coordinates. Represents true 3D coordinates. Represents the variance of the prediction. To calculate the predicted log-variance, a negative log-likelihood loss function is used to incorporate the uncertainty of the prediction into the training process.

[0123] Furthermore, in step 3, the loss function used to train the motion scene human pose estimation network model is the MSE loss function.

[0124] Step S4 includes the following steps:

[0125] Input the video sequence of human motion to be detected;

[0126] The 2D joint sequence to be detected is obtained using a 2D human posture detector;

[0127] The obtained 2D joint sequence is embedded with features and encoded at position, and then input into a spatiotemporal coding network to obtain the pose representation features for human motion adaptation.

[0128] The pose representation features are input into the uncertainty perception module. The uncertainty prediction branch predicts the degree of joint uncertainty, and the 3D coordinate prediction branch predicts the 3D features of the joints. Then, the features are refined through the uncertainty gating mechanism to obtain the final refined 3D features of human joints.

[0129] Finally, the refined features are mapped to 3D space through a linear regression layer to obtain the predicted 3D human pose results.

[0130] Figure 8 The above sequence shows the model's estimation results of human posture in motion scenarios. The images in the sequence above are 2D joint sequences of the human body. After passing through the network architecture of this invention, the predicted 3D human posture is shown in the sequence below, which achieves high accuracy in high-speed motion scenarios.

[0131] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0132] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0133] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the adaptive time modeling driven motion scene human pose estimation methods in the above embodiments.

[0134] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0135] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0138] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for human pose estimation in motion scenes driven by adaptive time modeling, characterized in that, Includes the following steps: S1. Construct a training dataset of human motion videos containing high-speed and low-speed motion, and extract and preprocess it to obtain 2D human joint sequence. S2. Construct a 3D human pose estimation network model that includes a spatiotemporal coding network, an adaptive time modeling module, a physical prior constraint module, an uncertainty perception module, and a pose regression layer. The time Transformer branch of the adaptive time modeling module integrates the adaptive time modeling module and can dynamically adjust the temporal receptive field according to the motion speed. S3. Input the preprocessed 2D human joint sequence into the 3D human pose estimation network model, and combine the physical prior constraint loss and uncertainty heteroscedasticity loss to complete the model training and obtain the trained network model. S4. Extract 2D human joint sequence from the human motion video to be detected and preprocess it. Input it into the trained network model. The pose regression layer outputs 3D human pose estimation results and joint confidence, realizing human pose estimation under different motion speed scenarios. Spatiotemporal coding networks are used to extract spatiotemporal features of 2D keypoint sequences, and a separate spatiotemporal attention mechanism is used to model the spatial and temporal dimensions separately. The adaptive temporal modeling module is used to understand the skeleton structure and motion patterns, learn the spatial relationships between joints and the changes of joints over time, and fuse spatial and temporal information to perform deep feature extraction. The physical prior constraint module is used to learn the physical constraints of human motion, including bone length prior prediction, joint angle range prediction, and adaptive constraint weight prediction, thereby providing reasonable guidance for posture estimation. The uncertainty perception module is used to predict the uncertainty of the joints. When the uncertainty is higher than the set value, it automatically reduces the uncertainty loss weight. When the uncertainty is lower than the set value, it automatically increases the uncertainty loss weight to more robustly predict the 3D joint coordinates and enhance the interpretability of the model. The pose regression layer fuses the features after spatiotemporal encoding, adaptive temporal modeling, and physical constraint enhancement, and outputs 3D joint coordinates and corresponding uncertainty estimates to obtain the final human pose estimation result.

2. The adaptive time modeling-driven method for human pose estimation in motion scenes according to claim 1, characterized in that, The spatiotemporal coding network adopts a separate spatiotemporal Transformer architecture, which includes two parallel branches. The processing order is spatial first and temporal first and spatial first, respectively. The outputs of the two branches are dynamically fused through a learnable attention fusion layer.

3. The adaptive time modeling-driven method for estimating human pose in motion scenes according to claim 2, characterized in that, The adaptive temporal modeling module is a Transformer-based network that learns the relationships between different joints in the same frame and the temporal relationships between the same joints in different frames by modeling the joints in time and space based on the input 2D human joint sequence.

4. The adaptive time modeling-driven method for human pose estimation in motion scenes according to claim 3, characterized in that, The adaptive temporal modeling module consists of a motion velocity estimator and a multi-scale temporal attention unit. The motion velocity estimator learns motion velocity features by calculating inter-frame feature differences and using an MLP network. It also assigns weights to different time scales through a dynamic router. The multi-scale temporal attention unit computes temporal attention in parallel at multiple time scales. For high-speed motion, a time downsampling rate less than a set value is used to capture local motion details. For low-speed motion, a time downsampling rate greater than a set value is used to preserve global motion dependencies. Through a dynamic fusion mechanism, the multi-scale temporal attention output is adaptively weighted and fused according to the routing weights output by the motion velocity estimator to obtain the temporal modeling features for motion perception.

5. The adaptive time modeling-driven method for estimating human pose in motion scenes according to claim 2, characterized in that, The uncertainty perception module is divided into two prediction heads: the mean prediction head and the uncertainty prediction head. The coordinates of each joint point follow a Gaussian distribution. The mean prediction head outputs the 3D joint coordinate prediction mean, and the uncertainty prediction head predicts the variance to represent the uncertainty of the joint point. By using heteroscedasticity loss during training, the model output is made more robust.

6. The adaptive time modeling-driven method for human pose estimation in motion scenes according to claim 1, characterized in that, During the training phase of S3, the adaptive time modeling module performs the following steps: The 2D key sequence with pre-encoded features is used as input, and the global features are input into the motion velocity estimation module to calculate the inter-frame differences and estimate the weights at different time scales. Another branch uses three different scales of temporal dimension downsampling to simulate receptive fields of different scales to adapt to different speeds of motion. Then it is input into temporal attention to learn the features of key points in the temporal dimension. Finally, it obtains features of the same size in the temporal dimension through temporal dimension upsampling. The weights of the three time scales obtained from motion speed perception and the features obtained from the three-scale time attention are dynamically weighted and fused to finally obtain adaptive time modeling features.

7. The adaptive time modeling-driven method for human pose estimation in motion scenes according to claim 1, characterized in that, During the training phase of S3, the uncertainty-aware module performs the following steps: The feature vectors obtained after the adaptive time modeling module are input into the physical prior network, and the model output is made more accurate by using bone length loss, joint angle loss and symmetry constraint loss that the left and right postures of the human body should be symmetrical. The obtained feature vector is input into the uncertainty perception module, which is divided into two branches: the mean prediction head and the variance prediction head. The mean prediction head is responsible for outputting the mean. The 3D coordinates of the key points are represented by the variance prediction head, which is responsible for outputting the variance. , representing the confidence level of each joint, utilizes uncertainty-aware loss to reduce the loss weight when the uncertainty is higher than a set value, and increase the loss weight when the uncertainty is lower than a set value to improve the robustness of the model.

8. The adaptive time modeling-driven method for human pose estimation in motion scenes according to claim 1, characterized in that, S1 specifically includes, S1.1 Collect videos containing different motion speeds from existing human motion datasets, including high-speed motion scenes and low-speed motion scenes; S1.2 Preprocess the collected videos, including resolution normalization and sequence segmentation by a fixed number of frames, to ensure that each training sample contains a complete motion cycle; S1.

3. Using a pre-trained high-quality 2D human pose detector, perform joint detection on each frame of the video to obtain a 2D joint coordinate sequence. S1.4 Post-processing of the detected 2D joint sequence includes coordinate normalization, normalizing the image coordinates to the range of [-1,1], and interpolating missing joints to ensure the quality of the input data.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method as described in any one of claims 1 to 8.

Images