A human action prediction method and device based on action-aware dynamic graph structure

Abstract

The application discloses a human action prediction method and device based on action-aware dynamic graph structure, and belongs to the technical field of computer vision and graph neural network. By acquiring a historical motion sequence, and performing mask joint reconstruction on the sequence, a dynamic attention graph for representing spatial dependency between joints under a current action mode is generated. The dynamic attention graph is fused with a shared adjacency matrix to construct an action-aware adjacency matrix. Based on the action-aware adjacency matrix, graph convolution is performed on the historical motion sequence to extract features rich in spatial topological information. Multi-scale time sequence convolution is performed on the spatial features to capture local details and long-range dependencies under different receptive fields, and time sequence features are extracted. Future motion sequences are predicted according to the time sequence features. The application realizes adaptive spatial modeling and fine-grained time sequence modeling of action modes, and improves the accuracy and naturalness of human action prediction.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and graph neural network technology, and in particular to a method, apparatus, device and storage medium for predicting human motion based on a motion-aware dynamic graph structure. Background Technology

[0002] Human motion prediction is a fundamental research topic in computer vision and graphics. Its core objective is to model the spatiotemporal evolution of human motion based on observed historical posture sequences, thereby accurately predicting future changes in human posture. This technology shows great promise in cutting-edge applications such as autonomous driving (e.g., predicting pedestrian movements), human-computer interaction (e.g., understanding human intentions), and virtual reality (e.g., generating realistic virtual characters), and has therefore attracted widespread attention from academia and industry.

[0003] In recent years, the advent of graph convolutional networks (GCNNs) has brought groundbreaking progress to human motion prediction tasks. Because the human skeleton naturally possesses graph structure properties, with joints considered as nodes and skeletal connections as edges, GCNNs can explicitly model the intrinsic topological relationships of the human skeleton through adjacency matrices. Compared to traditional recurrent neural networks (RNNs) or convolutional neural networks (CNNs), GCNNs have a significant advantage in capturing spatial dependencies between joints. Therefore, much research has focused on enhancing the ability to model joint relationships by improving the design of adjacency matrices.

[0004] Existing methods typically learn a uniform adjacency matrix from the data and then fix it for all action samples after training. These methods assume that different actions share the same spatial dependency patterns, thus abstracting the relationships between joints into a static, globally universal topology. However, this assumption deviates significantly from the actual characteristics of human movement. Human movements are highly diverse and complex, and the joint coordination patterns involved in different actions often differ significantly. For example, in the action of "sitting," the movements of the upper and lower limbs are relatively independent, and the connection between the left hand and the right leg is weak; while in the action of "walking," the limbs need to swing in coordination to maintain balance, and there is a strong coupling relationship between the arm and the opposite leg. If a uniform adjacency matrix is ​​used to encode all actions, the model will struggle to distinguish the differences in the strength of joint relationships between different actions, thus limiting its ability to express action-specific spatial dependencies.

[0005] In fact, constructing joint relationship representations that can adaptively adjust according to action patterns is one of the key paths to improving the performance of human motion prediction models. Action-aware spatial modeling helps models more accurately understand the motion semantics of the current action, capture the collaborative relationships of dynamic changes between joints, and thus generate more natural prediction results that conform to the laws of human movement. However, how to achieve such action-aware dynamic spatial modeling within a graph convolutional network framework remains a pressing technical challenge. Existing methods typically cannot dynamically adjust the graph structure based on the input action during the inference phase, resulting in insufficient generalization ability when facing complex or unseen actions. Therefore, a dynamic graph structure modeling method that can simultaneously consider spatial dependency generality and action specificity is urgently needed to further improve the accuracy and robustness of human motion prediction. Summary of the Invention

[0006] The present invention aims to at least partially solve one of the technical problems in the related art.

[0007] To address this, this invention proposes a human motion prediction method based on a motion-aware dynamic graph structure. The method involves acquiring historical motion sequences and performing masked joint reconstruction to generate a dynamic attention graph representing the spatial dependencies between joints in the current motion pattern. This dynamic attention graph is then fused with a shared adjacency matrix to construct a motion-aware adjacency matrix. Based on this matrix, graph convolution is performed on the historical motion sequences to extract spatial features. Multi-scale temporal convolution is then applied to these spatial features to capture local details and long-range dependencies, extracting temporal features. Finally, future motion sequences are predicted based on these temporal features, achieving accurate prediction of human motion and effectively improving the accuracy and naturalness of the prediction results.

[0008] Another objective of this invention is to provide a human motion prediction device based on a motion-aware dynamic graph structure.

[0009] The third objective of this invention is to provide a computer device.

[0010] The fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0011] To achieve the above objectives, this invention proposes a method for predicting human motion based on a motion-aware dynamic graph structure, comprising:

[0012] S1, Obtain the historical motion sequence, and perform masked joint reconstruction on the historical motion sequence to generate a dynamic attention map; S2, fuse the dynamic attention map with the shared adjacency matrix to obtain the action-aware adjacency matrix; S3, based on the action-aware adjacency matrix, performs graph convolution on historical motion sequences to extract spatial features; S4 performs multi-scale temporal convolution on spatial features to extract temporal features; S5 predicts future motion sequences based on temporal characteristics.

[0013] The human motion prediction method based on motion-aware dynamic graph structure according to an embodiment of the present invention may also have the following additional technical features: In one embodiment of the present invention, the step of acquiring historical motion sequences and performing masked joint reconstruction on the historical motion sequences to generate a dynamic attention map includes: S11, Obtain the historical motion sequence, randomly mask the joints in the historical motion sequence, and replace the masked joints with the corresponding joint average to generate the masked motion sequence; S12, input the masked motion sequence into the self-attention mechanism to calculate the dynamic attention map.

[0014] In one embodiment of the present invention, fusing the dynamic attention map with the shared adjacency matrix to obtain an action-aware adjacency matrix includes: S21, obtain the dynamic attention map and the shared adjacency matrix, and perform a weighted summation of the dynamic attention map and the shared adjacency matrix to generate the action-aware adjacency matrix.

[0015] In one embodiment of the present invention, the step of performing graph convolution on the historical motion sequence based on the action-aware adjacency matrix to extract spatial features includes: S31, embed the features of the historical motion sequence to obtain the initial node features; S32, the initial node features and the action-aware adjacency matrix are input into the graph convolutional layer, and the initial node features are aggregated into neighborhoods through the action-aware adjacency matrix to obtain spatial features.

[0016] In one embodiment of the present invention, the step of performing multi-scale temporal convolution on spatial features to extract temporal features includes: S41, the spatial features are transposed to obtain the temporal master sequence feature map, and the temporal master sequence feature map is input into multiple serial one-dimensional dilated convolutional layers for convolution processing; wherein, the one-dimensional dilated convolutional layers have different dilation rates to capture temporal dynamics under different receptive fields. S42, the output of the one-dimensional dilated convolutional layer is used as the input of the next one-dimensional dilated convolutional layer, and serial feature extraction is performed sequentially through multiple one-dimensional dilated convolutional layers with different dilation rates to obtain multi-scale fusion features; and the multi-scale fusion features are transposed to restore the spatial master-order layout to obtain temporal features.

[0017] In one embodiment of the present invention, predicting future motion sequences based on temporal features includes: S51, input the temporal features into the fully connected layer, perform dimensionality transformation and feature mapping, and generate the initial prediction sequence; S52 takes the initial prediction sequence as the input to the next network block, and performs iterative refinement through multiple cascaded network blocks to output the final predicted future motion sequence.

[0018] To achieve the above objectives, another aspect of the present invention proposes a human motion prediction device based on a motion-aware dynamic graph structure, comprising: The dynamic attention generation module acquires historical motion sequences and performs masked joint reconstruction on the historical motion sequences to generate a dynamic attention map. The adjacency matrix fusion module fuses the dynamic attention graph with the shared adjacency matrix to obtain the action-aware adjacency matrix; The graph convolutional spatial coding module extracts spatial features by performing graph convolution on historical motion sequences based on the action-aware adjacency matrix. The multi-scale temporal coding module performs multi-scale temporal convolution on spatial features to extract temporal features; The future sequence prediction module predicts future motion sequences based on temporal characteristics.

[0019] This invention discloses a human motion prediction method and apparatus based on a motion-aware dynamic graph structure. It generates a dynamic attention graph by introducing masked joint reconstruction and fuses it with a shared adjacency matrix to construct a motion-aware adjacency matrix. Combined with multi-scale temporal convolution to capture local details and long-range dependencies, it effectively solves the problem in existing technologies where fixed graph structures cannot adapt to motion heterogeneity and temporal information loss due to discrete cosine transform. It achieves integrated modeling from dynamic space modeling and multi-scale temporal encoding to future sequence prediction, significantly improving the accuracy and naturalness of human motion prediction and enhancing the model's generalization ability and robustness in complex motion scenarios.

[0020] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing a human motion prediction method based on a motion-aware dynamic graph structure as described in the first aspect embodiment.

[0021] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a human motion prediction method based on a motion-aware dynamic graph structure as described in the first aspect embodiment.

[0022] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0023] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a human motion prediction method based on a motion-aware dynamic graph structure according to an embodiment of the present invention; Figure 2 This diagram illustrates the main differences between another human motion prediction method based on a motion-aware dynamic graph structure according to an embodiment of the present invention and traditional methods. Figure 3 This is a network architecture diagram of another human motion prediction method based on a motion-aware dynamic graph structure according to an embodiment of the present invention; Figure 4 This is a visualization result of the "walking" action on the H3.6M dataset, based on another human motion prediction method based on a motion-aware dynamic graph structure according to an embodiment of the present invention. Figure 5 This is a visualization result of the "playing basketball" action in the CMU dataset, which is another human motion prediction method based on a motion-aware dynamic graph structure according to an embodiment of the present invention. Figure 6 This is a schematic diagram of a human motion prediction device based on a motion-sensing dynamic graph structure according to an embodiment of the present invention. Figure 7 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] The following description, with reference to the accompanying drawings, describes a method, apparatus, device, and storage medium for predicting human motion based on a motion-aware dynamic graph structure according to an embodiment of the present invention.

[0027] The core idea of ​​this invention is to construct a dynamic graph model covering multiple levels of the battery pack, from individual cells to clusters to the entire pack. This model integrates electrical connections and physical proximity to establish a spatial topology, and introduces temporal edges to characterize the system's dynamic operation. Based on this, an attribute-decoupled encoder is used to divide node features into health, anomaly, and trend subspaces. Orthogonal constraints and mutual information constraints are used to achieve feature decoupling and redundancy suppression. Furthermore, a cross-layer message passing mechanism establishes bidirectional information paths from bottom to top and from top to bottom. Gated attention is used to dynamically adjust the interaction intensity between levels, achieving deep fusion of multi-level features and collaborative updating of global representations. Finally, health status and lifetime prediction heads are set based on shared representations, and a joint loss function containing multiple constraints is constructed. This is combined with a sliding time window and edge weight update mechanism for integrated training. This transforms the traditionally separate state assessment and lifetime prediction into an intelligent prediction system that can fully exploit hierarchical relationships, suppress feature interference, and achieve multi-task collaborative optimization, significantly improving the accuracy, stability, and engineering applicability of battery pack health status and remaining lifetime predictions.

[0028] Example 1 To achieve the above invention, embodiments of the present invention provide a human motion prediction method based on a motion-aware dynamic graph structure, such as... Figure 1 As shown, it includes: S1: Obtain the historical motion sequence, perform masked joint reconstruction on the historical motion sequence, and generate a dynamic attention map.

[0029] Specifically, the first step is to acquire historical motion sequences, which consist of multiple consecutive frames of human pose data, each frame containing the three-dimensional coordinates of several joints. This step provides the basic input data for subsequent dynamic graph structure modeling and feature extraction, and is the starting point of the entire prediction process. By acquiring and preprocessing the original motion sequences, the integrity and standardization of the input data are ensured, laying the foundation for stable training and inference of the model.

[0030] Furthermore, to achieve dynamic graph structure modeling for action perception, this step introduces masked joint reconstruction as a self-supervised auxiliary task. Specifically, a random masking operation is performed on the joints in the input historical motion sequence, i.e., a subset of joints are randomly selected and their original coordinates are replaced with pre-calculated joint mean values, thereby constructing a damaged masked motion sequence. This operation forces the model to learn the intrinsic spatial dependencies and motion coordination patterns of the human skeleton even with missing information. Subsequently, the masked sequence is input into a self-attention mechanism, and through querying, key-value projection transformation, and attention weight calculation, the correlation strength between joints in the current action pattern is dynamically inferred, ultimately generating a dynamic attention graph. This attention graph can adaptively characterize the spatial dependency structure between joints for each input sample, providing a core basis for the subsequent construction of the action-aware adjacency matrix.

[0031] Furthermore, the mask probability directly affects the difficulty of the auxiliary task and the model's learning performance. It is typically set to empirical values ​​such as 0.15 or 0.20 to balance the degree of information loss with the feasibility of the reconstruction task. The joint mean is calculated based on the statistical average of the corresponding joint across all frames in the training set, ensuring the rationality and stability of the replacement values. The key vector dimension in the self-attention mechanism is used to scale the dot product result to stabilize gradient propagation; its value must match the feature dimension, commonly set to 64 or 128. These parameters work together in the generation process of the dynamic attention map, ensuring it accurately reflects the differences in joint relationships under different action modes.

[0032] Specifically, during the training phase, the model is guided to learn diverse spatial dependency patterns through a mask reconstruction task; during the inference phase, dynamic attention maps can still be generated in real time based on input actions, enabling the model to adapt to unseen or complex actions. This mechanism provides key technical support for human motion prediction in scenarios such as human-computer interaction, virtual reality, and autonomous driving, and is particularly suitable for applications requiring high-precision limb motion modeling.

[0033] Specifically, through the aforementioned masked joint reconstruction and dynamic attention generation mechanism, this step achieves adaptive modeling of action-specific spatial dependencies, effectively addressing the technical deficiency of traditional methods where fixed adjacency matrices cannot distinguish action heterogeneity. The dynamic attention map generated in this step provides rich spatial structure information for subsequent graph convolution and multi-scale temporal modeling, significantly improving the model's ability to express human motion and its prediction accuracy, laying a solid foundation for generating natural, accurate, and motion-compliant future pose sequences.

[0034] Furthermore, S1 includes: S11, obtain the historical motion sequence, randomly mask the joints in the historical motion sequence, and replace the masked joints with the corresponding joint average to generate the masked motion sequence.

[0035] Specifically, the historical motion sequence is first acquired as input data. This sequence consists of multiple consecutive frames of human pose, each frame containing the three-dimensional coordinate information of several joints, completely preserving the original spatial structure and temporal dependencies of human motion. The purpose of acquiring this sequence is to provide a data foundation for subsequent masking operations and reconstruction tasks, ensuring that the model can learn the spatial coordination relationships and temporal evolution patterns between joints from real motion data.

[0036] Furthermore, to model the spatial dependencies of actions, this step introduces a random masking mechanism to disrupt the historical motion sequence. Specifically, for each joint in the input historical motion sequence, it is independently determined whether it is masked using a preset masking probability. If a joint is selected, its original 3D coordinates are replaced with the joint mean value corresponding to that joint. This joint mean value is pre-calculated based on the statistical calculation of the joint's coordinates across all frames in all samples of the training set, reflecting the statistically typical position of the joint. Through the above operations, a masked motion sequence is generated. In this sequence, some joint information is replaced by masks, while the remaining joints remain unchanged, thus constructing a training sample with partially missing information but preserved overall structure.

[0037] Furthermore, the mask probability directly affects the difficulty of the auxiliary task and the model's learning performance. If the mask probability is too low, the model will struggle to learn effective spatial dependencies from the destruction; if it is too high, the reconstruction task will be too difficult, potentially leading to training instability. Typically, the mask probability is set to empirical values ​​such as 0.15 or 0.20 to balance the degree of information destruction with task feasibility. The joint mean is calculated based on the average coordinates of the corresponding joint across all samples and frames in the training set, ensuring the rationality and statistical stability of the replacement values. This mean is pre-calculated and fixed before training and is not updated during model training, thus providing a consistent benchmark for mask replacement.

[0038] Specifically, during the training phase, the model reconstructs complete joint positions based on partially masked sequences. This process forces the model to learn the intrinsic spatial structure of the human skeleton and the interdependencies between joints. During the inference phase, although masking is no longer performed, the spatial modeling capabilities learned during training are transferred to the processing of complete input sequences, enabling the model to dynamically model arbitrary motion patterns. This mechanism can be widely applied to scenarios requiring high-precision human motion modeling, such as character animation generation in virtual reality, intent understanding in human-computer interaction, and pedestrian trajectory prediction in autonomous driving.

[0039] Specifically, this step achieves controlled disruption of the input motion sequence, providing high-quality input data for the self-supervised mask reconstruction task. This operation effectively guides the model to focus on the spatial dependencies between joints, learn robust and generalizable motion representations, lays a solid foundation for the subsequent generation of dynamic attention maps, and thus improves the accuracy and naturalness of the entire human motion prediction model.

[0040] S12, input the masked motion sequence into the self-attention mechanism to calculate the dynamic attention map.

[0041] Specifically, the generated masked motion sequence is used as input to a self-attention mechanism, aiming to dynamically infer the spatial correlation strength between joints in the current action pattern through global dependency modeling capabilities. The self-attention mechanism overcomes the limitations of preset topological structures in graph convolution, directly and adaptively learning the dependency weights between any two joints based on the input data, thereby constructing a refined spatial relationship representation of the action. This step is the core of achieving dynamic graph structure modeling and provides a crucial basis for subsequent adjacency matrix fusion.

[0042] Specifically, the masked motion sequence is first embedded and positionally encoded using a fully connected layer to obtain an initialized node feature matrix. This feature matrix is ​​then projected onto the query space, key space, and value space, respectively, using a learnable projection matrix to achieve feature transformation. The query matrix and key matrix are dot-producted and divided by a scaling factor to stabilize the gradient, then normalized using the Softmax function to obtain the attention weight matrix, i.e., the dynamic attention graph. Each element in this graph represents the association strength between two corresponding joints in the current action mode. Finally, the attention weight matrix is ​​multiplied by the value matrix to complete feature aggregation based on global dependencies, outputting the attention-weighted node features. The entire computation process is entirely driven by the input data and requires no predefined graph structure priors.

[0043] Furthermore, the dimensionality settings of the query, key, and value projection matrices directly affect the model's expressive power and computational complexity. Typically, the feature dimension is set to the same value as the hidden layer dimension, such as 128 or 256, to ensure information capacity while controlling the number of parameters. The scaling factor is taken as the square root of the key vector dimension to prevent the dot product result from becoming too large, leading to softmax gradient saturation. The number of self-attention heads can be set to single or multiple heads as needed. The multi-head mechanism allows the model to capture diverse dependencies in different representation subspaces, further enhancing the expressive power of the dynamic attention map.

[0044] Specifically, during the training phase, the self-attention mechanism is guided to focus on key joint pairs with strong spatial dependencies by reconstructing the supervision signal of the task through masking, thereby learning the action-specific spatial structure. During the inference phase, this mechanism can still generate dynamic attention maps in real time based on the input action, enabling the model to adaptively model unseen or complex actions. This feature is particularly important in scenarios requiring fine-grained motion capture, such as real-time generation of character actions in virtual reality, understanding gesture intentions in human-computer interaction, and identifying abnormal gait in motion analysis.

[0045] Specifically, by applying the self-attention mechanism described above, this step achieves dynamic modeling of action-specific spatial dependencies, effectively overcoming the technical deficiency of traditional methods where fixed adjacency matrices cannot adapt to action heterogeneity. The generated dynamic attention graph can accurately reflect the differences in the strength of joint connections in different actions, such as the strong coupling of the limbs in the "walking" action and the relative independence of the upper limbs in the "sitting" action. This provides more accurate spatial structure information for subsequent graph convolution and temporal modeling, significantly improving the expressive power and prediction accuracy of the human action prediction model.

[0046] S2, the dynamic attention map is fused with the shared adjacency matrix to obtain the action-aware adjacency matrix.

[0047] Specifically, the dynamic attention map reflects the adaptive spatial dependencies between joints under the current input action pattern, exhibiting dynamic characteristics that change with the samples; the shared adjacency matrix, on the other hand, is a general topological structure learned from all data during training, capturing the basic joint relationships shared across actions. The purpose of fusing the two is to balance the detailed modeling of action-specific features with the stability constraints of general priors, providing a more complete graph structure input for subsequent graph convolution operations.

[0048] Specifically, first, a dynamic attention map generated by the masked joint reconstruction task is obtained. This map represents the association strength between any two joints in the current action in matrix form. Simultaneously, a shared adjacency matrix is ​​obtained. This matrix is ​​optimized through gradient descent during network training and eventually converges to a fixed-weight matrix, used to encode the basic connectivity and long-range dependencies of the human skeleton. The dynamic attention map and the shared adjacency matrix are then weighted and summed to obtain the action-aware adjacency matrix. This fusion process can be represented as multiplying each matrix by learnable weight parameters and then adding them together. These weight parameters balance the relative importance of dynamic dependencies and static priors, and their values ​​are adaptively adjusted during network training.

[0049] Furthermore, both the dynamic attention map and the shared adjacency matrix have the dimension of joints multiplied by joints to ensure consistency in spatial dimensions. The initial value of the learnable weight parameter is usually set to 0.5, indicating that dynamic and static information are equally important in the initial state; this parameter is optimized according to the task objective during backpropagation and eventually converges to a value that maximizes prediction performance. If a multi-head attention mechanism is used to generate multiple sets of dynamic attention maps, the results of multiple sets need to be aggregated before being fused with the shared adjacency matrix, or independent fusion weights can be set for each attention head to enhance the expressive power of the model.

[0050] Specifically, the action-aware adjacency matrix, as the core input of the graph convolutional module, directly participates in the neighborhood aggregation process of node features. During training, the fusion mechanism enables the model to learn general spatial structure priors from diverse action data, while adaptively adjusting joint relationships for each specific sample, thereby improving the model's ability to model different action patterns. During inference, the shared adjacency matrix remains fixed to provide stable structural priors, while the dynamic attention map is generated in real-time based on the input action, allowing the fused action-aware adjacency matrix to dynamically adjust with the input action. This mechanism is particularly suitable for applications requiring the processing of diverse action types, such as sports motion analysis, rehabilitation training monitoring, and complex human-computer interaction tasks.

[0051] Specifically, through the aforementioned fusion mechanism, this step achieves the complementary advantages of dynamic attention maps and shared adjacency matrices, constructing an action-aware adjacency matrix that possesses both action adaptability and structural stability. This matrix effectively solves the problem in traditional methods where fixed adjacency matrices cannot distinguish action heterogeneity, while avoiding the structural instability defects that may result from relying entirely on dynamic graphs. Graph convolution operations based on this action-aware adjacency matrix can more accurately aggregate joint features, improving the accuracy and robustness of spatial modeling and laying a solid foundation for subsequent temporal modeling and future sequence prediction.

[0052] Furthermore, S2 includes: S21, obtain the dynamic attention map and the shared adjacency matrix, and perform a weighted summation of the dynamic attention map and the shared adjacency matrix to generate the action-aware adjacency matrix.

[0053] Specifically, the dynamic attention map, generated by the preorder mask reconstruction task, adaptively characterizes the spatial dependencies between joints in response to the current input action, reflecting the joint coordination strength in a specific motion pattern. The shared adjacency matrix, a general topological structure learned from all data during training, encodes the basic connectivity relationships of the human skeleton and long-range dependencies shared across actions. The purpose of fusing the two is to combine the flexibility of dynamic information with the stability of static priors to construct a graph representation that is both action-adaptive and retains structural reliability.

[0054] Specifically, a dynamic attention map is first obtained from the masked joint reconstruction module. This map is stored as a two-dimensional matrix with the same dimension as the number of joints, and the matrix element values ​​represent the association strength between corresponding joint pairs. Simultaneously, this matrix is ​​randomly initialized; it is also a two-dimensional matrix of joints multiplied by the number of joints. Its element values ​​are optimized during network training using gradient descent, eventually converging to fixed weights. The dynamic attention map and the shared adjacency matrix are then weighted and summed—that is, corresponding elements of the two matrices are linearly combined, and the fused result is output as the action-aware adjacency matrix. The weight parameters used in the weighted summation are learnable scalars, whose values ​​are adaptively adjusted during network training to balance the relative importance of dynamic information and static priors.

[0055] Furthermore, the dynamic attention map and the shared adjacency matrix have strictly consistent dimensions, both being the product of the number of joints, to ensure spatial dimensional matching of the fusion operation. The initial value of the learnable weight parameter is typically set to 0.5, indicating that dynamic information and static priors are equally important in the initial state. This parameter is optimized based on the prediction loss during backpropagation, eventually converging to a value that maximizes model performance. If a multi-head attention mechanism is used to generate multiple sets of dynamic attention maps, the results from multiple sets need to be averaged or concatenated before being fused with the shared adjacency matrix, or independent fusion weights can be set for each attention head to enhance the model's ability to model different dependency patterns. The fused action-aware adjacency matrix needs to be normalized to ensure the numerical stability of graph convolution operations.

[0056] Specifically, the action-aware adjacency matrix generated in this step serves as the core input to the graph convolution module, directly participating in the neighborhood aggregation process of node features. During the training phase, the learnable weight parameters are updated synchronously with the shared adjacency matrix, enabling the model to learn a general spatial structure prior from diverse action data, while adaptively adjusting joint relationships for each specific sample. During the inference phase, the shared adjacency matrix remains fixed to provide a stable structural prior, while the dynamic attention map is generated in real time based on the input action. The learnable weight parameters use the values ​​after training convergence, allowing the fused action-aware adjacency matrix to dynamically adjust with the input action. This mechanism is particularly suitable for application scenarios that require processing diverse action types, such as switching between different sports in sports motion analysis, monitoring individual differences in patient movement patterns in rehabilitation training, and real-time understanding of gesture intentions in human-computer interaction.

[0057] Specifically, through the weighted summation operation described above, this step effectively fuses the dynamic attention map and the shared adjacency matrix, constructing an action-aware adjacency matrix that possesses both action adaptability and structural stability. This matrix effectively overcomes the limitation of traditional methods where fixed adjacency matrices cannot distinguish action heterogeneity, while avoiding the structural instability defects that may result from relying entirely on dynamic graphs. Performing graph convolution operations based on this action-aware adjacency matrix enables more accurate aggregation of joint features, improving the accuracy and robustness of spatial modeling, providing high-quality graph structure input for subsequent temporal modeling and future sequence prediction, thereby significantly improving the overall performance of the human action prediction model.

[0058] S3, based on the action-aware adjacency matrix, performs graph convolution on historical motion sequences to extract spatial features.

[0059] Specifically, the graph convolutional neural network updates the node representation by weightedly fusing the features of the center node and its neighboring nodes through a neighborhood aggregation mechanism. Unlike traditional graph convolutions that use a fixed adjacency matrix, the action-aware adjacency matrix used in this step integrates a dynamic attention map and a shared adjacency matrix. This allows for adaptive adjustment of aggregation weights based on the current input action, ensuring that the extracted spatial features include both general skeletal structure priors and joint coordination relationships under specific action patterns.

[0060] Specifically, the historical motion sequence is first embedded and positionally encoded using a fully connected layer to obtain an initial node feature matrix. The rows of this matrix correspond to time frames, the columns to joint nodes, and each element is the feature vector of the corresponding node. Then, the initial node features and the action-aware adjacency matrix are input into a graph convolutional layer. The core operation of the graph convolutional layer is neighborhood aggregation: for each joint node, its features are weighted and summed with the features of its neighboring nodes according to the weights in the action-aware adjacency matrix. This sum is then processed through a linear transformation and a non-linear activation function to obtain updated node features. This process can be formally represented as the product of the feature matrix and the action-aware adjacency matrix, multiplied by a learnable weight matrix. Multiple graph convolutional layers can be stacked to progressively expand the receptive field, capture higher-order neighbor information, and thus extract richer spatial semantic features.

[0061] Furthermore, the input and output feature dimensions of the graph convolutional layer need to be set according to the model capacity and task requirements, with common values ​​being 128, 256, or 512. The dimension of the learnable weight matrix is ​​determined by both the input and output feature dimensions, and the number of parameters needs to be controlled within a reasonable range to avoid overfitting. The number of graph convolutional layers directly affects the model's spatial modeling ability; too few layers make it difficult to capture long-range dependencies, while too many layers may lead to oversmoothing problems; typically, 2 to 4 layers are used. When stacking multiple graph convolutions, layer normalization and residual connections can be added after each layer to stabilize the training process and promote gradient propagation. The action-aware adjacency matrix remains unchanged during graph convolution, but it has already incorporated dynamic information in previous steps, so the aggregation operation of each layer reflects the characteristics of action awareness.

[0062] Specifically, the spatial features extracted in this step provide rich input information for the subsequent temporal modeling module. These features not only encode the individual attributes of each joint but also include its spatial dependencies with surrounding joints, enabling subsequent temporal convolutions to be modeled in a semantically richer feature space. This mechanism is particularly suitable for scenarios requiring precise capture of inter-joint coordinated movements, such as judging the technical compliance of athletes in sports motion analysis, monitoring patients' limb coordination in rehabilitation training, and generating realistic character animations in virtual reality. Through motion-aware graph convolution operations, the model can accurately distinguish the strength differences in joint relationships across different movements, thereby extracting more discriminative spatial features.

[0063] Specifically, this step achieves efficient modeling of the spatial structure of the human skeleton, extracting spatial features that contain both general topological priors and action-specific characteristics. These features effectively overcome the limitation of traditional methods where fixed adjacency matrices cannot adapt to action heterogeneity, enabling the model to more accurately characterize the collaborative patterns between joints in different actions. Subsequent multi-scale temporal modeling based on these spatial features further enhances the expressive power and prediction accuracy of human action prediction, laying a solid foundation for generating natural, accurate, and motion-compliant future posture sequences.

[0064] Furthermore, S3 includes: S31, embed the features of the historical motion sequence to obtain the initial node features.

[0065] Specifically, the original historical motion sequence consists of multiple consecutive frames of human poses, each frame containing the three-dimensional coordinate information of several joints, in the form of low-dimensional Euclidean coordinates. Directly using the original coordinates for graph convolution operations makes it difficult to fully explore the complex nonlinear relationships between joints, and the limited range of coordinate values ​​restricts the model's expressive power. Feature embedding operations, through learnable linear transformations, project the original coordinates of each joint into a high-dimensional feature space, providing richer semantic information for subsequent spatial dependency modeling and temporal feature extraction.

[0066] Specifically, the data format of the historical motion sequence is first reshaped, and the three-dimensional coordinates of each joint in each frame are unfolded into a feature vector, resulting in a two-dimensional matrix with the frame number multiplied by the number of joints. This matrix is ​​then input into a fully connected layer, whose input dimension is the original coordinate dimension and whose output dimension is the preset hidden layer dimension. The fully connected layer performs a linear transformation on the input using a learnable weight matrix and adds bias terms, mapping the coordinates of each joint to a high-dimensional feature space. To preserve the temporal positional information in the sequence, positional encoding is typically added after feature embedding. The encoded vector of the frame index is added to or concatenated with the embedded features, enabling the model to perceive the relative temporal position of each frame. After these operations, an initial node feature matrix is ​​obtained, with a dimension equal to the frame number multiplied by the hidden layer dimension. Each row corresponds to a frame, each column corresponds to a feature channel, and each element is the feature vector of each joint in the corresponding frame.

[0067] Furthermore, the hidden layer dimension of the feature embedding is a key parameter affecting model capacity and computational complexity, typically set to 128, 256, or 512 to balance expressive power and computational cost. The weight matrix of the fully connected layer is randomly initialized and optimized during training through backpropagation. If positional encoding is used, the dimension of the encoding vector must be consistent with the dimension of the embedded features to ensure dimension matching during addition operations. Positional encoding can be generated using a fixed-frequency sine and cosine function, or it can be set as a learnable embedding parameter. The former can be extrapolated to any length, while the latter is more flexible and can be adaptively adjusted according to the specific task.

[0068] Specifically, this step is the entry point for the entire human motion prediction process, and its output initial node features are directly input into the subsequent graph convolution module. The quality of feature embedding directly affects the degree to which the model retains the original motion information and the effectiveness of subsequent feature extraction. In different application scenarios, such as sports motion analysis, rehabilitation training monitoring, or virtual reality character animation generation, the source and format of historical motion sequences may differ. The design of the feature embedding layer needs to be universal and adaptable, capable of handling input data under different frame rates, different numbers of joints, and different coordinate systems. Through feature embedding, the original motion data is uniformly mapped to a standardized feature space, laying the foundation for subsequent modular processing.

[0069] Specifically, this step transforms the original low-dimensional joint coordinate data into high-dimensional node features rich in semantic information, providing a high-quality input representation for subsequent graph convolution and temporal modeling. This operation effectively enhances the model's expressive power, enabling subsequent graph convolution to perform neighborhood aggregation and feature updates in a richer feature space, thereby extracting more accurate spatial dependencies. Based on these initial node features, the entire human motion prediction model can more effectively learn the spatiotemporal evolution of human movement, ultimately improving the accuracy and naturalness of future pose sequence predictions.

[0070] S32, the initial node features and the action-aware adjacency matrix are input into the graph convolutional layer, and the initial node features are aggregated into neighborhoods through the action-aware adjacency matrix to obtain spatial features.

[0071] Specifically, the fundamental principle of graph convolutional neural networks lies in defining the connections between nodes using a graph structure and updating the feature representation of the central node by aggregating information from neighboring nodes. Unlike standard graph convolutions that use a fixed adjacency matrix, the action-aware adjacency matrix used in this step integrates a dynamic attention map and a shared adjacency matrix. This allows for adaptive adjustment of aggregation weights based on the current input action, ensuring that the neighborhood aggregation process reflects both general skeleton topological priors and joint collaboration relationships within specific action patterns. The initial node features provide the basic representation of each joint in high-dimensional space, and the combination of these two elements lays the foundation for extracting features rich in spatial semantics.

[0072] Specifically, the initial node feature matrix and the action-aware adjacency matrix are first input into the graph convolutional layer. The computation process of the graph convolutional layer can be divided into two stages: neighborhood aggregation and feature transformation. In the neighborhood aggregation stage, for each joint node, its own features are weighted and summed with the features of all neighboring nodes according to the weights defined in the action-aware adjacency matrix. The aggregation result reflects the contextual information of the node in a specific action pattern. In the feature transformation stage, the aggregated features are multiplied by a learnable weight matrix, a bias term is added, and then processed by a non-linear activation function to obtain the updated node features. This process can be formally represented as the product of the feature matrix and the action-aware adjacency matrix, multiplied by the learnable weight matrix, and finally outputting spatial features. Multiple graph convolutional layers can be stacked to gradually expand the receptive field, capture higher-order neighbor information, and thus extract richer spatial semantic features.

[0073] Furthermore, the input feature dimension of the graph convolutional layer is determined by the output dimension of the preceding feature embedding module. The output feature dimension needs to be set according to the model capacity and task requirements, with common values ​​being 128, 256, or 512. The dimension of the learnable weight matrix is ​​jointly determined by the input and output feature dimensions, and the number of parameters needs to be controlled within a reasonable range to avoid overfitting. The number of graph convolutional layers directly affects the model's spatial modeling ability. Too few layers make it difficult to capture long-range dependencies, while too many layers may lead to oversmoothing problems. Typically, 2 to 4 layers are used. When stacking multiple graph convolutions, layer normalization and residual connections can be added after each layer to stabilize the training process and promote gradient propagation. The action-aware adjacency matrix remains unchanged during graph convolution, but it has already incorporated dynamic information in the preceding steps. Therefore, the aggregation operation of each layer reflects the characteristics of action awareness. If a multi-head graph convolution mechanism is used, multiple sets of neighborhood aggregations with different parameters can be executed sequentially, and the results can be concatenated or summed to enhance the model's ability to model diverse dependency patterns.

[0074] Specifically, the spatial features extracted in this step provide rich input information for the subsequent temporal modeling module. These features not only encode the individual attributes of each joint but also include its spatial dependencies with surrounding joints, enabling subsequent temporal convolutions to be modeled in a semantically richer feature space. This mechanism is particularly suitable for scenarios requiring precise capture of inter-joint coordinated movements, such as judging the technical standardization of athletes' movements in sports motion analysis, where spatial features can accurately identify the degree of limb coordination; assessing whether patients' movement patterns are abnormal in rehabilitation training monitoring, where spatial features can reflect the mutual compensation mechanisms between joints; and generating realistic character animations in virtual reality, where spatial features can ensure the physical rationality and naturalness of limb movements. Through motion-aware graph convolution operations, the model can accurately distinguish the strength differences in joint relationships in different movements, thereby extracting more discriminative spatial features.

[0075] Specifically, this step achieves efficient modeling of the spatial structure of the human skeleton, extracting spatial features that contain both general topological priors and action-specific characteristics. These features effectively overcome the limitation of traditional methods where fixed adjacency matrices cannot adapt to action heterogeneity, enabling the model to more accurately characterize the collaborative patterns between joints in different actions. Subsequent multi-scale temporal modeling based on these spatial features further enhances the expressive power and prediction accuracy of human action prediction, providing high-quality spatial feature input for generating natural, accurate, and motion-compliant future posture sequences.

[0076] S4 performs multi-scale temporal convolution on spatial features to extract temporal features.

[0077] Specifically, the core challenge of human motion prediction lies in simultaneously modeling the spatial dependencies between joints and the dynamic changes of motion over time. Spatial features have encoded the cooperative relationships between joints in each frame, but have not yet fully depicted the evolution of these relationships in the temporal dimension. Multi-scale temporal convolution, by performing convolution operations on the temporal dimension at different receptive fields, can simultaneously capture the local fine-grained dynamics between adjacent frames and the global motion trends between long-distance frames, thereby constructing a temporal feature representation that includes both short-term motion details and long-term trajectory dependencies.

[0078] Specifically, the spatial features are first transposed, transforming the original spatially ordered feature map into a temporally ordered one, making the temporal dimension the primary axis of the convolution operation. Then, the transposed feature maps are fed into multiple sequential one-dimensional dilated convolutional layers for convolution. Each layer has a different dilation rate; layers with low dilation rates have smaller receptive fields, focusing on capturing high-frequency, fine-grained motion changes between adjacent frames, such as instantaneous acceleration and local swaying of joints; layers with high dilation rates have larger receptive fields, capable of spanning longer time intervals and modeling overall motion trajectories and long-term dependencies, such as periodic patterns and trends in movement. Each sequential branch performs convolution in turn, achieving the fusion of temporal features at different scales. Finally, the fused feature map is transposed again to restore the original spatially ordered layout, outputting the final temporal features.

[0079] Furthermore, the core parameters of multi-scale temporal convolution include kernel size, dilation rate set, and number of output channels. The kernel size is typically fixed at 3, considering the center frame and one frame before and after it in the temporal dimension to ensure the granularity of local modeling. The dilation rate set needs to cover the temporal range from local to global, commonly set as an increasing sequence such as [1, 2, 3] or [1, 2, 4] to exponentially expand the receptive field. If residual connections are used, the dimensions of the input and output features must be the same. Multiple temporal convolutional blocks can be stacked, with each block employing the same multi-scale structure to progressively refine temporal features and extract higher-level motion semantics.

[0080] Specifically, the temporal features extracted in this step are directly input into the subsequent prediction module to generate future motion sequences. The requirements for modeling the temporal sequence of human movements vary across different application scenarios. In sports motion analysis, it is necessary to simultaneously focus on the athlete's instantaneous bursts of movement and overall rhythm changes; multi-scale temporal convolution can effectively balance both. In rehabilitation training monitoring, patients' movements may exhibit slow changes and abnormal jerking; temporal features at different scales help identify these subtle anomalies. In human-computer interaction and virtual reality scenarios, it is necessary to respond to user actions in real time and generate natural and smooth feedback; multi-scale temporal convolution can capture the necessary temporal context while ensuring low latency. By flexibly adjusting the dilation rate set and the number of convolutional layers, this module can adapt to application scenarios with different frame rates, different motion complexities, and different real-time requirements.

[0081] Specifically, this step achieves efficient modeling of the temporal evolution of human motion, extracting temporal features that contain both local details and reflect the global structure. These features effectively overcome the information loss caused by discrete cosine transform in traditional methods, as well as the limitations of single-scale temporal modeling in simultaneously capturing details and trends. Predicting future sequences based on these temporal features significantly improves the model's ability to model complex motion patterns, generating more natural, accurate, and kinematically sound prediction results, providing crucial support for improving the overall performance of human motion prediction methods.

[0082] Furthermore, S4 includes: S41, the spatial features are transposed to obtain the temporal master sequence feature map, and the temporal master sequence feature map is input into multiple serial one-dimensional dilated convolutional layers for convolution processing; wherein, the one-dimensional dilated convolutional layers have different dilation rates to capture temporal dynamics under different receptive fields.

[0083] Specifically, the spatial features output by the graph convolution module are ordered primarily by the spatial dimension, meaning that rows of the feature matrix correspond to joint nodes and columns correspond to time frames. This layout facilitates spatial neighborhood aggregation operations but is not conducive to direct convolution processing along the temporal dimension. By transposing the feature matrix, the rows and columns are interchanged to obtain a temporally ordered feature map, making the temporal dimension the main axis of continuous storage and laying the data foundation for subsequent one-dimensional temporal convolution. Based on this, multiple sequential one-dimensional dilated convolutional layers with different dilation rates are employed, enabling the simultaneous capture of local fine-grained dynamics between adjacent frames and global motion trends spanning long distances within a single module, achieving multi-scale modeling of the temporal evolution of human motion.

[0084] Specifically, the spatial features output by the graph convolution module are first obtained. These features are typically stored as a three-dimensional tensor, with dimensions of batch size, number of joints, and number of feature channels, respectively. The temporal dimension is implicit in the batch dimension or explicitly represented by an additional dimension. To facilitate temporal convolution operations, the feature tensor needs to be reshaped and transposed so that the temporal dimension is explicitly expanded and becomes the main axis of the convolution operation, resulting in a temporal master-order feature map with dimensions of batch size, time step, and number of feature channels, respectively. Subsequently, this temporal master-order feature map is input into multiple sequential one-dimensional dilated convolutional layers. Each one-dimensional dilated convolutional layer uses a fixed-size kernel but sets different dilation rates. The dilation rate defines the interval between elements of the kernel. A dilation rate of 1 indicates standard convolution, where the receptive field covers consecutive time steps. When the dilation rate is greater than 1, the kernel samples in a skip manner along the temporal dimension, and the receptive field expands linearly with the dilation rate. Low-dilation convolutional branches focus on capturing high-frequency, fine-grained motion changes between adjacent frames, such as instantaneous joint acceleration and local sway amplitude—short-term dynamics. High-dilation convolutional branches, on the other hand, can span longer time intervals, modeling long-term dependencies such as overall motion trajectories, periodic patterns, and trend changes. Each convolutional operation is executed sequentially, with the output of the previous convolution serving as the input to the next, progressively extracting temporal features at different scales.

[0085] Furthermore, the core parameters of a one-dimensional dilated convolutional layer include the kernel size, the dilation rate set, and the number of output channels. The kernel size is typically fixed at 3, meaning it considers the center frame and one frame before and after it in the temporal dimension. This setting strikes a balance between local modeling granularity and the number of parameters. The dilation rate set needs to cover a temporal range from local to global. A common setting is an increasing sequence such as [1,2,3]. Dilation rate 1 covers consecutive frames, dilation rate 2 samples every other frame, and dilation rate 3 samples every two frames, thus exponentially expanding the receptive field. To capture a larger range of dependencies, sparser dilation rate combinations such as [1,2,4] or [1,3,5] can be used. The stride of the convolutional layer is typically set to 1, and padding operations are used to maintain the temporal length, facilitating multi-layer stacking and residual connections.

[0086] Specifically, this step is a core component of the multi-scale temporal coding module, whose output multi-scale temporal features are directly used for subsequent feature fusion and future sequence prediction. The requirements for human motion temporal modeling vary significantly across different application scenarios. In virtual reality and real-time human-computer interaction scenarios, the system needs to respond to user actions with low latency. In this case, the local dynamic information captured by the low dilation rate branch is particularly important, ensuring that the generated virtual character's actions remain synchronized with the user input. In sports motion analysis scenarios, it is necessary to focus on both the athlete's instantaneous explosive movements, such as joint angle changes at the moment of takeoff, and the overall movement rhythm and coordination. Multi-scale convolution can simultaneously meet both of these requirements. In rehabilitation training monitoring scenarios, patients' movements may exhibit slow, trend-like changes and localized abnormal jerking. Temporal features at different scales help to comprehensively assess rehabilitation progress and identify potential risks. By flexibly adjusting the dilation rate set and convolutional layer parameters, this module can adapt to application scenarios with different frame rates, different motion complexities, and different real-time requirements.

[0087] Specifically, this step achieves multi-level modeling of the temporal evolution of human motion, effectively capturing temporal dynamic information from local details to global structure. Compared with single-scale temporal modeling methods using a fixed receptive field, the multi-scale sequential design allows the model to simultaneously focus on short-term motion details and long-term motion trends, avoiding the information loss problem at a single scale. Compared with frequency domain transformation methods such as discrete cosine transform, dilated convolution operates directly in the time domain, preserving the local continuity of the original temporal signal and reducing information loss during the transformation process. Subsequent predictions based on these multi-scale temporal features can significantly improve the model's ability to model complex motion patterns, generating more natural, accurate, and kinematically consistent prediction results.

[0088] S42, the output of the one-dimensional dilated convolutional layer is used as the input of the next one-dimensional dilated convolutional layer, and serial feature extraction is performed sequentially through multiple one-dimensional dilated convolutional layers with different dilation rates to obtain multi-scale fusion features; and the multi-scale fusion features are transposed to restore the spatial master-order layout to obtain temporal features.

[0089] Specifically, the design intent of multi-scale sequential convolution is to capture multi-level temporal features ranging from local details to global trends, with each branch's output feature map encoding motion information at different scales. However, this information needs to be integrated before entering the subsequent prediction module to construct a unified temporal representation that simultaneously contains short-term dynamics and long-term dependencies. Sequential stacking, as a simple and effective feature extraction strategy, can achieve multi-level information abstraction and accumulation from local details to long-range dependencies by progressively expanding the receptive field while maintaining consistent feature map dimensions. After fusion, the data layout needs to be restored from temporal master order to spatial master order to ensure interface compatibility with subsequent graph convolution modules or prediction modules, forming a complete feature processing loop.

[0090] Specifically, the temporally dominant feature representation is first processed through a series of sequential convolutions. This process consists of a set of one-dimensional dilated convolutional layers with different dilation rates stacked sequentially, with the output of the previous layer directly serving as the input to the next. In this cascaded structure, temporal features are progressively aggregated and deepened as they pass through each layer: the first convolutional layer captures local fine-grained changes, and its output is then fed into the second layer to expand the receptive field, and so on, until the last convolutional layer completes the modeling of long-range dependencies. After continuous processing through all convolutional layers, the final feature map still maintains a temporally dominant layout (i.e., the time dimension is the first dimension). To restore the original spatial dominant layout, the feature map needs to be transposed, replacing the time dimension with the feature channel dimension, or rearranging the dimensional order according to the data format requirements defined by the model. The transposed feature map restores the spatial dominant format consistent with the output of the graph convolution module, and the resulting feature is the final temporal feature, which can be directly input into subsequent prediction modules or used as input to the next network block.

[0091] Furthermore, the sequential pass-through mechanism between layers does not introduce additional fusion parameters, but it requires strict matching of feature dimensions between adjacent convolutional layers; that is, the number of output channels of the previous convolutional layer must be consistent with the number of input channels of the next convolutional layer. This necessitates unified planning of the number of channels in intermediate hidden layers when designing the sequential convolutional chain. The choice of the number of feature channels needs to balance model capacity and computational cost, with common values ​​being 128, 256, or 512. If residual connections are used, it is necessary to ensure that the dimensions of the input spatial features and the output temporal features match. The transpose operation only changes the storage order of the data and does not change the actual value of the features, thus not affecting the expressive power of the model, but the correctness of the dimension index must be ensured at the framework level to avoid data misalignment. After multi-scale feature fusion, layer normalization or activation functions can be further introduced to enhance the stability and non-linear expressive power of the features.

[0092] Specifically, in virtual reality and real-time human-computer interaction scenarios, the system needs to generate continuous and natural motion feedback under low latency conditions. Multi-scale fusion features can ensure that the prediction results retain both the rapid response to the user's immediate actions and the coherence of the overall movement. In sports motion analysis scenarios, athletes' technical movements often contain both instantaneous force and overall rhythm characteristics. The fused temporal features can comprehensively reflect this information, providing rich evidence for subsequent posture assessment and feedback. In rehabilitation training monitoring scenarios, patients' movements may simultaneously exhibit a slow recovery trend and localized abnormal shaking. Multi-scale fusion mechanisms help distinguish between these two types of signals, avoiding mutual interference between information at a single scale. Through sequential stacking, the model can adaptively integrate contributions from different scales without the need for additional learning of complex fusion weights.

[0093] Specifically, this step achieves effective fusion of multi-scale temporal features and standardization of data layout. Serial stacking integrates local details and global trend information in a parameter-free manner, avoiding the computational overhead and overfitting risks associated with complex fusion modules. Transposition ensures data format compatibility with subsequent modules, enabling the entire network architecture to perform smooth end-to-end training and inference. Based on this multi-scale fusion feature for future sequence prediction, the model benefits simultaneously from detailed short-term dynamic characterization and long-term global control, significantly improving its ability to model complex human movements and ultimately generating more natural, accurate, and motion-compliant prediction results.

[0094] S5 predicts future motion sequences based on temporal characteristics.

[0095] Specifically, after the preceding spatial modeling and temporal encoding, the temporal features have fully integrated the dynamic spatial dependencies between joints and the multi-scale patterns of motion evolution over time. This includes both fine-grained motion details between local frames and reflects the overall long-term motion trend. The prediction stage aims to decode and generate a sequence of human poses for several future frames based on this semantically rich feature representation, ensuring that its spatial structure and temporal continuity conform to the motion patterns of the real human body. This step maps the abstract deep features back to the specific joint coordinate space, completing the entire reasoning process from historical observation to future prediction.

[0096] Specifically, the temporal features are first input into the prediction head module. The prediction head typically consists of fully connected layers, with its input dimension being the channel dimension of the temporal features and its output dimension being the number of predicted frames multiplied by the joint coordinate dimension. The fully connected layers use linear transformations to map the temporal features at each time step to the joint coordinate vector of the corresponding future frame, thus generating the entire future sequence at once. If a frame-by-frame generation strategy is adopted, a recurrent structure or autoregressive approach can be used, using the prediction result of the previous frame as the input of the next frame, iteratively generating future frames. In an architecture that stacks multiple network blocks, the output temporal features of the current block can be used as the input of the next network block. After gradual refinement through multiple cascaded blocks, the final predicted sequence is output. Each network block may contain residual connections to fuse shallow and deep features, mitigating the information decay problem in deep networks and ensuring the accuracy and stability of the prediction results.

[0097] Furthermore, the output dimension of the prediction head is jointly determined by the number of prediction frames and the joint coordinate dimension. The number of prediction frames is a hyperparameter set according to task requirements, commonly set to 10, 25, or 50 frames, corresponding to predictions of different time lengths in the future. The joint coordinate dimension is the number of joints multiplied by 3, outputting the three-dimensional coordinates of each joint. If a multi-block stacked architecture is used, the number of network blocks needs to be balanced between model capacity and computational cost, typically set to 4 to 8 blocks. The fusion coefficients in the residual connections can be set through grid search or empirically to balance the preservation of shallow features and the extraction of deep features. During the training phase, the loss function is calculated between the predicted output and the real future sequence, driving the parameter optimization of the entire network; during the inference phase, the prediction result is directly output without ground truth supervision.

[0098] Specifically, the future motion sequence generated in this step is the final output of the entire human motion prediction method, directly serving various downstream tasks. In virtual reality and gaming scenarios, the predicted sequence can be used to generate realistic virtual character animations, making user interaction more natural and fluid. In autonomous driving scenarios, predicting the future posture of pedestrians helps to anticipate their movement intentions, improving driving safety. In human-machine collaboration scenarios, robots can adjust their behavior based on predicted human motions, achieving safer and more efficient human-machine collaboration. In sports training and rehabilitation monitoring scenarios, the predicted sequence can be used to assess the technical standardization of athletes or the rehabilitation progress of patients, providing quantitative analysis and feedback. By adjusting the output dimension of the prediction head and the number of network blocks, this method can adapt to the differentiated needs of prediction duration and accuracy in different application scenarios.

[0099] Specifically, this module maps high-level temporal features to specific joint coordinates, enabling accurate prediction of future human motion. Based on the motion-aware spatial features and multi-scale temporal features extracted by the preceding module, the prediction results accurately reflect the joint coordination relationships and motion evolution patterns under different motion modes, effectively avoiding information loss and prediction bias caused by fixed graph structures and single temporal modeling in traditional methods. The output of this step directly reflects the performance of the entire human motion prediction method; its accuracy and naturalness are key indicators for evaluating the method's effectiveness. Through end-to-end training and inference, this method can generate future posture sequences that are highly consistent with real motion, providing reliable technical support for various application scenarios.

[0100] Furthermore, S5 includes: S51 inputs the temporal features into the fully connected layer, performs dimensional transformation and feature mapping, and generates the initial prediction sequence.

[0101] Specifically, the temporal features, extracted layer by layer by the preceding modules, have encoded the dynamic spatial dependencies between joints and the multi-scale evolution of motion over time. However, their data form remains an abstract vector in a high-dimensional feature space, which cannot be directly used as the output of human pose. The fully connected layer uses linear transformation to map the temporal features corresponding to each time step to the target output space, completing the transformation from semantic features to physical coordinates and generating a sequence of joint coordinates for future frames consistent with the format of the input historical sequence. This step is a crucial link in realizing the transition from feature representation to interpretable prediction results, laying the foundation for subsequent sequence refinement and final output.

[0102] Specifically, the temporal features are first acquired, typically stored as a three-dimensional tensor with dimensions in the order of batch size, time step, and number of feature channels, where the time step length matches the length of the historical sequence. These temporal features are then input to a fully connected layer. The input dimension of this fully connected layer equals the channel dimension of the temporal features, and the output dimension equals the number of predicted frames multiplied by the joint coordinate dimension. The fully connected layer performs a linear transformation on the input features using a learnable weight matrix, independently mapping the feature vector at each time step into a long vector containing the three-dimensional coordinate information of all joints in all future frames. Subsequently, the mapped long vectors are reshaped to restore the standard sequence format, i.e., batch size, number of predicted frames, and joint coordinate dimension, thus obtaining the initial prediction sequence. Each frame in this sequence corresponds to a future time point, and each frame contains the three-dimensional coordinates of all joints. The data format is completely consistent with the input historical motion sequence, facilitating subsequent processing and evaluation.

[0103] Furthermore, the input dimension of the fully connected layer is determined by the channel dimension of the temporal features, commonly set to 128, 256, or 512, depending on the model capacity and task complexity. The output dimension is determined by the number of predicted frames and the joint coordinate dimension. The number of predicted frames is a hyperparameter set according to application requirements, commonly set to 10, 25, or 50 frames, corresponding to predictions at different time lengths in the future. The joint coordinate dimension is the number of joints multiplied by 3, outputting the three-dimensional coordinates of each joint. The weight matrix of the fully connected layer is randomly initialized and optimized through backpropagation during training. If a stacked architecture of multiple network blocks is used, the fully connected layers in each block can be independently set with different output dimensions to achieve layer-by-layer refinement. During the reshaping operation, the accuracy of dimension calculation must be ensured to avoid data misalignment or indexing errors.

[0104] Specifically, in virtual reality and real-time human-computer interaction scenarios, the quality of the initial prediction sequence directly affects the system's response speed and interactive experience. If the initial prediction has high accuracy, the number of subsequent iterations can be reduced, lowering latency. In sports motion analysis scenarios, the initial prediction sequence can be used to quickly preview the future movement trends of athletes, providing coaches with immediate feedback. In rehabilitation training monitoring scenarios, the initial prediction sequence can assist in real-time assessment of whether a patient's movements deviate from the expected trajectory, issuing timely warnings. By adjusting the output dimension of the fully connected layer, this method can flexibly adapt to the prediction duration requirements of different application scenarios; for example, short-term predictions can be used for real-time control, while long-term predictions can be used for offline analysis.

[0105] Specifically, this step successfully decodes the abstract temporal features into a concrete sequence of future poses, completing the cross-domain transformation from feature space to physical space. While this initial prediction sequence may be further refined in subsequent modules, it already contains the model's understanding of the input's historical motion and a preliminary inference of future trends. Based on this initial prediction sequence, subsequent network blocks can gradually correct subtle biases and improve prediction accuracy through residual connections and feature fusion mechanisms. This step's design enables the entire human motion prediction method to possess end-to-end trainability and inference capabilities, providing a fundamental guarantee for generating natural, accurate, and motion-compliant future pose sequences.

[0106] S52 takes the initial prediction sequence as the input to the next network block, and performs iterative refinement through multiple cascaded network blocks to output the final predicted future motion sequence.

[0107] Specifically, while the initial prediction sequence generated by a single forward propagation already contains a preliminary estimate of future motion trends, its accuracy and naturalness still have room for improvement in complex action scenarios or long-term prediction tasks. By stacking multiple structurally similar but parameter-independent network blocks, each block performs residual learning and feature fusion based on the output of the previous block, gradually correcting joint position deviations and refining the motion trajectory, making the final prediction result closer to the real motion distribution. This iterative optimization mechanism embodies the idea of ​​hierarchical feature refinement in deep learning and is a key design for improving the model's expressive power and prediction performance.

[0108] Furthermore, the initial prediction sequence serves as the output of the first network block and the input of the second. Each network block contains a structure similar to steps S1 to S4: masked joint reconstruction generates a dynamic attention map, fusion yields an action-aware adjacency matrix, graph convolution extracts spatial features, multi-scale temporal convolution extracts temporal features, and a fully connected layer maps to generate the block's output prediction sequence. Unlike the initial prediction generation, the input to subsequent network blocks not only includes the prediction sequence from the previous block but also introduces shallow features through residual connections to alleviate information decay in deep networks. Residual connections weight and fuse the input and output features of the current block, allowing information to propagate directly through skip paths and gradients to flow more effectively in the reverse direction. Through the gradual refinement of multiple cascaded blocks, the final output prediction sequence shows significant improvements in spatial structure accuracy, temporal continuity, and motion naturalness.

[0109] Furthermore, the number of network blocks is a key hyperparameter affecting model performance and computational cost. Too few blocks make it difficult to sufficiently refine prediction results, while too many blocks may lead to oversmoothing or training difficulties; typically, 4 to 8 blocks are used. The internal structure of each network block remains consistent, but its parameters are initialized and optimized independently to learn different levels of refinement tasks. The fusion coefficients in the residual connections can be set through grid search or empirically, with common values ​​ranging from 0.1 to 0.3 to balance the preservation of shallow features with the contribution of deep features. During training, the output of each network block can be used to calculate the loss with the ground truth, employing a multi-level supervision strategy to accelerate convergence; during inference, only the output of the last block is needed as the final prediction result. If a shared parameter mechanism is used, multiple blocks can reuse the same parameters to reduce model capacity, but the refinement capability may be limited.

[0110] Specifically, the iterative optimization mechanism enables human motion prediction methods to adapt to application scenarios with varying accuracy requirements. In virtual reality and real-time human-computer interaction scenarios, where latency requirements are extremely high, the number of network blocks can be reduced or iterations can be truncated early to achieve a balance between accuracy and speed. In sports motion analysis and rehabilitation training monitoring scenarios, high-precision prediction results are typically needed to support quantitative evaluation; in this case, all network blocks can be fully refined to output the optimal prediction sequence. In offline processing scenarios, such as post-processing of motion capture data in film and animation production, the prediction quality can be further improved by increasing the number of network blocks, without being constrained by real-time requirements. The flexibility of this mechanism allows the same model architecture to adapt to various deployment environments without requiring redesign of the network structure.

[0111] Specifically, each network block supplements detailed information based on the preceding sequence, correcting spatial and temporal deviations to ensure that the final output future motion sequence achieves optimal levels of joint position accuracy, motion smoothness, and naturalness. This mechanism, in conjunction with dynamic attention maps, motion-aware adjacency matrices, and multi-scale temporal convolutions, constitutes a complete human motion prediction method, providing high-quality technical support for various application scenarios. Based on this final predicted sequence, downstream tasks can obtain reliable motion input, enabling realistic virtual character animations, accurate understanding of movement intentions, or scientific rehabilitation assessment feedback.

[0112] This invention discloses a human motion prediction method based on a motion-aware dynamic graph structure. It generates a dynamic attention graph by acquiring historical motion sequences and performing masked joint reconstruction. The dynamic attention graph is then fused with a shared adjacency matrix to construct a motion-aware adjacency matrix. Based on this matrix, graph convolution is performed to extract spatial features, and multi-scale temporal convolution is combined to extract temporal features. Finally, future motion sequences are predicted based on these temporal features. This method effectively solves the problem in existing technologies where fixed graph structures cannot adapt to motion heterogeneity and the loss of temporal information due to discrete cosine transform. It achieves integrated modeling from dynamic spatial modeling and multi-scale temporal encoding to future sequence prediction, significantly improving the accuracy and naturalness of human motion prediction and enhancing the model's generalization ability and robustness in complex motion scenarios.

[0113] Example 2 To achieve the above invention, embodiments of the present invention also provide another method for predicting human motion based on a motion-aware dynamic graph structure, including: Specifically, embodiments of the present invention introduce a masked joint reconstruction auxiliary task to dynamically construct a refined adjacency matrix that adapts to action patterns and can be adjusted based on input actions during the inference phase. In the auxiliary task, the model randomly masks some joints, forcing the network to identify the most representative joint pairs based on input motion and assign them higher weights. Simultaneously, embodiments of the present invention fuse the refined adjacency matrix generated by the auxiliary task with the action-shared adjacency matrix of a traditional graph convolutional network. The former changes with input actions during both training and inference phases, specifically designed to enhance the model's adaptability to specific action patterns, while the latter is optimized during training to capture universal underlying topological relationships across actions and remains fixed during inference, providing a stable structural prior for the model and ensuring the interpretability and stability of relationship modeling. This complementary fusion of the two achieves spatial relationship modeling for action perception.

[0114] Furthermore, besides spatial relationship modeling, temporal information encoding is also a key challenge in human motion prediction. Existing research often uses discrete cosine transform (DCT) to encode motion sequences from the time domain to the frequency domain. Although DCT excels at extracting global frequency domain features, information loss occurs during the transformation process (the main differences between the method of this invention and existing methods are detailed in [link to relevant documentation]). Figure 2 Therefore, embodiments of the present invention propose a cross-frame feature aggregation module. Specifically, several one-dimensional convolutional layers with a fixed kernel size of 3 are stacked sequentially, and progressively increasing dilation rates are set for each layer. This allows the model to refine temporal features in a fine-to-coarse manner: the lower-level convolutions with small dilation rates prioritize capturing high-frequency fine-grained motion changes between adjacent frames, and their outputs are then input into higher-level convolutions with large dilation rates. Based on the already modeled local patterns, long-range dependencies are further established, and the overall motion trajectory is characterized. Through this serial, hierarchical feature aggregation mechanism, a temporal representation that simultaneously contains local details and global structure can be efficiently constructed.

[0115] Furthermore, firstly, embodiments of the present invention provide a mathematical description of the problem. The input historical motion sequence is represented as a matrix. ,in Indicates the number of historical frames. Indicates the first The human pose vector of the frame. Here ,in Indicates the number of joints. The three-dimensional coordinates corresponding to each joint.

[0116] The objective of embodiments of the present invention is to predict future human pose sequences. ,in This indicates the length of the predicted sequence. The predicted sequence should approximate the actual future motion sequence as closely as possible. ,and and Having the same dimensions .

[0117] Figure 3 This diagram illustrates the network architecture of a human motion prediction method and device based on a motion-aware dynamic graph structure. The network is composed of stacked network blocks. Within each network block, the mask joint reconstruction task generates an adjacency matrix that refines the action, and fuses it with the action-shared adjacency matrix to achieve spatial modeling of action awareness. Meanwhile, the cross-frame feature aggregation module captures temporal features in a fine-to-coarse manner and promotes the propagation and fusion of features between different network blocks through residual connections.

[0118] Furthermore, to learn robust motion representations, embodiments of the present invention design a self-supervised auxiliary task—masked joint reconstruction. Its core idea is to force the model to learn the intrinsic structure of the motion sequence by disrupting and reconstructing the input signal. Specifically, for the input motion sequence... We use probability A portion of the joints is randomly masked. If a joint is masked, its original value is replaced with its pre-calculated joint mean. Mathematically, the masked input... Defined as: ; in ; in, Indicates the first The set of joints in a frame that are not masked.

[0119] First, the mask sequence Input a fully connected layer for location embedding to obtain the initial feature representation. Subsequently, a self-attention mechanism is introduced to dynamically infer an attention map for each motion sample. This attention map adaptively characterizes the inter-joint dependency strength on the global joint set for the current sample, thus forming a dynamic and refined adjacency matrix, denoted as . The process is represented as: ; ; ; ; ; in, For the first The learnable projection matrix of the block. Let be the dimension of the key vector. The resulting attention map. Explicitly characterizing the spatial dependency structure of action refinement, while This is the feature representation after attention aggregation.

[0120] Finally, MLP is used to further fuse information to obtain smoother and more efficient joint features. Its calculation form is: ; in, Representation layer normalization, This represents the activation function.

[0121] Furthermore, although existing GCN-based methods typically employ learnable adjacency matrices during the training phase... Instead of using a predefined topology, this approach models joint dependencies. However, the spatial structure obtained during inference is still shared across all actions, making it difficult to represent spatial dependency differences across different action patterns. To address this issue, an attention map generated by the mask reconstruction module is introduced into the GCN framework. This allows spatial relationships to be dynamically adjusted according to the input motion pattern, thereby constructing a dynamic spatial topology for motion perception. Specifically, the input motion sequence is first... The initial feature representation is obtained by embedding features and encoding their positions through a fully connected layer. Subsequently, in the... Within each block, the core operation is to update node features using an adjacency matrix that integrates dynamic and static information. The process can be formalized as follows: ; in, This is a dynamic attention map generated by the masked joint reconstruction module for the current input action. The action-shared adjacency matrix learned by traditional GCN These are learnable weight parameters used to balance the importance of dynamic and static dependencies. Indicates the first The input features of each block, The output features are represented. Through the above fusion mechanism, the model can adaptively adjust the joint relationships in specific action patterns while preserving stable structural priors, thereby achieving spatial modeling of motion perception.

[0122] Furthermore, to specifically handle features in the time dimension, the features output from the previous module are first processed. Transpose the graph to obtain a feature map ordered primarily by time. Subsequently, a set of different void ratios were used. A one-dimensional dilated convolutional layer is used for sequential modeling, allowing the model to progressively expand its temporal receptive field in a fine-to-coarse manner, thereby simultaneously capturing fine-grained changes in adjacent frames and the overall trend of long-term motion. Specifically, the output of the previous convolutional layer serves as the input to the next, aggregating dynamic information over a longer time range layer by layer. Finally, the fused features are transposed back to the original spatial layout to obtain the final output. The core convolution and aggregation processes of CFA can be represented as follows:

[0123] in, Represents the set of void ratios. Indicated by void ratio The corresponding one-dimensional convolution kernel, This represents the dilated convolution operation.

[0124] Furthermore, to alleviate the oversmoothing problem commonly found in deep graph convolutions, embodiments of the present invention introduce initial residual connections and identity mappings after the cross-frame feature aggregation module to preserve shallow feature information and stabilize the learning of deep representations. The residual connection takes the following form: ; as well as ; in, For the first The trainable parameter matrix of each block, and For hyperparameters, It is the identity matrix. Specifically, Used to control initial input features Characteristics of CFA output The fusion ratio enables information to be directly transmitted through jump paths, thereby alleviating the information degradation problem that may occur in deep networks; Used to control weighting coefficients The decay rate ensures that while deeper blocks aggregate lower-level features, they can still maintain the effective propagation of higher-level features.

[0125] Furthermore, for training the proposed method, the following loss function is defined. First, consistent with existing research, the average positional error per joint is used as the primary metric, and its loss is defined as: ; in, and These represent the predicted sequence length and the number of joints, respectively. Indicates the first [number]th ... Frame number The three-dimensional coordinates of each joint For the corresponding true value, Euler ( Norm.

[0126] Furthermore, to improve the geometric consistency of the predicted skeleton, a bone length loss is introduced. Bone length is obtained by calculating the three-dimensional Euclidean distance between joint pairs connected to the anatomical structure, and its loss function is expressed as: ; in, and These represent the predicted bone length and the actual bone length, respectively.

[0127] Furthermore, to ensure the assisted task can effectively learn the spatial relationships of the human skeleton, a reconstruction loss is introduced to measure the error between the reconstructed joints and the original joint positions, thereby maintaining local joint geometric consistency while learning the global skeletal structure. The reconstruction loss is defined as: ; in, The reconstructed joint positions represent the time steps. Next Predicted coordinates of each joint; These are the corresponding real coordinates in the historical input sequence. This represents the set of joints that are being masked. This represents the total number of mask joints, used to normalize the loss.

[0128] Finally, the total loss function used for training is: ; in, and This is a hyperparameter used to balance the weights of different loss terms.

[0129] Specifically, the human motion prediction method and device based on motion-aware dynamic graph structure achieved the following visualization results for "playing basketball" motion on the CMU dataset: Figure 4 and Figure 5 As shown. Figure 4 The visualization results of "walking" motion samples on the H3.6M dataset are shown in the red box. The proposed method predicts the leg spacing more accurately than the actual motion. Figure 5The visualization results of "playing basketball" action samples on the CMU dataset are shown in the red box. The proposed method's prediction of the arm swing trajectory is closer to the actual trajectory. Combining the two scenarios mentioned above, the proposed method performs particularly well in action prediction, especially in limb motion modeling (such as arm swing trajectory, leg movement amplitude, etc.). The prediction results are highly consistent with the actual movements, thus effectively reducing the prediction error of joint positions.

[0130] Another human motion prediction method based on a motion-aware dynamic graph structure in this invention generates a dynamic attention graph by introducing masked joint reconstruction and fusing it with a shared adjacency matrix to construct a motion-aware adjacency matrix. Combined with multi-scale temporal convolution to capture local details and long-range dependencies, this effectively solves the problem in existing technologies where fixed graph structures cannot adapt to motion heterogeneity and where temporal information is lost due to discrete cosine transform. It achieves integrated modeling from dynamic space modeling and multi-scale temporal encoding to future sequence prediction, significantly improving the accuracy and naturalness of human motion prediction and enhancing the model's generalization ability and robustness in complex motion scenarios.

[0131] Example 3 To achieve the above invention, such as Figure 6 As shown, this embodiment also provides a human motion prediction device 10 based on a motion-aware dynamic graph structure. The device 10 includes: The dynamic attention generation module 100 acquires historical motion sequences and performs masked joint reconstruction on the historical motion sequences to generate a dynamic attention map.

[0132] The adjacency matrix fusion module 200 fuses the dynamic attention map with the shared adjacency matrix to obtain the action-aware adjacency matrix.

[0133] The graph convolutional spatial coding module 300, based on the action-aware adjacency matrix, performs graph convolution on historical motion sequences to extract spatial features.

[0134] The multi-scale temporal coding module 400 performs multi-scale temporal convolution on spatial features to extract temporal features.

[0135] The future sequence prediction module 500 predicts future motion sequences based on temporal characteristics.

[0136] This invention discloses a human motion prediction device based on a motion-aware dynamic graph structure. A dynamic attention generation module acquires historical motion sequences and performs masked joint reconstruction to generate a dynamic attention graph. An adjacency matrix fusion module fuses the dynamic attention graph with a shared adjacency matrix to construct a motion-aware adjacency matrix. A graph convolutional spatial encoding module extracts spatial features, and a multi-scale temporal encoding module captures temporal dynamics. Finally, a future sequence prediction module outputs the prediction result. This effectively solves the problem in existing technologies where the fixed graph structure cannot adapt to motion heterogeneity and the loss of temporal information due to discrete cosine transform. It achieves integrated processing from dynamic spatial modeling and multi-scale temporal encoding to future sequence prediction, significantly improving the accuracy and naturalness of human motion prediction and enhancing the model's generalization ability and robustness in complex motion scenarios.

[0137] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 7 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads the executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the human motion prediction method based on motion-aware dynamic graph structure described above.

[0138] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a human motion prediction method based on a motion-aware dynamic graph structure as described in the foregoing embodiments.

[0139] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0140] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for predicting human motion based on a motion-aware dynamic graph structure, characterized in that, include: S1, Obtain the historical motion sequence, and perform masked joint reconstruction on the historical motion sequence to generate a dynamic attention map; S2, fuse the dynamic attention map with the shared adjacency matrix to obtain the action-aware adjacency matrix; S3, based on the action-aware adjacency matrix, performs graph convolution on historical motion sequences to extract spatial features; S4 performs multi-scale temporal convolution on spatial features to extract temporal features; S5 predicts future motion sequences based on temporal characteristics.

2. The method as described in claim 1, characterized in that, The step of acquiring historical motion sequences and performing masked joint reconstruction on the historical motion sequences to generate a dynamic attention map includes: S11, Obtain the historical motion sequence, randomly mask the joints in the historical motion sequence, and replace the masked joints with the corresponding joint average to generate the masked motion sequence; S12, input the masked motion sequence into the self-attention mechanism to calculate the dynamic attention map.

3. The method as described in claim 1, characterized in that, The process of fusing the dynamic attention map with the shared adjacency matrix to obtain the action-aware adjacency matrix includes: S21, obtain the dynamic attention map and the shared adjacency matrix, and perform a weighted summation of the dynamic attention map and the shared adjacency matrix to generate the action-aware adjacency matrix.

4. The method as described in claim 1, characterized in that, The method of extracting spatial features by performing graph convolution on historical motion sequences based on the action-aware adjacency matrix includes: S31, embed the features of the historical motion sequence to obtain the initial node features; S32, the initial node features and the action-aware adjacency matrix are input into the graph convolutional layer, and the initial node features are aggregated into neighborhoods through the action-aware adjacency matrix to obtain spatial features.

5. The method as described in claim 1, characterized in that, The step of performing multi-scale temporal convolution on spatial features to extract temporal features includes: S41, the spatial features are transposed to obtain the temporal master sequence feature map, and the temporal master sequence feature map is input into multiple serial one-dimensional dilated convolutional layers for convolution processing; wherein, the one-dimensional dilated convolutional layers have different dilation rates to capture temporal dynamics under different receptive fields. S42, the output of the one-dimensional dilated convolutional layer is used as the input of the next one-dimensional dilated convolutional layer, and serial feature extraction is performed sequentially through multiple one-dimensional dilated convolutional layers with different dilation rates to obtain multi-scale fusion features; and the multi-scale fusion features are transposed to restore the spatial master-order layout to obtain temporal features.

6. The method as described in claim 1, characterized in that, The prediction of future motion sequences based on temporal characteristics includes: S51, input the temporal features into the fully connected layer, perform dimensionality transformation and feature mapping, and generate the initial prediction sequence; S52 takes the initial prediction sequence as the input to the next network block, and performs iterative refinement through multiple cascaded network blocks to output the final predicted future motion sequence.

7. A human motion prediction device based on a motion-aware dynamic graph structure, characterized in that, include: The dynamic attention generation module acquires historical motion sequences and performs masked joint reconstruction on the historical motion sequences to generate a dynamic attention map. The adjacency matrix fusion module fuses the dynamic attention graph with the shared adjacency matrix to obtain the action-aware adjacency matrix; The graph convolutional spatial coding module extracts spatial features by performing graph convolution on historical motion sequences based on the action-aware adjacency matrix. The multi-scale temporal coding module performs multi-scale temporal convolution on spatial features to extract temporal features; The future sequence prediction module predicts future motion sequences based on temporal characteristics.

8. An electronic device, comprising: processor; The memory stores executable instructions; when the processor executes the instructions, it implements the human motion prediction method based on a motion-aware dynamic graph structure as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements a human motion prediction method based on a motion-aware dynamic graph structure as described in any one of claims 1-6.

Images