A method and system for predicting human motion sequences based on skeleton-enhanced Transformer

Abstract

The application provides a human motion sequence prediction method and system based on skeleton-enhanced Transformer, comprising the following steps: S1: sampling human motion to obtain original motion sequence data; S2: generating virtual joints based on real joints, adding the virtual joints to the sequence, constructing enhanced motion sequence data, and generating an enhanced skeleton graph; S3: inputting the enhanced skeleton graph into a graph convolution network, establishing the skeletal correlation between virtual joints and real joints, and optimizing feature extraction through a spatiotemporal attention mechanism; S4: decomposing the input motion sequence data and dividing it into a trend part and a residual part, wherein the trend part is used to capture the trend information of the overall motion, and the residual part is used to retain local details; S5: combining the extracted features with the trend part and the residual part to generate future motion sequence data to obtain a prediction result; the skeleton features are enhanced by simulating joint combinations and introducing virtual joints.

Description

Technical Field

[0001] This invention pertains to human motion sequence prediction methods, and particularly relates to a human motion sequence prediction method and system based on skeleton enhancement Transformer. Background Technology

[0002] The key to 3D human motion prediction lies in accurately modeling the spatial and temporal dependencies in motion sequences. In existing technologies, the potential feature relationships within the data have not been fully explored, especially when the data is incomplete (e.g., missing joints).

[0003] Existing methods suffer from the following shortcomings when dealing with latent feature relationships in the spatial dimension: Insufficient spatial feature relationship mining: Existing methods often fail to fully capture latent feature relationships when processing the spatial dimension of motion data, especially when faced with incomplete data (such as missing joints), limiting model performance; Poor balance between local and global feature modeling: GCN-based methods typically focus on modeling local features, while Transformer methods emphasize global features, lacking a method that effectively models both local and global spatiotemporal features simultaneously; Limited ability to handle data incompleteness: Due to pose overlap, environmental occlusion, or data noise, data in real-world applications may be incomplete, and most existing methods perform poorly when handling such incomplete data, making it difficult to maintain model performance and robustness; Over-smoothing of prediction trends: Many existing methods tend to make prediction results tend towards the average pose during long-term predictions, lacking detailed capture of motion trends and changes. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for predicting human motion sequences based on skeleton enhancement Transformer, which enhances skeleton features by introducing a combination of virtual joints and simulated real joints, thereby addressing the shortcomings of existing technologies in modeling spatiotemporal dependencies.

[0005] To achieve the above objectives, one aspect of this application provides a method for predicting human motion sequences based on a skeleton-enhanced Transformer, comprising the following steps: S1: sampling human movements to obtain raw motion sequence data; S2: generating virtual joints based on real joints and adding the virtual joints to the sequence to construct enhanced motion sequence data and generate an enhanced skeleton map; S3: inputting the enhanced skeleton map into a graph convolutional network to establish the skeletal correlation between virtual and real joints, and optimizing feature extraction through a spatiotemporal attention mechanism to obtain motion sequence data representing motion features, wherein the motion sequence data serves as input for subsequent decomposition processing; S4: decomposing the input motion sequence data and dividing it into a trend part and a residual part, wherein the trend part is used to capture the overall motion trend information, and the residual part is used to retain local details; S5: combining the extracted features with the trend part and the residual part to generate future motion sequence data to obtain a prediction result.

[0006] Preferably, S2 includes: the position of the virtual joint is calculated based on the average position of several real joints within a preset range, and the calculation formula is as follows:

[0007] ;

[0008] Among them, a i This represents joint data from a motion sequence, where c represents the enhancement coefficient. This represents Gaussian noise.

[0009] Preferably, S2 includes: adding Gaussian noise to simulate a real joint when generating a virtual joint.

[0010] Preferably, S3 includes: taking the enhanced action sequence data as input, and the calculation formula for the graph convolutional network is as follows:

[0011] ;

[0012] in, M represents augmented motion sequence data. s Let W represent the normalized adjacency matrix, σ represent the activation function, and W represent the normalized adjacency matrix. s This represents a trainable weight matrix.

[0013] Preferably, the spatiotemporal attention mechanism includes spatial attention computation and temporal attention computation, which includes the following steps:

[0014] S301: Use different learnable matrices W to perform linear transformations on the motion sequence data to obtain motion sequence data Q. t K t and V t The calculation formula is as follows:

[0015] ;

[0016] Among them, A t This represents the output of the graph convolution, traversing the time axis t;

[0017] S302: Q t With K t Multiplying the results yields the importance ranking between joints. Softmax is then used to normalize the importance ranking, which is then multiplied by V. t The magnitude of attention is obtained by multiplication, and its calculation formula is as follows:

[0018] ;

[0019] Where, d k Indicates the dimension of the data;

[0020] S303: The FFN method is used to perform a linear transformation through a learnable matrix, and then an activation function is used to activate the result of the linear transformation. The calculation formula is as follows:

[0021] ;

[0022] H represents the number of attention heads, used to indicate the number of heads that perform parallel computations in a multi-head attention mechanism;

[0023] S304: Move the time dimension of the motion sequence data to the outermost layer to obtain new motion sequence data;

[0024] S305: Use different learnable matrices W to perform linear transformations on the new motion sequence data to obtain motion sequence data Q. j K j and V j The calculation formula is as follows:

[0025] ;

[0026] Among them, A j This represents the output of the graph convolution, traversing the skeleton nodes;

[0027] S306: Q j With K j Multiplying the results yields the importance ranking between joints. Softmax is then used to normalize the importance ranking, which is then multiplied by V. j The magnitude of attention is obtained by multiplication, and its calculation formula is as follows:

[0028] ;

[0029] S307: The FFN method is used to perform a linear transformation through a learnable matrix, and then an activation function is used to activate the result of the linear transformation. The calculation formula is as follows:

[0030] .

[0031] Preferably, S4 includes:

[0032] The trend component is represented by the result of the moving average, MAB(A'), and its calculation formula is as follows:

[0033] ;

[0034] Let the residual result R represent the residual part, and its calculation formula is as follows:

[0035] .

[0036] Preferably, S5 includes:

[0037] S501: The result F is obtained by combining the extracted motion sequence data features with the trend component. e The calculation formula is as follows:

[0038] ;

[0039] S502: The extracted motion sequence data features are combined with the trend and residual components to obtain the final prediction result F. d The calculation formula is as follows:

[0040] ;

[0041] in, The result of the decoder;

[0042] S503: For F d Downsampling is performed to remove the virtual joints inserted between every two real joints, resulting in motion sequence data of the same size as the original motion sequence.

[0043] Another aspect of this application provides a human motion sequence prediction system based on skeleton enhancement Transformer. Using the aforementioned human motion sequence prediction method, it includes: a skeleton enhancement module configured to generate virtual joints and add the virtual joints to the sequence to construct enhanced motion sequence data; an encoder configured to extract features from the input motion sequence data and model them; a trend learning module configured to decompose the input motion sequence data, extract overall trend information, and incorporate the trend information during the decoding process; and a decoder configured to obtain the prediction result based on the extracted features combined with residual information.

[0044] Preferably, the encoder includes a multi-layer structure, which outputs the result F. e The input is repeatedly fed into the encoder to continue feature learning.

[0045] In another aspect, this application provides an electronic device including a processor and a memory, wherein the memory stores a computer program configured to be executed by the processor, the computer program including instructions for performing the above-described human motion sequence prediction method.

[0046] In another aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a computer, implements the above-described human motion sequence prediction method.

[0047] The technical solution provided in this application can achieve the following beneficial effects:

[0048] 1. By introducing virtual joints and combining graph convolutional networks with spatiotemporal attention mechanisms, this invention can more effectively capture the complex spatiotemporal dependencies in human movement, and can maintain high prediction accuracy even when faced with incomplete or noisy data.

[0049] 2. Due to the introduction of virtual joints and the application of trend learning methods, this invention demonstrates stronger robustness in dealing with common problems in real-world scenarios, such as missing data and noise. Simultaneously, the trend learning method ensures that predictions are not overly smoothed, thereby improving the model's generalization ability across different datasets.

[0050] 3. The trend learning method effectively captures overall trend information, enabling the prediction results to better reflect the real trend of motion changes and avoiding the common problem of prediction results tending to average posture. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 This is a flowchart of the human motion sequence prediction method provided in the embodiments of this application;

[0053] Figure 2 This is a flowchart of the spatiotemporal attention calculation method provided in the embodiments of this application;

[0054] Figure 3 This is a flowchart of the prediction result method provided in the embodiments of this application;

[0055] Figure 4 This is a schematic diagram of the human motion sequence prediction system provided in the embodiments of this application;

[0056] Figure 5 This is a schematic diagram of the skeleton enhancement module structure provided in an embodiment of this application;

[0057] Figure 6 These are schematic diagrams illustrating the visualization prediction results of different methods provided in the embodiments of this application;

[0058] Figure 7 yes Figure 6 Enlarged view of the dashed box;

[0059] Figure 8 yes Figure 6 Enlarged view within the black box. Detailed Implementation

[0060] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0061] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0062] This example provides a method for predicting human motion sequences based on skeleton-enhanced Transformers, such as... Figure 1 The steps shown are as follows:

[0063] S1: Sample human movements to obtain raw motion sequence data.

[0064] Specifically, this example uses the Human3.6M dataset, which contains approximately 3.6 million images. Eleven actors were recorded for the dataset, and these actors were numbered H1, H2, H3, H4, H5, H6, H7, H8, H9, H10, and H11. Only seven actors (H1, H5, H6, H7, H8, H9, and H11) have 3D human pose labels. The actors were filmed from four different angles in the Human3.6M dataset, with a video frame rate of 50Hz. Each actor performed 17 everyday actions, such as smoking, taking photos, and greeting others.

[0065] The Human3.6M dataset records the three-dimensional positions of human joints, represented by the three-dimensional coordinates of 32 joints per frame. In motion prediction, individual motion sequences are downsampled to 25 frames per second. For a single motion sequence, the data format is 25×32×3 (25 frames × 32 joints × 3D coordinates), and the overall data features are the result of temporal × spatial sorting.

[0066] S2: Generate virtual joints based on real joints, add the virtual joints to the sequence, construct enhanced motion sequence data, and generate an enhanced skeleton diagram.

[0067] In traditional human motion sequence prediction methods, the model only uses actual joint (real joint) data for prediction, which easily ignores the potential dependencies between joints. This example introduces virtual joints, which are generated by combining real joint data to simulate the potential relationships between joints, thereby better capturing the smoothness and interdependence of human motion.

[0068] By enhancing the skeleton graph, dependency connections between non-adjacent joints can be added to the skeleton model, forming long-distance dependencies across multiple joints. For example, adding a direct dependency between the upper and lower limbs helps the model understand how arm movements coordinate with leg postures. In addition to directly adjacent joint connections, connections between joints at different levels of the body can be added, such as directly associating distal joints like the head and ankles to capture overall body balance and posture changes. More refined dependency features can be introduced between local joints, such as capturing subtle joint motion changes through graph convolution operations. This helps the model accurately model minute local movements (such as changes in fingers or toes). These virtual joints enhance skeletal information, enabling the model to more comprehensively understand and capture subtle changes in motion sequences. This enhancement directly improves the model's predictive accuracy, especially when dealing with missing joint data; virtual joints provide additional compensatory information, ensuring the model's robustness.

[0069] Specifically, this example uses a 25×32×3 data input. The position of the virtual joint is calculated by averaging the positions of its neighboring real joints, using the following formula:

[0070] ; where a i This represents the joint data of the motion sequence, and 'c' represents the enhancement coefficient. According to human kinematics, the human skeleton is not linearly arranged. For example, the elbow joint between the shoulder and hand is positioned midway between the shoulder and hand during movement, but offset by a certain distance. Therefore, Gaussian noise is added to simulate this positional shift. To simulate real joints, the generated virtual joint positions are made closer to reality. The enhanced skeleton structure represents both real and virtual joints together for subsequent feature extraction and modeling. With c=1, the resulting data format is 25×63×3. Therefore, the enhanced sequence is:

[0071]

[0072] In this section, the original motion sequence data is expanded to reveal previously unexposed features, allowing subsequent models to improve motion prediction performance without any modifications.

[0073] S3: Input the enhanced skeleton graph into a graph convolutional network to establish the skeletal correlation between virtual joints and real joints, and optimize feature extraction through a spatiotemporal attention mechanism to obtain motion sequence data representing motion features. The motion sequence data is used as input for subsequent decomposition processing.

[0074] In one example, the augmented skeleton map is fed into a graph convolutional network (GCNN). The GCNN can effectively learn the association between the augmented skeleton map and the original skeleton map. Since the newly inserted virtual joints are not associated with real joints, the GCNN is used with the adjacency matrix M of the original motion sequence data. s Based on this, the model learns and establishes the skeletal relationship between virtual joints and real joints, allowing virtual joints to truly interact with real joints, rather than simply inserting node coordinates.

[0075] Specifically, to enhance motion sequences Using (25×63×3) as input, the formula for calculating graph convolution is as follows:

[0076]

[0077] in, M represents augmented motion sequence data. s Let W represent the normalized adjacency matrix of the graph, σ represent the activation function, and W represent the normalized adjacency matrix of the graph. sThe trainable weight matrix is ​​represented by (25×63×3). To further optimize the feature extraction process, a spatiotemporal attention mechanism is introduced into the model.

[0078] The spatiotemporal attention mechanism enhances the model's ability to model spatiotemporal dependencies by weighting different time steps and spatial features. Specifically, for a given original motion sequence data A, a graph convolutional network is used to compute the result A^. Taking the enhanced motion sequence data (25×63×3) as an example, different matrices W are used to linearly transform the motion sequence data to obtain the motion sequence data Q. t K t and V t This process does not change the dimensions of the data, but simply uses different learnable matrices W to transform the motion sequence data to obtain multiple motion sequence data, which are used to establish the importance between pairs in subsequent modeling.

[0079] d k This is the dimension of the data, 63×3, used to represent Q. t K t Compressing the results to prevent the inner product from becoming too large can accelerate convergence. t K t Preliminary results on the importance of joints were obtained, and then normalized using softmax. These results were then compared with V. t The magnitude of attention is obtained by multiplication. However, the attention calculation process uses linear calculations between learnable matrices, which lacks nonlinear effects to further simulate the modeling between joints. Therefore, the FFN method is adopted, which involves linear transformation of learnable matrices and then activation of the result of the linear transformation using an activation function (ReLU).

[0080] After spatial attention calculation, the motion sequence data is in the form of 25×63×3. In order to perform attention calculation on the temporal features, the dimensions of the motion sequence data are transformed, and the temporal dimension is moved to the outermost layer for interaction, resulting in a data form of 63×3×25. Other calculation methods are the same as those for spatial attention.

[0081] In one example, such as Figure 2 As shown, the spatiotemporal attention mechanism includes the following computational steps:

[0082] S301: Use different learnable matrices W to perform linear transformations on the motion sequence data to obtain motion sequence data Q. t K t and V t The calculation formula is as follows:

[0083] ;

[0084] Among them, A t This represents the output of graph convolution;

[0085] S302: Q t With K t Multiplying the results yields the importance ranking between joints. Softmax is then used to normalize the importance ranking, which is then multiplied by V. t The magnitude of attention is obtained by multiplication, and its calculation formula is as follows:

[0086] ;

[0087] Where, d k Indicates the dimension of the data;

[0088] S303: The FFN method is used to perform a linear transformation through a learnable matrix, and then an activation function is used to activate the result of the linear transformation. The calculation formula is as follows:

[0089] ;

[0090] S304: Move the time dimension of the motion sequence data to the outermost layer to obtain new motion sequence data;

[0091] S305: Use different learnable matrices W to perform linear transformations on the new motion sequence data to obtain motion sequence data Q. j K j and V j The calculation formula is as follows:

[0092] ;

[0093] Among them, A j This represents the output of graph convolution;

[0094] S306: Q j With K j Multiplying the results yields the importance ranking between joints. Softmax is then used to normalize the importance ranking, which is then multiplied by V. j The magnitude of attention is obtained by multiplication, and its calculation formula is as follows:

[0095] ;

[0096] S307: The FFN method is used to perform a linear transformation through a learnable matrix, and then an activation function is used to activate the result of the linear transformation. The calculation formula is as follows:

[0097] .

[0098] Here, A' represents the output of the spatiotemporal attention mechanism. This mechanism enables the model to effectively distinguish spatial features at different timestamps and temporal features of the same joint at different times, thus capturing the dynamic changes and patterns of the action more comprehensively. By combining graph convolutional networks and the spatiotemporal attention mechanism, this model demonstrates stronger capabilities in modeling complex motion data. The data obtained in this section is in the form of 63×3×25.

[0099] S4: Decompose the input motion sequence data and divide it into a trend part and a residual part. The trend part is used to capture the trend information of the overall motion, and the residual part is used to preserve local details.

[0100] In time series tasks, features are often divided into trend and residual components to better extract and understand the inherent structure of motion sequence data. This decomposition method is widely used in time series modeling and prediction. To prevent the prediction results from converging to the average pose, a trend learning method is introduced into the model. The trend learning method decomposes the input motion sequence data into a trend component and a residual component, as shown in the following equation:

[0101] The trend component is represented by the result of the moving average, MAB(A'), and its calculation formula is as follows:

[0102] ;

[0103] Let the residual result R represent the residual part, and its calculation formula is as follows:

[0104] ;

[0105] The trend part is used to capture the trend information of the overall movement, while the residual part is used to preserve local details. For human movement, the trend part can represent the overall movement pattern of a certain action. For example, when a person starts to bend over from standing, the overall movement process is the trend. The residual part can represent subtle movement changes or noise, such as the movement of fingers in a waving hand. The residual part is particularly important for the reproduction of local movements.

[0106] By modeling the trend part, graph convolutional networks can capture smooth, slow motion changes over long time scales. The motion sequence data obtained using the spatiotemporal attention mechanism is in the form of 63×3×25. The result of the moving average, MAB(A'), is used to represent the trend part of human motion, while the residual result R (complete motion - moving average result) is the residual part. Thus, the trend part and the residual part are obtained. The trend part is additively fused with the initial motion sequence data to comprehensively model the global motion, while the residual part is additively fused with the decoder result to allow different motion output results to get rid of the average pose learned by the model and have different action details. Compared with previous methods, by capturing the residual part, the local details of action prediction are more complete.

[0107] S5: Based on the extracted features and trend components and residual components, future motion sequence data are generated to obtain prediction results.

[0108] Features were previously extracted from motion sequence data. The extracted features were then combined with the trend components, which are rich in global information, to obtain the result F. e The result F e The process will continue iteratively to learn features, combining the results of multiple training iterations (the extracted features) with the residual to generate the future motion sequence and obtain the final prediction result F. d (25×63×3), in order to obtain a data format of the same size as the original motion sequence data, further processing of F... d Downsampling, which involves removing the simulated nodes inserted between every two joints, yields 25×32×3 motion sequence data. By combining graph convolutional networks with spatiotemporal attention mechanisms and trend learning methods, complex spatiotemporal relationships can be better handled, and more accurate future motion sequence data can be generated.

[0109] In one example, such as Figure 3 As shown, the prediction results include the following steps:

[0110] S501: The result F is obtained by combining the extracted motion sequence data features with the trend component. e The calculation formula is as follows:

[0111] ;

[0112] S502: The extracted motion sequence data features are combined with the residual part to obtain the final prediction result F. d The calculation formula is as follows:

[0113] ;

[0114] The motion sequence data extracted by S502 has the following features: F e The result of multiple training cycles.

[0115] S503: For F d Downsampling is performed to remove the virtual joints inserted between every two real joints, resulting in motion sequence data of the same size as the original motion sequence.

[0116] In one example, the SE-Former model was tested on several commonly used 3D human motion prediction benchmark datasets (such as Human3.6M, CMU-MoCap, and 3DPW) with the enhancement coefficient c set to 1 and the loss coefficient λ. o and λ e All values ​​were set to 1. The model was trained over 50 training epochs with a batch size of 24 and an initial learning rate of 0.003. The learning rate was decreased by 0.9 after each iteration. Training was performed using an NVIDIA RTX-3090 GPU with the Adam optimizer. Input actions were scaled by 1 / 100, and output actions were rescaled accordingly. Experiments were conducted on the fifth topic of Human3.6M and all test samples in the CMU-Mocap test folder. Table 1 below shows the short-term and long-term mean joint position error results on the CMU-Mocap dataset. The experimental results show that, compared with other methods, SE-Former (Ours' method) achieves a lower mean joint position error (MPJPE) in both short-term and long-term motion prediction tasks, demonstrating its significant improvement in prediction accuracy.

[0117]

[0118] Table 1

[0119] Furthermore, experiments show that this invention exhibits stronger robustness when handling incomplete data (missing joint information). To simulate incomplete data during data acquisition, the input motion sequence data was randomly masked by 50%, and 50% of the joint coordinates were randomly selected and set to 0. The same random seed was used in different control methods to maintain consistency in the incompleteness condition of the input motion sequence data. Table 2 below shows the experimental data for handling incomplete motion sequences. The experimental results show that the model can still maintain high prediction accuracy when facing this type of data, and the average decrease is much better than that of traditional methods.

[0120]

[0121] Table 2

[0122] Table 3 below shows the experimental data of the model structure ablation, the experimental results and... Figure 4The results show that the introduced trend learning method effectively prevents the prediction results from converging to the average pose. Especially in long-term prediction scenarios, SE-Former can better capture the overall motion trend and local dynamics, improving the reliability and accuracy of the prediction results.

[0123]

[0124] Table 3

[0125] Figure 4 Visualized prediction results of different methods are shown with a frame interval of 80 milliseconds. Compared with the baseline (GT), the LTD prediction is more accurate. After 320 milliseconds, the LTD prediction has entered the static "average pose" (dashed box). The predictions of hand movements by STSGCN and AuxFormer become smooth (solid box). The method of this invention predicts more accurate movements, and the limb features (hand pointing, arm bending) are closer to the real results.

[0126] Figure 5 and Figure 6 This indicates a comparison of visualized prediction results. Figure 5 As shown in the dashed box, after 320 milliseconds, the LTD prediction has entered a static "average pose," meaning that the LTD method does not have the ability to prevent the model from tending towards the average pose. Figure 6 As shown in the black box, the predictions of hand features by the STSGCN and AuxFormer methods have deviated from the actual results, and the local capabilities have decreased. In the fourth row of visualization results, the method of this invention tends to be in motion within the given time range, and the local limb movements are more consistent with the actual results. The experimental results prove the feasibility and superiority of the technical solution of this invention in practical applications.

[0127] like Figure 7 As shown, this example provides a human motion sequence prediction system based on skeleton enhancement Transformer. Using the human motion sequence prediction method provided in the above example, it includes a skeleton enhancement module, an encoder, a trend learning module, and a decoder. The skeleton enhancement module is configured to generate virtual joints and add the virtual joints to the sequence to construct enhanced motion sequence data. The encoder is configured to extract features from the input motion sequence data and model them. The trend learning module is configured to decompose the input motion sequence data, extract the overall trend information, and incorporate the trend information during the decoding process. The decoder is configured to obtain the prediction result based on the extracted features and residual information.

[0128] Specifically, such as Figure 8As shown, the skeleton enhancement module is used to enhance the original motion sequence data, improve the model's ability to learn complex motions, form new topological relationships between virtual joints and real joints, enrich the feature space of the skeleton, and improve the correlation and feature diversity between joints by adding virtual joints.

[0129] The trend learning module decomposes motion sequence data into trend and residual components to capture overall trends and local details, improving prediction accuracy. It smooths the original sequence using moving averages or low-pass filtering, extracting the global trend, such as the overall upward trend when the arm is raised. By removing the trend component, it retains local rapid changes or detailed movements, such as sudden pauses or accelerations during the raising process. The features extracted by the encoder are embedded and fused with the trend and residual information to form a comprehensive feature for decoding.

[0130] The entire model employs an encoder-decoder architecture. The encoder is responsible for extracting features from the input augmented motion sequence data. During the encoding process, the extracted features are combined with the globally informative trend components to obtain the result F. e This allows the network to focus more on the overall motion trend. Because the encoder has a multi-layered structure, the result F... e The input will continue to be fed into the encoder for further feature learning. That is, the output of each encoder layer will continue to be used as the input of the next layer, realizing progressive feature extraction. It captures spatial information from local to global at different levels, improves feature representation capability, and the feature embedding representation after multi-layer encoding contains local temporal information and global trend information, providing rich input for the decoder.

[0131] The decoder then combines the features output by the encoder with the residual to generate future motion sequence data and obtain the final prediction result F. d During the decoding process, the generated motion sequence data is guaranteed to have both an overall trend and local changes. The generated motion sequence data is decoded into joint data (including real joints and virtual joints) in the skeleton diagram to show the future motion trajectory.

[0132] In the encoder-decoder architecture, the encoder is responsible for extracting features from the input enhanced motion sequence data and introducing a trend component at each layer. It uses deep learning to learn local and global features. The decoder combines the encoder's output with the residual component to generate the final prediction result. The encoder-decoder architecture can capture complex motion features, including the temporal evolution and spatial structure changes of the action, and effectively cope with diverse motion patterns.

[0133] This example provides an electronic device corresponding to the above example, including a processor and a memory, wherein the memory stores a computer program configured to be executed by the processor, the computer program including instructions for performing the human motion sequence prediction method of the present invention.

[0134] This example provides a computer-readable storage medium corresponding to the above example, the computer-readable storage medium storing a computer program, which, when executed by a computer, implements the human motion sequence prediction method of the present invention.

[0135] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A human motion sequence prediction method based on skeleton-enhanced Transformer, characterized in that, The method comprises the following steps: S1: sampling human motion to obtain original motion sequence data; S2: generating virtual joints based on real joints, adding the virtual joints to the sequence, constructing enhanced motion sequence data, and generating an enhanced skeleton graph; S3: inputting the enhanced skeleton graph into a graph convolution network to establish the skeletal correlation between virtual joints and real joints, and optimizing feature extraction through a spatiotemporal attention mechanism to obtain motion sequence data representing motion features, which is used as input for subsequent decomposition processing; S4: decomposing the input motion sequence data into a trend part and a residual part, the trend part being used to capture the trend information of the overall motion, and the residual part being used to retain local details; S5: generating future motion sequence data according to the extracted features combined with the trend part and the residual part to obtain a prediction result.

2. The human motion sequence prediction method according to claim 1, characterized by, S2 comprises: The position of the virtual joint is calculated based on the average position of several real joints within its preset range, and the calculation formula is as follows: ； where a i represents the original skeleton joint information, c represents an enhancement coefficient, represents Gaussian noise.

3. The human motion sequence prediction method according to claim 2, characterized by, S2 comprises: When generating the virtual joint, Gaussian noise is added to simulate the real joint.

4. The human motion sequence prediction method according to claim 1, characterized by, S3 comprises: taking the enhanced motion sequence data as input, and the calculation formula of the graph convolution network is as follows: ； wherein, denotes enhanced motion sequence data, M s denotes a normalized adjacency matrix, σ denotes an activation function, W s denotes a trainable weight matrix.

5. The human motion sequence prediction method according to claim 4, characterized in that, The spatiotemporal attention mechanism comprises spatial attention calculation and temporal attention calculation, which comprises the following steps: S301: Linearly transform the motion sequence data using different learnable matrices W to obtain motion sequence data Q t , K t , and V t , whose calculation formula is as follows: ； wherein A t represents the result of graph convolution output, A t is distinguished from the original data in order to traverse the time axis t; S302: Q t and K t The importance degree result between joints is obtained by multiplying, and the importance degree conversion is normalized by using softmax, and then multiplied by V t The size of attention is obtained, and the calculation formula is as follows: ； wherein d k represents the dimension of the data; S303: using the FFN method to linearly convert through a learnable matrix, and then using an activation function to activate the result of linear conversion, and the calculation formula is as follows: ； Where H represents the number of attention heads, indicating the number of parallel computing heads in the multi-head attention mechanism; S304: taking the time dimension of the motion sequence data to the outermost layer to obtain new motion sequence data; S305: Linearly transform the new motion sequence data using different learnable matrix W to obtain motion sequence data Q j , K j and V j , whose calculation formula is as follows: ； wherein A j represents the result of graph convolution output, A j is to distinguish from the original data for traversing the skeleton node; S306: Q j With K j The importance degree result between joints is obtained by multiplication, and the importance degree conversion normalized by using softmax is multiplied by V j The size of attention is obtained, and the calculation formula is as follows: ； S307: using the FFN method to linearly convert through a learnable matrix, and then using an activation function to activate the result of linear conversion, and the calculation formula is as follows: 。 6. The human motion sequence prediction method of claim 1, wherein S4 Comprise: The trend part is represented by the result MAB(A') of the moving average, and the calculation formula is as follows: ； The residual part is represented by the result R of the residual, and the calculation formula is as follows: 。 7. The human motion sequence prediction method of claim 1, wherein S5 Comprise: S501: The extracted motion sequence data features are combined with the trend part to obtain a result F e The calculation formula is as follows: ； S502: The extracted motion sequence data features are combined with the residual part to obtain a final prediction result F d The calculation formula is as follows: ； wherein is the result of the decoder; S503: to F d Down-sampling is performed to remove the virtual joints inserted between every two real joints, and a motion sequence data of the same size as the original motion sequence is obtained.

8. A human motion sequence prediction system based on skeleton-augmented Transformer, using the human motion sequence prediction method of any one of claims 1-7. Comprise: The skeleton enhancement module is configured to generate virtual joints and add the virtual joints to the sequence to construct enhanced motion sequence data; The encoder is configured to extract features and model the input motion sequence data; The trend learning module is configured to decompose the input motion sequence data to extract overall trend information, and integrate the trend information in the decoding process; The decoder is configured to obtain a prediction result according to the extracted features combined with residual information.

9. The human motion sequence prediction system of claim 8, wherein, The encoder comprises a multi-layer structure, resulting F e Looping the input into the encoder continues feature learning.

10. An electronic device, comprising: The computer readable storage medium stores a computer program, which is executed by a computer, and realizes the human motion sequence prediction method according to any one of claims 1-7.

11. A computer readable storage medium characterized by, The computer readable storage medium stores a computer program, which is executed by a computer, and realizes the human motion sequence prediction method according to any one of claims 1-7.

Images