Human motion sequence prediction method and system based on skeleton enhanced Transform

Abstract

The invention provides a human motion sequence prediction method and system based on skeleton enhanced Transform, and the method comprises the following steps: S1, carrying out the sampling of human motions, and obtaining original motion sequence data; s2, generating a virtual joint based on a real joint, adding the virtual joint into a sequence, constructing enhanced motion sequence data, and generating an enhanced skeleton graph; s3, inputting the enhanced skeleton graph into a graph convolutional network, establishing skeleton relevance between a virtual joint and a real joint, and optimizing feature extraction through a space-time attention mechanism; s4, the input motion sequence data is decomposed and divided into a trend part and a residual part, the trend part is used for capturing trend information of overall motion, and the residual part is used for retaining local details; s5, the extracted features are combined with the trend part and the residual part to generate future motion sequence data, and a prediction result is obtained; skeleton features are enhanced by simulating joint combinations and introducing virtual joints.

Description

Technical Field

[0001] The present invention relates to a human motion sequence prediction method, and in particular to a human motion sequence prediction method and system based on a skeleton-enhanced Transformer. Background Art

[0002] The key to 3D human motion prediction lies in accurately modeling the spatial and temporal dependencies in motion sequences. In the prior art, the potential feature relationships in the data have not been fully explored, especially when the data is incomplete (e.g., missing joints).

[0003] Existing methods have the following deficiencies when dealing with latent feature relationships in the spatial dimension. For example, the spatial feature relationship mining is insufficient: when dealing with the spatial dimension of motion data, existing methods often cannot fully capture the latent feature relationships, especially when faced with incomplete data (such as missing joints), the performance of the model will be limited; the balance between local and global feature modeling is poor: GCN-based methods usually focus on modeling local features, while Transformer methods focus on global features, and lack a method to effectively model local and global spatiotemporal features at the same time; limited ability to deal with data incompleteness: due to posture overlap, environmental occlusion or data noise, the data in actual applications may be incomplete, and most existing methods perform poorly when dealing with such incomplete data, making it difficult to maintain the performance and robustness of the model; excessive smoothing of prediction trends: many existing methods tend to make the prediction results tend to the average posture when predicting for a long time, and lack detailed capture of motion trends and changes. Summary of the invention

[0004] The purpose of this application is to provide a human motion sequence prediction method and system based on skeleton-enhanced Transformer, which enhances skeleton features by introducing virtual joints and simulating real joint combinations to solve the shortcomings of the existing technology in modeling spatiotemporal dependency relationships.

[0005] In order to achieve the above-mentioned purpose, one aspect of the present application provides a human motion sequence prediction method based on skeleton-enhanced Transformer, comprising the following steps: S1: sampling human body movements to obtain original motion sequence data; S2: generating virtual joints based on real joints, and adding the virtual joints to the sequence, constructing enhanced motion sequence data, and generating an enhanced skeleton graph; S3: inputting the enhanced skeleton graph into a graph convolutional network, establishing the skeletal correlation between virtual joints and real joints, and optimizing feature extraction through a spatiotemporal attention mechanism; S4: decomposing the input motion sequence data and dividing it into a trend part and a residual part, the trend part being used to capture the trend information of the overall motion, and the residual part being used to retain local details; S5: generating future motion sequence data based on the extracted features combined with the trend part and the residual part to obtain prediction results.

[0006] Preferably, S2 includes: the position generated by the virtual joint is calculated based on the average position of several real joints within its preset range, and the calculation formula is as follows: ; Among them, a i represents the motion sequence joint data, and c represents the enhancement coefficient.

[0007] Preferably, S2 includes: when generating virtual joints, adding Gaussian noise to simulate real joints.

[0008] Preferably, S3 includes: taking the enhanced action sequence data as input, the calculation formula of the graph convolution network is as follows: ; in, represents enhanced motion sequence data, M s represents the normalized adjacency matrix, σ represents the activation function, W s Represents a trainable weight matrix.

[0009] Preferably, the spatiotemporal attention mechanism includes spatial attention calculation and temporal attention calculation, which includes the following steps: S301: Use different learnable matrices W to linearly transform the motion sequence data to obtain the motion sequence data Q t , K t and V t , and its calculation formula is as follows: ; Among them, A t Represents the result of graph convolution output, traversing the time axis t; S302:Q t With K tMultiply them together to get the importance of the joints, use softmax to normalize the importance, and then add V t Multiplying together gives the size of attention, which is calculated as follows: ; Among them, d k Represents the dimension of the data; S303: Use the FFN method to perform linear transformation through a learnable matrix, and then use an activation function to activate the result of the linear transformation. The calculation formula is as follows: ; S304: moving the time dimension of the motion sequence data to the outermost layer to obtain new motion sequence data; S305: Use different learnable matrices W to perform linear transformation on the new motion sequence data to obtain motion sequence data Q j , K j and V j , and its calculation formula is as follows: ; Among them, A j Represents the result of graph convolution output, traversing the skeleton nodes; S306:Q j With K j Multiply them together to get the importance of the joints, use softmax to normalize the importance, and then add V j Multiplying together gives the size of attention, which is calculated as follows: ; S307: Use the FFN method to perform linear transformation through a learnable matrix, and then use an activation function to activate the result of the linear transformation. The calculation formula is as follows: .

[0010] Preferably, S4 includes: The trend part is represented by the moving average result MAB(A'), and its calculation formula is as follows: ; The residual result R is used to represent the residual part, and its calculation formula is as follows: .

[0011] Preferably, S5 includes: S501: The extracted motion sequence data features are combined with the trend part to obtain the result F e , and its calculation formula is as follows: ; S502: The extracted motion sequence data features are combined with the trend part and the residual part to obtain the final prediction result F d , and its calculation formula is as follows: ; S503: F d Down sampling is performed to remove the virtual joints inserted between every two real joints to obtain motion sequence data of the same size as the original motion sequence.

[0012] On the other hand, the present application provides a human motion sequence prediction system based on a skeleton-enhanced Transformer, using the above-mentioned human motion sequence prediction method, including: a skeleton enhancement module, configured to generate virtual joints, and add the virtual joints to the sequence to construct enhanced motion sequence data; an encoder, configured to extract features of the input motion sequence data and model it; a trend learning module, configured to decompose the input motion sequence data, extract overall trend information, and integrate the trend information into the decoding process; a decoder, configured to obtain a prediction result based on the extracted features combined with the residual information.

[0013] Preferably, the encoder comprises a multi-layer structure, e The loop is input into the encoder to continue feature learning.

[0014] On the other hand, the present application provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing the above-mentioned human motion sequence prediction method.

[0015] On the other hand, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the above-mentioned human motion sequence prediction method is implemented.

[0016] The technical solution provided by this application can achieve the following beneficial effects: 1. By introducing virtual joints and combining graph convolutional networks with spatiotemporal attention mechanisms, the present invention can more effectively capture the complex spatiotemporal dependencies in human motion, especially in the case of incomplete or noisy data, while still maintaining a high prediction accuracy.

[0017] 2. Due to the introduction of virtual joints and the application of trend learning methods, the present invention shows stronger robustness in dealing with common problems such as missing data and noise in actual scenarios. At the same time, the trend learning method ensures that the prediction will not be over-smoothed, thereby improving the generalization ability of the model on different data sets.

[0018] 3. The trend learning method effectively captures the overall trend information, so that the prediction results can better reflect the real trend of motion changes and avoid the common problem that the prediction results tend to the average posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0020] Figure 1 is a flow chart of a method for predicting a human motion sequence provided by an embodiment of the present application; Figure 2 is a flow chart of the spatiotemporal attention calculation method provided in an embodiment of the present application; Figure 3 is a flow chart of the prediction result method provided in the embodiment of the present application; Figure 4 is a schematic diagram of the structure of a human motion sequence prediction system provided in an embodiment of the present application; Figure 5 It is a schematic diagram of the structure of the skeleton reinforcement module provided in the embodiment of the present application; Figure 6 It is a schematic diagram of the visual prediction results of different methods provided in the embodiments of the present application; Figure 7 yes Figure 6 Enlarged view of the dashed box in the middle; Figure 8 yes Figure 6 Enlarged image of the black box. DETAILED DESCRIPTION

[0021] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0022] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0023] This example provides a human motion sequence prediction method based on skeleton-enhanced Transformer. Figure 1 The following steps are shown: S1: Sample human motion and obtain original motion sequence data.

[0024] Specifically, this example takes the Human3.6M dataset as an example. The Human3.6M dataset includes about 3.6 million images, and a total of 11 actors are recorded in the dataset. Among them, the 11 actors are numbered as H1, H2, H3, H4, H5, H6, H7, H8, H9, H10 and H11, and only H1, H5, H6, H7, H8, H9 and H11 have 3D human posture labels. In the Human3.6M dataset, the actors are filmed and recorded from 4 angles, and the video recording frame rate is 50Hz. Each actor performs 17 daily actions, such as smoking, taking pictures, and greeting.

[0025] The Human3.6M dataset records the 3D positions of human joints, represented by the 3D coordinates of 32 joint points per frame. In motion prediction, a single motion sequence is downsampled to 25 frames per second. For a single motion sequence, the data format is 25×32×3 (25 frames×32 joints×3D coordinates), and the overall data features are the result of time×space sorting.

[0026] S2: Generate virtual joints based on real joints, add the virtual joints to the sequence, construct enhanced motion sequence data, and generate an enhanced skeleton graph.

[0027] In traditional human motion sequence prediction methods, the model only uses the data of actual joints (real joints) for prediction, which easily ignores the potential dependencies between joints. This example introduces virtual joints, which are generated by combining the data of real joints to simulate the potential relationships between joints, thereby better capturing the smoothness and interdependence of human motion.

[0028] By enhancing the skeleton graph, dependency connections of non-adjacent joints can be added to the skeleton model to form long-distance dependencies across multiple joints. For example, adding a direct dependency between the upper and lower limbs helps the model understand how arm movements are coordinated with leg postures. In addition to directly adjacent joint connections, connections between joints at different levels of the body can also be added, such as directly associating distal joints such as the head and ankles to capture overall body balance and posture changes. More refined dependency features are introduced between local joints, such as capturing small changes in joint movement through graph convolution operations, which helps the model accurately model small local movements (such as changes in fingers or toes). These virtual joints enhance the skeleton information, allowing the model to more fully understand and capture subtle changes in motion sequences. This enhancement directly improves the prediction accuracy of the model, especially in the case of missing data for some joints, where virtual joints provide additional compensation information to ensure the robustness of the model.

[0029] Specifically, this example uses the data format of 25×32×3 as input, and the position of the virtual joint is calculated by averaging the positions of multiple real joints adjacent to it. The calculation formula is as follows: ; Among them, a i represents the joint data of the motion sequence, and c represents the enhancement coefficient. According to the knowledge of human kinematics, the human bones are not arranged linearly. For example, the elbow joint between the shoulder and the palm is located between the shoulder and the hand during movement and is offset by a certain position. Therefore, in order to simulate the position offset, Gaussian noise is added at the same time. To simulate the real joints, the generated virtual joint positions are closer to the actual situation. The enhanced skeleton structure represents the real joints and virtual joints together for subsequent feature extraction and modeling. When c=1, the data format obtained is 25×63×3. That is, the enhanced sequence is: In this section, the original motion sequence data is expanded to present previously unexposed features, and the subsequent model can improve the performance of motion prediction without any modification.

[0030] S3: Input the enhanced skeleton graph into a graph convolutional network to establish the skeletal association between virtual joints and real joints, and optimize feature extraction through a spatiotemporal attention mechanism.

[0031] In one example, the enhanced skeleton graph is input into the graph convolutional network, which can effectively learn the association between the enhanced skeleton graph and the original skeleton graph. Since the newly inserted virtual joints are not associated with the real joints, the graph convolutional network is used to learn the adjacency matrix M of the original motion sequence data.s Based on this, the model learns and establishes the skeletal associations between virtual joints and real joints, allowing the virtual joints to truly interact with the real joints instead of simply inserting the node coordinates.

[0032] Specifically, to enhance motion sequences (25×63×3) as input, the calculation formula of graph convolution is as follows: in, represents enhanced motion sequence data, M s represents the normalized adjacency matrix of the graph, σ represents the activation function, and W s Represents a trainable weight matrix, and the resulting data format is (25×63×3). In order to further optimize the feature extraction process, the spatiotemporal attention mechanism is introduced into the model.

[0033] The spatiotemporal attention mechanism enhances the model's ability to model spatiotemporal dependencies by weighting different time steps and spatial features. Specifically, for a given original motion sequence data A, the result A^ is calculated by the graph convolutional network. Taking the enhanced motion sequence data (25×63×3) as an example, different matrices W are used to linearly transform the motion sequence data to obtain the motion sequence data Q t , K t and V t This process does not change the dimension of the data, but only uses different learnable matrices W to transform the motion sequence data to obtain multiple motion sequence data, which are used to establish the importance between each other in subsequent modeling.

[0034] d k is the dimension of the data, which is 63×3, and is used to convert Q t K t The result of Q is compressed to prevent the inner product result from being too large, which can accelerate convergence. t K t Get the preliminary importance results between joints, use softmax to normalize the importance conversion, and then add V t Multiply to get the size of attention, and in the process of attention calculation, linear calculations between learnable matrices are used, lacking the effect of nonlinearity to further simulate the modeling between joints. Therefore, the FFN method is adopted, that is, linear transformation of the learnable matrix is ​​performed, and then the activation function (ReLu) is used to activate the result of the linear transformation.

[0035] After the spatial attention is calculated, the motion sequence data is in the form of 25×63×3. In order to be able to perform attention calculation on the time features, the dimensions of the motion sequence data are converted, and the time dimension is brought to the outermost layer for interaction to obtain a data format of 63×3×25. The other calculation methods are the same as the spatial attention form.

[0036] In one example, if Figure 2 As shown, the spatiotemporal attention mechanism includes the following calculation steps: S301: Use different learnable matrices W to linearly transform the motion sequence data to obtain the motion sequence data Q t , K t and V t , and its calculation formula is as follows: ; Among them, A t Represents the result of graph convolution output; S302:Q t With K t Multiply them together to get the importance of the joints, use softmax to normalize the importance, and then add V t Multiplying together gives the size of attention, which is calculated as follows: ; Among them, d k Represents the dimension of the data; S303: Use the FFN method to perform linear transformation through a learnable matrix, and then use an activation function to activate the result of the linear transformation. The calculation formula is as follows: ; S304: moving the time dimension of the motion sequence data to the outermost layer to obtain new motion sequence data; S305: Use different learnable matrices W to perform linear transformation on the new motion sequence data to obtain motion sequence data Q j , K j and V j , and its calculation formula is as follows: ; Among them, A j Represents the result of graph convolution output; S306:Q j With K j Multiply them together to get the importance of the joints, use softmax to normalize the importance, and then add V j Multiplying together gives the size of attention, which is calculated as follows: ; S307: Use the FFN method to perform linear transformation through a learnable matrix, and then use an activation function to activate the result of the linear transformation. The calculation formula is as follows: .

[0037] Among them, A' is the result of the spatiotemporal attention mechanism output. The spatiotemporal attention mechanism enables the model to effectively distinguish the spatial features of different timestamps and the temporal features of the same joint at different times, so as to more comprehensively capture the dynamic changes and patterns of the action. By combining the graph convolutional network and the spatiotemporal attention mechanism, this model has shown stronger ability in modeling complex motion data. The data format obtained in this part is 63×3×25.

[0038] S4: Decompose the input motion sequence data and divide it into a trend part and a residual part, wherein the trend part is used to capture the trend information of the overall motion, and the residual part is used to retain local details.

[0039] In time series tasks, features are often divided into trend parts and residual parts in order to better extract and understand the intrinsic structure of motion sequence data. This decomposition method is widely used in time series modeling and prediction. In order to prevent the prediction results from converging to the average posture, the model introduces a trend learning method. The trend learning method decomposes the input motion sequence data into a trend part and a residual part. The decomposition method is shown in the following formula: The trend part is represented by the moving average result MAB(A'), and its calculation formula is as follows: ; The residual result R is used to represent the residual part, and its calculation formula is as follows: ; Among them, the trend part is used to capture the trend information of the overall movement, and the residual part is used to retain local details. For human movement, the trend part can represent the overall movement pattern of a certain action. For example, a person bends down from standing. The overall movement process is the trend; the residual part can represent subtle movement changes or noise, such as the swinging of fingers in waving. The residual part is particularly important for the restoration of local movements.

[0040] By modeling the trend part, the graph convolutional network can capture smooth and slow motion changes over a long time scale. The motion sequence data obtained using the spatiotemporal attention mechanism is in the form of 63×3×25. The moving average result MAB(A') is used to represent the trend part of the human body motion, and the residual result R (complete motion-moving average result) is the residual part. So far, the trend part and the residual part are obtained. The trend part and the initial motion sequence data are additively fused to comprehensively model the global motion, and the residual part and the decoder result are additively fused to make different motion output results break away from the average posture learned by the model to have different motion details. Compared with previous methods, by capturing the residual part, the local details of motion prediction will be more complete.

[0041] S5: Generate future motion sequence data based on the extracted features combined with the trend part and the residual part to obtain the prediction result.

[0042] Previously, the features of the motion sequence data were extracted, and the extracted features were combined with the trend part with rich global information to obtain the result F e , the result is F e The feature learning will continue in a loop, and the results of multiple loop training (extracted features) will be combined with the residual part to generate the future motion sequence to obtain the final prediction result F d (25×63×3), in order to obtain the same size of data as the original motion sequence data, F d Downsampling is performed, that is, the simulated nodes inserted between every two joints are removed to obtain 25×32×3 motion sequence data. By combining the graph convolutional network-spatiotemporal attention mechanism and trend learning method, it can better handle complex spatiotemporal relationships and generate more accurate future motion sequence data.

[0043] In one example, if Figure 3 As shown, the prediction results include the following steps: S501: The extracted motion sequence data features are combined with the trend part to obtain the result F e , and its calculation formula is as follows: ; S502: The extracted motion sequence data features are combined with the residual part to obtain the final prediction result F d , and its calculation formula is as follows: ; The motion sequence data feature extracted in S502 is F e The results of multiple training cycles.

[0044] S503: F dDown sampling is performed to remove the virtual joints inserted between every two real joints to obtain motion sequence data of the same size as the original motion sequence.

[0045] In an example, the SE-Former model was tested on several commonly used 3D human motion prediction benchmark datasets (such as Human3.6M, CMU-MoCap, and 3DPW), with the enhancement coefficient c set to 1 and the loss coefficient λ o and λ e are all set to 1. The model was trained for 50 training cycles with a batch size of 24 and an initial learning rate of 0.003. The learning rate was decreased by 0.9 after each iteration. An NVIDIA RTX-3090 GPU was used for training, and the Adam optimizer was used. The input action was scaled by 1 / 100, and the output action was rescaled accordingly. Experiments were conducted on the fifth subject of Human3.6M and all test samples in the CMU-Mocap test folder. Table 1 below shows the results of the short-term and long-term predicted average joint position error on the CMU-Mocap dataset. The experimental results show that compared with other methods, Ours (the method of the present invention) SE-Former achieved lower mean joint position error (MPJPE) in both short-term and long-term motion prediction tasks, demonstrating its significant improvement in prediction accuracy.

[0046] Table 1 In addition, the experiment also shows that the present invention exhibits stronger robustness when processing incomplete data (missing some joint information). In order to simulate the situation of incomplete data during data acquisition, the input motion sequence data is randomly masked by 50%, and 50% of the joint coordinates are randomly selected and set to 0. The same random seed is set in different control methods to keep the incomplete conditions of the input motion sequence data consistent. Table 2 below shows the experimental data for processing incomplete motion sequences. The experimental results show that the model can still maintain a high prediction accuracy when facing such data, and the average decrease is much better than the traditional method.

[0047] Table 2 Table 3 below shows the model structure ablation experimental data, experimental results and Figure 4 It shows that the introduced trend learning method effectively prevents the prediction results from converging to the average posture. Especially in long-term prediction scenarios, SE-Former can better capture the overall motion trend and local dynamics, improving the credibility and accuracy of the prediction results.

[0048] Table 3 Figure 4 The visualization prediction results of different methods are shown, with a frame interval of 80 milliseconds. Compared with the baseline (GT), the action is completed more accurately. After 320 milliseconds, the prediction of LTD has entered the static "average posture" (dashed box); the predictions of STSGCN and AuxFormer for hand movements become smooth (solid box); the method of the present invention predicts more accurate movements, and the limb features (hand pointing, arm bending) are closer to the real results.

[0049] Figure 5 and Figure 6 Represents the comparison of visual prediction results, Figure 5 It is shown that in the dashed box, after 320 milliseconds, the prediction of LTD has entered the static "average posture", that is, the LTD method does not have the ability to suppress the model from tending to the average posture. Figure 6 It shows that in the black box, the prediction of hand features in the STSGCN and AuxFormer methods has deviated from the actual results, and the local ability has decreased. In the fourth row of visualization results, the method of the present invention has been tending to the motion state within the given time range and the local movement of the limbs is more consistent with the actual results. The experimental results prove the feasibility and superiority of the technical solution of the present invention in practical applications.

[0050] like Figure 7 As shown, this example provides a human motion sequence prediction system based on skeleton-enhanced Transformer, using the human motion sequence prediction method provided by the above example, including a skeleton enhancement module, an encoder, a trend learning module and a decoder. The skeleton enhancement module is configured to generate virtual joints, and add the virtual joints to the sequence to construct enhanced motion sequence data; the encoder is configured to extract features of the input motion sequence data and model it; the trend learning module is configured to decompose the input motion sequence data, extract the overall trend information, and integrate the trend information into the decoding process; the decoder is configured to obtain a prediction result based on the extracted features combined with the residual information.

[0051] Specifically, Figure 8 As shown in the figure, the skeleton enhancement module is used to enhance the original motion sequence data, improve the model's ability to learn complex motions, form new topological relationships between virtual joints and real joints, enrich the feature space of the skeleton, and enhance the correlation and feature diversity between joint points by adding virtual joints.

[0052] The trend learning module decomposes the motion sequence data into a trend part and a residual part to capture the overall trend and local detail information and improve the accuracy of the prediction. It uses a sliding average or low-pass filter to smooth the original sequence and extract the global trend part, such as the overall upward trend when the arm is raised. By removing the trend part, the local rapid changes or detailed movements are retained, such as sudden pauses or accelerations during the raising process. The feature embedding extracted by the encoder is fused with the trend and residual information to form a comprehensive feature for decoding.

[0053] The whole model adopts an encoder-decoder architecture. The encoder is responsible for extracting features from the input enhanced motion sequence data. During the encoding process, the extracted features are combined with the trend part with rich global information to obtain the result F e , so that the network pays more attention to the overall motion trend. Since the encoder has a multi-layer structure, the result F e The input will continue to be looped into the encoder to continue feature learning. That is, the output result of each layer of encoder will continue to be the input of the next layer, realizing progressive feature extraction, capturing local to global spatial information at different levels, and improving feature expression capabilities. The feature embedding representation after multi-layer encoding contains local timing information and global trend information, providing rich input for the decoder.

[0054] The decoder generates future motion sequence data based on the features output by the encoder and combines them with the residual part to obtain the final prediction result F. d ,During the decoding process, it is ensured that the generated motion ,sequence data has both the overall trend and the local changes are maintained, ,and the generated motion sequence data is decoded into the joint point data (including ,real joints and virtual joints) in the skeleton graph to show the ,future motion trajectory.

[0055] In the encoder-decoder architecture, the encoder is responsible for feature extraction of the input enhanced motion sequence data, and introduces the trend part at each layer to deeply learn local and global features. The decoder combines the output of the encoder with the residual part to generate the final prediction result. The encoder-decoder architecture can capture complex motion features, including the temporal evolution of the action and the changes in the spatial structure, and effectively respond to diverse motion patterns.

[0056] This example provides an electronic device corresponding to the above example, including a processor and a memory, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing the human motion sequence prediction method of the present invention.

[0057] This example provides a computer-readable storage medium corresponding to the above example, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the human motion sequence prediction method of the present invention is implemented.

[0058] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A human motion sequence prediction method based on skeleton-enhanced Transformer, characterized in that: The following steps are involved: S1: Sample human motion and obtain original motion sequence data; S2: Generate virtual joints based on real joints, add the virtual joints to the sequence, construct enhanced motion sequence data, and generate an enhanced skeleton graph; S3: inputting the enhanced skeleton graph into a graph convolutional network, establishing the skeleton association between virtual joints and real joints, and optimizing feature extraction through a spatiotemporal attention mechanism; S4: decomposing the input motion sequence data and dividing it into a trend part and a residual part, wherein the trend part is used to capture the trend information of the overall motion, and the residual part is used to retain local details; S5: Generate future motion sequence data based on the extracted features combined with the trend part and the residual part to obtain the prediction result.

2. The human motion sequence prediction method according to claim 1, characterized in that: S2 includes: The position generated by the virtual joint is calculated based on the average position of several real joints within its preset range, and its calculation formula is as follows: Among them, a i represents the original skeleton joint information, and c represents the enhancement coefficient.

3. The human motion sequence prediction method according to claim 2, characterized in that: S2 includes: When generating virtual joints, Gaussian noise is added to simulate real joints.

4. The human motion sequence prediction method according to claim 1, characterized in that: S3 includes: Taking the enhanced action sequence data as input, the calculation formula of the graph convolutional network is as follows: A^=SGCN(A * )=σ(M s A * W s ); Among them, A * represents enhanced motion sequence data, M s represents the normalized adjacency matrix, σ represents the activation function, W s Represents a trainable weight matrix.

5. The human motion sequence prediction method according to claim 4, characterized in that: The spatiotemporal attention mechanism includes spatial attention calculation and temporal attention calculation, which includes the following steps: S301: Use different learnable matrices W to linearly transform the motion sequence data to obtain the motion sequence data Q t , K t and V t , and its calculation formula is as follows: Q t ,K t ,V t =A t W Q ,A t W K ,A t W V ; Among them, A t Represents the result of graph convolution output; S302:Q t With K t Multiply them together to get the importance of the joints, use softmax to normalize the importance, and then add V t Multiplying together gives the size of attention, which is calculated as follows: Among them, d k Represents the dimension of the data; S303: Use the FFN method to perform linear transformation through a learnable matrix, and then use an activation function to activate the result of the linear transformation. The calculation formula is as follows: S304: moving the time dimension of the motion sequence data to the outermost layer to obtain new motion sequence data; S305: Use different learnable matrices W to perform linear transformation on the new motion sequence data to obtain motion sequence data Q j , K j and V j , and its calculation formula is as follows: Q j ,K j ,V j =A j W Q ,A j W K ,A j W V ; Among them, A j Represents the result of graph convolution output; S306:Q j With K j Multiply them together to get the importance of the joints, use softmax to normalize the importance, and then add V j Multiplying together gives the size of attention, which is calculated as follows: S307: Use the FFN method to perform linear transformation through a learnable matrix, and then use an activation function to activate the result of the linear transformation. The calculation formula is as follows:

6. The method for predicting human motion sequence according to claim 1, characterized in that S4 include: The trend part is represented by the moving average result MAB(A'), and its calculation formula is as follows: The residual result R is used to represent the residual part, and its calculation formula is as follows: R = A'-MAB(A').

7. The method for predicting human motion sequence according to claim 1, characterized in that S5 include: S501: The extracted motion sequence data features are combined with the trend part to obtain the result F e , and its calculation formula is as follows: F e =A′+MAB(A′); S502: The extracted motion sequence data features are combined with the residual part to obtain the final prediction result F d , and its calculation formula is as follows: F d =A″+R; S503: F d Down sampling is performed to remove the virtual joints inserted between every two real joints to obtain motion sequence data of the same size as the original motion sequence.

8. A human motion sequence prediction system based on skeleton-enhanced Transformer, using the human motion sequence prediction method according to any one of claims 1 to 7, characterized in that: include: A skeleton enhancement module is configured to generate virtual joints and add the virtual joints to a sequence to construct enhanced motion sequence data; An encoder is configured to extract features and model the input motion sequence data; a trend learning module, configured to decompose the input motion sequence data, extract the overall trend information, and incorporate the trend information into the decoding process; The decoder is configured to obtain a prediction result based on the extracted features combined with the residual information.

9. The human motion sequence prediction system according to claim 7, characterized in that: The encoder includes a multi-layer structure, and the result F e The loop is input into the encoder to continue feature learning.

10. An electronic device, characterized in that: It comprises a processor and a memory, wherein a computer program is stored in the memory, and the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the human motion sequence prediction method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the human motion sequence prediction method according to any one of claims 1 to 7 is implemented.

Images