Human motion prediction method and system based on space-time modeling of diffusion model

Abstract

The invention provides a human body motion prediction method and system based on space-time modeling of a diffusion model, relates to the technical field of human body motion analysis, and aims to solve the problems that a noise prediction technology in existing human body motion prediction cannot fully capture close association of space-time interaction, and prediction is completed only by depending on a single prediction result of the model. The real condition of the motion sequence cannot be accurately and faithfully reflected, and the calculation complexity is high. According to the method, a noise prediction network model is constructed by combining a plurality of space-time crossing fusion graph convolutional networks with a stacked motion residual learning network, accurate prediction of human motion noise is realized, and human motion is accurately predicted according to a noise prediction result. According to the method, the problems existing in the noise prediction technology in existing human motion prediction are solved, the spatial topological relation between joints is fully considered, effective information of data can be fully utilized, meanwhile, the calculation efficiency is considered, the requirement of real-time application is met, and then the accuracy of human motion prediction is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of human motion analysis, and in particular relates to a method and system for predicting human motion based on spatiotemporal modeling of a diffusion model. Background Art

[0002] Human Motion Prediction (HMP) is a technology that predicts future 3D human skeleton motion based on historical motion data. This technology has broad applications in fields such as virtual reality, augmented reality, human-computer interaction, and autonomous driving. Its core goal is to build a model that accurately predicts subsequent motions through in-depth analysis of historical human motion data, thereby providing high-precision motion prediction support for related applications.

[0003] Currently, existing human motion prediction models typically employ diffusion models, such as those described in patents CN119169205A and CN118470115A. Diffusion models are a recent advancement in generative modeling, demonstrating significant advantages in generating high-quality, multimodal, and long sequences of human motion. By learning the distributional characteristics of human motion data and incorporating the inherent constraints of motion sequences, these models can accurately predict subsequent human motion given given observation conditions. This characteristic makes them highly valuable for human motion prediction.

[0004] However, existing methods for human motion prediction based on diffusion models lack spatiotemporal motion constraints and data integration, resulting in diverse and inaccurate prediction results that fail to reflect actual motion patterns. Furthermore, the algorithms are complex and inefficient. Spatial constraints refer to the relative positional relationships of joints at each time step, reflecting the geometric structure and interactions between joints. Temporal constraints refer to the dependencies between different moments, reflecting the dynamic evolution of joint motion. For example, some methods use simple multi-layer perceptrons for spatial modeling. While this reduces computational complexity, it fails to fully utilize the topological information between joints, resulting in poor learning of spatial dependencies. Other methods adopt a separate modeling strategy, using a spatial Transformer for joint modeling during the initial denoising phase and a temporal Transformer for temporal modeling. While this approach simplifies the complexity of spatiotemporal modeling to some extent, the independent processing of spatial and temporal modeling fails to fully capture the close connections between spatiotemporal interactions, thus affecting prediction accuracy. Furthermore, the high computational complexity of the Transformer makes it difficult to meet real-time performance requirements, limiting its deployment in practical applications.

[0005] Furthermore, existing methods rely solely on a single prediction from the model to complete the prediction task, which limits the full utilization of the effective information in the input, resulting in the prediction results being unable to accurately and faithfully reflect the true state of the input sequence. In summary, when using diffusion models to achieve faithful human motion prediction, there are at least two major challenges: one is the spatiotemporal modeling of human motion, and the other is the effective integration of spatiotemporal constraints into the diffusion model. Summary of the Invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a human motion prediction method and system based on spatiotemporal modeling of a diffusion model. The method combines multiple spatiotemporal cross-fusion graph convolutional networks with a stacked motion residual learning network to construct a noise prediction network model, thereby achieving accurate prediction of human motion noise and accurately predicting human motion based on the noise prediction results.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: A first aspect of the present invention provides a method for predicting human motion based on spatiotemporal modeling of a diffusion model, comprising: Obtain the spectrum of the observed motion sequence and the spectrum of the pure noise sequence; Inputting the observed motion sequence spectrum and the pure noise sequence spectrum into a trained noise prediction network model to obtain predicted noise; According to the predicted noise, based on the standard normal distribution, the current denoised motion sequence spectrum is sampled and generated, the noisy observed motion sequence spectrum is calculated, and the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism, and the iterative denoising is repeated until a new complete motion sequence spectrum is obtained; Performing inverse discrete cosine transform on the new complete motion sequence spectrum to obtain a predicted motion sequence; Among them, the noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network; the improved stacked motion residual learning network includes a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks and a second linear mapping layer connected in sequence; the motion sequence spectrum is input into the first linear mapping layer for linear mapping; the output of the first linear mapping layer is sent to the cascaded spatiotemporal cross-fusion graph convolutional network to obtain the forward prediction result and backward prediction result of each spatiotemporal cross-fusion graph convolutional network; each forward prediction result is added to obtain the final noise feature; the noise feature passes through the second linear mapping layer to obtain the predicted noise.

[0008] As an implementation method, the frequency spectrum of the observed motion sequence is obtained, and the specific process is as follows: performing padding processing on the observed motion sequence to obtain a padded observed motion sequence; The padded observed motion sequence is subjected to discrete cosine transform to obtain the observed motion sequence spectrum.

[0009] As an implementation method, the observed motion sequence spectrum and the pure noise sequence spectrum are input into a trained noise prediction network model. The specific process is as follows: Inputting the observed motion sequence spectrum into a conditional encoder to obtain input conditions of a stacked motion residual learning network; The input conditions of the stacked motion residual learning network and the pure noise sequence spectrum are input into the improved stacked motion residual learning network to obtain predicted noise.

[0010] As an implementation method, the observed motion sequence spectrum is input into a conditional encoder, wherein the conditional encoder includes an observation encoder and a time encoder. The specific process is as follows: Encoding the observed motion sequence spectrum by an observation encoder to obtain an encoded observed motion sequence spectrum; The denoising time step is encoded by a temporal encoder and processed by a single-layer perceptron to obtain the processed denoising time step; The input conditions of the stacked motion residual learning network are obtained according to the encoded observed motion sequence spectrum and the processed denoised time steps.

[0011] As an implementation method, the input conditions of the stacked motion residual learning network and the pure noise sequence spectrum are input into the improved stacked motion residual learning network. The specific process is as follows: According to the predicted noise and the pure noise sequence spectrum, sampling is performed to generate a denoised motion sequence spectrum, and a noisy observed motion sequence spectrum is calculated; According to the current denoised motion sequence spectrum and the noisy observed motion sequence spectrum, the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism; Input the motion sequence spectrum of the current denoising time step into the first linear mapping layer for linear mapping to obtain the mapped motion sequence spectrum of the current denoising time step; Inputting the mapped motion sequence spectrum of the current denoising time step and the input conditions of the stacked motion residual learning network into the first spatiotemporal cross-fusion graph convolutional network to obtain a first forward prediction result and a first backward prediction result of the first spatiotemporal cross-fusion graph convolutional network; After subtracting the mapped motion sequence spectrum of the current denoised time step from the current backward prediction result, the spectrum is iteratively input into the remaining spatiotemporal cross-fusion graph convolutional network with the input conditions of the stacked motion residual learning network until all backward and forward prediction results are obtained. Each forward prediction result is added together to obtain the final noise feature; The final noise features are linearly mapped through the second linear mapping layer to obtain the predicted noise.

[0012] As an implementation method, the spatiotemporal cross-fusion graph convolutional network includes two parallel branch networks, and the left branch network includes a cross attention module and a left double-transposed graph convolutional network; The right branch network includes a right doubly transposed graph convolutional network.

[0013] As an implementation method, the mapped motion sequence spectrum of the current denoising time step and the input conditions of the stacked motion residual learning network are input into the first spatiotemporal cross-fusion graph convolutional network. The specific process is as follows: In the first spatiotemporal cross-fusion graph convolutional network, the input conditions of the stacked motion residual learning network and the motion sequence spectrum of the current denoising time step after the mapping are input into the first left branch network, and encoding and feature extraction are performed through the cross attention module and the first left double transposed graph convolutional network to obtain the first left branch noise estimate; The motion sequence spectrum of the mapped denoised time step is input into the first right branch network, and encoded and feature extracted by the first right double transposed graph convolutional network to obtain the first right branch noise estimation; The first left branch noise estimate and the first right branch noise estimate are concatenated and passed through a linear mapping layer to obtain first backward and first forward prediction results.

[0014] As an implementation method, encoding and feature extraction are performed through a cross-attention module and a first left double-transposed graph convolutional network, wherein the double-transposed graph convolutional network includes two consecutive graph convolutions and transpositions and a linear mapping layer. The specific process is as follows: Inputting the input conditions of the stacked motion residual learning network and the motion sequence spectrum of the mapped denoised time step into the cross attention module to obtain enhanced data; The enhanced data is input into the first graph convolution layer to extract the observed spectrum features and the noise spectrum features; Transposing the observed spectrum characteristics and the noise spectrum characteristics to obtain transposed observed spectrum characteristics and noise spectrum characteristics; The transposed noise spectrum features are extracted through a linear mapping layer to obtain the first left branch noise estimation.

[0015] As an implementation method, the motion sequence spectrum of the denoising time step is calculated as follows: ; in, , n represents the length of the observation sequence, m represents the length of the prediction sequence, is the transpose operation, ⊙ represents the Hadamard product, represents the denoised motion sequence spectrum, represents the spectrum of the noisy observed motion sequence, represents the motion sequence spectrum at the denoised time step, is the discrete cosine transform Inverse discrete cosine transform.

[0016] A second aspect of the present invention provides a method for predicting human motion based on spatiotemporal modeling of a diffusion model, comprising: A data acquisition module is used to obtain the observed motion sequence spectrum and the pure noise sequence spectrum; A noise prediction network model module is used to input the observed motion sequence spectrum and the pure noise sequence spectrum into a trained noise prediction network model to obtain predicted noise; The noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network; the improved stacked motion residual learning network includes a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks and a second linear mapping layer connected in sequence; the motion sequence spectrum is input into the first linear mapping layer for linear mapping; the output of the first linear mapping layer is sent to the cascaded spatiotemporal cross-fusion graph convolutional network to obtain the forward prediction result and backward prediction result of each spatiotemporal cross-fusion graph convolutional network; each forward prediction result is added to obtain the final noise feature; the noise feature passes through the second linear mapping layer to obtain the predicted noise; A motion prediction module is configured to generate a current denoised motion sequence spectrum by sampling based on the predicted noise and a standard normal distribution, calculate the spectrum of the observed motion sequence with noise, and obtain the motion sequence spectrum of the next denoised time step through a masking mechanism, and repeat iterative denoising until a new complete motion sequence spectrum is obtained; Performing inverse discrete cosine transform on the new complete motion sequence frequency spectrum to obtain a predicted motion sequence.

[0017] One or more of the above technical solutions have the following beneficial effects: In this embodiment, an efficient noise prediction network (NPN) is constructed to extract effective spatiotemporal information from the input motion sequence. This information is then integrated into the NPN through a reasonable fusion mechanism, thereby achieving accurate noise prediction and improving the accuracy of human motion prediction.

[0018] In this embodiment, the improved stacked motion residual learning network model can not only more effectively integrate spatiotemporal constraint information, but also significantly improve the accuracy and robustness of prediction through the residual learning mechanism.

[0019] In this embodiment, a spatiotemporal cross-fusion graph convolutional network (STCF-GCN module) is embedded in an improved stacked motion residual learning framework (SMRLearning), enabling efficient integration of spatiotemporal constraint information into the diffusion model, fully capturing the close correlation between spatiotemporal interactions, and improving prediction accuracy. Using a layer-by-layer residual correction method to gradually optimize the prediction results, the improved stacked motion residual learning framework (SMRLearning) consists of multiple layers of stacked submodules, each of which generates both forward and backward predictions. The final prediction result is the integration of the forward predictions of all submodules, while the backward prediction is used to represent the information that has been effectively utilized in the input. The input of each submodule is the difference between the current input and its backward prediction, thereby capturing residual motion data that has not yet been modeled. In this way, each submodule focuses on processing residual information not modeled by the previous module, gradually optimizing the prediction results, and generating motion predictions that are more faithful and accurate than existing methods.

[0020] In this embodiment, a human motion prediction method based on spatiotemporal modeling of a diffusion model improves prediction accuracy, specifically by 4% improvement in average displacement error and final displacement error compared to existing technologies. Furthermore, the structure is more compact and memory usage is lower. The design of the spatiotemporal cross-fusion graph convolutional network (STCF-GCN module) not only maintains modeling capabilities but also takes into account computational efficiency and effectively reduces computational complexity by 36% compared to existing technologies.

[0021] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0023] Figure 1 This is a flow chart of a method for predicting human motion based on spatiotemporal modeling of a diffusion model according to the first embodiment of the present invention; Figure 2 This is a training diagram of a noise prediction network model according to the first embodiment; Figure 3 This is an improved stacked motion residual learning network model of the first embodiment; Figure 4 This is the spatiotemporal cross-fusion graph convolutional network model of the first embodiment; Figure 5 This is the double-transposed graph convolutional network model of the first embodiment. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0025] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0026] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0027] Example 1 This embodiment discloses a method for predicting human motion based on spatiotemporal modeling of a diffusion model.

[0028] To more clearly illustrate this embodiment, a process for implementing human motion prediction based on spatiotemporal modeling of a diffusion model can be specifically described as follows: A method for predicting human motion based on spatiotemporal modeling of a diffusion model, comprising: S1. Obtain the spectrum of the observed motion sequence and the spectrum of the pure noise sequence; S2, inputting the observed motion sequence spectrum and the pure noise sequence spectrum into the trained noise prediction network model to obtain predicted noise; Among them, the noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network; the improved stacked motion residual learning network includes a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks and a second linear mapping layer connected in sequence; the motion sequence spectrum is input into the first linear mapping layer for linear mapping; the output of the first linear mapping layer is sent to the cascaded spatiotemporal cross-fusion graph convolutional network to obtain the forward prediction result and backward prediction result of each spatiotemporal cross-fusion graph convolutional network; each forward prediction result is added to obtain the final noise feature; the noise feature passes through the second linear mapping layer to obtain the predicted noise; S3. Based on the predicted noise and standard normal distribution, the current denoised motion sequence spectrum is sampled and generated. The spectrum of the observed motion sequence with noise is calculated. The spectrum of the motion sequence at the next denoised time step is obtained through a masking mechanism. The denoising is repeated iteratively until a new complete motion sequence spectrum is obtained. The new complete motion sequence spectrum is subjected to inverse discrete cosine transform to obtain the predicted motion sequence.

[0029] like Figure 1 As shown, in step S1, the observed motion sequence spectrum and the pure noise sequence spectrum are obtained.

[0030] S1-1. Obtain an observed motion sequence.

[0031] In this embodiment, an Intel Real Sense D435 camera is used to synchronously capture 0.5 second (25 frames) of human motion RGB images and depth images at a frequency of 50 Hz.

[0032] For the collected RGB images, the MediaPipe algorithm is used to extract the human joint skeleton and obtain the two-dimensional joint coordinates.

[0033] Then, the two-dimensional joint coordinates in the RGB image are mapped to the corresponding depth image to obtain the depth value of each joint point.

[0034] Based on the camera intrinsic parameters, the two-dimensional joint coordinates are converted into three-dimensional space coordinates to obtain the three-dimensional position of the joint point.

[0035] Finally, the three-dimensional joint coordinates of each frame are arranged in time sequence to generate the observed motion sequence .

[0036] S1-2, sample the spectrum of pure noise sequence.

[0037] In this embodiment, the initial value is sampled from the standard normal distribution, that is, the pure noise sequence spectrum. .

[0038] S1-3. Before performing discrete cosine transform on the observed motion sequence, a padding process is first performed to obtain a padded observed motion sequence.

[0039] In this embodiment, in order to observe the motion sequence To predict human motion prediction, fill the observed motion sequence to obtain the filled observed motion sequence , the formula is: (1) in, represents the observed motion sequence after padding, n represents the length of the observed sequence, and m represents the length of the predicted sequence.

[0040] S1-3. Perform discrete cosine transform on the observed motion sequence to obtain a spectrum of the observed motion sequence.

[0041] Discrete cosine transform (DCT) is used to encode temporal information in human motion prediction (HMP).

[0042] Given Joint coordinates Step Sequence , its DCT coefficient The calculation formula is: (2) in, .

[0043] At the same time, given a set of DCT coefficients , its original sequence can be reconstructed by inverse discrete cosine transform (IDCT) : (3) In this embodiment, for the observed motion sequence after filling Perform DCT encoding and extract the low-frequency components, and the observed motion sequence spectrum is obtained , the formula is: (4) in, Represents the spectrum of the observed motion sequence.

[0044] After the above steps, the discrete cosine transform (DCT) is used to convert the time domain information into frequency domain information by representing the temporal changes of the joints as linear combinations of the DCT basis, effectively capturing the motion patterns of individual joints. Furthermore, by discarding high-frequency information, a more compact representation is provided, effectively capturing the inherent smoothness of human motion.

[0045] like Figure 1 As shown, in step S2, the observed motion sequence spectrum and the pure noise sequence spectrum are input into the trained noise prediction network model to obtain the predicted noise.

[0046] S2-1. Construct a noise prediction network model.

[0047] The key to the diffusion model of human body motion prediction is the accuracy of noise prediction. In this embodiment, an efficient noise prediction model is constructed.

[0048] For the noise prediction network (NPN), an improved stacked motion residual learning (Stacked Motion Residual Network, SMRNet) is designed. This network gradually optimizes the motion prediction results through the residual learning mechanism, significantly improving the accuracy and robustness of the prediction.

[0049] In this embodiment, a noise prediction network model is designed, which includes a conditional encoder and an improved stacked motion residual learning network. The improved stacked motion residual learning network includes N spatiotemporal cross-fusion graph convolutional networks (STCF-GCN modules) and two linear mapping layers.

[0050] The conditional encoder includes an observation encoder and a time encoder.

[0051] In order to integrate the temporal and spatial information in motion, a new spatiotemporal cross-fusion graph convolutional network (STCF-GCN module) is designed. The network is designed using a double transposed graph convolutional network (DTGCN). The module uses a motion spectrum perturbed by noise after linear mapping. The diffusion condition generated by the conditional encoder is Input, output noise characteristics.

[0052] Each spatiotemporal cross-fusion graph convolutional network consists of two parallel branch networks.

[0053] The left branch network includes a crisscross attention module and a double-transposed graph convolutional network.

[0054] The right branch network consists of a double-transposed graph convolutional network.

[0055] Each dual transposed graph convolutional network (DTGCN) consists of two graph convolutional layers (GCN), two transposition operations (Trans), and a linear mapping layer.

[0056] In this way, the model can fully utilize information from different sources and leverage the advantages of the Dual Transposed Graph Convolutional Network (DTGCN) in terms of spatiotemporal constraints, thereby achieving more comprehensive and accurate noise estimation. The core of this method lies in the left branch's deep exploration of motion patterns in historical data through a motion attention mechanism, while the right branch focuses on capturing the characteristics of current noise. The two complement each other and ultimately achieve accurate noise prediction.

[0057] like Figure 2 As shown, S2-2, the noise prediction network model is trained.

[0058] 1) Obtain the original complete motion sequence to be trained and the observed motion sequence to be trained.

[0059] In this embodiment, an original complete motion sequence is obtained from two commonly used data sets for human body prediction (Human3.6M and HumanEva-I), and an observed motion sequence is extracted from the original complete motion sequence.

[0060] Get a 3D skeleton input sequence containing N frames, that is, the original complete motion sequence, recorded as .

[0061] From the original complete motion sequence, obtain the observed motion sequence, denoted as ,in Indicates the Frame of human motion, it will The 3D coordinates of the joints are arranged vertically into a vector, and . Predict the next frame( ),Right now ,in This requires a deep understanding of the input sequence to generate a motion sequence that is seamless and faithful to the input.

[0062] 2) Fill the observed motion sequence to be trained according to formula (1) to obtain the filled observed motion sequence to be trained.

[0063] 3) Performing discrete cosine transform on the padded observed motion sequence to be trained to obtain the spectrum of the observed motion sequence to be trained and the spectrum of the original complete motion sequence to be trained.

[0064] In this embodiment, for the observed motion sequence after filling and the original complete motion sequence , perform DCT encoding on both of them, and extract the former low-frequency components, and the observed motion sequence spectrum is obtained and the original complete motion sequence spectrum , the formula is: (5) in, represents the spectrum of the observed motion sequence to be trained, Represents the original complete motion sequence spectrum to be trained.

[0065] 4) Add noise to the original complete motion sequence spectrum to be trained to obtain the noisy original motion sequence spectrum.

[0066] In this embodiment, the diffusion process is to the original complete sequence spectrum to be trained. Adding Noise , get the noisy motion spectrum to be trained , the formula is: (6) in, , is the total number of diffusion steps, , , , is a predefined variance scheduler.

[0067] 5) Input the noisy motion spectrum to be trained into the noise prediction network model.

[0068] A noise prediction network model (NPN) is trained using the Human3.6M and HumanEva-I datasets.

[0069] By training a noise prediction network (NPN) To predict noise , whose goal is to minimize the loss function, the formula is: (7) in, Indicates loss in noise and the expected value at time step t, averaging the prediction loss of the training data, The smaller, The closer , the optimization effect is the best, and the noise prediction network model predicts the noise more accurately.

[0070] Finally, we can obtain a well-trained and accurate noise prediction network.

[0071] Experiments were conducted on commonly used human motion prediction datasets (Human3.6M and HumanEva-I). The results demonstrate that the proposed method can produce more accurate and faithful human motion prediction results. Compared with existing methods, this method improves both the mean displacement error and the final displacement error by approximately 4%, while also achieving a smaller model size, lower computational complexity, and faster inference speed.

[0072] like Figure 3 As shown in the figure, a unique spatiotemporal cross-fusion graph convolutional network (STCF-GCN module) is designed within the stacked motion residual learning network to efficiently model the spatiotemporal constraints of human motion. This STCF-GCN module employs a dual-branch structure, each containing a dual-transposed graph convolutional network (DTGCN) module. This module simultaneously captures joint topology in the spatial dimension and motion evolution in the temporal dimension, thereby comprehensively modeling spatiotemporal constraints.

[0073] S2-3. Input the observed motion sequence spectrum and the pure noise sequence spectrum into the trained noise prediction network model.

[0074] In this embodiment, the motion sequence spectrum is observed and pure noise sequence spectrum are input into the trained noise prediction network model.

[0075] The observed motion sequence spectrum and the pure noise sequence spectrum are input into the trained noise prediction network model, where the trained noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network. The specific process is as follows: like Figure 3 As shown, S2-3-1, the observed motion sequence spectrum is input into the conditional encoder to obtain the input condition of the stacked motion residual learning network.

[0076] The observed motion sequence spectrum is input into the conditional encoder, where the conditional encoder includes an observation encoder and a time encoder. The specific process is: (1) The observed motion sequence spectrum is encoded by the observation encoder to obtain the encoded observed motion sequence spectrum.

[0077] In this embodiment, the observation encoder is used to observe the spectrum The encoded spectrum of the observed motion sequence is obtained. First, the observed spectrum o is mapped to high-dimensional features, which are then fed into the self-attention module. The self-attention mechanism dynamically calculates the correlation between the spectra to adjust the weight information and extract more effective features. Finally, the spectrum is fed into the Dual Transposed Graph Convolutional Network (DTGCN) for spatiotemporal feature fusion.

[0078] (2) The denoising time step is encoded by a time encoder and processed by a single-layer perceptron to obtain the processed denoising time step.

[0079] In this embodiment, the denoising time step t is encoded by a temporal encoder, which is first encoded by a sinusoidal position encoding and processed by a single-layer perceptron.

[0080] (3) Based on the encoded observed motion sequence spectrum and the processed denoised time step, the input conditions of the stacked motion residual learning network are obtained.

[0081] The processed denoised time step is fused with the encoded observed motion sequence spectrum, and the overall formula is: (8) in, and is a single hidden layer perceptron, is a sinusoidal position code, Represents the input condition of the stacked motion residual learning network, i.e., the diffusion condition, which is input into each spatiotemporal cross-fusion graph convolutional network (STCF-GCN module) in the improved stacked motion residual learning framework (SMRLearning).

[0082] like Figure 3 As shown, S2-3-2, the input conditions of the stacked motion residual learning network and the pure noise sequence spectrum are input into the improved stacked motion residual learning network to obtain the predicted noise.

[0083] The input conditions of the stacked motion residual learning network and the pure noise sequence spectrum are input to the improved stacked motion residual learning network, where the stacked motion residual learning network includes two linear mapping layers and several spatiotemporal cross-fusion graph convolutional networks. The specific process is as follows: (1) Based on the predicted noise and pure noise sequence spectra, the denoised motion sequence spectrum is sampled and generated, and the noisy observed motion sequence spectrum is calculated.

[0084] Specifically, the initial value is sampled based on the standard normal distribution, that is, the spectrum of the pure noise sequence , iteratively generate denoised motion sequence spectrum , the formula is: (9) in, ,and .

[0085] For each step , by observing the motion sequence spectrum Adding Noise Calculate the spectrum of the noisy observed motion sequence using the formula: (10) in, represents the spectrum of the noisy observed motion sequence, .

[0086] (2) Based on the denoised motion sequence spectrum and the noisy observed motion sequence spectrum, the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism.

[0087] In this embodiment, according to the denoised motion sequence spectrum and the spectrum of the noisy observed motion sequence , the final motion sequence spectrum of the next de-voiced time step is obtained through the mask mechanism , the formula is: (11) in, , n represents the length of the observation sequence, m represents the length of the prediction sequence, is the transpose operation, ⊙ represents the Hadamard product, represents the denoised motion sequence spectrum, represents the spectrum of the noisy observed motion sequence, represents the motion sequence spectrum at the denoised time step, is the discrete cosine transform Inverse discrete cosine transform.

[0088] (3) The motion sequence spectrum of the denoised time step is input into the first linear mapping layer for linear mapping to obtain the motion sequence spectrum of the denoised time step after mapping.

[0089] like Figure 4As shown in Figure 4, (4) the motion sequence spectrum of the mapped denoised time step and the input conditions of the stacked motion residual learning network are input into each spatiotemporal cross-fusion graph convolutional network to obtain the forward prediction results and backward prediction results of each spatiotemporal cross-fusion graph convolutional network.

[0090] The motion sequence spectrum of the mapped denoised time step and input conditions Input into the first spatiotemporal cross-fusion graph convolutional network to obtain the first forward prediction result and the first backward prediction result .

[0091] In this embodiment, the motion sequence spectrum and input conditions of the mapped denoised time step are input into each spatiotemporal cross-fusion graph convolutional network.

[0092] First, the input condition and the motion sequence spectrum of the mapped denoised time step are input into the first spatiotemporal cross-fusion graph convolutional network to obtain the first forward prediction result and the first backward prediction result.

[0093] The left and right branches process information from different sources respectively to achieve comprehensive and accurate noise estimation. In the left branch, a cross-attention module is introduced, whose purpose is to extract valuable information from historical motion data to enhance the understanding of the underlying motion pattern. This goal is achieved by using movement information of

[0094] The specific process is: 1) The input condition and the mapped denoised time step motion sequence spectrum are input to the first left branch network, and encoded and feature extracted through the cross attention module and the first left double transposed graph convolutional network to obtain the first left branch noise estimate.

[0095] like Figure 5 As shown, in this embodiment, encoding and feature extraction are performed through a cross attention module and a first left double transposed graph convolutional network, wherein the double transposed graph convolutional network includes two consecutive graph convolutions and transpositions and a linear mapping layer. The specific process is as follows: The input condition and the motion sequence spectrum of the mapped denoised time step are input into the cross-attention module to obtain the enhanced data.

[0096] The enhanced data is input into the first graph convolution layer to extract the observed spectrum features and the noise spectrum features.

[0097] In this embodiment, the graph convolutional network aggregates information between nodes through an adjacency matrix, enabling efficient feature learning and graph embedding. This property makes it particularly suitable for representing human joints as a graph structure to capture the spatial relationships between joints. However, in actual human motion prediction tasks, in addition to the dependencies between joints, it is also necessary to fully consider the temporal constraints during the movement of each joint.

[0098] A dual graph-based modeling strategy is employed. First, human joints are represented as a graph structure, and then each joint is modeled as a separate graph structure, capturing the temporal dynamics and sequential constraints of its motion pattern. The first layer of graph convolution primarily encodes joint information to characterize the spatial dependencies between joints. The second layer of graph convolution focuses on extracting frequency-domain features to effectively characterize dynamic changes during motion.

[0099] The forward propagation process is expressed as:

[0100] (12) in, Represents the input motion spectrum features, including observation spectrum features and noise spectrum features; and Representing the temporal adjacency matrix and the joint adjacency matrix respectively, the trainable adjacency matrix is ​​used to enable the network to adapt to different graph structures more flexibly, thereby better capturing the spatiotemporal relationship; is a trainable weight parameter; Represents a matrix transpose operation.

[0101] Transpose the observed spectrum characteristics and the noise spectrum characteristics to obtain the transposed observed spectrum characteristics and the noise spectrum characteristics .

[0102] The transposed noise spectrum features are extracted through a linear mapping layer to obtain the first left branch noise estimation.

[0103] By applying different adjacency matrices to the two layers of GCN, the module is able to simultaneously consider the correlation between temporal features and joint features, thereby better capturing the complex relationships in spatiotemporal sequences.

[0104] A linear output layer is used to extract the deep features in the motion sequence. The formula is: (13) The partial motion similarity between the denoised result and the current observation sequence is evaluated through the cross-attention module to generate noise estimates. These noise estimates are then input into the dual transposed graph convolutional network (DTGCN) for spatiotemporal modeling, and the final output is recorded as the left branch noise estimate , the formula is: (14) in, The query is used, the diffusion condition c serves as the key and value, and Linear represents the corresponding linear projection layer. The scaling factor d is used to maintain numerical stability during training. Cross-attention is used to measure the local motion similarity between the denoising result of the previous step and the current observation sequence, while the dual-transposed graph convolutional network (DTGCN) is responsible for enforcing spatiotemporal constraints to ensure data consistency.

[0105] Among them, in the first spatiotemporal cross-fusion graph convolutional network, when the input is When , according to formula (14), is the noise estimate of the first left branch.

[0106] 2) The motion sequence spectrum of the mapped denoised time step is input into the first right branch network, and encoded and feature extracted by the first right double transposed graph convolutional network to obtain the first right branch noise estimation.

[0107] The right branch focuses on the noise characteristics of the current moment and directly processes the noisy motion spectrum data By encoding and extracting features from these data, a noise feature estimate is generated, which is recorded as the right branch noise estimate , the formula is: (15) Among them, in the first spatiotemporal cross-fusion graph convolutional network, when the input is When , according to formula (15), is the first right branch noise estimate.

[0108] 3) The left branch noise estimate and the right branch noise estimate are concatenated and passed through the linear mapping layer to obtain the backward and forward prediction results respectively.

[0109] In this embodiment, the results of the left and right branches are and are combined to generate the final output of the spatiotemporal cross-fusion graph convolutional network (STCF-GCN module), as follows:

[0110] (16) in, and Representing the The linear projection layer of the module is used to generate backward and forward prediction results. represents the backward prediction result, represents the forward prediction result, represents the left branch noise estimate, represents the right branch noise estimate.

[0111] Among them, in the first spatiotemporal cross-fusion graph convolutional network, is the first backward prediction result, is the first forward prediction result.

[0112] Then, the motion sequence spectrum of the mapped denoised time step is subtracted from the first backward prediction result, as shown in formula (17), and input into the second spatiotemporal cross-fusion graph convolutional network with the input condition of the stacked motion residual learning network to obtain the second forward prediction result. and the second backward prediction result .

[0113] (17) in, Represents the input of the current layer of spatiotemporal cross-fusion graph convolutional network, Represents the input of the next layer of spatiotemporal cross-fusion graph convolutional network.

[0114] Among them, when the second spatiotemporal cross-fusion graph convolutional network is input, according to formula (17), the input of the second spatiotemporal cross-fusion graph convolutional network is , is the second backward prediction result, is the second forward prediction result.

[0115] Finally, the input is iteratively fed into each spatiotemporal cross-fusion graph convolutional network until all backward and forward prediction results are obtained.

[0116] According to formula (17), the input of each spatiotemporal cross-fusion graph convolutional network is obtained in turn, and the input is cyclically input to N spatiotemporal cross-fusion graph convolutional networks in turn until N forward prediction results and N backward prediction results are obtained. According to formula (16), all forward prediction results and backward prediction results are obtained.

[0117] Among them, N is the number of layers of the spatiotemporal cross fusion graph convolutional network (STCF-GCN module), which is adjusted according to the size of the dataset. For example, the Human3.6M dataset has 8 layers and the Humaneva-I has 4 layers.

[0118] (5) Each forward prediction result is added together to obtain the final noise feature.

[0119] In this embodiment, each corresponding noise feature is added together to obtain the final noise feature z, which is expressed as follows: (18) in, Represents the forward prediction result, and N represents the number of layers of the spatiotemporal cross fusion graph convolutional network (STCF-GCN module).

[0120] (6) The final noise features are linearly mapped through the second linear mapping layer to obtain the predicted noise.

[0121] By the noise characteristics Perform linear mapping and finally get the predicted noise, the formula is: (19) in, represents the prediction noise.

[0122] Following the above steps, an improved stacked motion residual learning framework optimizes prediction results layer by layer, with specialized learning performed on the residuals at each layer, achieving highly accurate human motion prediction. Specifically, the improved stacked motion residual learning (SMRNet) employs a multi-layered architecture, with each layer focused on capturing motion features at a different level. Through layer-by-layer optimization, the model effectively reduces prediction error while retaining important motion information. This layered residual learning approach not only improves prediction accuracy but also enhances the model's adaptability to complex human motion patterns.

[0123] like Figure 1 As shown, in step S3, according to the predicted noise, based on the standard normal distribution, the current denoised motion sequence spectrum is sampled and generated, the noisy observed motion sequence spectrum is calculated, and the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism, and the iterative denoising is repeated until a new complete motion sequence spectrum is obtained; The new complete motion sequence spectrum is subjected to inverse discrete cosine transform to obtain the predicted motion sequence.

[0124] (1) According to the predicted noise, based on the standard normal distribution, the current denoised motion sequence spectrum is sampled and the noisy observed motion sequence spectrum is calculated.

[0125] (2) Based on the current denoised motion sequence spectrum and the noisy observed motion sequence spectrum, the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism.

[0126] Repeat steps (1) and (2) for iterative denoising until a new complete motion sequence spectrum is obtained.

[0127] Iterate the time step T according to formula (9), (10), (11) until the new complete motion sequence spectrum is obtained .

[0128] The above steps help preserve the original observation sequence and ensure smooth continuity between the last observation frame and the first prediction frame.

[0129] like Figure 1 As shown, the new complete motion sequence spectrum is subjected to inverse discrete cosine transform to obtain the predicted motion sequence.

[0130] The predicted motion sequence is: = (20) in, Represents the new complete motion sequence spectrum, that is, the predicted motion spectrum. The predicted motion sequence is the inverse discrete cosine transform The last m frames of .

[0131] After the above steps, the future m-frame motion sequence can be predicted based on the observed motion sequence.

[0132] Example 2 The purpose of this embodiment is to provide a human motion prediction system based on spatiotemporal modeling of a diffusion model, including: A data acquisition module is used to obtain the observed motion sequence spectrum and the pure noise sequence spectrum; A noise prediction network model module is used to input the observed motion sequence spectrum and the pure noise sequence spectrum into a trained noise prediction network model to obtain predicted noise; The noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network; the improved stacked motion residual learning network includes a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks and a second linear mapping layer connected in sequence; the motion sequence spectrum is input into the first linear mapping layer for linear mapping; the output of the first linear mapping layer is sent to the cascaded spatiotemporal cross-fusion graph convolutional network to obtain the forward prediction result and backward prediction result of each spatiotemporal cross-fusion graph convolutional network; each forward prediction result is added to obtain the final noise feature; the noise feature passes through the second linear mapping layer to obtain the predicted noise; A motion prediction module is configured to generate a current denoised motion sequence spectrum by sampling based on the predicted noise and a standard normal distribution, calculate the spectrum of the observed motion sequence with noise, and obtain the motion sequence spectrum of the next denoised time step through a masking mechanism, and repeat iterative denoising until a new complete motion sequence spectrum is obtained; Performing inverse discrete cosine transform on the new complete motion sequence frequency spectrum to obtain a predicted motion sequence.

[0133] Based on providing a human motion prediction system based on spatiotemporal modeling of a diffusion model, the method steps in embodiment 1 are implemented. The steps involved in the apparatus of the above embodiment correspond to those of the method embodiment 1. For the specific implementation, please refer to the relevant description part of embodiment 1.

[0134] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0135] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A method for predicting human motion based on spatiotemporal modeling of a diffusion model, characterized in that: include: Obtain the spectrum of the observed motion sequence and the spectrum of the pure noise sequence; Inputting the observed motion sequence spectrum and the pure noise sequence spectrum into a trained noise prediction network model to obtain predicted noise; According to the predicted noise, based on the standard normal distribution, the current denoised motion sequence spectrum is sampled and generated, the noisy observed motion sequence spectrum is calculated, and the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism, and the iterative denoising is repeated until a new complete motion sequence spectrum is obtained; Performing inverse discrete cosine transform on the new complete motion sequence spectrum to obtain a predicted motion sequence; The noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network; The improved stacked motion residual learning network includes a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks and a second linear mapping layer connected in sequence; the motion sequence spectrum is input into the first linear mapping layer for linear mapping; the output of the first linear mapping layer is sent to the cascaded spatiotemporal cross-fusion graph convolutional network to obtain the forward prediction result and backward prediction result of each spatiotemporal cross-fusion graph convolutional network; each forward prediction result is added to obtain the final noise feature; the noise feature passes through the second linear mapping layer to obtain the predicted noise.

2. The method for predicting human motion based on spatiotemporal modeling of a diffusion model according to claim 1, characterized in that: Obtain the spectrum of the observed motion sequence. The specific process is: performing padding processing on the observed motion sequence to obtain a padded observed motion sequence; The padded observed motion sequence is subjected to discrete cosine transform to obtain the observed motion sequence spectrum.

3. The method for predicting human motion based on spatiotemporal modeling of a diffusion model according to claim 1, characterized in that: The observed motion sequence spectrum and the pure noise sequence spectrum are input into the trained noise prediction network model. The specific process is as follows: Inputting the observed motion sequence spectrum into a conditional encoder to obtain input conditions of a stacked motion residual learning network; The input conditions of the stacked motion residual learning network and the pure noise sequence spectrum are input into the improved stacked motion residual learning network to obtain predicted noise.

4. The method for predicting human motion based on spatiotemporal modeling of a diffusion model as claimed in claim 3, characterized in that: The observed motion sequence spectrum is input into a conditional encoder, wherein the conditional encoder includes an observation encoder and a time encoder. The specific process is as follows: Encoding the observed motion sequence spectrum by an observation encoder to obtain an encoded observed motion sequence spectrum; The denoising time step is encoded by a temporal encoder and processed by a single-layer perceptron to obtain the processed denoising time step; The input conditions of the stacked motion residual learning network are obtained according to the encoded observed motion sequence spectrum and the processed denoised time steps.

5. The method for predicting human motion based on spatiotemporal modeling of a diffusion model as claimed in claim 3, characterized in that: The input conditions of the stacked motion residual learning network and the pure noise sequence spectrum are input into the improved stacked motion residual learning network. The specific process is as follows: According to the predicted noise and the pure noise sequence spectrum, sampling is performed to generate a denoised motion sequence spectrum, and a noisy observed motion sequence spectrum is calculated; According to the current denoised motion sequence spectrum and the noisy observed motion sequence spectrum, the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism; Input the motion sequence spectrum of the current denoising time step into the first linear mapping layer for linear mapping to obtain the mapped motion sequence spectrum of the current denoising time step; Inputting the mapped motion sequence spectrum of the current denoising time step and the input conditions of the stacked motion residual learning network into the first spatiotemporal cross-fusion graph convolutional network to obtain a first forward prediction result and a first backward prediction result of the first spatiotemporal cross-fusion graph convolutional network; After subtracting the mapped motion sequence spectrum of the current denoised time step from the current backward prediction result, the spectrum is iteratively input into the remaining spatiotemporal cross-fusion graph convolutional network with the input conditions of the stacked motion residual learning network until all backward and forward prediction results are obtained. Each forward prediction result is added together to obtain the final noise feature; The final noise features are linearly mapped through the second linear mapping layer to obtain the predicted noise.

6. The method for predicting human motion based on spatiotemporal modeling of a diffusion model as claimed in claim 5, characterized in that: The spatiotemporal cross-fusion graph convolutional network consists of two parallel branch networks. The left branch network includes a cross-attention module and a left double-transposed graph convolutional network. The right branch network includes a right doubly transposed graph convolutional network.

7. A method for predicting human motion based on spatiotemporal modeling of a diffusion model according to any one of claims 5 to 6, characterized in that: The mapped motion sequence spectrum of the current denoising time step and the input conditions of the stacked motion residual learning network are input into the first spatiotemporal cross-fusion graph convolutional network. The specific process is as follows: In the first spatiotemporal cross-fusion graph convolutional network, the input conditions of the stacked motion residual learning network and the motion sequence spectrum of the current denoising time step after the mapping are input into the first left branch network, and encoding and feature extraction are performed through the cross attention module and the first left double transposed graph convolutional network to obtain the first left branch noise estimate; The motion sequence spectrum of the mapped denoised time step is input into the first right branch network, and encoded and feature extracted by the first right double transposed graph convolutional network to obtain the first right branch noise estimation; The first left branch noise estimate and the first right branch noise estimate are concatenated and passed through a linear mapping layer to obtain first backward and first forward prediction results.

8. The method for predicting human motion based on spatiotemporal modeling of a diffusion model as claimed in claim 7, characterized in that: Encoding and feature extraction are performed through the cross-attention module and the first left double-transposed graph convolutional network. The double-transposed graph convolutional network includes two consecutive graph convolution and transposition layers and a linear mapping layer. The specific process is as follows: Inputting the input conditions of the stacked motion residual learning network and the motion sequence spectrum of the mapped denoised time step into the cross attention module to obtain enhanced data; The enhanced data is input into the first graph convolution layer to extract the observed spectrum features and the noise spectrum features; Transposing the observed spectrum characteristics and the noise spectrum characteristics to obtain transposed observed spectrum characteristics and noise spectrum characteristics; The transposed noise spectrum features are extracted through a linear mapping layer to obtain the first left branch noise estimation.

9. The method for predicting human motion based on spatiotemporal modeling of a diffusion model according to claim 1, wherein: The motion sequence spectrum of the denoising time step is calculated as follows: ； in, , n represents the length of the observation sequence, m represents the length of the prediction sequence, is the mask vector, ⊙ represents the Hadamard product, represents the denoised motion sequence spectrum, represents the spectrum of the noisy observed motion sequence, represents the motion sequence spectrum at the denoised time step, is the discrete cosine transform, Inverse discrete cosine transform.

10. A human motion prediction system based on spatiotemporal modeling of diffusion model, characterized in that: include: A data acquisition module is used to obtain the observed motion sequence spectrum and the pure noise sequence spectrum; A noise prediction network model module is used to input the observed motion sequence spectrum and the pure noise sequence spectrum into a trained noise prediction network model to obtain predicted noise; The noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network; the improved stacked motion residual learning network includes a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks and a second linear mapping layer connected in sequence; the motion sequence spectrum is input into the first linear mapping layer for linear mapping; the output of the first linear mapping layer is sent to the cascaded spatiotemporal cross-fusion graph convolutional network to obtain the forward prediction result and backward prediction result of each spatiotemporal cross-fusion graph convolutional network; each forward prediction result is added to obtain the final noise feature; the noise feature passes through the second linear mapping layer to obtain the predicted noise; A motion prediction module is configured to generate a current denoised motion sequence spectrum by sampling based on the predicted noise and a standard normal distribution, calculate the spectrum of the observed motion sequence with noise, and obtain the motion sequence spectrum of the next denoised time step through a masking mechanism, and repeat iterative denoising until a new complete motion sequence spectrum is obtained; Performing inverse discrete cosine transform on the new complete motion sequence spectrum to obtain a predicted motion sequence.

Images