A human motion prediction method and system based on diffusion model space-time modeling

Abstract

The application provides a human motion prediction method and system based on a diffusion model space-time modeling, relates to the technical field of human motion analysis, and aims to solve the problem that noise prediction technology in existing human motion prediction cannot fully capture the close correlation of space-time interaction, only relies on a single prediction result of a model to complete prediction, cannot accurately and faithfully reflect the real situation of a motion sequence, and has high computational complexity.The method combines a plurality of space-time cross fusion graph convolution networks and a stacked motion residual learning network to construct a noise prediction network model, accurately predicts human motion noise, and accurately predicts human motion according to the noise prediction result.The application solves the problem of noise prediction technology in existing human motion prediction, fully considers the spatial topological relationship between joints, can fully utilize effective information of data, simultaneously considers the calculation efficiency, meets the demand of real-time application, and further improves the accuracy of human motion prediction.

Description

Technical Field

[0001] This invention belongs to the field of human motion analysis technology, and in particular relates to a method and system for predicting human motion based on spatiotemporal modeling of diffusion model. Background Technology

[0002] Human Motion Prediction (HMP) is a technology that predicts future 3D human skeletal movements based on historical motion data. This technology has wide applications in various fields, such as virtual reality, augmented reality, human-computer interaction, and autonomous driving. Its core objective is to build models that can accurately predict subsequent movements through in-depth analysis of historical human motion data, thereby providing high-precision motion prediction support for related applications.

[0003] Currently, existing human motion prediction models typically employ diffusion models, such as those in patents CN119169205A and CN118470115A. Among these, diffusion models represent a recent advancement in generative modeling, demonstrating significant advantages in generating high-quality, multimodal, and long-sequence human motion data. By learning the distribution characteristics of human motion data and incorporating the inherent constraints of motion sequences, this model can accurately predict subsequent human motion given specific observation conditions. This characteristic makes it highly valuable for human motion prediction tasks.

[0004] However, existing human motion prediction methods based on diffusion models have shortcomings in spatiotemporal motion constraints and data integration, leading to diverse and inaccurate prediction results that fail to reflect real motion patterns. Furthermore, these methods suffer from high algorithm complexity and low efficiency. Spatial constraints refer to the relative positional relationships of joints at each time step, reflecting the geometric structure and interactions between joints; temporal constraints refer to the dependencies between different moments, reflecting the dynamic evolution of joint motion. For example, some methods use simple multilayer perceptrons for spatial modeling, which reduces computational complexity but fails to fully utilize the topological information between joints, resulting in poor learning of spatial dependencies. Other methods employ a separate modeling strategy, using a spatial Transformer for joint modeling in the initial noise reduction phase and then a temporal Transformer for temporal modeling. While this approach simplifies the complexity of spatiotemporal modeling to some extent, the independent processing of spatial and temporal modeling fails to fully capture the close relationship between spatiotemporal interactions, thus affecting prediction accuracy. Moreover, the high computational complexity of the Transformer makes it difficult to meet real-time performance requirements, limiting its deployment in practical applications.

[0005] Furthermore, existing methods rely solely on a single prediction from the model to complete the prediction task, which limits the full utilization of effective information in the input, resulting in predictions that fail to accurately and faithfully reflect the true state of the input sequence. In summary, achieving faithful human motion prediction using diffusion models presents at least two major challenges: first, spatiotemporal modeling of human motion; and second, effectively integrating spatiotemporal constraints into the diffusion model. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention provides a method and system for predicting human motion based on spatiotemporal modeling of diffusion model. This method combines multiple spatiotemporal cross-fusion graph convolutional networks with stacked motion residual learning networks to construct a noise prediction network model, thereby achieving accurate prediction of human motion noise and accurately predicting human motion based on the noise prediction results.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] The first aspect of this invention provides a method for predicting human motion based on spatiotemporal modeling of a diffusion model, comprising:

[0009] Obtain the spectrum of the observed motion sequence and the spectrum of the pure noise sequence;

[0010] The observed motion sequence spectrum and the pure noise sequence spectrum are input into the trained noise prediction network model to obtain the predicted noise;

[0011] Based on the predicted noise and a standard normal distribution, the current denoised motion sequence spectrum is sampled and generated. The noisy observed motion sequence spectrum is calculated, and the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism. The denoising process is repeated iteratively until a new complete motion sequence spectrum is obtained.

[0012] The inverse discrete cosine transform is performed on the spectrum of the new complete motion sequence to obtain the predicted motion sequence;

[0013] The noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network. The improved stacked motion residual learning network includes a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks, and a second linear mapping layer connected in sequence. The motion sequence spectrum is input to the first linear mapping layer for linear mapping. The output of the first linear mapping layer is fed into the cascaded spatiotemporal cross-fusion graph convolutional networks to obtain the forward and backward prediction results of each spatiotemporal cross-fusion graph convolutional network. The forward prediction results are added together to obtain the final noise features. The noise features are then passed through the second linear mapping layer to obtain the predicted noise.

[0014] As one implementation method, the spectrum of the observed motion sequence is obtained, and the specific process is as follows:

[0015] The observed motion sequence is filled to obtain the filled observed motion sequence;

[0016] The spectrum of the observed motion sequence is obtained by performing a discrete cosine transform on the filled-in observed motion sequence.

[0017] As one implementation method, the observed motion sequence spectrum and the pure noise sequence spectrum are input into a trained noise prediction network model. The specific process is as follows:

[0018] The observed motion sequence spectrum is input into the conditional encoder to obtain the input conditions for the stacked motion residual learning network;

[0019] The input conditions of the stacked motion residual learning network and the spectrum of the pure noise sequence are input into the improved stacked motion residual learning network to obtain the predicted noise.

[0020] As one implementation method, the observed motion sequence spectrum is input to a conditional encoder, wherein the conditional encoder includes an observation encoder and a time encoder, and the specific process is as follows:

[0021] The spectrum of the observed motion sequence is encoded by an observation encoder to obtain the encoded spectrum of the observed motion sequence.

[0022] The denoising time step is encoded by a time encoder and then processed by a single-layer perceptron to obtain the processed denoising time step.

[0023] The input conditions for the stacked motion residual learning network are obtained based on the spectrum of the encoded observed motion sequence and the processed denoised time step.

[0024] As one implementation method, the input conditions of the stacked motion residual learning network and the spectrum of the pure noise sequence are input into the improved stacked motion residual learning network. The specific process is as follows:

[0025] Based on the predicted noise and the pure noise sequence spectrum, a denoised motion sequence spectrum is generated by sampling, and the noisy observed motion sequence spectrum is calculated.

[0026] Based on the current denoised motion sequence spectrum and the noisy observed motion sequence spectrum, the motion sequence spectrum for the next denoised time step is obtained through a masking mechanism.

[0027] The motion sequence spectrum of the current denoised time step is input into the first linear mapping layer for linear mapping to obtain the mapped motion sequence spectrum of the current denoised time step.

[0028] The mapped motion sequence spectrum of the current denoised time step and the input conditions of the stacked motion residual learning network are input into the first spatiotemporal cross-fusion graph convolutional network to obtain the first forward prediction result and the first backward prediction result of the first spatiotemporal cross-fusion graph convolutional network.

[0029] The difference between the mapped motion sequence spectrum of the current denoised time step and the current backward prediction result is used to iteratively input the input conditions of the stacked motion residual learning network into the residual spatiotemporal cross-fusion graph convolutional network until all backward and forward prediction results are obtained.

[0030] The results of each forward prediction are summed to obtain the final noise features;

[0031] The predicted noise is obtained by linearly mapping the final noise features through a second linear mapping layer.

[0032] As one implementation, the spatiotemporal cross-fusion graph convolutional network includes two parallel branch networks. The left branch network includes a cross attention module and a left double transpose graph convolutional network.

[0033] The right branch network includes the right double transpose graph convolutional network.

[0034] As one implementation method, the mapped motion sequence spectrum of the current denoised time step and the input conditions of the stacked motion residual learning network are input into the first spatiotemporal cross-fusion graph convolutional network. The specific process is as follows:

[0035] In the first spatiotemporal cross-fusion graph convolutional network, the input conditions of the stacked motion residual learning network and the motion sequence spectrum of the current denoised time step after mapping are input to the first left branch network. The first left branch noise estimate is obtained by encoding and feature extraction through the cross attention module and the first left double transpose graph convolutional network.

[0036] The motion sequence spectrum of the denoised time step after mapping is input into the first right branch network, and the first right double transpose graph convolutional network is used for encoding and feature extraction to obtain the noise estimate of the first right branch.

[0037] The first left branch noise estimate and the first right branch noise estimate are concatenated and passed through a linear mapping layer to obtain the first backward and first forward prediction results.

[0038] As one implementation, encoding and feature extraction are performed using a cross-attention module and a first left double-transposed graph convolutional network. Each double-transposed graph convolutional network includes two consecutive graph convolutions and transposes, as well as a linear mapping layer. The specific process is as follows:

[0039] The input conditions of the stacked motion residual learning network and the motion sequence spectrum of the denoised time step after mapping are input into the cross-attention module to obtain the enhanced data;

[0040] The enhanced data is input into the first graph convolutional layer to extract the observed spectral features and noise spectral features;

[0041] The observed spectral features and noise spectral features are transposed to obtain the transposed observed spectral features and noise spectral features;

[0042] The first left branch noise estimate is obtained by extracting the noise spectrum features of the transposed noise through a linear mapping layer.

[0043] As one implementation method, the motion sequence spectrum of the denoised time step is calculated using the following formula:

[0044] ;

[0045] in, Where n represents the length of the observed sequence and m represents the length of the predicted sequence. It's a transpose operation; ⊙ represents the Hadamard product. Represents the spectral density of the denoised motion sequence. This represents the spectrum of a noisy observed motion sequence. The spectrum of the motion sequence at the denoised time step. It is the discrete cosine transform Inverse Discrete Cosine Transform.

[0046] A second aspect of the present invention provides a method for predicting human motion based on spatiotemporal modeling of a diffusion model, comprising:

[0047] The data acquisition module is used to acquire the spectrum of the observed motion sequence and the spectrum of the pure noise sequence;

[0048] A noise prediction network model module is used to input the observed motion sequence spectrum and the pure noise sequence spectrum into a trained noise prediction network model to obtain the predicted noise.

[0049] The noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network. The improved stacked motion residual learning network includes a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks, and a second linear mapping layer connected in sequence. The motion sequence spectrum is input to the first linear mapping layer for linear mapping. The output of the first linear mapping layer is fed into the cascaded spatiotemporal cross-fusion graph convolutional networks to obtain the forward and backward prediction results of each network. Each forward prediction result is summed to obtain the final noise feature. The noise feature is then passed through the second linear mapping layer to obtain the predicted noise.

[0050] The motion prediction module is used to sample and generate the current denoised motion sequence spectrum based on the predicted noise and a standard normal distribution, calculate the noisy observed motion sequence spectrum, obtain the motion sequence spectrum of the next denoised time step through a masking mechanism, and repeat the iterative denoising until a new complete motion sequence spectrum is obtained.

[0051] The inverse discrete cosine transform is performed on the spectrum of the new complete motion sequence to obtain the predicted motion sequence.

[0052] The above one or more technical solutions have the following beneficial effects:

[0053] In this embodiment, by constructing an efficient Noise Prediction Network (NPN), effective spatiotemporal information is extracted from the input motion sequence, and this information is integrated into the Noise Prediction Network (NPN) through a reasonable fusion mechanism, thus achieving accurate prediction of noise and improving the accuracy of human motion prediction.

[0054] In this embodiment, the improved stacked motion residual learning network model not only more effectively integrates spatiotemporal constraint information, but also significantly improves the accuracy and robustness of prediction through the residual learning mechanism.

[0055] In this embodiment, a spatiotemporal cross-fusion graph convolutional network (STCF-GCN module) is embedded into an improved stacked motion residual learning framework (SMRLearning). This enables the efficient integration of spatiotemporal constraint information into the diffusion model, fully capturing the close correlation of spatiotemporal interactions and improving prediction accuracy. A layer-by-layer residual correction method is employed to progressively optimize the prediction results. The improved stacked motion residual learning framework (SMRLearning) consists of multiple stacked sub-modules, each generating both forward and backward predictions. The final prediction result is the ensemble of the forward predictions from all sub-modules, while the backward prediction represents the information in the input that has been effectively utilized. The input to each sub-module is the difference between the current input and its backward prediction, thereby capturing residual motion data that has not yet been modeled. In this way, each sub-module focuses on processing residual information not modeled by the previous module, progressively optimizing the prediction results and generating more faithful and accurate motion predictions than existing methods.

[0056] In this embodiment, the human motion prediction method based on spatiotemporal modeling of diffusion model improves prediction accuracy, specifically by 4% improvement in average displacement error and final displacement error compared to existing technologies. Simultaneously, the structure is more compact, with lower memory usage. The design of the spatiotemporal cross-fusion graph convolutional network (STCF-GCN module) ensures modeling capabilities while maintaining computational efficiency and effectively reduces computational complexity by 36% compared to existing technologies.

[0057] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0058] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0059] Figure 1 This is a flowchart of a human motion prediction method based on spatiotemporal modeling of a diffusion model, as described in Embodiment 1.

[0060] Figure 2 This is a training diagram of a noisy prediction network model in this embodiment.

[0061] Figure 3 This is an improved stacking motion residual learning network model in this embodiment.

[0062] Figure 4 This is the spatiotemporal cross-fusion graph convolutional network model of this embodiment;

[0063] Figure 5 This is the double transposed graph convolutional network model of this embodiment. Detailed Implementation

[0064] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0065] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0066] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0067] Example 1

[0068] This embodiment discloses a method for predicting human motion based on spatiotemporal modeling of a diffusion model.

[0069] To more clearly illustrate this embodiment, a process for predicting human motion based on spatiotemporal modeling using a diffusion model can be specifically described as follows:

[0070] A method for predicting human motion based on spatiotemporal modeling of a diffusion model, comprising:

[0071] S1. Obtain the spectrum of the observed motion sequence and the spectrum of the pure noise sequence;

[0072] S2. Input the observed motion sequence spectrum and the pure noise sequence spectrum into the trained noise prediction network model to obtain the predicted noise;

[0073] The noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network. The improved stacked motion residual learning network comprises a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks, and a second linear mapping layer connected in sequence. The motion sequence spectrum is input to the first linear mapping layer for linear mapping. The output of the first linear mapping layer is fed into the cascaded spatiotemporal cross-fusion graph convolutional networks to obtain the forward and backward prediction results of each network. Each forward prediction result is summed to obtain the final noise features. These noise features are then processed by the second linear mapping layer to obtain the predicted noise.

[0074] S3. Based on the predicted noise and the standard normal distribution, sample and generate the current denoised motion sequence spectrum, calculate the noisy observed motion sequence spectrum, and obtain the motion sequence spectrum of the next denoised time step through a masking mechanism. Repeat the iterative denoising until a new complete motion sequence spectrum is obtained.

[0075] The predicted motion sequence is obtained by performing an inverse discrete cosine transform on the spectrum of the new complete motion sequence.

[0076] like Figure 1 As shown, in step S1, the spectrum of the observed motion sequence and the spectrum of the pure noise sequence are obtained.

[0077] S1-1, Obtain the observed motion sequence.

[0078] In this embodiment, an Intel RealSense D435 camera is used to synchronously acquire RGB images and depth images of human motion for 0.5 seconds (25 frames) at a frequency of 50Hz.

[0079] The MediaPipe algorithm is used to extract the human joint skeleton from the acquired RGB images and obtain two-dimensional joint coordinates.

[0080] Subsequently, the two-dimensional joint coordinates in the RGB image are mapped to the corresponding depth image to obtain the depth value of each joint point.

[0081] Based on camera intrinsic parameters, the two-dimensional joint coordinates are converted into three-dimensional spatial coordinates to obtain the three-dimensional position of the joint points.

[0082] Finally, the coordinates of the three-dimensional key points in each frame are arranged in chronological order to generate the observed motion sequence. .

[0083] S1-2, Sampling pure noise sequence spectrum.

[0084] In this embodiment, initial values ​​are sampled from a standard normal distribution, i.e., the spectrum of a pure noise sequence. .

[0085] S1-3. Before performing discrete cosine transform on the observed motion sequence, a filling process is performed to obtain the filled observed motion sequence.

[0086] In this embodiment, in order to observe the motion sequence To predict human motion, the observed motion sequence is filled in to obtain the filled observed motion sequence. The formula is:

[0087] (1)

[0088] in, This represents the observed motion sequence after filling, where n represents the length of the observed sequence and m represents the length of the predicted sequence.

[0089] S1-3. Perform discrete cosine transform on the observed motion sequence to obtain the spectrum of the observed motion sequence.

[0090] Discrete cosine transform (DCT) is used to encode temporal information in human motion prediction (HMP).

[0091] Given the first Joint coordinates Step sequence Its DCT coefficient The calculation formula is:

[0092] (2)

[0093] in, .

[0094] Meanwhile, given a set of DCT coefficients The original sequence can be reconstructed using the inverse discrete cosine transform (IDCT). :

[0095] (3)

[0096] In this embodiment, for the filled observation motion sequence Perform DCT encoding and extract the front end. The low-frequency components are used to obtain the spectrum of the observed motion sequence. The formula is:

[0097] (4)

[0098] in, This represents the spectrum of the observed motion sequence.

[0099] Following the steps described above, the Discrete Cosine Transform (DCT) is used to represent the time variations of a joint as a linear combination of DCT bases, thereby converting time-domain information into frequency-domain information and effectively capturing the motion patterns of individual joints. Furthermore, by discarding high-frequency information, a more compact representation is provided, effectively capturing the inherent smoothness of human motion.

[0100] like Figure 1 As shown, in step S2, the observed motion sequence spectrum and the pure noise sequence spectrum are input into the trained noise prediction network model to obtain the predicted noise.

[0101] S2-1. Construct a noise prediction network model.

[0102] The key to the diffusion model of human motion prediction is the accuracy of noise prediction. In this embodiment, an efficient noise prediction model is constructed.

[0103] For Noise Prediction Networks (NPN), an improved Stacked Motion Residual Network (SMRNet) is designed. This network optimizes motion prediction results step by step through a residual learning mechanism, which significantly improves the accuracy and robustness of prediction.

[0104] In this embodiment, a noise prediction network model is designed, which includes a conditional encoder and an improved stacked motion residual learning network. The improved stacked motion residual learning network includes N spatiotemporal cross-fusion graph convolutional networks (STCF-GCN modules) and two linear mapping layers.

[0105] The condition encoder includes an observation encoder and a time encoder.

[0106] To fuse temporal and spatial information in motion, a novel spatiotemporal cross-fusion graph convolutional network (STCF-GCN module) is designed. This network employs a dual transposed graph convolutional network (DTGCN) design, and the module uses the linearly mapped, noisy perturbation motion spectrum. The diffusion conditions generated by the conditional encoder are Input and output noise characteristics.

[0107] Each spatiotemporal cross-fusion graph convolutional network includes two parallel branch networks.

[0108] The left branch network includes a cross-attention module and a double transpose graph convolutional network.

[0109] The right branch network includes a double transposed graph convolutional network.

[0110] Each Double Transposed Graph Convolutional Network (DTGCN) consists of two graph convolutional layers (GCN), two transpose operations (Trans), and one linear mapping layer.

[0111] In this way, the model can fully utilize information from different sources and leverage the advantages of the Double Transposed Graph Convolutional Network (DTGCN) in terms of spatiotemporal constraints, thereby achieving more comprehensive and accurate noise estimation. The core of this method lies in the left branch's in-depth mining of motion patterns in historical data through a motion attention mechanism, while the right branch focuses on capturing current noise characteristics. The two complement each other, ultimately achieving accurate noise prediction.

[0112] like Figure 2 As shown in Figure S2-2, the noise prediction network model is trained.

[0113] 1) Obtain the original complete motion sequence to be trained and the observed motion sequence to be trained.

[0114] In this embodiment, an original complete motion sequence is obtained from two commonly used datasets for human prediction (Human3.6M and HumanEva-I), and the observed motion sequence is extracted from the original complete motion sequence.

[0115] Obtain a 3D skeleton input sequence containing N frames, i.e., the original complete motion sequence, denoted as . .

[0116] The observed motion sequence is obtained from the original complete motion sequence, denoted as... ,in Indicates the first The human motion in the frame, it The 3D coordinates of each joint are arranged vertically as a vector, and Predicting what will happen next. frame( ),Right now ,in This represents the i-th frame of the predicted human motion. This requires a deep understanding of the input sequence to generate a motion sequence that seamlessly integrates with and faithfully replicates the input.

[0117] 2) Fill the observed motion sequence to be trained according to formula (1) to obtain the filled observed motion sequence to be trained.

[0118] 3) Perform discrete cosine transform on the filled observed motion sequence to be trained to obtain the spectrum of the observed motion sequence to be trained and the original complete motion sequence to be trained.

[0119] In this embodiment, for the filled observation motion sequence and the original complete motion sequence Both were DCT encoded separately, and the first two were extracted. The low-frequency components are used to obtain the spectrum of the observed motion sequence. and the spectrum of the original complete motion sequence The formula is:

[0120] (5)

[0121] in, This represents the spectrum of the observed motion sequence used for training. This represents the spectrum of the original complete motion sequence to be trained.

[0122] 4) Add noise to the spectrum of the original complete motion sequence to be trained to obtain the spectrum of the original motion sequence with noise.

[0123] In this embodiment, the diffusion process diffuses the spectrum of the original complete sequence to be trained. Add noise The noisy motion spectrum to be trained is obtained. The formula is:

[0124] (6)

[0125] in, , It is the total number of diffusion steps. , , , It is a predefined variance scheduler.

[0126] 5) Input the noisy motion spectrum to be trained into the noise prediction network model.

[0127] A noise prediction network model (NPN) was trained using the Human3.6M and HumanEva-I datasets.

[0128] By training a noise prediction network (NPN) To predict noise Its goal is to minimize the loss function, as shown in the formula:

[0129] (7)

[0130] in, Indicates the loss in noise The expected value at time step t is used to average the prediction loss over the training data. The smaller, The closer The better the optimization effect, the more accurate the noise prediction network model will be in predicting noise.

[0131] Ultimately, we can obtain a well-trained and accurate noise prediction network.

[0132] Experiments were conducted on commonly used human motion prediction datasets (Human3.6M and HumanEva-I). Experimental results show that the proposed method can generate more accurate and faithful human motion prediction results. Compared with existing methods, the proposed method improves both the average displacement error and the final displacement error by about 4%, while having a smaller model size, lower computational complexity, and faster inference speed.

[0133] like Figure 3 As shown, a unique spatiotemporal cross-fusion graph convolutional network (STCF-GCN module) is designed in the stacked motion residual learning network to efficiently model the spatiotemporal constraints of human motion. The STCF-GCN module adopts a two-branch structure, with each branch containing a dual transposed graph convolutional network (DTGCN) module, which can simultaneously capture the joint topological relationships in the spatial dimension and the motion evolution laws in the temporal dimension, thereby achieving comprehensive modeling of spatiotemporal constraints.

[0134] S2-3. Input the observed motion sequence spectrum and the pure noise sequence spectrum into the trained noise prediction network model.

[0135] In this embodiment, the spectrum of the observed motion sequence will be... The spectrum of the pure noise sequence is input into the trained noise prediction network model.

[0136] The observed motion sequence spectrum and the pure noise sequence spectrum are input into a trained noise prediction network model, which includes a conditional encoder and an improved stacked motion residual learning network. The specific process is as follows:

[0137] like Figure 3As shown in S2-3-1, the observed motion sequence spectrum is input to the conditional encoder to obtain the input conditions for the stacked motion residual learning network.

[0138] The observed motion sequence spectrum is input into the conditional encoder, which includes an observation encoder and a time encoder. The specific process is as follows:

[0139] (1) The spectrum of the observed motion sequence is encoded by the observation encoder to obtain the encoded spectrum of the observed motion sequence.

[0140] In this embodiment, the observation encoder is used to analyze the observed spectrum. Encoding is performed to obtain the spectrum of the observed motion sequence. First, the observed spectrum o is mapped to high-dimensional features, and then input into a self-attention module. The self-attention mechanism dynamically calculates the correlation between the spectra to adjust the weight information, thereby extracting more effective features. Finally, it is input into a dual transposed graph convolutional network (DTGCN) for spatiotemporal feature fusion.

[0141] (2) The denoising time step is encoded by a time encoder and processed by a single-layer perceptron to obtain the processed denoising time step.

[0142] In this embodiment, the denoising time step t is encoded by a time encoder. First, t is encoded by a sinusoidal position encoder and then processed by a single-layer perceptron.

[0143] (3) Based on the spectrum of the encoded observed motion sequence and the denoised time step, the input conditions of the stacked motion residual learning network are obtained.

[0144] The denoised time step is fused with the coded spectrum of the observed motion sequence, and the overall formula is:

[0145] (8)

[0146] in, and It is a single-hidden-layer perceptron. It is a sinusoidal position code. The input conditions, i.e., the diffusion conditions, represent the input conditions of the stacked motion residual learning network, which are input to each spatiotemporal cross-fusion graph convolutional network (STCF-GCN module) in the improved stacked motion residual learning framework (SMRLearning).

[0147] like Figure 3 As shown in S2-3-2, the input conditions of the stacked motion residual learning network and the spectrum of the pure noise sequence are input into the improved stacked motion residual learning network to obtain the predicted noise.

[0148] The input conditions and the spectrum of the pure noise sequence are fed into the improved stacked motion residual learning network, which includes two linear mapping layers and several spatiotemporal cross-fusion graph convolutional networks. The specific process is as follows:

[0149] (1) Based on the predicted noise and pure noise sequence spectrum, sample to generate the denoised motion sequence spectrum, and calculate the noisy observed motion sequence spectrum.

[0150] Specifically, the initial values ​​are sampled based on a standard normal distribution, i.e., the spectrum of a pure noise sequence. Iterative generation of denoised motion sequence spectra The formula is:

[0151] (9)

[0152] in, ,and .

[0153] For each step By analyzing the spectrum of the observed motion sequence Add noise The formula for calculating the spectrum of a noisy observed motion sequence is:

[0154] (10)

[0155] in, This represents the spectrum of a noisy observed motion sequence. .

[0156] (2) Based on the denoised motion sequence spectrum and the noisy observed motion sequence spectrum, the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism.

[0157] In this embodiment, based on the denoised motion sequence spectrum and the spectrum of noisy observed motion sequences The final motion sequence spectrum for the next de-voicing time step is obtained through a masking mechanism. The formula is:

[0158] (11)

[0159] in, Where n represents the length of the observed sequence and m represents the length of the predicted sequence. It's a transpose operation; ⊙ represents the Hadamard product. Represents the spectral density of the denoised motion sequence. This represents the spectrum of a noisy observed motion sequence. The spectrum of the motion sequence at the denoised time step. It is the discrete cosine transform Inverse Discrete Cosine Transform.

[0160] (3) Input the motion sequence spectrum of the denoised time step into the first linear mapping layer for linear mapping to obtain the motion sequence spectrum of the denoised time step after mapping.

[0161] like Figure 4 As shown, (4) the motion sequence spectrum of the denoised time step after mapping and the input conditions of the stacked motion residual learning network are input into each spatiotemporal cross-fusion graph convolutional network to obtain the forward prediction result and backward prediction result of each spatiotemporal cross-fusion graph convolutional network.

[0162] The mapped denoised time step motion sequence spectrum and input conditions The input is fed into the first spatiotemporal cross-fusion graph convolutional network to obtain the first forward prediction result. and the first backward prediction result .

[0163] In this embodiment, the motion sequence spectrum of the denoised time step and the input conditions are input into each spatiotemporal cross-fusion graph convolutional network.

[0164] First, the input conditions and the motion sequence spectrum of the denoised time step after mapping are input into the first spatiotemporal cross-fusion graph convolutional network to obtain the first forward prediction result and the first backward prediction result.

[0165] The left and right branches process information from different sources to achieve comprehensive and accurate noise estimation. In the left branch, a cross-attention module is introduced to extract valuable information from historical motion data, thereby enhancing the understanding of underlying motion patterns. This is achieved by aggregating information from diffusion conditions. This goal is achieved through motion information.

[0166] The specific process is as follows:

[0167] 1) Input the input conditions and the motion sequence spectrum of the denoised time step after mapping into the first left branch network. Encode and extract features through the cross attention module and the first left double transpose graph convolutional network to obtain the noise estimate of the first left branch.

[0168] like Figure 5 As shown, in this embodiment, encoding and feature extraction are performed using a cross-attention module and a first left double-transposed graph convolutional network. Each double-transposed graph convolutional network includes two consecutive graph convolutions and transposes, as well as a linear mapping layer. The specific process is as follows:

[0169] The input conditions and the motion sequence spectrum of the denoised time step after mapping are input into the cross-attention module to obtain the enhanced data.

[0170] The enhanced data is input into the first graph convolutional layer to extract the observed spectral features and noise spectral features.

[0171] In this embodiment, the graph convolutional network aggregates information between nodes through an adjacency matrix, thereby achieving efficient feature learning and graph embedding. This characteristic makes it particularly suitable for representing human joints as a graph structure to capture the spatial relationships between joints. However, in practical human motion prediction tasks, in addition to the dependencies between joints, the temporal constraints during the movement of each joint also need to be fully considered.

[0172] A graph-based dual modeling strategy is adopted. First, human joints are represented as graph structures, and then each joint is modeled as a separate graph structure, thereby capturing the temporal dynamics and sequence constraints in its motion patterns. In the first graph convolutional layer, joint information is primarily encoded to characterize the spatial dependencies between joints; in the second graph convolutional layer, the focus is on extracting frequency domain features to effectively represent the dynamic changes during motion.

[0173] The forward propagation process can be represented by the following formula:

[0174]

[0175] (12)

[0176] in, The input motion spectrum features include observed spectrum features and noise spectrum features; and These represent the temporal adjacency matrix and the joint adjacency matrix, respectively. The use of trainable adjacency matrices allows the network to adapt more flexibly to different graph structures, thereby better capturing spatiotemporal relationships. These are trainable weight parameters; This represents the matrix transpose operation.

[0177] The observed spectral features and noise spectral features are transposed to obtain the transposed observed spectral features and noise spectral features. .

[0178] The first left branch noise estimate is obtained by extracting the noise spectrum features of the transposed noise through a linear mapping layer.

[0179] By applying different adjacency matrices to the two layers of GCN, this module can simultaneously consider the correlation between temporal features and joint features, thereby better capturing the complex relationships in spatiotemporal sequences.

[0180] Deep features are extracted from motion sequences using a linear output layer, as shown in the formula:

[0181] (13)

[0182] A cross-attention module is used to evaluate the partial motion similarity between the denoised result and the current observation sequence to generate noise estimates. These noise estimates are then fed into a Double Transposed Graph Convolutional Network (DTGCN) for spatiotemporal modeling, and the final output is denoted as the left branch noise estimate. The formula is:

[0183] (14)

[0184] in, As a query, the diffusion condition c serves as both the key and value, and Linear represents the corresponding linear projection layer. The scaling factor d is used to maintain numerical stability during training. Cross-attention measures the local motion similarity between the previous denoising result and the current observation sequence, while the Dual Transposed Graph Convolutional Network (DTGCN) is responsible for imposing spatiotemporal constraints to ensure data consistency.

[0185] In the first spatiotemporal cross-fusion graph convolutional network, when the input is... At that time, according to formula (14). This is the noise estimation for the first left branch.

[0186] 2) Input the motion sequence spectrum of the denoised time step after mapping into the first right branch network, and encode and extract features through the first right double transpose graph convolutional network to obtain the noise estimate of the first right branch.

[0187] The right branch focuses on the noise characteristics at the current moment, directly processing the noisy motion spectrum data. By encoding and extracting features from this data, a noise feature estimate is generated, denoted as the right-branch noise estimate. The formula is:

[0188] (15)

[0189] In the first spatiotemporal cross-fusion graph convolutional network, when the input is... At that time, according to formula (15). This is the noise estimate for the first right branch.

[0190] 3) The left branch noise estimate and the right branch noise estimate are concatenated and passed through a linear mapping layer to obtain the backward and forward prediction results.

[0191] In this embodiment, the results generated by the left and right branches and These are combined to generate the final output of the Spatiotemporal Cross-Fusion Graph Convolutional Network (STCF-GCN module), as shown in the formula:

[0192]

[0193] (16)

[0194] in, and Representing the first The linear projection layer of each module is used to generate backward and forward predictions. This indicates the result of backward prediction. This indicates the result of the forward prediction. This represents the left branch noise estimation. This represents the right branch noise estimation.

[0195] In the first spatiotemporal cross-fusion graph convolutional network... This is the first backward prediction result. This is the result of the first forward prediction.

[0196] Then, the difference between the motion sequence spectrum of the denoised time step after mapping and the first backward prediction result is calculated as shown in Equation (17). This difference, along with the input conditions of the stacked motion residual learning network, is then input into the second spatiotemporal cross-fusion graph convolutional network to obtain the second forward prediction result. Second backward prediction results .

[0197] (17)

[0198] in, This represents the input to the current layer's spatiotemporal cross-fusion graph convolutional network. This represents the input to the next layer of the spatiotemporal cross-fusion graph convolutional network.

[0199] Among them, when the second spatiotemporal cross-fusion graph convolutional network is input, according to formula (17), the input of the second spatiotemporal cross-fusion graph convolutional network is... , This is the result of the second backward prediction. This is the result of the second forward prediction.

[0200] Finally, the inputs are iteratively fed into each spatiotemporal cross-fusion graph convolutional network until all backward and forward prediction results are obtained.

[0201] According to formula (17), the input of each spatiotemporal cross-fusion graph convolutional network is obtained sequentially, and the input is cyclically fed into N spatiotemporal cross-fusion graph convolutional networks until N forward prediction results and N backward prediction results are obtained. According to formula (16), all forward prediction results and backward prediction results are obtained.

[0202] Where N is the number of layers in the Spatiotemporal Cross-Fusion Graph Convolutional Network (STCF-GCN module), which is adjusted according to the size of the dataset. For example, the Human3.6M dataset has 8 layers, and the Humaneva-I dataset has 4 layers.

[0203] (5) Add each forward prediction result together to obtain the final noise features.

[0204] In this embodiment, each corresponding noise feature is summed to obtain the final noise feature z, as shown in the formula:

[0205] (18)

[0206] in, This represents the forward prediction result, and N represents the number of layers in the spatiotemporal cross-fusion graph convolutional network (STCF-GCN module).

[0207] (6) The final noise features are linearly mapped through the second linear mapping layer to obtain the predicted noise.

[0208] By analyzing noise characteristics By performing a linear mapping, the predicted noise is finally obtained, as shown in the formula:

[0209] (19)

[0210] in, This represents the predicted noise.

[0211] Following the steps outlined above, the improved stacked motion residual learning framework optimizes prediction results layer by layer, with specialized learning performed on the residuals of each layer, thereby achieving high-precision human motion prediction. Specifically, the improved stacked motion residual learning (SMRNet) employs a multi-layered structure, with each layer focusing on capturing different levels of motion features. Through layer-by-layer optimization, the model effectively reduces prediction errors while preserving important motion information. This hierarchical residual learning method not only improves prediction accuracy but also enhances the model's adaptability to complex human motion patterns.

[0212] like Figure 1As shown, in step S3, based on the predicted noise and the standard normal distribution, the current denoised motion sequence spectrum is sampled and generated, the noisy observed motion sequence spectrum is calculated, and the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism. The denoising is iterated repeatedly until a new complete motion sequence spectrum is obtained.

[0213] The predicted motion sequence is obtained by performing an inverse discrete cosine transform on the spectrum of the new complete motion sequence.

[0214] (1) Based on the predicted noise and the standard normal distribution, sample and generate the current denoised motion sequence spectrum, and calculate the noisy observed motion sequence spectrum.

[0215] (2) Based on the current denoised motion sequence spectrum and the noisy observed motion sequence spectrum, the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism.

[0216] Repeat steps (1) and (2) for iterative denoising until a new complete motion sequence spectrum is obtained.

[0217] According to formulas (9), (10), and (11), the iterative time step T is used until a new complete motion sequence spectrum is obtained. .

[0218] The above steps help preserve the original observation sequence and ensure a smooth continuity between the last observed frame and the first predicted frame.

[0219] like Figure 1 As shown, the inverse discrete cosine transform is performed on the spectrum of the new complete motion sequence to obtain the predicted motion sequence.

[0220] The predicted motion sequence is given by the following formula:

[0221] = (20)

[0222] in, This represents the spectrum of the new complete motion sequence, i.e., the predicted motion spectrum. The predicted motion sequence is the result of the inverse discrete cosine transform. The last m frames.

[0223] After the above steps, the motion sequence of the next m frames can be predicted based on the observed motion sequence.

[0224] Example 2

[0225] The purpose of this embodiment is to provide a human motion prediction system based on spatiotemporal modeling of a diffusion model, including:

[0226] The data acquisition module is used to acquire the spectrum of the observed motion sequence and the spectrum of the pure noise sequence;

[0227] A noise prediction network model module is used to input the observed motion sequence spectrum and the pure noise sequence spectrum into a trained noise prediction network model to obtain the predicted noise.

[0228] The noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network. The improved stacked motion residual learning network includes a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks, and a second linear mapping layer connected in sequence. The motion sequence spectrum is input to the first linear mapping layer for linear mapping. The output of the first linear mapping layer is fed into the cascaded spatiotemporal cross-fusion graph convolutional networks to obtain the forward and backward prediction results of each network. Each forward prediction result is summed to obtain the final noise feature. The noise feature is then passed through the second linear mapping layer to obtain the predicted noise.

[0229] The motion prediction module is used to sample and generate the current denoised motion sequence spectrum based on the predicted noise and a standard normal distribution, calculate the noisy observed motion sequence spectrum, obtain the motion sequence spectrum of the next denoised time step through a masking mechanism, and repeat the iterative denoising until a new complete motion sequence spectrum is obtained.

[0230] The inverse discrete cosine transform is performed on the spectrum of the new complete motion sequence to obtain the predicted motion sequence.

[0231] This invention provides a human motion prediction system based on spatiotemporal modeling using a diffusion model, implementing the method steps in Embodiment 1.

[0232] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1.

[0233] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0234] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for predicting human motion based on spatiotemporal modeling of a diffusion model, characterized in that, include: Obtain the spectrum of the observed motion sequence and the spectrum of the pure noise sequence; The observed motion sequence spectrum and the pure noise sequence spectrum are input into the trained noise prediction network model to obtain the predicted noise; Based on the predicted noise, the current denoised motion sequence spectrum is generated by sampling according to the standard normal distribution. The noisy observed motion sequence spectrum is calculated, and the motion sequence spectrum of the next denoised time step is obtained through a masking mechanism. The denoising process is repeated iteratively until a new complete motion sequence spectrum is obtained. The inverse discrete cosine transform is performed on the spectrum of the new complete motion sequence to obtain the predicted motion sequence; The noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network. The conditional encoder includes an observation encoder and a time encoder. The observation encoder encodes the spectrum of the observed motion sequence to obtain the encoded spectrum of the observed motion sequence. The time encoder encodes the denoising time step and processes it through a single-layer perceptron to obtain the processed denoising time step. The input conditions of the stacked motion residual learning network are obtained based on the encoded spectrum of the observed motion sequence and the processed denoising time step. The improved stacked motion residual learning network includes a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks, and a second linear mapping layer connected in sequence. The motion sequence spectrum is input to the first linear mapping layer for linear mapping. The output of the first linear mapping layer is fed into the cascaded spatiotemporal cross-fusion graph convolutional networks to obtain the forward and backward prediction results of each spatiotemporal cross-fusion graph convolutional network. The forward prediction results are added together to obtain the final noise features. The noise features are then passed through the second linear mapping layer to obtain the predicted noise.

2. The human motion prediction method based on spatiotemporal modeling of a diffusion model as described in claim 1, characterized in that, The specific process for obtaining the spectrum of the observed motion sequence is as follows: The observed motion sequence is filled to obtain the filled observed motion sequence; The spectrum of the observed motion sequence is obtained by performing a discrete cosine transform on the filled-in observed motion sequence.

3. The human motion prediction method based on spatiotemporal modeling of a diffusion model as described in claim 1, characterized in that, The observed motion sequence spectrum and the pure noise sequence spectrum are input into the trained noise prediction network model. The specific process is as follows: The observed motion sequence spectrum is input into the conditional encoder to obtain the input conditions for the stacked motion residual learning network; The input conditions of the stacked motion residual learning network and the spectrum of the pure noise sequence are input into the improved stacked motion residual learning network to obtain the predicted noise.

4. The human motion prediction method based on spatiotemporal modeling of a diffusion model as described in claim 3, characterized in that, The input conditions of the stacked motion residual learning network and the spectrum of the pure noise sequence are input into the improved stacked motion residual learning network. The specific process is as follows: Based on the predicted noise and the pure noise sequence spectrum, a denoised motion sequence spectrum is generated by sampling, and the noisy observed motion sequence spectrum is calculated. Based on the current denoised motion sequence spectrum and the noisy observed motion sequence spectrum, the motion sequence spectrum for the next denoised time step is obtained through a masking mechanism. The motion sequence spectrum of the current denoised time step is input into the first linear mapping layer for linear mapping to obtain the mapped motion sequence spectrum of the current denoised time step. The mapped motion sequence spectrum of the current denoised time step and the input conditions of the stacked motion residual learning network are input into the first spatiotemporal cross-fusion graph convolutional network to obtain the first forward prediction result and the first backward prediction result of the first spatiotemporal cross-fusion graph convolutional network. The difference between the mapped motion sequence spectrum of the current denoised time step and the current backward prediction result is used to iteratively input the input conditions of the stacked motion residual learning network into the residual spatiotemporal cross-fusion graph convolutional network until all backward and forward prediction results are obtained. The results of each forward prediction are summed to obtain the final noise features; The predicted noise is obtained by linearly mapping the final noise features through a second linear mapping layer.

5. The human motion prediction method based on spatiotemporal modeling of a diffusion model as described in claim 4, characterized in that, The spatiotemporal cross-fusion graph convolutional network includes two parallel branch networks. The left branch network includes a cross attention module and a left double transpose graph convolutional network. The right branch network includes the right double transpose graph convolutional network.

6. A method for predicting human motion based on spatiotemporal modeling of a diffusion model as described in any one of claims 4-5, characterized in that, The mapped motion sequence spectrum of the current denoised time step and the input conditions of the stacked motion residual learning network are fed into the first spatiotemporal cross-fusion graph convolutional network. The specific process is as follows: In the first spatiotemporal cross-fusion graph convolutional network, the input conditions of the stacked motion residual learning network and the motion sequence spectrum of the current denoised time step after mapping are input to the first left branch network. The first left branch noise estimate is obtained by encoding and feature extraction through the cross attention module and the first left double transpose graph convolutional network. The motion sequence spectrum of the denoised time step after mapping is input into the first right branch network, and the first right double transpose graph convolutional network is used for encoding and feature extraction to obtain the noise estimate of the first right branch. The first left branch noise estimate and the first right branch noise estimate are concatenated and passed through a linear mapping layer to obtain the first backward and first forward prediction results.

7. The human motion prediction method based on spatiotemporal modeling of a diffusion model as described in claim 6, characterized in that, Encoding and feature extraction are performed using a cross-attention module and a first left-hand double-transposed graph convolutional network. Each double-transposed graph convolutional network consists of two consecutive graph convolutions and transposes, plus a linear mapping layer. The specific process is as follows: The input conditions of the stacked motion residual learning network and the motion sequence spectrum of the denoised time step after mapping are input into the cross-attention module to obtain the enhanced data; The enhanced data is input into the first graph convolutional layer to extract the observed spectral features and noise spectral features; The observed spectral features and noise spectral features are transposed to obtain the transposed observed spectral features and noise spectral features; The first left branch noise estimate is obtained by extracting the noise spectrum features of the transposed noise through a linear mapping layer.

8. The human motion prediction method based on spatiotemporal modeling of a diffusion model as described in claim 1, characterized in that, The spectrum of the motion sequence at the denoised time step is calculated using the following formula: ； in, Where n represents the length of the observed sequence and m represents the length of the predicted sequence. It is a mask vector, and ⊙ represents the Hadamard product. Represents the spectrum of a denoised motion sequence. This represents the spectrum of a noisy observed motion sequence. The spectrum of the motion sequence at the denoised time step. It is the discrete cosine transform. Inverse Discrete Cosine Transform.

9. A human motion prediction system based on spatiotemporal modeling of a diffusion model, characterized in that, include: The data acquisition module is used to acquire the spectrum of the observed motion sequence and the spectrum of the pure noise sequence; A noise prediction network model module is used to input the observed motion sequence spectrum and the pure noise sequence spectrum into a trained noise prediction network model to obtain the predicted noise. The noise prediction network model includes a conditional encoder and an improved stacked motion residual learning network. The conditional encoder includes an observation encoder and a time encoder. The observation encoder encodes the spectrum of the observed motion sequence to obtain the encoded spectrum of the observed motion sequence. The time encoder encodes the denoising time step and processes it through a single-layer perceptron to obtain the processed denoising time step. The input conditions of the stacked motion residual learning network are obtained based on the encoded spectrum of the observed motion sequence and the processed denoising time step. The improved stacked motion residual learning network comprises a first linear mapping layer, multiple cascaded spatiotemporal cross-fusion graph convolutional networks, and a second linear mapping layer connected in sequence. The motion sequence spectrum is input to the first linear mapping layer for linear mapping. The output of the first linear mapping layer is fed into the cascaded spatiotemporal cross-fusion graph convolutional networks to obtain the forward and backward prediction results of each network. Each forward prediction result is summed to obtain the final noise features. These noise features are then processed by the second linear mapping layer to obtain the predicted noise. The motion prediction module is used to sample and generate the current denoised motion sequence spectrum based on the predicted noise and a standard normal distribution, calculate the noisy observed motion sequence spectrum, obtain the motion sequence spectrum of the next denoised time step through a masking mechanism, and repeat the iterative denoising until a new complete motion sequence spectrum is obtained. The inverse discrete cosine transform is performed on the spectrum of the new complete motion sequence to obtain the predicted motion sequence.

Images