A video human behavior prediction method based on residual diffusion theory and skeleton points

Abstract

The application discloses a video human behavior prediction method based on residual diffusion theory and skeleton points, comprising the following steps: S10, acquiring an original group motion sequence, decomposing the multi-person sequence into individual trajectories, and then converting the time-domain motion sequence of each individual into frequency coefficients through discrete cosine transform (DCT); S20, based on the frequency domain representation, introducing a residual term to construct a forward diffusion process; S30, defining a weighted residual noise as a learning target, and realizing conversion from noise to residual noise; S40, determining an optimal step through transformation, realizing acceleration and unified training and reasoning; S50, applying physical constraints to single-person prediction, introducing a weighted combined biomechanical regularization loss function to constrain the physical feasibility of the generated sequence; and S60, synthesizing a group sequence and outputting a predicted sequence. The application utilizes deep learning technology, extracts human skeleton point information, combines a residual diffusion mechanism, predicts human behavior in a video sequence or a real-time scene, and realizes high-precision prediction of future behavior.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a video human behavior prediction method based on residual diffusion theory and skeletal points. Background Technology

[0002] In existing technologies, a common background technique is the skeletal point behavior recognition method based on convolutional neural networks (CNNs). This method identifies and predicts simple human actions, such as walking or waving, by analyzing sequences of human skeletal points, and is widely used in security monitoring. Its implementation process includes: first, using a pose estimation tool to extract the coordinates of key human skeletal points from video frames; then, inputting these point sequences into a CNN model for feature extraction and spatiotemporal modeling; finally, using a softmax classifier to output the behavior category or short-term prediction. While it can effectively identify simple actions in static environments, it suffers from insufficient robustness, easily leading to prediction bias. Specifically, because this method mainly relies on convolutional layers to extract local features from skeletal point sequences without introducing noise processing or diffusion mechanisms, in complex dynamic scenes (such as changes in lighting, occlusion, or background interference), the extracted features are easily affected by noise, leading to increased prediction bias. This deficiency stems from its implementation focusing only on spatiotemporal modeling while neglecting the smoothing effect of residual mechanisms on noise, making it difficult to maintain high-accuracy predictions in practical applications such as intelligent monitoring.

[0003] Another common technique is the spatiotemporal sequence prediction method based on recurrent neural networks (RNNs). This method aims to predict the evolutionary trend of human behavior using historical skeletal data, and is suitable for medical or interactive systems. Its implementation involves: collecting time-series data of human skeletal points; then processing the sequence through a variant of the RNN, the Long Short-Term Network (LSTM), to capture long-term dependencies; finally, combining fully connected layers to generate behavior prediction outputs, such as motion trajectories for the next few frames. This method can handle sequence dependencies, but it consumes significant computational resources, suffers from latency issues in real-time applications, and does not consider noise robustness caused by diffusion mechanisms. Specifically, this method processes the time-series data of skeletal points through LSTM units, relying on layer-by-layer recursive computation to model long-term dependencies. This leads to an exponential increase in computational complexity when dealing with high-dimensional data or long sequence inputs, and it does not integrate diffusion theory to optimize noise propagation paths. Consequently, in real-time interactive systems, model updates lag, affecting response speed. This deficiency stems from its implementation emphasizing sequence dependencies without considering the optimization of residual diffusion, resulting in low prediction efficiency in resource-constrained environments. Summary of the Invention

[0004] To address the aforementioned issues and overcome the shortcomings of existing technologies in terms of insufficient robustness and low real-time prediction efficiency in complex dynamic scenarios, this invention proposes a video human behavior prediction method based on residual diffusion theory and skeletal points. By utilizing deep learning technology, this method extracts human skeletal point information and combines it with the residual diffusion mechanism to predict human behavior in video sequences or real-time scenarios, achieving high-precision prediction of future behavior.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: a video human behavior prediction method based on residual diffusion theory and skeletal points, comprising the following steps:

[0006] S10, Obtain the original group motion sequence, decompose the multi-person sequence into individual trajectories, and then convert the time-domain motion sequence of each individual into frequency coefficients through Discrete Cosine Transform (DCT).

[0007] S20, Construct the diffusion process: Based on the frequency domain representation, introduce residual terms to construct the forward diffusion process;

[0008] S30, Residual Noise Learning: Define weighted residual noise as the learning target to achieve the conversion from noise to residual noise;

[0009] S40, Smooth Equivalent Transformation Acceleration: Determines the optimal step through transformation, thereby accelerating and unifying training inference;

[0010] S50, Biomechanical Regularization Constraint: Imposing physical constraints on individual predictions by introducing a weighted combination of biomechanical regularization loss functions to constrain the physical feasibility of the generated sequences;

[0011] S60, synthesizes the population sequence and outputs the predicted sequence.

[0012] Furthermore, the time-domain motion sequence of each individual is converted into frequency coefficients using a Discrete Cosine Transform (DCT), with the following transformation formula:

[0013] ;

[0014] in, For frequency coefficients indexed by frequency k, Let k be the temporal motion sequence of each individual with time index t; t is the time index, ranging from 1 to F, representing a specific frame position in the temporal sequence; k is the frequency index, ranging from 1 to F, representing a specific frequency component in the frequency domain; F is the total number of frames in the entire motion sequence, i.e., the sum of the observed frame number O and the predicted frame number (F - O); within the frame, the observed past O frames and the predicted future (F - O) frames are integrated into a complete sequence with a total length of F; in practice, the first K < F low-frequency components are retained for compressed representation, while the basic motion energy is preserved.

[0015] Furthermore, the forward diffusion process, which involves introducing residual terms, includes:

[0016] Residual terms: ;

[0017] in, Represents the true ground frequency coefficient. Indicates degenerate input;

[0018] Forward diffusion process incorporates R:

[0019] ;

[0020] Where: probability distribution ;

[0021] ;

[0022] in, Indicates given initial clean data Given the residual term R, the entire noise sequence from time step 1 to T The joint probability distribution of z; t z represents the noisy data vector at time step t in the diffusion process. t-1 This represents the noisy data vector from the previous time step t-1; α t This represents the noise scheduling parameter, which controls the proportion of noise added at each step, α. t Within the range (0,1), it decreases as t increases, ensuring that noise gradually dominates; β t represents the noise variance scheduling parameter, the noise intensity added at time step t, typically in the range (0,1); I represents the identity matrix, used to define the covariance matrix of the Gaussian distribution, indicating that the noise is independent in each dimension;

[0023] Reparameterization yields:

[0024] ;

[0025] in, α represents the time step from time step 1 to t. S The cumulative product is used to reparameterize the entire process; α s s represents the noise scheduling parameters under index s; z0 represents the true frequency coefficients, i.e., the clean data vector at the beginning of the diffusion process; It is a small positive number;

[0026] Substitute: .

[0027] Furthermore, the weighted residual noise :

[0028] ;

[0029] Minimize the training objective:

[0030] ;

[0031] in, Due to the condition of the mask U-Net predictions.

[0032] Furthermore, during the acceleration process of the smooth equivalent transformation,

[0033] coefficient For large Approaching zero, therefore we find:

[0034] ;

[0035] i is the iteration coefficient, representing the range from 1 to T;

[0036] get:

[0037] ;

[0038] in, This is the noisy data vector at the optimal acceleration time step T′ of the diffusion process; it represents the intermediate noisy state of the frequency domain motion sequence. is the cumulative noise scheduling parameter, which is the cumulative scaling factor at time step T′.

[0039] Furthermore, the biomechanical regularization loss function For weighted combinations: ;

[0040] in, This represents the i-th regular term; Indicates weight;

[0041] Instead of measuring only on real ground data, regularization terms are applied directly to the denoised output sequence, thereby constraining physical properties in real time during the generation process.

[0042] Furthermore, the regularization terms include bone length consistency, bone length ratio, bone vector angle, velocity smoothing, velocity upper limit, relative distance and collision, and bone constraint combination;

[0043] The bone length consistency effect is to penalize the length drift of the same bone over time and maintain the approximate constancy of bone length in human anatomy.

[0044] The bone length ratio is used to characterize the relative proportional deviation between the predicted bone length and the actual bone length on the ground by the logarithmic bone length difference. It maintains scale invariance when changing across datasets or units, so that the overall scaling is converted into translational deviation, which is then canceled out by the difference operation.

[0045] The function of bone vector angle is to minimize the angle between the predicted bone vector and the actual bone vector on the ground;

[0046] The velocity smoothing effect uses second-order differences as smoothing priors to ensure the continuity of the trajectory on the second derivative;

[0047] The velocity capping function applies hinge loss to the portion of the frame-by-frame velocity norm that exceeds a threshold.

[0048] Relative distance and collision: The function is to constrain the consistency of the relative distance matrix between joints, while preventing joints or limbs from intersecting and colliding, and ensuring the spatial physical rationality in multi-person or single-person sequences;

[0049] Bone constraint combination: Combine the first three bone-related constraints to form a joint regularization term.

[0050] Furthermore,

[0051] The bone length consistency: ;

[0052] in, This represents the bone length vector of the current frame t. For reference bone length vector,

[0053] The bone length ratio: ;

[0054] in, This represents the bone length vector of the predicted sequence. Represents the bone length vector of the actual ground sequence;

[0055] Bone vector angle: ;

[0056] in, and The bone vectors of the predicted sequence and the ground truth sequence are used respectively. The cosine similarity is calculated using the dot product and norm. Converted to angle, It is a small positive number;

[0057] Speed ​​smoothing: ;

[0058] in, The second-order difference of the velocity in frame t is obtained through two consecutive difference operations. ,in Let be the joint velocity vector of frame t. Let be the joint velocity vector of frame t-1. Let be the joint velocity vector for frame t-2;

[0059] Speed ​​limit: ;

[0060] in, This represents the joint velocity vector in frame t. Its Euclidean norm, The preset speed limit threshold, Ensure that only the portion exceeding the threshold is penalized;

[0061] Relative distance and collision: ;

[0062] in, and These are the distance matrices for the predicted sequence and the ground truth sequence, respectively; the matrix elements are the Euclidean distances between joints. The first part minimizes the matrix bias, and the second part is the hinge loss penalty for the negative distance. The minimum allowable distance threshold;

[0063] Bone restraint combination: .

[0064] Furthermore, the total loss:

[0065] ;

[0066] in, ; It is an adjustable hyperparameter. It is weighted residual noise. Due to the condition of the mask U-Net predictions.

[0067] The beneficial effects of adopting this technical solution are:

[0068] This invention addresses the issue of decreased prediction accuracy in skeletal behavior recognition methods based on Convolutional Neural Networks (CNNs) under noise interference (such as changes in lighting or occlusion). By introducing residual diffusion theory, this invention optimizes the feature extraction process and enhances the model's robustness to complex scenes. Simultaneously, addressing the shortcomings of recurrent neural network (RNN)-based spatiotemporal sequence prediction methods, such as high computational complexity and insufficient real-time performance, this invention combines the residual diffusion mechanism with Discrete Cosine Transform (DCT) to reduce the dimensionality of skeletal point data, thereby reducing computational resource consumption and improving the prediction speed and accuracy of the model in real-time applications. The purpose of this invention is to achieve high-precision and high-efficiency human behavior prediction, meeting the real-time requirements in dynamic environments. Attached Figure Description

[0069] Figure 1 This is a schematic diagram of a video human behavior prediction method based on residual diffusion theory and skeletal points according to the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings.

[0071] In this embodiment, see Figure 1 As shown, this invention proposes a video human behavior prediction method based on residual diffusion theory and skeletal points, including the following steps:

[0072] S10, Obtain the original group motion sequence, decompose the multi-person sequence into individual trajectories, and then convert the time-domain motion sequence of each individual into frequency coefficients through Discrete Cosine Transform (DCT).

[0073] The temporal motion sequence of each individual The discrete cosine transform (DCT) is used to convert the coefficients to frequency coefficients to reduce dimensionality and emphasize the low-frequency dominant mode.

[0074] The improvement in this data processing lies in the fact that frequency domain operations naturally suppress high-frequency noise (such as jitter) and improve the smoothness of the generated data.

[0075] Specifically, the time-domain motion sequence of each individual is converted into frequency coefficients using Discrete Cosine Transform (DCT). The transformation formula is as follows:

[0076] ;

[0077] in, For frequency coefficients indexed by frequency k, Let k be the temporal motion sequence of each individual with time index t; t is the time index, ranging from 1 to F, representing a specific frame position in the temporal sequence; k is the frequency index, ranging from 1 to F, representing a specific frequency component in the frequency domain; F is the total number of frames in the entire motion sequence, i.e., the sum of the observed frame number O and the predicted frame number (F - O); within the frame, the observed past O frames and the predicted future (F - O) frames are integrated into a complete sequence with a total length of F; in practice, the first K < F low-frequency components are retained for compressed representation, while the basic motion energy is preserved.

[0078] S20, Constructing the diffusion process: Based on the frequency domain representation, a residual term is introduced to construct the forward diffusion process, realizing the core improvement of directly initiating the inverse process from the observation of noise degradation.

[0079] Specifically, the process of introducing residual terms to construct the forward diffusion process includes:

[0080] Residual terms: ;

[0081] in, Represents the true ground frequency coefficient. Indicates degenerate input;

[0082] Forward diffusion process incorporates R:

[0083] ;

[0084] Where: probability distribution ;

[0085] ;

[0086] in, Indicates given initial clean data Given the residual term R, the entire noise sequence from time step 1 to T The joint probability distribution of z; t z represents the noisy data vector at time step t in the diffusion process. t-1 This represents the noisy data vector from the previous time step t-1; α t This represents the noise scheduling parameter, which controls the proportion of noise added at each step, α. t Within the range (0,1), it decreases as t increases, ensuring that noise gradually dominates; β t represents the noise variance scheduling parameter, the noise intensity added at time step t, typically in the range (0,1); I represents the identity matrix, used to define the covariance matrix of the Gaussian distribution, indicating that the noise is independent in each dimension;

[0087] Reparameterization yields:

[0088] ;

[0089] in, α represents the time step from time step 1 to t. S The cumulative product is used to reparameterize the entire process; α s s represents the noise scheduling parameters under index s; z0 represents the true frequency coefficients, i.e., the clean data vector at the beginning of the diffusion process; It is a small positive number;

[0090] Substitute: .

[0091] The improvement in this process lies in bridging degradation and clean data in the residual terms, allowing the reverse process to proceed from... Startup reduces the number of sampling steps (to 5–20 steps) and supports real-time prediction.

[0092] S30, Residual Noise Learning: Define weighted residual noise as the learning target to achieve the conversion of noise into residual noise and improve prediction continuity.

[0093] Specifically, the weighted residual noise :

[0094] ;

[0095] Minimize the training objective:

[0096] ;

[0097] in, Due to the condition of the mask U-Net predictions;

[0098] This invention will learn from noise The shift to weighted residual noise supports mask completion: future frequencies are masked and noise is added, and the spectrum is completed by denoising, ensuring continuity and diversity of category switching based on the observed frequency.

[0099] S40, Smooth Equivalent Transformation Acceleration: Determining the Optimal Step Through Transformation This enables faster and more unified training inference.

[0100] Specifically, during the acceleration process of the smooth equivalent transformation, the coefficients For large Approaching zero, therefore we find:

[0101] ;

[0102] in, This is the optimal step; i is the iteration coefficient, representing the steps from 1 to T; The coefficient corresponding to the noise scheduling under the i-th iteration coefficient;

[0103] get:

[0104] ;

[0105] in, This is the noisy data vector at the optimal acceleration time step T′ of the diffusion process; it represents the intermediate noisy state of the frequency domain motion sequence. is the cumulative noise scheduling parameter, which is the cumulative scaling factor at time step T′.

[0106] The improvement of this transformation lies in its approximate Gaussian prior, which allows for the direct reuse of existing noisy scheduling instead of redesigning it, reducing performance loss. This stage ensures that from... Minimize losses, build upon the diffusion from the previous stage, and set the stage for constraints in the next stage.

[0107] S50, Biomechanical Regularization Constraint: Imposing physical constraints on individual predictions by introducing a weighted combination of biomechanical regularization loss functions to constrain the physical feasibility of the generated sequence, ensuring that the predicted motion conforms to human anatomy and dynamics, thereby avoiding unreasonable artifacts such as joint extension, twisting, shaking, teleportation or collision.

[0108] Specifically, the biomechanical regularization loss function... For weighted combinations: ;

[0109] in, This represents the i-th regular term; Indicates weight;

[0110] Weight Through globally learnable vectors Soft assignment is achieved through gradient descent optimization and batch statistics-based MLP output. This weighting mechanism ensures all regularization terms are non-negative and sum to 1, avoiding scale explosion or instability caused by negative weights. It also supports visual monitoring and cross-dataset adaptation (e.g., automatically adjusting weights on datasets with different units or scales). The improvement lies in that this soft assignment mechanism allows the model to adaptively emphasize specific constraints during training, improving generalization ability, and "pulling" the predicted sequence back into the physically feasible region in the early training phase, reducing error accumulation and out-of-distribution collapse during the sampling phase.

[0111] Instead of measuring only on real ground data, regularization terms are applied directly to the denoised output sequence, thereby constraining physical properties in real time during the generation process.

[0112] The regularization terms include bone length consistency, bone length ratio, bone vector angle, velocity smoothing, velocity upper limit, relative distance and collision, and bone constraint combination. The combination of these regularization terms ensures comprehensive constraints on the predicted sequence at the anatomical (bone length, angle) and dynamic (velocity, collision) levels, which are directly applied to the denoised output sequence and can effectively "pull back" the predicted physical feasible domain in the early stages of training.

[0113] The bone length consistency effect is to penalize the length drift of the same bone over time, maintain the approximate constancy of bone length in human anatomy, and avoid unreasonable "scaling" or deformation artifacts in the predicted sequence.

[0114] The bone length consistency:

[0115] ;

[0116] in, This represents the bone length vector of the current frame t. To use the bone length vector as a reference, the first frame of the sequence or the overall average bone length is usually taken as the benchmark to ensure temporal consistency.

[0117] The bone length ratio is used to characterize the relative proportional deviation between the predicted bone length and the actual bone length on the ground by using the logarithmic bone length difference. It maintains scale invariance when changing across datasets or units, so that the overall scaling is converted into translational deviation, which is then canceled out by the difference operation, thus avoiding the generalization problem caused by absolute scale sensitivity.

[0118] The bone length ratio: ;

[0119] in, This represents the bone length vector of the predicted sequence. The vector represents the bone length of the ground truth sequence, and the logarithmic operation ensures the relativity of the ratio.

[0120] The role of bone vector angle is to minimize the angle between the predicted bone vector and the actual bone vector on the ground, suppress unreasonable joint folding, twisting or abnormal bending, and ensure that movement conforms to the biomechanical constraints of the human joint range of motion.

[0121] Bone vector angle: ;

[0122] in, and The bone vectors of the predicted sequence and the ground truth sequence are used respectively. The cosine similarity is calculated using the dot product and norm. Converted to angle, It is a small positive constant.

[0123] The speed smoothing effect uses second-order difference as a smoothing prior to ensure the continuity of the trajectory on the second derivative, reduce unnatural dynamics such as jitter, "sudden braking" or "sudden turning" in the predicted sequence, and improve the smoothness and realism of the motion.

[0124] Speed ​​smoothing: ;

[0125] in, The second-order difference of the velocity in frame t is obtained through two consecutive difference operations. ,in Let be the joint velocity vector of frame t.

[0126] The function of the velocity upper limit is to apply hinge loss to the portion of the frame-by-frame velocity norm that exceeds the threshold, suppressing unreasonable "teleportation" or overspeed motion, and ensuring that the predicted sequence conforms to the physiological velocity limit of human movement.

[0127] Speed ​​limit:

[0128] in, This represents the joint velocity vector in frame t. Its Euclidean norm, The preset speed limit threshold, Ensure that only the portion exceeding the threshold is penalized.

[0129] Relative distance and collision: The function is to constrain the consistency of the relative distance matrix between joints, while preventing joints or limbs from intersecting and colliding, and ensuring the spatial physical rationality in multi-person or single-person sequences.

[0130] Relative distance and collision: ;

[0131] in, and These are the distance matrices for the predicted sequence and the ground truth sequence, respectively; the matrix elements are the Euclidean distances between joints. The first part minimizes the matrix bias, and the second part is the hinge loss penalty for the negative distance. This is the minimum allowed distance threshold.

[0132] Bone restraint combination: The purpose of this term is to combine the first three skeletal-related constraints into a joint regularization term, further enhancing the overall stability of the skeletal structure and avoiding local artifacts caused by insufficient individual constraints.

[0133] Total loss:

[0134] ;

[0135] in, ; This represents an adjustable hyperparameter used to control the biomechanical regularization loss. The weighting proportion in the total loss. It is a positive real number, typically determined during training through grid search, cross-validation, or auto-tuning (e.g., 0.1 to 1.0) to balance generation quality and physical feasibility.

[0136] S60, synthesizes the population sequence and outputs the predicted sequence.

[0137] The key point of this invention is to apply the residual noise diffusion model to the field of group behavior prediction. Through the organic integration of group decomposition, frequency domain processing, residual noise learning and biomechanical constraints, it achieves efficient, diverse, continuous and physically reasonable prediction of multi-person movement sequences.

[0138] This invention decomposes multi-person motion sequences into the independent trajectories of individual individuals for prediction, reducing computational complexity from quadratic to linear (scaling linearly with the number of individuals P). After prediction, it implicitly captures multi-person interaction dynamics through shared context aggregation. The key advantage of this strategy lies in its first application of this decomposition method to a residual noise diffusion framework, achieving scalability and real-time performance in group prediction. Existing group prediction techniques often rely on complex graph networks or global modeling, leading to inefficiency. Compared to existing single-person prediction methods, this invention extends to multi-person scenarios, avoiding the explosive complexity of interaction modeling.

[0139] This invention transforms group behavior prediction into a mask completion task in the frequency domain. It uses Discrete Cosine Transform (DCT) to compress skeletal data into low-frequency coefficients (preserving K < F components) and applies mask noise to future segments during inference for completion. The innovation lies in integrating frequency domain completion into the group prediction framework, supporting smooth generation of multi-person sequences and rare category switching (e.g., switching from group walking to dispersed sitting). Unlike existing time-domain prediction methods, this invention suppresses high-frequency jitter through frequency domain operations, improving prediction continuity, and requires only a single end-to-end loss optimization.

[0140] This invention introduces residual terms into the forward diffusion process, redefining the conditional distribution and enabling the inverse process to be initiated directly from noise-degraded observations. The core innovation lies in applying this residual diffusion mechanism to group behavior prediction for the first time, bridging the gap between multi-person observation data and the predicted sequence, significantly reducing the number of sampling steps (to 5-20 steps), and improving the predictability and controllability of multi-person interactions. Unlike existing diffusion models in image restoration, this invention extends to dynamic group sequences, addressing the discontinuities and efficiency issues in multi-person prediction.

[0141] This invention defines weighted residual noise as the objective, minimizes the expected loss, and performs learning conditionally based on multi-person masked observations. The innovation lies in applying residual noise learning to population prediction, supporting continuous completion of multi-person sequences and diverse class switching. Unlike existing separate noise / residual prediction methods, this invention unifies the training / inference process, improving the generalization ability of population dynamics.

[0142] This invention determines the optimal step through a smooth equivalent transformation, enabling direct reuse of noisy scheduling and accelerated multi-person sampling. Its key advantage lies in being the first to apply this transformation to group behavior prediction, eliminating biases in multi-person sequences and unifying the preceding and following processes, supporting real-time multi-agent generation. Compared to existing technologies that require redesigning scheduling, this invention simplifies the implementation of group prediction.

[0143] This invention designs a weighted loss, including bone length consistency, ratio, angle, velocity smoothing, upper limit, relative distance collision, and combination terms, which are directly applied to the multi-person denoising output. The innovation lies in the first-time integration of biomechanical constraints into the population prediction framework. Adaptive weights (global + batch MLP) ensure the physical plausibility of multi-person interactions and prevent collision / scaling artifacts. Unlike existing methods lacking multi-person physical constraints, this invention improves the realism and adaptability of population sequences.

[0144] This invention has achieved significant technical effects and advantages in the field of group behavior prediction, mainly reflected in prediction accuracy, diversity, physical rationality, sampling efficiency, and generalization ability. These effects are verified through experimental comparison data on standard benchmark datasets, and further illustrated by theoretical derivation and analysis of methodological characteristics. The standard datasets used in the experiments include:

[0145] MuPoTS is a multi-person motion tracking and segmentation dataset containing synchronized motion sequences of multiple individuals in complex scenes. The data originates from a video capture system. The dataset provides annotated 2D / 3D skeletal joint positions, covering common group behaviors such as standing, walking, and interaction. It records at 25 fps and contains approximately 2000 sequences. In our experiments, we used a fully annotated subset (approximately 1500 sequences) for training and testing, setting the number of observation frames as the baseline and the number of prediction frames as 100 frames (approximately 4 seconds). This dataset is suitable for evaluating the accuracy (ADE, FDE) and diversity (APD) of multi-person predictions, and is appropriate for validating the performance of this invention in group interaction scenarios.

[0146] CMU-Mocap is a motion capture dataset containing high-precision 3D human motion data recorded by an optical motion capture system. It includes approximately 2600 sequences of multiple individuals performing tasks such as dance, movement, and daily activities, at a frame rate of 120 fps. In our experiments, we downsampled to 25 fps, using approximately 2000 sequences, observing 25 frames, and predicting 100 frames. This dataset emphasizes the continuity and physical plausibility of multi-person collaborative movements, making it suitable for evaluating MPJPE and FPD metrics, and ideal for verifying the biomechanical regularization effect of this invention.

[0147] Human3.6M is a large-scale single-person motion dataset generated by seven professional actors performing 11 activities (including walking, eating, and greeting) in an indoor scene. It contains approximately 3.6 million frames of 3D human pose data at a frame rate of 50 fps. In our experiments, we downsampled to 25 fps, using training data (approximately 2.5 million frames) from five themes (S1, S5, S6, S7, and S8), and the remaining themes as test data. The dataset consisted of 25 observation frames and 100 prediction frames. This dataset provides high-precision ground-based data, suitable for evaluating the accuracy (ADE, FDE) and multimodal performance (MMADE, MMFDE) of single-person predictions, and serves as a key benchmark for validating the single-person prediction advantages of this invention.

[0148] HumanEva-I is a small but high-precision single-person motion dataset containing synchronized video and motion capture data for six actions (walking, running, punching, etc.) across three themes (S1, S2, S3), totaling approximately 40 sequences at a frame rate of 60fps. In our experiments, we downsampled to 25fps, observing 25 frames and predicting 100 frames. This dataset emphasizes the dynamic details and physical constraints of actions, making it suitable for evaluating action diversity (APD) and distribution similarity (FPD), and is an ideal choice for testing the generalization ability of this invention on complex single-person actions.

[0149] These datasets are all public benchmarks containing diverse multi-person / single-person motion sequences. Experimental conditions were: observation frames O=25 (Human3.6M and HumanEva-I) or equivalent benchmark settings, prediction frames FO=100 (2 seconds), and the model was trained for 1000 epochs on an NVIDIA RTX 4090 GPU using the PyTorch framework with the AdamW optimizer and a learning rate of 1e-4. Evaluation metrics included Average Displacement Error (ADE), Final Displacement Error (FDE), Multimodal Average Displacement Error (MMADE), Multimodal Final Displacement Error (MMFDE), Average Pairwise Distance (APD, for diversity), and Average Joint Position Error (MPJPE), as well as Fraser Distance (FPD, for distribution similarity).

[0150] Experimental results show that the MP-ResDiff model of this invention outperforms or approaches existing methods in all metrics, as detailed below:

[0151] On the multi-user datasets MuPoTS and CMU-Mocap, this invention demonstrates superior performance in ADE, FDE, and FPD metrics. For example, in the 1-second, 2-second, and 3-second predictions on the MuPoTS dataset, the ADE scores are 5.59, 6.71, and 8.24 (outperforming DuMMF's 4.05, 6.78, and 9.05, etc.), the FDE scores are 6.22, 8.38, and 16.58 (outperforming Multi-Range Transformers' 7.49, 13.77, and 20.59), and the FPD scores are 6.01, 10.85, and 18.5 (outperforming the baseline). On the MPJPE metric, the scores for 1-second, 2-second, and 3-second predictions are 9.0, 15.6, and 20.1 (outperforming EMPMP's 9.2, 16.0, and 20.1, respectively). These data indicate that this invention improves accuracy and distribution similarity in both short-term and long-term predictions, with an average improvement of 10%-20%. In theory, this is due to the fact that the residual noise diffusion process is started directly from the noise degradation observation, reducing the number of sampling steps (only 5-20 steps vs. the traditional 50-100 steps), and ensuring continuity through DCT frequency domain completion.

[0152] On the single-person datasets Human3.6M and HumanEva-I, this invention leads in APD, ADE, FDE, MMADE, and MMFDE metrics. For example, on Human3.6M, ADE and FDE are 0.308 and 0.478, respectively (superior to HumanMAC's 0.369 and 0.480), and MMADE and MMFDE are 0.511 and 0.561, respectively (superior to LDPM's 0.430 and 0.473). On HumanEva-I, APD is 7.633 (higher than the baseline 6.554, indicating better diversity), and ADE and FDE are 0.214 and 0.264, respectively (superior to the baseline 0.209 and 0.223). Average improvements of 5%–15% are achieved, particularly in diversity (16% improvement in APD) and multimodal accuracy. Biomechanical regularization constraints (such as bone length consistency and velocity smoothing) ensure physical feasibility, and theoretical analysis shows that these terms are achieved through weighted loss. Pulling back the prediction to a reasonable domain early in training reduces artifacts.

[0153] Furthermore, this invention boasts high sampling efficiency, requiring only 20-50 steps of DDIM sampling (vs. traditional 1000 steps), with inference latency scaling linearly with the number of individuals P. It exhibits strong generalization ability, maintaining a leading position in cross-dataset validation. Overall, this invention achieves efficient, accurate, and reasonable population prediction through the integration of residual diffusion, frequency domain completion, and biomechanical constraints, making it suitable for applications such as animation and autonomous driving.

[0154] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A video human behavior prediction method based on residual diffusion theory and skeletal points, characterized in that, Including the following steps: S10, Obtain the original group motion sequence, decompose the multi-person sequence into individual trajectories, and then convert the time-domain motion sequence of each individual into frequency coefficients through Discrete Cosine Transform (DCT). S20, Construct the diffusion process: Based on the frequency domain representation, introduce residual terms to construct the forward diffusion process; S30, Residual Noise Learning: Define weighted residual noise as the learning target to achieve the conversion from noise to residual noise; S40, Smooth Equivalent Transformation Acceleration: Determines the optimal step through transformation, thereby accelerating and unifying training inference; S50, Biomechanical Regularization Constraint: Imposing physical constraints on individual predictions by introducing a weighted combination of biomechanical regularization loss functions to constrain the physical feasibility of the generated sequences; S60, synthesizes the population sequence and outputs the predicted sequence; The forward diffusion process, which involves introducing residual terms, includes: Residual terms: ; in, Represents the true ground frequency coefficient. Indicates degenerate input; Forward diffusion process incorporates R: ； Where: probability distribution ; ； in, Indicates given initial clean data Given the residual term R, the entire noise sequence from time step 1 to T The joint probability distribution of z; t z represents the noisy data vector at time step t in the diffusion process. t-1 This represents the noisy data vector from the previous time step t-1; α t This represents the noise scheduling parameter, which controls the proportion of noise added at each step, α. t Within the range (0,1), it decreases as t increases, ensuring that noise gradually dominates; β t represents the noise variance scheduling parameter, the noise intensity added at time step t, in the range (0,1); I represents the identity matrix, used to define the covariance matrix of the Gaussian distribution, indicating that the noise is independent in each dimension; Reparameterization yields: ； in, α represents the time step from time step 1 to t. S The cumulative product is used to reparameterize the entire process; α s s represents the noise scheduling parameters under index s; z0 represents the true frequency coefficients, i.e., the clean data vector at the beginning of the diffusion process; It is a small positive number; Substitute: ; Wherein, the weighted residual noise : ； Minimize the training objective: ； in, Due to the condition of the mask U-Net predictions.

2. The video human behavior prediction method based on residual diffusion theory and skeletal points according to claim 1, characterized in that, The time-domain motion sequence of each individual is converted into frequency coefficients using Discrete Cosine Transform (DCT). The transformation formula is as follows: ； in, For frequency coefficients indexed by frequency k, Let k be the temporal motion sequence of each individual with time index t; t is the time index, ranging from 1 to F, representing the frame position in the temporal sequence; k is the frequency index, ranging from 1 to F, representing the frequency components in the frequency domain; F is the total number of frames in the entire motion sequence, i.e., the sum of the observed frame number O and the predicted frame number (F - O); within the frame, the observed past O frames and the predicted future (F - O) frames are integrated into a complete sequence with a total length of F; in practice, the first K < F low-frequency components are retained for compressed representation, while the basic motion energy is preserved.

3. The video human behavior prediction method based on residual diffusion theory and skeletal points according to claim 1, characterized in that, During the acceleration process of the smooth equivalent transformation: ； in, This is the noisy data vector at the optimal acceleration time step T′ of the diffusion process; it represents the intermediate noisy state of the frequency domain motion sequence. is the cumulative noise scheduling parameter, which is the cumulative scaling factor at time step T′.

4. The video human behavior prediction method based on residual diffusion theory and skeletal points according to claim 1, characterized in that, The biomechanical regularization loss function For weighted combinations: ; in, This represents the i-th regular term; Indicates weight; Instead of measuring only on real ground data, regularization terms are applied directly to the denoised output sequence, thereby constraining physical properties in real time during the generation process.

5. The video human behavior prediction method based on residual diffusion theory and skeletal points according to claim 4, characterized in that, The regularization terms include bone length consistency, bone length ratio, bone vector angle, velocity smoothing, velocity limit, relative distance and collision, and bone constraint combination; The bone length consistency effect is to penalize the length drift of the same bone over time and maintain the approximate constancy of bone length in human anatomy. The bone length ratio is used to characterize the relative proportional deviation between the predicted bone length and the actual bone length on the ground by the logarithmic bone length difference. It maintains scale invariance when changing across datasets or units, so that the overall scaling is converted into translational deviation, which is then canceled out by the difference operation. The function of bone vector angle is to minimize the angle between the predicted bone vector and the actual bone vector on the ground; The velocity smoothing effect uses second-order differences as smoothing priors to ensure the continuity of the trajectory on the second derivative; The velocity capping function applies hinge loss to the portion of the frame-by-frame velocity norm that exceeds a threshold. Relative distance and collision: The function is to constrain the consistency of the relative distance matrix between joints, while preventing joints or limbs from intersecting and colliding, and ensuring the spatial physical rationality in multi-person or single-person sequences; Bone constraint combination: Combine the first three bone-related constraints to form a joint regularization term.

6. The video human behavior prediction method based on residual diffusion theory and skeletal points according to claim 5, characterized in that, The bone length consistency: ; in, This represents the bone length vector of the current frame t. For reference bone length vector, The bone length ratio: ; in, This represents the bone length vector of the predicted sequence. Represents the bone length vector of the actual ground sequence; Bone vector angle: ; in, and The bone vectors of the predicted sequence and the ground truth sequence are used respectively. The cosine similarity is calculated using the dot product and norm. Converted to angle, It is a small positive number; Speed ​​smoothing: ; in, The second-order difference of the velocity in frame t is obtained through two consecutive difference operations. ,in Let be the joint velocity vector of frame t. Let be the joint velocity vector of frame t-1. Let be the joint velocity vector for frame t-2; Speed ​​limit: ; in, This represents the joint velocity vector in frame t. Its Euclidean norm, The preset speed limit threshold, Ensure that only the portion exceeding the threshold is penalized; Relative distance and collision: ; in, and These are the distance matrices for the predicted sequence and the ground truth sequence, respectively; the matrix elements are the Euclidean distances between joints. The first part minimizes the matrix bias, and the second part is the hinge loss penalty for the negative distance. The minimum allowable distance threshold; Bone restraint combination: .

7. The video human behavior prediction method based on residual diffusion theory and skeletal points according to claim 6, characterized in that, Total loss: ； in, ; It is an adjustable hyperparameter.

Images