Diversified human motion prediction system and method based on multi-level feature extraction

Abstract

The application provides a diversified human motion prediction system and method based on multi-level feature extraction, and relates to the technical field of computer vision and image processing.The application adopts a deep learning module based on a Transformer and a GCN, adopts a multi-layer Transformer module to further optimize motion samples, and according to the grouping condition of human body joints, a Transformer independent generator on each layer is responsible for calculating the feature map of one or more human body joints.Such a design makes each Transformer module only focus on the feature information of a specific part of the body, and the self-attention mechanism can capture the connection and difference between the key points of the specific part.In addition, the graph convolution network can effectively capture the feature information between nodes.The combined network of the two can bring better improvement effect.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and relates to a diversified human motion prediction system and method based on multi-level feature extraction. Background Technology

[0002] Human motion prediction is the process of predicting future human movements based on 3D human pose information from a past time series. This field holds a crucial position in computer vision as a method for understanding and simulating human behavior. Human motion prediction plays a vital role in human-computer interaction, animation creation, and autonomous driving. However, the complexity and subjectivity of human movements, the multiple potential possibilities of human actions, and the difficulty in obtaining prior knowledge of future human intentions pose significant challenges to human motion prediction.

[0003] Human motion prediction primarily follows two research approaches: deterministic motion prediction and diverse motion prediction. Deterministic motion prediction only predicts the most likely future motion, without considering the diversity of future actions. This research performs exceptionally well in short-term predictions, but in long-term predictions, because human motion is subject to subjective control and can exhibit diversity, deterministic motion prediction struggles to achieve ideal results. Diverse human motion prediction aims to predict a variety of possible future actions. Comparatively, diverse motion prediction is more challenging in long-term predictions. Diverse motion prediction has significant research value in many safety-related applications. For example, in autonomous driving applications, the multiple possibilities of pedestrian actions that need to be monitored can be calculated using diverse motion prediction methods to determine the possible range of pedestrian movement, thereby ensuring pedestrian safety.

[0004] Most existing work focuses on using deep generative models to effectively learn data distributions, such as using variational autoencoders (VAEs) and adversarial neural networks to model multimodal data distributions. However, these previous works have struggled to achieve a high level of both diversity and accuracy simultaneously. In fact, these two metrics can easily influence and constrain each other; for example, improved accuracy may lead to a decrease in diversity. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, this invention proposes a diversified human motion prediction system and method based on multi-level feature extraction. The diversified human motion prediction system can further improve both the diversity and accuracy of diversified motion prediction, optimize the mutual constraints between diversity and prediction accuracy, and maintain both accuracy and diversity at a high level overall.

[0006] The technical solution of this invention is as follows:

[0007] A diversified human motion prediction system based on multi-level feature extraction includes a preprocessing module, a training module, and a motion prediction module.

[0008] The preprocessing module extracts a 2.5-second 3D human motion sequence from the Human3.6M training data at the original frame rate (50Hz). It divides the pose data of the first 25 frames (0.5 seconds) of the 3D human motion sequence as past time series and divides the pose data of the last 100 frames (2 seconds) of the 3D human motion sequence as labels for predicting future motion poses. Then, it randomly generates two sets of latent variables using a Gaussian distribution.

[0009] In the Human3.6M public dataset, the positional data of human motion exists in the form of 17 human body 3D joint coordinates, namely: Hip, Left Knee, Left Ankle, Left Foot, Right Knee, Right Ankle, Right Foot, Spine, Neck, Head, Left Shoulder, Left Elbow, Left Wrist, Left Hand, Right Shoulder, Right Elbow, and Right Wrist.

[0010] The training module uses the past time-series pose data and latent variables processed in the preprocessing module as input to the diversified human motion prediction deep neural network. This diversified human motion prediction deep neural network mainly consists of three generators: an initialization generator G1, a two-stage generator G2, and a two-layer Transformer generator G3. The initialization generator G1, composed of multiple graph convolutional layers, is used to perform the first stage of feature extraction on the past motion sequences and generate an initial feature map X1. The initial feature map X1 is concatenated with the sum of two sets of latent variables. The concatenated feature map is then input to the two-stage generator G2, which is constructed from multiple graph convolutional layers and outputs the second-stage feature map X2. The 17 joints are divided into upper body and lower body groups. Upper body joints include: spine, neck, head, left shoulder, left elbow, left wrist, left hand, right shoulder, right elbow, and right wrist. Lower body joints include: hip, left knee, left ankle, left foot, right knee, right ankle, and right foot. A two-stage feature map X2 is input into a two-layer Transformer generator G3. In G3, each layer uses an independent generator G3 to extract personalized features from the corresponding joint group. After each independent generator completes its calculation, it only updates the feature map of the corresponding joint group and then updates the feature map. Proceed to the next level of calculation. The final layer's output... The final future motion sequence is synthesized by combining the two-stage feature map X2. The output of the deep neural network for predicting diverse human motion is a complete sample of future motions containing N possibilities, where the number N can be arbitrarily set. For the loss function, a defined reconstruction error function is used, requiring only that one of the future motions be as close as possible to Y. g Y g This represents the pose data of frames (2 seconds) following the division of 100 3D human motion sequences in the training data used as labels; the reconstruction loss function. Defined as: The loss function uses Euclidean distance as the distance metric D; k is a variable representing one of the N possible future motion samples generated. To better constrain other predicted actions, constraints are also imposed on other predicted motions. A method based on searching for similar past motion samples using a distance threshold is proposed, and future motion labels with a certain degree of similarity to past motions are statistically grouped into... As pseudo-labels, where Q represents the number of pseudo-labels. The loss for other predicted motions is expressed as: To improve the diversity of motion predictions, the enhanced diversity loss was revised to be specific to each body part. The enhanced diversity loss can be expressed as: The total loss function is defined as: Where λ d , λ r and λ mm These represent the hyperparameters corresponding to the loss function.

[0011] The motion prediction module is used to load the diverse human motion prediction model trained by the training module, input the required past time series 3D human posture data into the trained diverse human motion prediction model network model, and obtain N possible future time series human motion samples.

[0012] The method for predicting diverse human movements using the above system includes the following steps:

[0013] Step 1: Preprocess the 3D pose estimation dataset to obtain past motion sequences and future motion sequences with coordinate training labels as training data, and generate latent variables through Gaussian distribution.

[0014] Step 2: Input the 3D human posture information from past time series and latent variables into a deep neural network for predicting diverse human motion, and finally output N possible future motion sequences.

[0015] Step 3: Load the model saved after training in Step 2, input the pose of a past time series into the trained network, and obtain motion pose data in various future sequences.

[0016] Furthermore, step 1 specifically includes:

[0017] The training dataset was constructed using data from the Human3.6M public dataset, specifically frames S1, S5, S6, S7, and S8, while frames S9 and S11 were used as the test dataset. From the Human3.6M training data, a 2.5-second 3D human motion sequence was extracted at the original frame rate (50Hz). The pose data of the first 25 frames (0.5 seconds) of each 3D human motion sequence were used as past time series, and the pose data of the last 100 frames (2 seconds) of each 3D human motion sequence were used as labels for predicting future motion poses. Two sets of latent variables were then randomly generated using a Gaussian distribution.

[0018] Furthermore, step 2 specifically includes:

[0019] Step 2.1: The initialization generator G1 module in the deep neural network for predicting diverse human motion performs the first stage of feature extraction on the past motion sequence Q to generate the initial feature map X1; the initialization generator G1 is defined as a GCN network with multiple graph convolutional layers, which can be formally represented as:

[0020] X1=G1(X) (1)

[0021] Step 2.2: Merge the latent variables from different normal distributions, then concatenate the initial feature map X1 with the latent variables. The concatenated tensor is passed through a two-stage generator G2 to output the two-stage feature map X2. The two sets of latent variables can be represented as [Z 1 Z 2 The entire process can be represented as:

[0022]

[0023] Step 2.3: Divide the 17 joints into two groups: upper body and lower body. The upper body joints include: Spine, Neck, Head, Left Shoulder, Left Elbow, Left Wrist, Left Hand, Right Shoulder, Right Elbow, and Right Wrist. The lower body joints include: Hip, Left Knee, Left Ankle, Left Foot, Right Knee, Right Ankle, and Right Foot.

[0024] The two-stage feature map X2 is input into a two-layer Transformer generator G3. In the two-layer Transformer generator G3, each layer uses an independent generator G3 to extract personalized features from the corresponding keypoint group, thereby improving the diversity and accuracy of sampling. This invention integrates the generators of each layer... Designed as a Transformer-based encoder, human keypoints are explicitly embedded as tokens to learn the constraints between them. The Transformer encoder learns keypoints and temporal features; each module of the Transformer encoder contains a multi-head self-attention module (MSA) and a multilayer sensor module (MLP), and employs the LayerNorm normalization method. The multi-head self-attention module (MSA) is an extension of the self-attention module (SA) with h "heads", represented as:

[0025]

[0026]

[0027] Where i∈{1,2}, These are the learnable parameters of the three linear projection layers; The weight parameters are: d is the dimension of the module's hidden layer, h represents the number of multi-head attention "heads", β is a parameter, and in practical applications, ε is set to...

[0028] In the two-layer Transformer generator G3, the computation results of each layer only update the feature map of the corresponding joint. Then it will proceed to the next layer to continue calculating the next group of joints. The output of the last layer is... Combine X2 with the final future action sequence. This series of operations is defined as:

[0029]

[0030]

[0031]

[0032] Step 2.4: Regarding the loss function, a defined reconstruction error function was used, which only requires that one of the future motions be as close as possible to the label Y. g The loss function for reconstruction Defined as:

[0033]

[0034] The loss function uses Euclidean distance as the distance metric D, Y. g This represents the pose data of the last frame (2 seconds) of the training data divided into 100 3D human motion sequences, which are used as labels; k is a variable representing one of the N possible future motion samples generated.

[0035] To better constrain other predicted actions, constraints should also be imposed on other predicted movements. A method based on searching for similar past motion samples using a distance threshold is proposed, and future motions with a certain degree of similarity to past movements are statistically labeled. As pseudo-labels, where Q represents the number of pseudo-labels. The loss is calculated for other predicted actions. This is represented as shown in formula (9):

[0036]

[0037] For each pseudo-tag, The constraint generator must have at least one motion sample to cover. To improve the diversity of predicted motions, diversity-enhanced loss is used. As shown in formula (10):

[0038]

[0039] Where σ is the normalization factor and N represents the number of future motion samples generated.

[0040] Minimizing the Euclidean distance between predicted samples promotes greater diversity in the predicted samples. To improve the diversity of motion predictions, the enhanced diversity loss is reformulated to be specific to each body part. For i∈{1,2}, the enhanced diversity loss can be expressed as:

[0041]

[0042] The total loss function is defined as follows:

[0043]

[0044] Where, λ d , λ r and λ mm These represent the hyperparameters corresponding to the loss function.

[0045] Furthermore, step 3 specifically includes:

[0046] Load the model_best trained in step 2, input the past time series pose data and latent variables into the model, and output the predicted motion pose data for multiple future time series and the corresponding evaluation metrics.

[0047] The beneficial effects of this invention are as follows: This invention employs a deep learning module based on Transformer and GCN, using multi-layered Transformer modules to further optimize motion samples. Based on the grouping of human joints, each Transformer layer's independent generator is responsible for calculating feature maps of one or more human joints. This design allows each Transformer module to focus solely on the feature information of a specific body part, with a self-attention mechanism automatically capturing the connections and differences between key points in that specific part. Furthermore, graph convolutional networks can effectively capture feature information between nodes. The combination of these two technologies results in a significantly improved network performance. Attached Figure Description

[0048] Figure 1 This is a flowchart of the diversified human motion prediction method based on multi-level feature extraction described in this embodiment of the invention.

[0049] Figure 2 This is a framework diagram of the deep neural network for predicting diverse human motion based on multi-level feature extraction as described in this embodiment of the invention.

[0050] Figure 3 This is a schematic diagram of a single-layer Transformer in an embodiment of the present invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0052] This embodiment uses Windows as the development environment, PyCharm as the development platform, and Python as the development language. It employs the multi-level feature extraction-based diversified human motion prediction method of this invention to achieve predictions of multiple possibilities for future human motion.

[0053] like Figure 1 As shown, a method for predicting diverse human motion based on multi-level feature extraction includes the following steps:

[0054] Step 1: Preprocess the 3D pose estimation dataset to obtain past motion sequences as training data and future motion sequences with coordinate training labels. This includes the following steps:

[0055] The training dataset was constructed using data from the Human3.6M public dataset, specifically frames S1, S5, S6, S7, and S8, while frames S9 and S11 were used as the test dataset. From the Human3.6M training data, a 2.5-second 3D human motion sequence was extracted at the original frame rate (50Hz). The pose data of the first 25 frames (0.5 seconds) of each 3D human motion sequence were used as past time series, and the pose data of the last 100 frames (2 seconds) of each 3D human motion sequence were used as labels for predicting future motion poses. Two sets of latent variables were then randomly generated using a Gaussian distribution.

[0056] Step 2: Input the past time series pose data and latent variable z into the deep neural network for predicting diverse human motions and train it. Use multiple loss functions that measure diversity and accuracy. After training, obtain the model weights for predicting diverse human motions. The overall network framework is as follows: Figure 2 As shown; specifically including:

[0057] Step 2.1: The initialization generator G1 module in the deep neural network for predicting diverse human motion performs the first stage of feature extraction on the past motion sequence X to generate the initial feature map X1; the initialization generator G1 is defined as a GCN network with multiple graph convolutional layers, as shown in formula (1):

[0058] X1=G1(X) (1)

[0059] Step 2.2: Merge the latent variables with different normal distributions, then concatenate the initial feature map X1 with the latent variables. The concatenated tensor is passed through a two-stage generator G2 composed of multiple graph convolutional layers to output the two-stage feature map X2. The two sets of latent variables can be represented as [Z 1 Z 2 The entire process can be represented as:

[0060]

[0061] Step 2.3: Divide the 17 joints into two groups: upper body and lower body. The upper body joints include: Spine, Neck, Head, Left Shoulder, Left Elbow, Left Wrist, Left Hand, Right Shoulder, Right Elbow, and Right Wrist. The lower body joints include: Hip, Left Knee, Left Ankle, Left Foot, Right Knee, Right Ankle, and Right Foot. The two-stage feature map X2 is input into the multi-layer Transformer generator G3.

[0062] In the two-layer Transformer generator G3, each layer uses an independent generator G3 to extract personalized features from the corresponding keypoint group, thereby improving the diversity and accuracy of sampling. This invention integrates the generator G3 of each layer... (i) Designed as a Transformer-based encoder, human keypoints are explicitly embedded as tokens to learn the constraints between them. The Transformer encoder learns keypoints and temporal features; each module of the Transformer encoder contains a multi-head self-attention module (MSA) and a multilayer sensor module (MLP), and employs the LayerNorm normalization method. The multi-head self-attention module (MSA) is an extension of the self-attention module (SA) with h "heads", represented as:

[0063]

[0064]

[0065] in These are the learnable parameters of three linear projection layers. The weight parameters are represented by d, which is the dimension of the module's hidden layer, and h, which represents the number of multi-head attention "heads". β is a parameter; in practical applications, ε is set to...

[0066] In the two-layer Transformer generator G3, the computation results of each layer only update the feature map of the corresponding joint. Then it will proceed to the next layer to continue calculating the next group of joints. The output of the last layer is... Then combine it with X2 to synthesize the final future action sequence. This series of operations is defined as:

[0067]

[0068]

[0069] Step 2.4: Output of the last layer Then combine it with X2 to form the final future action sequence, as shown below:

[0070]

[0071] Step 2.5: Use the preprocessed past time series pose data and latent variables as input to train the deep neural network. For the loss function, a defined reconstruction error function is used, requiring that one of the future motions be as close as possible to the pose data Y of the last frame (2 seconds) of the 100 3D human motion sequences divided into labeled training data. g The loss function for reconstruction The definition is shown in formula (8):

[0072]

[0073] The loss function uses Euclidean distance as the distance metric D, and k is a variable representing one of the N possible future motion samples generated.

[0074] To better constrain other predicted actions, constraints should also be imposed on other predicted movements. A method based on searching for similar past motion samples using a distance threshold is proposed, and future motions with a certain degree of similarity to past movements are statistically labeled. As pseudo-labels, where Q represents the number of pseudo-labels. The loss is calculated for other predicted actions. This is represented as shown in formula (9):

[0075]

[0076] For each pseudo-tag, The constraint generator must have at least one motion sample to cover it. To improve the diversity of predicted motions, a diversity-enhancing loss is used. As shown in formula (10):

[0077]

[0078] Where σ is the normalization factor and N represents the number of future motion samples generated.

[0079] Minimizing the Euclidean distance between predicted samples promotes greater diversity in the predicted samples. To improve the diversity of motion predictions, the enhanced diversity loss is reformulated to be specific to each body part. The enhanced diversity loss can be expressed as:

[0080]

[0081] The total loss function is defined as shown in formula (12):

[0082]

[0083] Where λ d ,λ r and λ mm These represent the hyperparameters corresponding to the loss function.

[0084] Step 4: Load the model trained and saved in Step 3, input the pose of a past time series into the trained network, and obtain motion pose data in various future sequences;

[0085] Specifically, load the model_best trained in step 3, input the past time series pose data and latent variables into the model, and output the predicted motion pose data for multiple future time series and the corresponding evaluation metrics.

[0086] Evaluation metrics: (1) To measure the diversity of predicted motion, Average Pairwise Distance (APD) is used: the average L2 distance between all motion sample pairs is calculated and defined as: (2) To measure the reconstruction accuracy of the entire sequence, Average Displacement Error (ADE) is used: This is based on the actual ground motion Y... g The average L2 distance between the nearest sample and the nearest sample over all time steps, defined as: (3) To measure the reconstruction accuracy of the last future pose, the Final Displacement Error (FDE) is used: the final true ground pose Y. g The L2 distance between [T] and the final pose of the closest sample is defined as: min i ∥Y i [T]-Y g [T]∥2. (4) Multi-Modal ADE (MMADE): The multimodal version of ADE. (5) Multi-Modal FDE (MMFDE): The multimodal version of FDE.

[0087] In summary, APD (Advanced Perspective Distributed ...

[0088] Table 1

[0089]

[0090] Based on the above steps, the model in this invention is compared with the ERD model, acLSTM model, Pose-Knows model, MT-VAE model, HP-GAN model, BoM model, GMVAE model, DeLiGAN model, DSF model, DCT model, DLow model, and GSPS model. As shown in Table 1, on the public datasets Human3.6M and HumanEva-I, the method proposed in this invention outperforms the other methods in most metrics.

[0091] Corresponding to the method for predicting diverse human motions based on multi-level feature extraction in this invention, this invention also provides a system for predicting diverse human motions based on multi-level feature extraction, including a preprocessing module, a training module, and a motion prediction module.

[0092] Preprocessing Module: In the Human3.6M public dataset, this system uses data from S1, S5, S6, S7, and S8 as the training dataset, and S9 and S11 as the test dataset. In the Human3.6M public dataset, the positional data of human motion exists in the form of 17 human body 3D joint coordinates, namely: Hip, Left Knee, Left Ankle, Left Foot, Right Knee, Right Ankle, Right Foot, Spine, Neck, Head, Left Shoulder, Left Elbow, Left Wrist, Left Hand, Right Shoulder, Right Elbow, and Right Wrist. From the Human3.6M training data, a 2.5-second 3D human motion sequence was extracted at the original frame rate (50Hz). The pose data of the first 25 frames (0.5 seconds) of the 3D human motion sequence were divided as past time series, and the pose data of the last 100 frames (2 seconds) of the 3D human motion sequence were divided as labels for predicting future motion poses. Then, two sets of latent variables were randomly generated using a Gaussian distribution.

[0093] Training Module: The pre-processed past time-series pose data and latent variables are used as input to the diversified human motion prediction deep neural network. The diversified human motion prediction deep neural network mainly consists of three generators: an initialization generator G1, a two-stage generator G2, and a two-layer Transformer generator G3. The initialization generator G1, composed of multiple graph convolutional layers, is used for the first stage of feature extraction from the past motion sequences, generating an initial feature map X1. The initial feature map X1 is concatenated with the sum of two sets of latent variables, and then the concatenated feature map is input to the two-stage generator G2. G2, constructed from multiple graph convolutional layers, outputs the second-stage feature map X2. The 17 joints are divided into two groups: upper body and lower body. Upper body joints include: spine, neck, head, left shoulder, left elbow, left wrist, left hand, right shoulder, right elbow, and right wrist. Lower body joints include: hip, left knee, left ankle, left foot, right knee, right ankle, and right foot. Two-stage feature maps (X2) are input into a multi-layer Transformer generator (G3). In the two-layer Transformer generator G3, each layer uses an independent generator (G3) to extract personalized features from the corresponding joint group. After each independent generator completes its calculation, it only updates the feature map of the corresponding joint group and then updates the feature map. Proceed to the next level of calculation. The final layer's output... The final future motion sequence is synthesized by combining the two-stage feature map X2. The output of the diversified 3D human motion prediction network is a complete sample of future motion containing N possibilities, where the number N can be arbitrarily set. For the loss function, a defined reconstruction error function is used, requiring only that one of the future motions be as close as possible to the pose data Y of the last frame (2 seconds) of the 100 3D human motion sequences divided from the labeled training data. g The loss function for reconstruction Defined as: The loss function uses Euclidean distance as the distance metric D, where k is a variable representing one of the N possible future motion samples generated. To better constrain other predicted actions, constraints should also be imposed on other predicted motions. A method based on searching for similar past motion samples using a distance threshold is proposed, and future motion labels with a certain degree of similarity to past motions are statistically grouped into... As pseudo-labels, where Q represents the number of pseudo-labels. The loss for other predicted motions is expressed as: To improve the diversity of motion predictions, the enhanced diversity loss was revised to be specific to each body part. The enhanced diversity loss can be expressed as: The total loss function is defined as: Where λ d ,λ r and λ mm These represent the hyperparameters corresponding to the loss function.

[0094] Motion prediction module: Used to load the diverse human motion prediction model trained by the training module, input the required past time series 3D human posture data into the trained diverse human motion prediction model network model, and obtain N possible future time series human motion samples.

[0095] For the diverse human motion prediction system based on multi-level feature extraction in this embodiment of the invention, since it corresponds to the diverse human motion prediction method based on multi-level feature extraction in the above embodiment, the description is relatively simple. For related similarities, please refer to the description of the diverse human motion prediction method based on multi-level feature extraction in the above embodiment, which will not be described in detail here.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A diversified human motion prediction system based on multi-level feature extraction, characterized in that, The system includes a preprocessing module, a training module, and a motion prediction module; The preprocessing module extracts a 2.5-second 3D human motion sequence from the Human3.6M training data at the original frame rate, divides the pose data of the first 25 frames of the 3D human motion sequence as past time series, and divides the pose data of the last 100 frames of the 3D human motion sequence as labels for predicting future motion poses. Then, two sets of latent variables are randomly generated using a Gaussian distribution. In the Human3.6M public dataset, the positional data of human motion exists in the form of 17 human body 3D joint coordinates, namely: root, left knee, left ankle, left toe, right knee, right ankle, right toe, spine, neck, head, left shoulder, left elbow, left wrist, left hand, right shoulder, right elbow and right wrist; The training module uses the past time-series pose data and latent variables processed in the preprocessing module as input to the diversified human motion prediction deep neural network. This network consists of an initialization generator G1, a two-stage generator G2, and a two-layer Transformer generator G3. The initialization generator G1, composed of multiple graph convolutional layers, performs the first-stage feature extraction on the past motion sequence and generates an initial feature map X1. The initial feature map X1 is concatenated with the sum of two sets of latent variables, and the concatenated feature map is input to the two-stage generator G2, which, constructed from multiple graph convolutional layers, outputs the second-stage feature map X2. The 17 joints are divided into upper body and lower body groups. The second-stage feature map X2 is input to the two-layer Transformer generator G3. In G3, each layer uses an independent generator G3 to extract personalized features from the corresponding joint group. After each independent generator completes its calculation, it only updates the feature map of the corresponding joint group and updates the feature map. Proceed to the next layer of calculation; the final layer output. The final future motion sequence is synthesized by combining the two-stage feature map X2; the output of the deep neural network for predicting diverse human motion is a complete sample of future motions containing N possibilities, where the number of N is arbitrarily set; the loss function uses a defined reconstruction error function that only requires that one of the future motions be as close as possible to Y. g Y g This represents the pose data of frames following the division of 100 3D human motion sequences in the training data used as labels; the reconstruction loss function. Defined as: The loss function uses Euclidean distance as the distance metric D; k is a variable representing one of the N possible future motion samples generated; and a method based on a distance threshold to search for similar past motion samples is proposed, which statistically identifies future motion labels with a certain degree of similarity to past motion samples. As pseudo-labels, where Q represents the number of pseudo-labels; the loss for other predicted motions is expressed as: To improve the diversity of motion predictions, the enhanced diversity loss was revised to be a body part-specific loss, expressed as: The total loss function is defined as: Where λ d , λ r and λ mm These represent the hyperparameters corresponding to the loss function; The motion prediction module is used to load the diverse human motion prediction model trained by the training module, input the required past time series 3D human posture data into the trained diverse human motion prediction model network model, and obtain N possible future time series human motion samples.

2. The diversified human motion prediction system based on multi-level feature extraction according to claim 1, characterized in that, The upper body joints include: spine, neck, head, left shoulder, left elbow, left wrist, left hand, right shoulder, right elbow, and right wrist; the lower body joints include: root, left knee, left ankle, left toe, right knee, right ankle, and right toe.

3. A method for predicting diverse human movements using the system described in claim 1 or 2, characterized in that, The method includes the following steps: Step 1: Preprocess the 3D pose estimation dataset to obtain past motion sequences and future motion sequences with coordinate training labels as training data, and generate latent variables through Gaussian distribution; Step 2: Input the 3D human pose information from past time series and latent variables into a deep neural network for predicting diverse human motion, and finally output N possible future motion sequences. Step 3: Load the model saved after training in Step 2, input the pose of a past time series into the trained network, and obtain motion pose data in various future sequences.

4. The method according to claim 3, characterized in that, Step 1 specifically includes: The training dataset was constructed using data from the Human3.6M public dataset, specifically S1, S5, S6, S7, and S8, while S9 and S11 were used as the test dataset. From the Human3.6M training data, a 2.5-second 3D human motion sequence was extracted at the original frame rate of 50Hz. The pose data of the first 25 frames of the 3D human motion sequence were divided into past time series, and the pose data of the last 100 frames of the 3D human motion sequence were divided into 100 frames as labels for predicting future motion poses. Then, two sets of latent variables were randomly generated using a Gaussian distribution.

5. The method according to claim 3 or 4, characterized in that, Step 2 specifically includes: Step 2.1: The initialization generator G1 module in the deep neural network for predicting diverse human motion performs the first stage of feature extraction on the past motion sequence X, generating an initial feature map X1; the initialization generator G1 is defined as a GCN network with multiple graph convolutional layers, formally represented as: X1=G1(X) (1) Step 2.2: Merge the latent variables with different normal distributions, then concatenate the initial feature map X1 with the latent variables. The concatenated tensor is passed through the two-stage generator G2 to output the two-stage feature map X2; the two sets of latent variables are represented as [Z 1 Z 2 The entire process can be represented as follows: Step 2.3: The two-stage feature map X2 is input into the two-layer Transformer generator G3. In the two-layer Transformer generator G3, each layer uses an independent generator G3 to extract personalized features from the corresponding keypoint group; the generators of each layer... Designed as a Transformer-based encoder, human keypoints are explicitly embedded as tokens to learn the constraints between them. The Transformer encoder is used to learn keypoints and temporal features. Each module of the Transformer encoder contains a multi-head self-attention module (MSA) and a multi-layer sensor module (MLP), and uses the LayerNorm normalization method. The multi-head self-attention module (MSA) is an extension of the self-attention module (SA) with h "heads", represented as follows: Where i∈{1,2}, These are the learnable parameters of the three linear projection layers; The weight parameters are: d is the dimension of the module's hidden layer, h represents the number of multi-head attention "heads", β is a parameter, and in practical applications, ε is set to... In the two-layer Transformer generator G3, the computation results of each layer only update the feature map of the corresponding joint. Then it will proceed to the next layer to continue calculating the next key group; the output of the last layer is... Combined with X2, the final future action sequence is synthesized; this series of operations is defined as: Step 2.4: The loss function uses the defined reconstruction error function, which only requires that one of the future motions be as close as possible to the label Y. g The loss function for reconstruction Defined as: The loss function uses Euclidean distance as the distance metric D, Y. g This represents the pose data of the frames following 100 3D human motion sequences in the training data used as labels; k is a variable representing one of the N possible future motion samples generated. To better constrain other predicted actions, constraints should also be imposed on other predicted movements. A method based on searching for similar past motion samples using a distance threshold is proposed, and future motions with a certain degree of similarity to past movements are statistically labeled. As pseudo-labels, where Q represents the number of pseudo-labels; the loss for other predicted actions. This is represented as shown in formula (9): For each pseudo-tag, The constraint generator must have at least one motion sample to cover; to improve the diversity of predicted motions, diversity-enhanced loss is used. As shown in formula (10): Where σ is the normalization factor and N represents the number of future motion samples generated; To improve the diversity of motion prediction, an enhanced diversity loss is formulated, making it a loss specific to each body part; the enhanced diversity loss is expressed as: The total loss function is defined as follows: Where, λ d , λ r and λ mm These represent the hyperparameters corresponding to the loss function.

Images