A 3D human motion prediction method based on uncertainty loss function

Abstract

This paper discloses a 3D human motion prediction method based on an uncertainty loss function. This method uses an attention mechanism to model spatial information, a convolutional neural network to model temporal information, and a convolutional network on the temporal channel to decode feature maps and generate a prediction sequence. Furthermore, this paper proposes a new loss function formulation that effectively accounts for noise and dependencies in the generated sequence. This method effectively improves the accuracy of 3D human motion prediction.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a 3D human motion prediction method based on an uncertainty loss function. Background Art

[0002] The task of predicting human motion based on 3D skeletons can be described as follows: given a sequence of motions represented by human joints in multiple consecutive frames, predict the sequence of human joints in the next few frames. Human motion prediction has important applications in autonomous driving, human-computer interaction, and monitoring systems. However, human motion has rich semantics and uncertainties that accumulate over time, and motion sequences contain information in both spatial and temporal dimensions. This requires human motion prediction models to model both time and space and fully consider the accumulation of motion uncertainty and the dependencies between motions in the prediction.

[0003] In the existing technology, the existing solutions are as follows: use a graph convolutional encoder to learn features of the input human 3D skeleton; establish a classifier to perform behavior recognition on the input human 3D skeleton; establish a non-autoregressive decoder to predict the human 3D skeleton at future moments; use a shared graph convolutional encoder and classifier to further perform behavior recognition on the predicted human 3D skeleton.

[0004] However, the above solution requires additional action category information as supervision information, and requires multiple encoders and classifiers to learn features, which results in high model complexity. At the same time, the settings of the model and optimization process do not take into account the prediction error accumulation and dependency. Summary of the Invention

[0005] Motion prediction has important applications in many human-computer interaction, autonomous driving, and monitoring systems. However, due to the rich semantics, accumulated uncertainty, and multidimensional temporal and spatial information of human motion, accurate and high-quality motion prediction is a major challenge. The present invention aims to address the technical issues of the existing technology in achieving accurate and high-quality motion prediction and provide a 3D human motion prediction method based on an uncertainty loss function.

[0006] The specific technical solutions adopted in the present invention are as follows:

[0007] A 3D human motion prediction method based on uncertainty loss function, comprising:

[0008] Inputting the observed motion sequence into a trained 3D human motion prediction model to obtain a predicted motion sequence; the motion sequence is composed of multiple consecutive frames of human joint points;

[0009] The 3D human motion prediction model adopts an encoder-decoder structure; the encoder is composed of a cascade of multiple coding layers, and residual connections are added between adjacent coding layers; each coding layer is composed of a spatial information extraction layer and a temporal information extraction layer. First, the spatial information extraction layer calculates the attention map of the current coding layer input frame by frame, and adds the obtained attention maps and performs graph convolution to extract spatial information. Then, the temporal information extraction layer uses a one-dimensional convolutional neural network to further extract temporal information from the spatial information. The current coding layer input is added with the extracted temporal information after 1×1 convolution and used as the final output of the current coding layer; the decoder is cascaded by multiple convolutional layers and a fully connected layer. It takes the high-dimensional spatial feature map output by the encoder as input, and multiple convolutional layers perform convolution on the time channel in sequence to transform the time dimension to the same number of frames as the predicted action sequence. Then, the fully connected layer maps the high-dimensional features to three-dimensional space, thereby outputting the three-dimensional coordinates of each joint point in the predicted frame of the human joint point at each predicted moment;

[0010] The total loss function used in the training of the 3D human motion prediction model is composed of a weighted balance loss function and a salient loss function; in the balance loss function, a noise factor is assigned to each predicted frame as a learnable weight to balance the noise difference between frames; in the salient loss function, a higher weight is set for the first predicted frame than for other predicted frames to increase the importance of the first predicted frame to the overall prediction performance.

[0011] Preferably, the total loss function L used in the 3D human motion prediction model training is as follows:

[0012] L=λL a +(1-λ)L s

[0013] Where: λ is the weight; L a To balance the loss function, L s To highlight the loss function, the two calculation formulas are:

[0014]

[0015]

[0016] Where: t represents the noise factor corresponding to the predicted frame of the t-th frame, which is a learnable parameter in training; ω represents the prominence factor, which is several times the value of T out Hyperparameters of xt and They represent the t-th frame prediction in the predicted action sequence; T out Indicates the number of frames of the predicted action sequence.

[0017] Preferably, the weight λ is set to 0.1.

[0018] Preferably, the T out =10,ω=100.

[0019] Preferably, the T out =25,ω=400.

[0020] Preferably, the action sequence input to the encoder is represented as a matrix of T×N×C dimensions, where T is the number of observed action sequence frames, N is the number of human joints in each frame, and C represents the position coordinate dimension of each joint in three-dimensional space.

[0021] Preferably, the encoder comprises n coding layers in total. In the first coding layer, the input dimension of the spatial information extraction layer is T×N×C l , C l is the feature dimension output by the previous network layer. The output dimension of the spatial information extraction layer is T×N×N. The convolution kernel size of the one-dimensional convolutional neural network used in the temporal information extraction layer is 1×n, where n≥3. For the first encoding layer, its input is the observed action sequence, C1=C. After the action sequence of the input encoder passes through n layers of encoding layers, the final output is a high-dimensional spatial feature map of the action sequence, and its final output dimension is T×N×C n+1 .

[0022] Preferably, in the decoder, the input is a high-dimensional spatial feature map of the action sequence, and the first convolution layer in the decoder performs convolution on the time channel to convert the dimension of the high-dimensional spatial feature map from T×N×C n+1 Convert to T out ×N×C n+1 , where T out Represents the number of frames in the predicted action sequence; the remaining convolutional layers in the decoder keep the feature map dimension unchanged when performing convolution on the temporal channel.

[0023] Preferably, the decoder comprises four convolutional layers in total.

[0024] Preferably, the convolution kernel size in the one-dimensional convolutional neural network is preferably 1×7.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1) The method of the present invention proposes to use the self-attention mechanism to generate sample correlation graphs to extract spatial information, which has stronger modeling capabilities in the spatial dimension and improves the prediction accuracy.

[0027] 2) The method of the present invention designs a new loss function form to optimize the training process, balancing the model's attention to each time point in the prediction sequence, so that the model can learn more structured knowledge and make higher-quality predictions.

[0028] 3) This method has a simple model structure, does not rely on additional data input, has fast inference speed and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the end-to-end 3D human motion prediction model structure composed of encoder-decoder.

[0030] Figure 2 Schematic diagram of extracting spatial information through the self-attention mechanism in the spatial information extraction layer. DETAILED DESCRIPTION

[0031] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments. The technical features of each embodiment of the present invention may be combined accordingly, provided that there is no conflict between them.

[0032] To generate predicted action sequences from observed action sequences, the core of the present method is an end-to-end encoder-decoder model architecture. This model first uses an encoder based on an attention mechanism and a convolutional network to extract spatial and temporal information and construct a feature map. This is then followed by a lightweight encoder to generate the predicted sequence. Furthermore, the present invention proposes learnable weights and fixed weight optimization training in the loss function, enabling the model to generate more accurate and higher-quality predicted sequences. The specific implementation process of the present invention is described in detail below.

[0033] In a preferred embodiment of the present invention, a 3D human motion prediction method based on an uncertainty loss function is provided, wherein the observed motion sequence is input into a trained 3D human motion prediction model to obtain a predicted motion sequence.

[0034] In the task of human motion prediction, the action sequence represented by the input multiple frames of human joints contains information in two dimensions, time and space, so it is necessary to extract the information in the two dimensions separately. Therefore, the 3D human motion prediction model of the present invention designs a self-attention network to generate a sample-related graph. The network takes the observation sequence as input, calculates the attention graph frame by frame, and then adds the obtained attention graph as the graph of a single sample. Based on this graph, the spatial information is extracted through graph convolution. In the time dimension, a convolutional neural network based on 1 convolution kernel is used to extract time information. Each information extraction block includes a spatial information extraction block and a time information extraction block. The overall encoder is composed of a plurality of information extraction blocks stacked together, and residual connections are added between blocks. An information extraction block is equivalent to a coding layer of the encoder, and the encoder finally outputs the feature map of the observed sequence in the high-dimensional space. After the above-mentioned encoder obtains the high-dimensional spatial feature map, the 3D human motion prediction model of the present invention can further use a lightweight decoder to generate the observation sequence. The decoder takes the high-dimensional space feature map as input. The convolution layer constructs the observation sequence and extracts the features in the high-dimensional space. The fully connected layer then maps the high-dimensional features into the three-dimensional space to obtain the three-dimensional coordinates of the joint points.

[0035] like Figure 1 As shown, the encoder-decoder structure used in the 3D human motion prediction model in an embodiment of the present invention is shown.

[0036] The encoder of this 3D human motion prediction model is composed of multiple cascaded coding layers, and residual connections are added between adjacent coding layers. Each coding layer consists of a spatial information extraction layer and a temporal information extraction layer. First, the spatial information extraction layer calculates the attention map of the current coding layer input frame by frame, and then adds the obtained attention maps and performs graph convolution to extract spatial information. The spatial information extraction process is as follows: Figure 2 As shown in the figure; then the time information extraction layer uses a one-dimensional convolutional neural network to further extract time information from the spatial information, and the current coding layer input is added with the extracted time information after 1:1 convolution and used as the final output of the current coding layer.

[0037] The decoder of this 3D human motion prediction model is composed of a cascade of multiple convolutional layers and a fully connected layer. It takes the high-dimensional spatial feature map output by the encoder as input. Multiple convolutional layers perform convolution on the temporal channel to transform the time dimension to the same number of frames as the predicted motion sequence. The fully connected layer then maps the high-dimensional features into three-dimensional space, thereby outputting the three-dimensional coordinates of each joint point in the predicted frame at each prediction moment. In this embodiment of the present invention, the decoder contains a total of four convolutional layers and one fully connected layer.

[0038] In the embodiment of the present invention, since the action sequence is represented as the motion trajectory of the human joints in multiple consecutive frames, the action sequence input to the encoder is represented as a matrix of T×N×C dimensions, where T is the number of frames of the observed action sequence, N is the number of human joints in each frame, and C represents the position coordinate dimension of each joint in three-dimensional space, that is, C=3. Therefore, assuming that the encoder contains n layers of coding layers, the input dimension of the spatial information extraction layer in the lth layer of coding layers is T×N×C. l , C l is the feature dimension output by the previous network layer. The output dimension of the spatial information extraction layer is T×N×N. The convolution kernel size of the one-dimensional convolutional neural network used in the temporal information extraction layer is 1×n, where n≥3 and n is preferably 7. In the above encoders, all encoders are cascaded, and the input of the l-th encoding layer is the output of the l-1 encoding layer. However, it is particularly important to note that for the first encoding layer, its input is the observed action sequence of the original input encoder, C1=C. After the action sequence of the input encoder passes through n layers of encoding layers, the high-dimensional spatial feature map of the action sequence is finally output, and its final output dimension is T×N×C n+1 In the decoder, the input is the high-dimensional spatial feature map of the action sequence. The first convolutional layer in the decoder performs convolution on the time channel to reduce the dimension of the high-dimensional spatial feature map from T×N×C n+1 Convert to T out ×N×C n+1 , where T out Represents the number of frames in the predicted action sequence; the remaining convolutional layers in the decoder keep the feature map dimension unchanged when performing convolution on the temporal channel.

[0039] Therefore, the data processing flow of the entire 3D human motion prediction model is as follows: the original input ×N×C dimensional action sequence first passes through each encoding layer in the encoder in turn; and each encoding layer contains a spatial information extraction layer and a temporal information extraction layer, which is the output of the previous layer (T×N×C l ) is input, and after spatial information extraction and temporal information extraction, the output (T×N×C l+1 ) is added to the output of the previous layer after 1×1 convolution as the final output of the encoding layer. The observation sequence output by the encoder is the feature map of the high-dimensional space (T×N×C n+1 ), and then input it into the decoder for decoding to generate the predicted action sequence. The decoder contains four convolutional layers and one fully connected layer. The four convolutional layers perform convolution on the time channel. The first convolutional layer converts the input feature map (T×N×C n+1 ) is converted into an output feature map (T out ×N×C n+1 ), the fully connected layer gets the final output (T out×N×C, C=3).

[0040] In addition, in order to balance the model's attention to each time point in the prediction sequence, so that the model can learn more structured knowledge and make higher-quality predictions, the present invention involves a new loss function form to optimize the training process.

[0041] Specifically, the total loss function used in training the 3D human motion prediction model is an uncertainty loss function, which is a weighted combination of a balance loss function and a prominence loss function. The balance loss function assigns a noise factor as a learnable weight to each predicted frame to balance the noise differences between frames. The prominence loss function, on the other hand, gives the first predicted frame a higher weight than other predicted frames to increase its importance to the overall prediction performance.

[0042] The above total loss function L is as follows:

[0043] L=λL a +(1-λ)L s

[0044] Where: λ is the weight; L a To balance the loss function, L s To highlight the loss function, the two calculation formulas are:

[0045]

[0046]

[0047] Where: t represents the noise factor corresponding to the predicted frame of the t-th frame, which is a learnable parameter in training; ω represents the prominence factor, which is several times the value of T out Hyperparameters of x t and They represent the t-th frame prediction in the predicted action sequence; T out Indicates the number of frames of the predicted action sequence.

[0048] The noise factor σ is set as a learnable model parameter, which is dynamically learned during the training process. The noise factor is used as a weight to balance the training of each frame. The frame with more noise will be given a smaller weight. The prominence factor ω is set as a fixed hyperparameter value and is set to a large value (it needs to be much larger than T out ), in order to emphasize the importance of the first frame of the prediction for the overall prediction effect. a and L s The weighted weight λ can be reasonably optimized according to actual conditions, and is preferably set to 0.1.

[0049] In order to demonstrate the effect of the above-mentioned 3D human motion prediction method based on uncertainty loss function on a specific dataset, the method is applied to the Human3.6M dataset for testing. In the experiment, the above-mentioned 3D human motion prediction model is trained with two prediction steps, namely, T and out = 10 frames (400ms in the future) of short-term prediction model and T out = Long-term prediction model of 25 frames (next 1000ms). Number of prediction steps T out In the short-term prediction model of 10 frames, ω is set to 100 and the number of prediction steps is T out In the long-term prediction model for 25 frames, ω is set to 400. The weight λ is optimized to 0.1.

[0050] To demonstrate the effectiveness of the present invention, the 3D human motion prediction model based on the present invention was compared with other existing motion prediction models on the same dataset. Table 1 shows the prediction performance of the 3D human motion prediction model based on the present invention on the Human3.6M dataset, including the average error for 15 motions.

[0051] Table 1 Comparison of the method of the present invention with other methods

[0052]

[0053] It should be noted that the results for 80, 160, 320, and 400ms in Table 1 above are comparisons from a short-term prediction model, while the results for 560ms and 1000ms are comparisons from a long-term prediction model. Among the existing models used for comparison, Res-sup is a baseline RNN-based method, while DMGNN and PGBIG are GCN-based methods that have previously achieved the best results. The specific structures and implementations of these existing models are referenced below:

[0054] Res-sup: Julieta Martinez, Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEEconference on computer vision and pattern recognition, pages 2891–2900, 2017.

[0055] DMGNN: Maosen Li, Siheng Chen, Yangheng Zhao, Ya Zhang, Yanfeng Wang, and Qi Tian. Dynamic multiscale graph neural networks for 3d skeleton based human motion prediction. In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition, pages 214–223, 2020.

[0056] PGBIG: Tiezheng Ma, Y ​​ongwei Nie, Chengjiang Long, Qing Zhang, and GuiqingLi. Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition, pages 6437–6446, 2022.

[0057] From the comparison in Table 1, it can be seen that in the prediction results of the method of the present invention, the average error of a single frame is significantly improved compared with the model of the prior art, and the prediction accuracy of the model is higher than that of the existing model.

[0058] In this example, to further verify the advantages of the total loss function of the present invention, an ablation experiment was conducted on the two loss terms in the total loss function. This was also compared with the average loss function commonly used in existing models (i.e., the error weights for all samples in the loss function are the same). The final comparison results are shown in Table 2.

[0059] Table 2 Comparison of loss functions

[0060]

[0061] Similarly, the results of 80, 160, 320, and 400 ms in Table 1 above are the comparison results obtained by the short-term prediction model, and 560 ms and 1000 ms are the comparison results obtained by the long-term prediction model.

[0062] As can be seen from Table 2, the total loss function that introduces the balanced loss function and the prominent loss function in the present invention can achieve better prediction accuracy than the single loss term and the average loss function. This is because the total loss function balances the model's attention to each time point in the prediction sequence, allowing the model to learn more structured knowledge.

[0063] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A 3D human motion prediction method based on uncertainty loss function, characterized in that: include: Inputting the observed motion sequence into a trained 3D human motion prediction model to obtain a predicted motion sequence; the motion sequence is composed of multiple consecutive frames of human joint points; The 3D human motion prediction model adopts an encoder-decoder structure; the encoder is composed of a cascade of multiple coding layers, and residual connections are added between adjacent coding layers; each coding layer is composed of a spatial information extraction layer and a temporal information extraction layer. First, the spatial information extraction layer calculates the attention map of the current coding layer input frame by frame, and adds the obtained attention maps and performs graph convolution to extract spatial information. Then, the temporal information extraction layer uses a one-dimensional convolutional neural network to further extract temporal information from the spatial information. The current coding layer input is added with the extracted temporal information after 1×1 convolution and used as the final output of the current coding layer; the decoder is cascaded by multiple convolutional layers and a fully connected layer. It takes the high-dimensional spatial feature map output by the encoder as input, and multiple convolutional layers perform convolution on the time channel in sequence to transform the time dimension to the same number of frames as the predicted action sequence. Then, the fully connected layer maps the high-dimensional features to three-dimensional space, thereby outputting the three-dimensional coordinates of each joint point in the predicted frame of the human joint point at each predicted moment; The total loss function used in the training of the 3D human motion prediction model is a weighted combination of a balance loss function and a prominence loss function. The balance loss function assigns a noise factor as a learnable weight to each prediction frame to balance the noise differences between frames. The prominence loss function sets a higher weight for the first prediction frame than for other prediction frames to increase its importance to the overall prediction performance. The total loss function L used in the 3D human motion prediction model training is as follows: L=λL a +(1-λ)L s Where: λ is the weight; L a To balance the loss function, L s To highlight the loss function, the two calculation formulas are: Where: t represents the noise factor corresponding to the predicted frame of the t-th frame, which is a learnable parameter in training; ω represents the prominence factor, which is several times the value of T out Hyperparameters of x t and They represent the t-th frame prediction in the predicted action sequence; T out Indicates the number of frames of the predicted action sequence.

2. The 3D human motion prediction method based on uncertainty loss function according to claim 1, wherein The weight λ is set to 0.

1.

3. The 3D human motion prediction method based on uncertainty loss function according to claim 1, wherein The T out =10,ω=100.

4. The 3D human motion prediction method based on uncertainty loss function according to claim 1, wherein The T out =25,ω=400.

5. The 3D human motion prediction method based on uncertainty loss function according to claim 1, wherein: The action sequence input to the encoder is represented as a matrix of t×n×c dimensions, where t is the number of frames of the observed action sequence, N is the number of human joints in each frame, and C represents the position coordinate dimension of each joint in three-dimensional space.

6. The 3D human motion prediction method based on uncertainty loss function according to claim 5, characterized in that: The encoder contains n coding layers. In the first coding layer, the input dimension of the spatial information extraction layer is T×N×C l , C l is the feature dimension output by the previous network layer. The output dimension of the spatial information extraction layer is T×N×N. The convolution kernel size of the one-dimensional convolutional neural network used in the temporal information extraction layer is 1×n, where n≥3. For the first encoding layer, its input is the observed action sequence, C1=C. After the action sequence of the input encoder passes through n layers of encoding layers, the final output is a high-dimensional spatial feature map of the action sequence, and its final output dimension is T×N×C n+1 .

7. The 3D human motion prediction method based on uncertainty loss function according to claim 6, characterized in that: In the decoder, the input is the high-dimensional spatial feature map of the action sequence. The first convolution layer in the decoder performs convolution on the time channel to reduce the dimension of the high-dimensional spatial feature map from T×N×C n+1 Convert to T out ×N×C n+1 , where T out Represents the number of frames in the predicted action sequence; the remaining convolutional layers in the decoder keep the feature map dimension unchanged when performing convolution on the temporal channel.

8. The 3D human motion prediction method based on uncertainty loss function according to claim 1, wherein: The decoder contains four convolutional layers in total.

9. The 3D human motion prediction method based on uncertainty loss function according to claim 1, wherein: The convolution kernel size in the one-dimensional convolutional neural network is 1×7.

Images