A method for predicting human motion in a three-dimensional space based on a scene

Abstract

The application discloses a human action prediction method for scene perception in a three-dimensional space, and aims at human skeleton data acquired by a sensor or a game engine. Firstly, a space-time graph convolutional neural network of a self-learning adjacency matrix is used to extract motion features, the natural connection between human skeleton nodes is considered in space by using the adjacency matrix, and the connection of the same joints at different time steps is modeled by one-dimensional convolution in time. Then, a deep convolutional network is constructed on a three-dimensional scene point cloud to obtain a semantic segmentation result of a corresponding time RGB image as a scene feature. After that, a cross-channel self-attention model is used to realize feature fusion of the motion feature and the scene feature. Finally, the fused features are used for human action prediction in a three-dimensional scene. Compared with a traditional action prediction method, the method comprehensively considers the relationship between human motion and a scene, improves the prediction accuracy, and can be used for motion prediction problems of human-computer interaction, automatic driving and some scene monitoring systems.

Description

Technical Field

[0001] The present invention relates to the technical field of human motion prediction, and in particular to a human motion prediction method based on scene perception in three-dimensional space. Background Art

[0002] Human motion prediction in three-dimensional space involves predicting future postures and trajectories given a set of historical motion postures and trajectories. Human motion prediction has many important real-life applications in fields such as human-computer interaction, autonomous driving, and surveillance systems. For example, in autonomous driving, it facilitates precise maneuvering and motion planning; in surveillance systems, predicting pedestrian postures can help prevent traffic accidents.

[0003] Despite significant progress in human motion prediction, due to the random nature of human motion, predicting human motion in three-dimensional space remains a challenging task. Most current work performs motion prediction independently of the motion context, focusing solely on historical human motion. However, human motion is influenced by the surrounding environment, such as when avoiding collisions or ascending and descending stairs. Ignoring the surrounding environment can lead to inaccurate predictions. Understanding the environment, constraining motion, and obtaining contextually reliable poses is crucial for predicting future motion.

[0004] Therefore, integrating the motion environment into the task of motion prediction is a feasible technical route. Zhe Cao et al. conducted a preliminary exploration of this issue and proposed GPP-NET based on the motion background. [1] , developed a three-stage computational framework (Goal, Path, Pose) that includes three-dimensional posture and position in a three-dimensional scene. Based on this, Abduallah Mohamed et al. integrated several effective components in the human motion prediction task, avoided the use of deep loop architecture, and proposed the Skeleton-Graph Model [2] By incorporating background images as visual signals and establishing a connection between scene and motion, this approach resolves ambiguity in certain prediction scenarios and avoids the repeated accumulation of errors in long-term prediction problems. However, considering that directly using background images is inefficient in describing spatial geometry, and that simple connection operations do not allow motion to interact well with the environment, existing methods still have relatively high average joint position errors and relatively poor overall performance in predicting motion trajectories.

[0005] [1]Zhe Cao,Hang Gao,Karttikeya Mangalam,Qi-Zhi Cai,Minh V o,andJitendra Malik.Long-term human motion prediction with scene context.InEuropean Conference on Computer Vision,pages 387–404.Springer,2020.1,2,3,5,6,7

[0006] [2]Abduallah Mohamed,Huancheng Chen,Zhangyang Wang,and ChristianG.Claudel.Skeleton-graph:Long-term 3d motion prediction from 2d observationsusing deep spatio-temporal graph cnns.In ICCVW,2021.2,3,4,5,6,7 Summary of the Invention

[0007] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a human motion prediction method based on scene perception in three-dimensional space. Taking into account that the future movement of the human body is jointly driven by historical movement and the current scene, the present invention focuses on the joint reasoning of human motion and scene, and is used for human-computer interaction, autonomous driving and monitoring systems in some scenarios. It has a significant performance improvement over existing methods in predicting motion posture and trajectory.

[0008] In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0009] A method for predicting human motion based on scene perception in three-dimensional space, which incorporates scene information into predicting future human motion, specifically includes the following steps:

[0010] Step 1: For the human motion skeleton data obtained by the sensor or game engine, in each time step t, the human motion skeleton data is modeled as a spatiotemporal graph G t =(V t ,E t ); where V t is a point on the space-time graph, E t is the edge of the space-time graph; the point set V t Contains all joint points of a skeleton action sequence; formally, the edge set E t It consists of two subsets, the adjacency matrix A tThe weight of the edge of the space-time graph is defined, which represents the natural connection of the human joints, E f represents the same joint in continuous time, then for a particular joint i, all edges in E f represent its trajectory over time;

[0011] Step 2: Based on the processing of human motion skeleton data in step 1, the natural connection of human joints is considered in space using the adjacency matrix, and the connection of the same joint at different times is modeled by one-dimensional convolution in time. The human motion features f m are extracted by constructing a space-time graph convolutional neural network.

[0012] Step 3: In order to integrate scene information into the prediction of human future motion, a visual feature extractor ψ based on PointConv is used to encode the visual feature f s =ψ(X 1:n ), where X 1:n ={X1,X2,…,X n} represents the point cloud stream from time 1 to time n. The specific operation is to perform semantic segmentation on the RGB image corresponding to the time point by using the point cloud at n time points, and the semantic segmentation result is used as the visual feature f s .

[0013] Step 4: Multi-modal feature fusion with attention perception is performed on the human motion features and visual features obtained in steps 2 and 3 respectively in order to utilize information from different modalities. A cross-modal self-attention model is deployed to capture the correlation between the input human motion features f m and visual features f s and establish communication between multi-modal information. Based on the attention mechanism, the fused features f are represented as follows:

[0014] f=cross_trans(f s ,f m )

[0015] Step 5: The fused features obtained in step 4 are used to predict the future motion of humans, M H+F =cross_trans(h,f);M H+Frepresents the predicted obtained whole motion sequence, wherein H represents the time of historical motion, F represents the time of predicted future motion, f represents the fusion feature of motion and scene, h is an implicit vector containing output time-space encoding, facilitating the output as an array of the required shape; in order to make the predicted obtained motion sequence more natural, a skeleton consistency loss L is introduced to measure the gap between the predicted obtained motion sequence and the real motion sequence; the skeleton consistency loss L mainly constrains the normal length of the skeleton and the angle between the joints; for the normal length of the skeleton, the mean square error L2 is used for constraint, and for the angle between the joints, the cosine similarity L c is used for constraint, so that L = λ1L2 + λ2L c , wherein λ1, λ2 are the weights of the two constraints.

[0016] The adjacency matrix representing the natural connection of human joints in step 1 has a fixed weight, but the model is self-learning to find its optimal weight; the specific method is: since the shape of the adjacency matrix is J*J, wherein J is the number of human joints, the adjacency matrix is regarded as a two-dimensional image with T channels, that is, T*J*J, in time T, and is input to a convolutional neural network, which can learn the relationship between each joint in human motion, and the self-learning adjacency matrix can better discover the interaction between the joints.

[0017] The visual feature in step 3 can integrate scene information into the predicted future motion of the human body, providing guidance for the prediction process with scene information, so that the predicted human motion is constrained by the scene, and the prediction result is more reliable.

[0018] The visual feature in step 3 is obtained by performing semantic segmentation on the RGB image corresponding to the point cloud at n time points, and the semantic segmentation result is used as the visual feature, which is more intuitive than the feature obtained by directly inputting the RGB image.

[0019] Step 4 fuses human motion features and visual features, and the specific operation is: a cross-modal self-attention model is deployed to fuse motion features and visual features, instead of simply connecting or stacking, which helps the interaction between the two features and better plays the guiding role of visual features on human motion.

[0020] The differences (innovations) of the present application compared with the prior art are as follows:

[0021] 1) Compared with the prior art, the present application pays more attention to the influence of the scene on the motion, and focuses on the joint reasoning of human motion and scene, which has a greater performance improvement in predicting future posture and trajectory than the prior art.

[0022] 2) The present application considers that the use of only background RGB images is inefficient in describing spatial geometry and cannot effectively extract visual features, and therefore adopts a point cloud-based visual extractor for semantic segmentation, which can better extract features and learn potential information in the background.

[0023] 3) The present application extracts human motion features and visual features, and fuses the two features through a cross-modal self-attention model, instead of using simple connection or stacking as in the prior art, which helps the interaction of the two features and can better play the guiding role of visual features on human motion. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The flowchart of the present application for human action prediction in three-dimensional space;

[0025] Figure 2 The connection mode of human joint points in and between frames;

[0026] Figure 3a and Figure 3b are the standard adjacency matrix and the self-learning adjacency matrix, respectively;

[0027] Figure 4 is the human motion feature extraction process;

[0028] Figure 5a , Figure 5b and Figure 5c are the input scene, the real segmentation result of the input scene and the segmentation result of the present application, respectively;

[0029] Figure 6a and Figure 6b are the real motion skeleton sequence and the prediction result of the present application, respectively. DETAILED DESCRIPTION

[0030] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0031] The total algorithm flowchart is as follows Figure 1The human motion skeleton data and the motion environment data are separated according to the data set type. For the human motion skeleton data, the communication connection between human joints is established in depth by using a self-learning adjacency matrix in space, and the motion trajectory of the same joint at different time steps is modeled by one-dimensional convolution in time, and the human motion features are extracted by using a spatio-temporal graph convolutional neural network. For the motion environment data, a deep convolutional network is constructed on the three-dimensional scene point cloud to obtain the semantic segmentation result of the corresponding time background image as the scene feature. Then the two kinds of features extracted are interacted through a cross-modal self-attention model to realize the multi-modal feature fusion with attention perception. Finally, the human action prediction in the three-dimensional scene is carried out by using the fused features.

[0032] 1. Spatio-temporal graph convolutional neural network of self-learning adjacency matrix

[0033] The skeleton sequence is usually represented by 2D or 3D coordinates of each human joint in each frame. In the present application, a self-learning adjacency matrix is used to form a hierarchical representation of the skeleton sequence, and a non-directional spatio-temporal graph G t =(V t ,E t ) is constructed on the skeleton sequence with J joints and time T. Wherein V t is the point of the spatio-temporal graph, E t is the edge of the spatio-temporal graph, as shown in Figure 2 . The point set V t contains all the joint nodes of a skeleton motion sequence; formally, the edge set E t is composed of two subsets, the adjacency matrix A t defines the weight of the graph edge, representing the natural connection of human joints, E f represents the same joint at consecutive times, so for a particular joint i, all edges in E f represent its trajectory over time.

[0034] As an adjacency matrix representing the natural connection of human joints, its weight is not fixed in the present application, but the model is self-learning to find its optimal weight. Since the shape of the adjacency matrix is T*J*J, it can be regarded as a two-dimensional image with T channels. The original adjacency matrix is input into a convolutional neural network, so that the relationship between each joint node in human motion can be learned, and the self-learning adjacency matrix can better discover the interaction between joint nodes. The heat map of the self-learning adjacency matrix is shown in Figure 3a and Figure 3b , Figure 3a is a standard adjacency matrix, Figure 3bThe learned adjacency matrix is shown in the figure. As can be seen from the figure, the learned adjacency matrix is not composed of only 0 or 1, but is composed of numbers between 0 and 1, and the size of the number reflects the strength of the relationship between different joints, which can intuitively reflect the correlation between the joints.

[0035] 2. Motion feature extraction

[0036] The human motion feature f is extracted by constructing a spatio-temporal graph convolutional neural network. m The natural connection of the human joints is considered in space by using the adjacency matrix, and the connection of the same joints at different times is modeled by one-dimensional convolution in time. The human motion feature extraction process is shown in Figure 4 As can be seen from the figure, the human motion feature extraction process is to input the two-dimensional human motion skeleton data into the adjacency matrix and the graph convolutional neural network in parallel, so that the self-learning of the adjacency matrix and the updating of the graph convolutional neural network can be performed at the same time, and the speed of motion feature extraction is accelerated.

[0037] 3. Visual feature extraction

[0038] 4. In order to integrate scene information into the prediction of human future motion, the visual feature extractor ψ based on PointConv is used to encode the visual feature f s =ψ(X 1:n ), wherein X 1:n ={X1,X2,…,X n} represents the point cloud stream from time 1 to time n. The specific operation is to perform semantic segmentation on the RGB image corresponding to the time by using the point cloud at n times, and the result is used as the visual feature. The segmentation results are shown in Figure 5a 、 Figure 5b and Figure 5c , each figure from top to bottom is three scenes. As can be seen from the figure, the segmentation result of the present application is basically consistent with the true segmentation result of the input scene, and different objects in the actual scene can be segmented by the point cloud, and the semantic segmentation result is good.

[0039] Compared with the prior art, the human motion feature and the visual feature are extracted, and the cross-modal self-attention model is used to fuse the two kinds of features, which is helpful for the interaction of the two kinds of features, and can better play the guiding role of the visual feature on the human motion.

[0040] The cross-modal self-attention model is mainly used to capture the input human motion feature f m and the visual feature f scorrelation and establish communication between multi-modal information, based on attention mechanism, the fusion feature f is represented as follows:

[0041] f = cross_trans(f s ,f m )

[0042] Where f s ,f m are visual features and motion features respectively.

[0043] To predict the future motion of human beings using the fusion feature, an implicit vector h containing the time and space encoding of the output is introduced to facilitate the output to be an array of the required shape. The formula can be expressed as:

[0044] M H+F = cross_trans(h,f)

[0045] M H+F represents the entire motion sequence obtained by prediction, where H represents the time of historical motion, F represents the time of predicted future motion, f represents the fusion feature of motion and scene, and h is an implicit vector containing the time and space encoding of the output, facilitating the output to be an array of the required shape.

[0046] Due to the complexity of human motion and the natural connection of human joints, predicting human future motion in a three-dimensional scene will cause unnatural motion problems, such as abnormal distance between joint nodes or abnormal position of key joints. In order to reduce the occurrence of abnormalities and make the predicted skeleton sequence more natural, the invention introduces a skeleton consistency loss L to constrain the length of the skeleton (the distance between two joint nodes) and the angle between joints in the predicted three-dimensional skeleton sequence.

[0047] For the normal length of the skeleton, the invention uses mean square error L2 for constraint. Specifically, the difference between the actual skeleton length and the prediction result is calculated, and the formula is as follows:

[0048]

[0049] Where T is time, J is the number of joints, i and j are joint numbers, P represents the actual joint, represents the predicted joint, and represent the three-dimensional space coordinates of the i-th joint and the j-th joint in the actual joint at time t, respectively, and and represent the three-dimensional space coordinates of the i-th joint and the j-th joint in the predicted joint at time t, respectively.

[0050] For the angle between joints, the invention uses cosine similarity L cThe constraint is performed. Specifically, the difference between the cosine value of the actual two joints and the predicted result is calculated, and the formula is as follows:

[0051]

[0052] Wherein T is time, J is the number of joints, i is the joint number, C is the cosine similarity, P represents the actual joint, represents the predicted joint, and respectively represent the three-dimensional space coordinates of the i-th joint and the next joint in the actual joint at time t, and and respectively represent the three-dimensional space coordinates of the i-th joint and the next joint in the predicted joint at time t.

[0053] Then the skeleton consistency constraint L can be defined as:

[0054] L=λ1L2+λ2L c

[0055] Wherein λ1, λ2 are the weights of the two constraints, and λ1 is 0.1 and λ2 is 0.0005 through experiments.

[0056] The motion prediction comparison chart is shown in Figure 6a and Figure 6b , Figure 6a is the real motion skeleton sequence, Figure 6b is the prediction result of the application, and it can be seen from the chart that the difference between the skeleton sequence predicted by the application and the actual motion skeleton sequence is small, the relative position of the joint conforms to the human motion law and the action is natural, especially in the position of the joint, still has good prediction effect.

Claims

1. A method for predicting human motion based on scene perception in three-dimensional space, characterized by: Integrating scene information into predicting the future movement of the human body includes the following steps: Step 1: For the human motion skeleton data obtained by the sensor or game engine, in each time step t, the human motion skeleton data is modeled as a spatiotemporal graph G t =(V t ,E t ); where V t is a point on the space-time graph, E t is the edge of the space-time graph; the point set V t Contains all joint points of a skeleton action sequence; formally, the edge set E t It consists of two subsets, the adjacency matrix A t The edge weights of the spatiotemporal graph are defined to represent the natural connections of human joints, E f represents the same joint in continuous time, then for a specific joint i, E f All edges in represent its trajectory over time; Step 2: Using the human motion skeleton data processed in step 1, the natural connection of human joints is considered in space using the adjacency matrix, and the connection of the same joints at different times is modeled through one-dimensional convolution in time. The human motion features f are extracted by constructing a spatiotemporal graph convolutional neural network. m ; Step 3: In order to incorporate scene information into the prediction of the future motion of the human body, a PointConv-based visual feature extractor ψ is used to encode the visual features f s =ψ(X 1:n ), where X 1:n ={X1,X2,…,X n } represents the point cloud flow from time 1 to time n. The specific operation is to perform semantic segmentation on the RGB image at the corresponding time through the point cloud at n times, and use the semantic segmentation result as the visual feature f s ; Step 4: Perform attention-aware multimodal feature fusion based on the human motion features and visual features obtained in steps 2 and 3; deploy a cross-modal self-attention model to capture the input human motion features f m and visual features f s correlation and establish communication between multimodal information; Step 5: Use the fusion features obtained in step 4 to predict the future movement of the human body, M H+F =cross_trans(h,f);M H+F Represents the entire predicted motion sequence, where H represents the time of historical motion, F represents the time of predicted future motion, f represents the fusion feature of motion and scene, and h is a latent vector containing the output time-space encoding; the skeleton consistency loss L is introduced to measure the gap between the predicted motion sequence and the real motion sequence; the skeleton consistency loss L is used to constrain the normal length of the skeleton and the angle between the joints; for the normal length of the skeleton, the mean square error L2 is used for constraint, and for the angle between the joints, the cosine similarity L is used. c Constraint, let L = λ1L2 + λ2L c , where λ1 and λ2 are the weights of the two constraints.

2. The method for predicting human motion based on scene perception in three-dimensional space according to claim 1, characterized in that: The adjacency matrix representing the natural connections of human joints described in step 1 has unfixed weights, but the model is allowed to self-learn and discover its optimal weights. The specific method is as follows: since the shape of the adjacency matrix is ​​J*J, where J is the number of human joints, within time T, the adjacency matrix is ​​regarded as a two-dimensional image with T channels, that is, T*J*J, and is input into a convolutional neural network, which can learn the connection between each joint point in human movement. Using the self-learned adjacency matrix can better discover the interactivity between joint points.

Images