A human behavior prediction method in a scene point cloud

Abstract

A human behavior prediction method in a scene point cloud. In the first step, the surrounding scene where the human body is located is represented in the form of a three-dimensional point cloud through hardware three-dimensional scanning or software three-dimensional reconstruction; in the second step, a feature encoder with a multi-attention modeling mechanism is designed to extract action dependence inside the human skeleton data, geometric structure dependence inside the scene point cloud, and human-scene interaction dependence between the two; in the third step, the multi-attention coding features extracted are sent to a decoder to predict the reasonable motion posture and trajectory of the human body at a future time; compared with the prior art, the best prediction effect is realized under the least parameter amount, and the algorithm explainability in the human behavior prediction field is enhanced.

Description

Technical Field

[0001] This invention relates to the field of human behavior prediction technology, specifically to a method for predicting future human behavior based on historical human movement information and current scene structure information. Background Technology

[0002] Most current behavior prediction algorithms are based on a known history of human movement information to predict future movement information [1,2,3,4,5]. Such models can be understood as human behavior prediction algorithms driven by motion data, thus ignoring the positional movement information of the human body in the scene. Considering that human movement in real life is often constrained by the layout and structure of the scene, such as the human body's posture and trajectory being constrained by multiple factors from the stair steps and handrails during the action of climbing stairs. Therefore, how to effectively perceive the scene information in the process of predicting the future movement of the human body is a key issue. However, how to design an effective human behavior prediction system with scene awareness still needs further exploration. In 2020, GPP-Net [6] proposed using background images to represent scene information, using heatmaps to represent human movement information, and extracting the representations from them through convolutional neural networks, thereby introducing the contextual information of the scene into the prediction process of future human behavior. Similarly, SocialPool[7] uses 3D skeleton data to represent human motion information and background images to represent scene information, and extracts their respective representations through recurrent neural networks. Recently, Skeleton-Graph[8] has made further improvements on the basis of GPP-Net, using 3D skeleton data to represent human motion information and graph convolutional networks to model the representations. Although the specific networks used by such models are not exactly the same, they all have the following two common points: (1) using RGB-based image or video data to represent scene information; (2) separating scene modeling and human motion modeling into two parallel information flow branches. These two aspects also make these models have shortcomings in terms of model effectiveness, robustness and interpretability: (1) over-reliance on scene appearance information, making it difficult to capture robust scene representations. Under the condition that the scene layout structure remains unchanged, subtle changes in the color, shape or texture of objects in the scene will cause changes in the final extracted scene representation. (2) It is difficult to give full play to the constraining role of scene information in human behavior and movement. The main role of scene information is to use the structure of the current scene as a trajectory prior knowledge to guide the prediction of future human movement. (3) Lack of modeling for human-scene interaction. The two-stream framework used in previous algorithms cannot analyze the interaction dependencies between the human body and the scene.

[0003] [1]Dang,L.;Nie,Y.;Long,C.;Zhang,Q.;and Li,G.2021.MSR-GCN:Multi-ScaleResidual Graph Convolution Networks for Human Motion Prediction.In ICCV.

[0004] [2]Sofianos,T.;Sampieri,A.;Franco,L.;and Galasso,F.2021.Space-Time-Separable Graph Convolutional Network for Pose Forecasting.In ICCV.

[0005] [3]Zhong,C.;Hu,L.;Zhang,Z.;Ye,Y.;and Xia,S.2022.Spatial-TemporalGating-Adjacency GCN for Human Motion Prediction.In CVPR.

[0006] [4]Li,M.;Chen,S.;Zhao,Y.;Zhang,Y.;Wang,Y.;and Tian,Q.2020.DynamicMultiscale Graph Neural Networks for 3D Skeleton Based Human MotionPrediction.In CVPR.

[0007] [5]Li,M.;Chen,S.;Zhao,Y.;Zhang,Y.;Wang,Y.;and Tian,Q.2021.MultiscaleSpatio-Temporal Graph Neural Networks for 3D Skeleton-Based MotionPrediction.IEEE Trans.Image Process.

[0008] [6]Cao,Z.;Gao,H.;Mangalam,K.;Cai,Q.;Vo,M.;and Malik,J.2020.Long-TermHuman Motion Prediction with Scene Context.In ECCV.

[0009] [7] Adeli, V.; Adeli, E.; Reid, I.; Niebles, JC; and Rezatofighi, H. 2020. Socially and Contextually Aware Human Motion and Pose Forecasting. IEEERobotics Autom. Lett.

[0010] [8]Mohamed,A.;Chen,H.;Wang,Z.;and Claudel,CG2021.Skeleton-Graph:Long-Term 3D Motion Prediction From 2D Observations Using Deep Spatio-Temporal Graph CNNs.In ICCVW. Summary of the Invention

[0011] To address the problems existing in the prior art, the present invention aims to provide a method for predicting human behavior in scene point clouds. Firstly, the human behavior prediction task is redefined as a joint reasoning problem based on 3D historical human motion information and 3D current scene structure information. A multi-attention encoding network is designed to jointly model human motion patterns, scene geometric patterns, and human-scene interaction patterns. Results show that this method achieves higher prediction accuracy with fewer parameters, while also improving the interpretability of the prediction results.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] A method for predicting human behavior in scene point clouds, the first step of which is to represent scene information as 3D point cloud data through hardware 3D scanning or software 3D reconstruction. Human motion information is represented as a three-dimensional skeleton sequence. Where N s N represents the number of points contained in the scene point cloud S. bThe first step involves defining the number of joints in each human skeleton and the second step involves cropping the scene point cloud S by selecting key regions, since human motion is less susceptible to the influence of distant scene objects. This reduces redundancy. The third step considers the different types of dependencies between the human skeleton B and the scene point cloud S, namely the motion features within the human skeleton B, the geometric features within the scene point cloud S, and the human-scene interaction features between the human skeleton and the scene point cloud. Therefore, an encoder with a multi-attention modeling mechanism is designed to jointly model these different dependencies using both self-attention and cross-attention mechanisms, extracting multi-attention features. The fourth step involves a decoder composed of a temporal convolutional network that decodes the extracted multi-attention features, ultimately outputting the predicted future human motion posture and trajectory.

[0014] The human behavior prediction method in scene point cloud constructs a new scene perception paradigm based on 3D point cloud. Through 3D scanning or 3D reconstruction, the human action prediction task of scene perception is redefined as a joint reasoning problem based on historical 3D human motion information and current 3D scene structure information, thereby effectively predicting future human action sequences that are both coherent and smooth in their own movements and compatible with the surrounding environment.

[0015] The human behavior prediction method in the scene point cloud reduces the redundancy of scene point cloud data by selecting key regions: based on the continuous characteristics of human movement, the range of human movement in the future usually expands linearly with the continuation of movement time; therefore, the range of human movement in the scene in the past time period is used as an anchor point, and the key regions in the scene are cropped out by linear scaling according to the ratio of the predicted duration of future movement to the known duration of historical movement.

[0016] The method for predicting human behavior in scene point clouds constructs an encoder with a multi-attention modeling mechanism that includes self-attention and cross-attention layers. Since there are three different types of dependencies between the human skeleton B and the scene point cloud S, namely human motion features, scene structure features, and human-scene interaction features, the designed encoder must be able to simultaneously and jointly model these three different modes. Specifically, the human skeleton B and the scene point cloud S are input into two independent self-attention layers respectively. For the self-attention model of the (l+1)th layer processing the scene point cloud, C... l The input scene feature dimension of the self-attention model is represented as follows: S l Linear transformations are performed using three independent multilayer perceptrons to obtain scene point query items. Scene point key value item and scene point value items In the self-attention encoding mechanism of scene structure and Similarity as a basis The coefficients of the weighted summation:

[0017]

[0018] Simultaneously, h different values ​​are obtained by transforming them all at once through a multi-head attention mechanism. The scene structure feature output of the k-th self-attention head is represented as: The ability to represent scene structural features is improved by merging the outputs from h self-attention heads.

[0019]

[0020] in, To output the mapping matrix, for the (l+1)th layer self-attention model processing human joints, its input features are: B l Linear transformations are performed using three independent multilayer perceptrons to obtain the keypoint query items. Key value item of key point Key value item and The similarity between them as a basis The coefficients of the weighted summation:

[0021]

[0022] The human motion feature output of the k-th self-attention head is represented as: The ability to represent human motion features is improved by merging the outputs from h self-attention heads:

[0023]

[0024] After the self-attention modeling of the scene point cloud and human joints at layer l+1, the scene structural features are transformed from... Updated to Human movement characteristics are formed by Updated to C l+1 Indicates the feature dimension at the time of output;

[0025] Unlike the self-attention layer, the cross-attention layer uses human skeletal data. This will be considered a cross-attention query. Used to measure the interaction between each body joint and the scene point cloud. Therefore, the cross-attention mechanism is defined as:

[0026]

[0027]

[0028] Where: the human-scene interaction feature output of the kth cross-attention head is represented as: Improving human-scene interaction features by merging the outputs from h cross-attention heads. l+1 The ability to represent;

[0029] Finally, the above scene structural features S l+1 Human movement characteristics B l+1 Human-scene interaction features I l+1 The signals are merged into a multi-attention encoding and fed into a decoder composed of a temporal convolutional network to predict the future human motion posture and trajectory that are reasonable in terms of self-motion and scene interaction. Compared with existing inventions, this invention has the following advantages:

[0030] Traditional scene-aware human behavior prediction methods use RGB pixel-based images or videos as scene information input. Because such inputs are heavily influenced by scene appearance information—for example, even with an unchanged scene structure, changes in the color, texture, and shape of objects within the scene will alter the final extracted scene representation—these algorithms struggle to extract robust scene information. Secondly, the primary role of scene information in human behavior prediction is to use the scene's geometric structure as prior trajectory information to constrain predicted future human actions. Therefore, compared to scene input based on RGB images, the human behavior prediction framework based on 3D point cloud scenes in this invention can effectively perceive scene structure information. Furthermore, this invention addresses the shortcomings of existing methods in the interpretability of human behavior prediction results by exploring how 3D historical human actions and 3D scene structure influence future human actions. Attached Figure Description

[0031] Figure 1 This is a diagram illustrating the algorithm framework for predicting human behavior in point clouds based on the present invention.

[0032] Figure 2 This is a schematic diagram of the prediction results of the present invention on the GTA-IM dataset.

[0033] Figure 3 This diagram illustrates the performance comparison of different algorithms on the GTA-IM dataset. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0035] The overall framework of the algorithm is as follows Figure 1 As shown, the human behavior prediction method in scene point cloud of the present invention has the following steps: First, in the 3D scene point cloud, the key region selection module selects key regions closely related to human interaction based on the input human motion data to reduce the redundancy of the input scene; Second, the multi-attention encoding network jointly extracts human motion patterns, scene geometry, and human-scene interaction correlations from the input 3D human motion data and 3D scene point cloud data; Third, the encoding network predicts future human 3D actions based on the extracted multi-attention encoding features.

[0036] 1. The key area selection module crops the input 3D scene point cloud.

[0037] Due to the continuous nature of human movement, a preliminary estimate of the approximate range of motion within the next two seconds can be made based on a single second of input 3D human motion data. Therefore, the scene can be roughly divided into areas within the human's motion range and areas outside of it. This allows areas that directly interact with human movement to be cropped from the scene input, reducing redundancy and the influence of irrelevant objects in the scene.

[0038] 2. Modeling Multiple Dependencies in Multi-Encoding Networks

[0039] Considering the input scene point cloud data and human motion 3D skeleton data There are three dependencies between the human skeleton B and the scene point cloud S: the action features within the human skeleton B, the geometric structure features within the scene point cloud S, and the human-scene interaction features between the human skeleton and the scene point cloud. Therefore, an effective encoder network is designed to effectively model these three dependencies. The multi-attention encoding network designed in this invention contains three modules: two self-attention modules that receive scene input and human action input respectively to model their respective internal patterns, and a cross-attention module that simultaneously receives scene and human action data as input to jointly model the interaction pattern between them. In the (l+1)th layer of the scene self-attention network, SelfAtt(·) dynamically infers the paired structural associations between scene points and aggregates the contextual information between nodes based on the associations between scene points, thereby achieving flexible and dynamic information propagation. Specifically, the (l+1)th layer of the scene self-attention layer SelfAtt(·) uses the output features S of the lth layer... l As input, and S l Scene point query items are obtained through three independent multilayer perceptrons. Scene point key value item and scene point value items In the self-attention encoding mechanism of scene structure and Similarity as a basis The coefficients of the weighted summation:

[0040]

[0041] Simultaneously, h different values ​​are obtained by transforming them all at once through a multi-head attention mechanism. The scene structure feature output of the k-th self-attention head is represented as: The scene structure feature S is improved by merging the outputs from h self-attention heads. l+1 The ability to represent:

[0042]

[0043] in, The output is a mapping matrix. Finally, the (l+1)th layer of the scene self-attention network encodes the input scene features S. l Updated to S l+1 Similarly, for the (l+1)th layer self-attention model processing human joints, its input data is the output B of the lth layer. l B l Linear transformations are performed using three independent multilayer perceptrons to obtain the keypoint query items. Key value item of key point Key value item and The similarity between them as a basis The coefficients of the weighted summation:

[0044]

[0045] The human motion feature output of the k-th self-attention head is represented as: Improve human motion characteristics B by merging the outputs from h self-attention heads. l+1 The ability to represent:

[0046]

[0047] After self-attention modeling of the scene point cloud and human joints at layer l+1, the scene structural features are derived from S l Updated to S l+1 Human movement characteristics are determined by B l Updated to B l+1 .

[0048] Considering the interaction and constraints between scene point clouds and human motion within the scene, a cross-attention network, CrossAtt(·), is designed to extract the interaction features between scene points and keypoints. In the (l+1)th layer of cross-attention encoding, the human skeleton data B... l It will then be converted into a key query item. Scene point cloud data S l It is then converted into a scene key-value item. and scene value items This allows for the measurement of the interaction between each body joint and the scene point cloud. l+1 Therefore, the cross-attention mechanism is defined as:

[0049]

[0050] Where: the human-scene interaction feature output of the kth cross-attention head is represented as: MultiHead(·) improves human-scene interaction features by merging the outputs from h cross-attention heads. l+1 The ability to represent:

[0051]

[0052] 3. Decoder network predicts future human movements

[0053] By decoding the multi-attention encoded features obtained from the encoder network, prediction of future human actions is achieved. Compared to recurrent network architectures (such as RNNs and GRUs) that output human actions at one time step at a time, convolutional networks can directly output a sequence of human actions. Therefore, the decoder designed based on convolutional networks can not only accelerate the inference time of human behavior prediction in real-world scenarios but also avoid the problem of error accumulation. Ultimately, it achieves the prediction of future human actions based on human actions over the past T seconds and the current scene geometry. Human movement posture and trajectory within seconds. Figure 2 The prediction results of the method of the present invention are shown, wherein the first 5 frames of the skeleton sequence are known historical human motion data, and the last 10 frames of the skeleton sequence are the future human motion results predicted by the method of the present invention. Figure 3 By comparing the prediction results of the method of the present invention in Figure (c) with those of the GPP-Net method in Figure (a) and the Skeleton-Graph method in Figure (b), it can be found that the method of the present invention predicts the future human motion more accurately and reasonably.

Claims

1. A method for predicting human behavior in scene point clouds, characterized in that: The first step is to represent the scene information as 3D point cloud data through hardware 3D scanning or software 3D reconstruction. Human motion information is represented as a three-dimensional skeleton sequence. Where N s N represents the number of points contained in the scene point cloud S. b T represents the number of joints in each human skeleton, and T represents the number of skeleton frames in the motion sequence. The second step involves cropping the scene point cloud S by selecting key regions, since human motion is not easily affected by distant objects in the scene. This reduces redundancy. The third step considers the different types of dependencies between the human skeleton B and the scene point cloud S: the motion features within the human skeleton B, the geometric features within the scene point cloud S, and the human-scene interaction features between them. Therefore, an encoder with a multi-attention modeling mechanism is designed to jointly model these different dependencies using both self-attention and cross-attention mechanisms, extracting multi-attention features. The fourth step involves a decoder composed of a temporal convolutional network that decodes the extracted multi-attention features, ultimately outputting the predicted future human motion posture and trajectory. An encoder with a multi-attention modeling mechanism, including self-attention and cross-attention layers, is constructed. Since there are three different types of dependencies between the human skeleton B and the scene point cloud S, namely human motion features, scene structure features, and human-scene interaction features, the designed encoder must be able to simultaneously and jointly model these three different modes. Specifically, the human skeleton B and the scene point cloud S are input into two independent self-attention layers respectively. For the self-attention model of the (l+1)th layer processing the scene point cloud, C... l The input scene feature dimension of the self-attention model is represented as follows: S l Linear transformations are performed using three independent multilayer perceptrons to obtain scene point query items. Scene point key value item and scene point value items In the self-attention encoding mechanism of scene structure and Similarity as a basis The coefficients of the weighted summation: Simultaneously, h different values ​​are obtained by transforming them all at once through a multi-head attention mechanism. The scene structure feature output of the k-th self-attention head is represented as: The ability to represent scene structural features is improved by merging the outputs from h self-attention heads. in, To output the mapping matrix, for the (l+1)th layer self-attention model processing human joints, its input features are: B l Linear transformations are performed using three independent multilayer perceptrons to obtain the keypoint query items. Key value item of key point Key value item and The similarity between them as a basis The coefficients of the weighted summation: The human motion feature output of the k-th self-attention head is represented as: The ability to represent human motion features is improved by merging the outputs from h self-attention heads: After the self-attention modeling of the scene point cloud and human joints at layer l+1, the scene structural features are transformed from... Updated to Human movement characteristics are formed by Updated to C l+1 Indicates the feature dimension at the time of output; Unlike the self-attention layer, the cross-attention layer uses human skeletal data. This will be considered a cross-attention query. Used to measure the interaction between each body joint and the scene point cloud. Therefore, the cross-attention mechanism is defined as: Where: the human-scene interaction feature output of the kth cross-attention head is represented as: Improving human-scene interaction features by merging the outputs from h cross-attention heads. l+1 The ability to represent; Finally, the above scene structural features S l+1 Human movement characteristics B l+1 Human-scene interaction features I l+1 The signals are merged into a multi-attention encoding and fed into a decoder composed of a temporal convolutional network to predict the future human motion posture and trajectory that are reasonable and consistent with the scene interaction.

2. The method for predicting human behavior in scene point clouds according to claim 1, characterized in that: A novel scene perception paradigm based on 3D point clouds is constructed. Through 3D scanning or 3D reconstruction, the task of human motion prediction in scene perception is redefined as a joint reasoning problem based on historical 3D human motion information and current 3D scene structure information. This effectively predicts future human motion sequences that are both coherent and smooth in their own movements and compatible with the surrounding environment.

3. The method for predicting human behavior in scene point clouds according to claim 1, characterized in that: Redundancy in scene point cloud data can be reduced by selecting key regions: Based on the continuous nature of human movement, the range of human movement usually expands linearly with the continuation of movement time in the future; therefore, the range of human movement in the scene in the past time period is used as an anchor point, and the scene is linearly scaled according to the ratio of the predicted duration of future movement to the known duration of historical movement, thereby cropping out the key regions in the scene.

Images