A multi-person action prediction method based on the combination of self-relation and group relationship

Abstract

The present invention proposes a multi-person action prediction method based on the combination of self-relationship and group relationship, including obtaining a training data set; the training data set includes a human action sequence; constructing a basic human action prediction model, the human action prediction model includes: a group interaction representation learning module, a self-relationship representation learning module, and a synchronous Transformer; inputting the training data set into the basic human action prediction model for training to obtain a human action prediction model; wherein the loss function used in the training process is a final loss function obtained by weighted combination of the average position error of each joint and the average speed error of each joint; and realizing multi-person action prediction according to the human action prediction model. Through the method proposed by the present invention, the future joint point coordinates of the target can be predicted according to the observed joint point coordinates, and the multi-person action can be accurately predicted.

Description

Technical Field

[0001] The invention belongs to the field of action prediction. Background Art

[0002] Multi-person action prediction plays an important role in fields such as autonomous driving, human-computer interaction, and monitoring systems. The task of multi-person action prediction is to predict the future joint coordinates of the target based on the observed joint coordinates. Compared with single-person action prediction, multi-person action prediction has more practical application significance because in real life, it is more common for multiple people to exist in a scene at the same time. However, due to the complex connections and interactions between multiple targets, multi-person action prediction is also more challenging. Previous Transformer-based methods have achieved good results in multi-person action prediction tasks, but these methods all input the joints of all people in a scene as a whole into a Transformer network based on the self-attention mechanism to learn the connections between the joints, including the relationship within the own joints (self-relationship) and the relationship between the own joints and the joints of others (group interaction relationship). Moreover, these methods model and learn the self-relationship and group relationship together, and cannot explicitly express and distinguish the self-relationship and group relationship.

[0003] Although the Transformer-based method helps to solve complex dependency problems to a certain extent, there is still a problem to consider: the effect of Transformer when processing a single-person internal relationship with less joint point data is usually not as good as the effect of group interaction relationship in a multi-person scene with relatively more data sources. Specifically, when the data source is insufficient, Transformer cannot play the expected effect, especially for the action prediction task based on the human skeleton joint point. In addition, if the present invention ignores the kinematic laws and roughly treats the three-dimensional skeleton data as ordinary coordinate points, it will produce imperfect, unrealistic predictions, resulting in reduced accuracy. Summary of the invention

[0004] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the purpose of the present invention is to propose a multi-person action prediction method based on the combination of self-relationship and group relationship, so as to realize multi-person action prediction.

[0006] To achieve the above-mentioned purpose, the first embodiment of the present invention proposes a multi-person action prediction method based on the combination of self-relationship and group relationship, comprising:

[0007] Acquire a training data set; the training data set includes a human motion sequence;

[0008] Constructing a basic human motion prediction model, the human motion prediction model includes: a group interaction representation learning module, a self-relation representation learning module, and a synchronous Transformer;

[0009] The training data set is input into the basic human motion prediction model for training to obtain a human motion prediction model; wherein the loss function used in the training process is a final loss function obtained by weighted combination of the average position error of each joint and the average velocity error of each joint;

[0010] Multi-person action prediction is achieved according to the human action prediction model.

[0011] In addition, the multi-person action prediction method based on the combination of self-relationship and group relationship according to the above embodiment of the present invention may also have the following additional technical features:

[0012] Furthermore, in one embodiment of the present invention, obtaining a training data set includes:

[0013] The observed human action sequence is defined as Represents the length of the observed action sequence, each The three-dimensional joint coordinates representing each joint point of the action of the nth person at time t;

[0014] Define the velocity coordinates of the nth person at time t It is obtained by differentiating the position coordinates. The formula is as follows:

[0015]

[0016] The position coordinates and the velocity coordinates Stitch together to get a new coordinate Finally, a new sequence is obtained As training data.

[0017] Furthermore, in one embodiment of the present invention, the step of constructing a basic human motion prediction model includes:

[0018] Acquire group interaction representation through the group interaction representation learning module;

[0019] Acquire the self-relation representation through the self-relation representation learning module;

[0020] The interaction between the self-relation representation and the group interaction representation is enhanced by the synchronous Transformer.

[0021] Further, in one embodiment of the present invention, the acquiring the group interaction representation through the group interaction representation learning module includes:

[0022] Use a two-layer multilayer perceptron as the encoder to expand the training dataset to a higher dimension F encoder :

[0023]

[0024] Among them, Z 1 , Z 2 It represents the training data generated by the concatenation of position and speed, and the superscripts 1 and 2 represent the first person and the second person, respectively;

[0025] F encoder Input is fed into a Transformer consisting of four stacked layers of cross attention.

[0026]

[0027]

[0028] in, is the output of the Transformer at layer l, PE(·) represents the position encoding operation, MH(·) represents the multi-head attention calculation, Q 1 represents the query of the first person, V 2 With K 2 The key and value representing the second person, represents the scaling factor;

[0029] Get group interaction representation Represents a splicing operation.

[0030] Further, in one embodiment of the present invention, the obtaining the self-relation representation through the self-relation representation learning module includes:

[0031] The encoder representation F encoder Input is a self-relation representation learning module composed of 13 layers of graph convolution blocks:

[0032]

[0033]

[0034] in, represents the output of the l-th graph convolutional layer of the n-th person, Drop(·) represents random dropout, σ(·) represents the activation function, and A l represents the adjacency matrix of the lth graph convolutional layer, BN(·) represents batch normalization, and W l represents the learnable parameter matrix;

[0035] Get self-relationship representation Represents a splicing operation.

[0036] Furthermore, in one embodiment of the present invention, the step of enhancing the interaction between the self-relation representation and the group interaction representation by the synchronous Transformer includes:

[0037] The self-relation representation is projected into a new dimension through a secondary projection:

[0038]

[0039] Where W1, W2, W3 are learnable parameters, and ⊙ represents the Hadamard product;

[0040] Representing their own relationships Interacting with the group representation F so-out Add relative position encoding:

[0041]

[0042]

[0043] in, and Represents the input of the lth block, and Represents the output of the lth block;

[0044] The two representations are fused through the variant attention mechanism, where the variant attention mechanism is expressed as:

[0045]

[0046] Among them, MHA(·) is a variant of the attention mechanism;

[0047] Update the interaction representation and relationship representation through the update module:

[0048]

[0049]

[0050] Among them, f IU With f RU They are all composed of two layers of MLP;

[0051] The representation obtained by synchronous Transformer The decoder obtains the final prediction result on the position vector The final prediction result on the velocity vector

[0052] Furthermore, in one embodiment of the present invention, the step of inputting the training data set into the basic human motion prediction model for training to obtain the human motion prediction model further includes:

[0053] The average per-joint position error loss and the average per-joint velocity error loss are combined into the final loss function in a weighted manner; where

[0054] The average per-joint position error loss is defined as:

[0055]

[0056] Where P is the number of frames in the predicted pose sequence, J is the number of joints used in training, represents the true value of the 3D coordinate of the j-th joint in the t-th frame, represents the predicted coordinates, ||‖2 represents the l2 norm;

[0057] The average per-joint velocity error loss is defined as:

[0058]

[0059] in, is the velocity information of the jth joint in the true value sequence of the tth frame, Represents the predicted speed information;

[0060] The final loss function is expressed as:

[0061]

[0062] To achieve the above-mentioned purpose, the second embodiment of the present invention proposes a multi-person action prediction device based on the combination of self-relationship and group relationship, comprising the following modules:

[0063] An acquisition module, used to acquire a training data set; the training data set includes a human body motion sequence;

[0064] A construction module is used to construct a basic human motion prediction model, wherein the human motion prediction model includes: a group interaction representation learning module, a self-relation representation learning module, and a synchronous Transformer;

[0065] A training module, used for inputting the training data set into the basic human motion prediction model for training to obtain a human motion prediction model; wherein the loss function used in the training process is a final loss function obtained by weighted combination of the average position error of each joint and the average velocity error of each joint;

[0066] The prediction module is used to realize multi-person action prediction according to the human action prediction model.

[0067] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the multi-person action prediction method based on the combination of self-relationships and group relationships as described above.

[0068] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, a multi-person action prediction method based on the combination of self-relationships and group relationships as described above is implemented.

[0069] The multi-person action prediction method based on the combination of self-relationship and group relationship proposed in the embodiment of the present invention, on the one hand, proposes a processing method for explicitly representing individual relationship and group relationship; and proposes to learn group interaction relationship through cross-attention mechanism; explicitly constructs the intrinsic relationship of individual skeletal joints through GCN; on the other hand, a novel synchronous Transformer based on a variant of the attention mechanism is proposed, which can synchronously fuse the extracted intrinsic relationship of individual skeletal joints and the interaction relationship between groups to produce smoother and more accurate actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0071] Figure 1 A flowchart of a multi-person action prediction method based on the combination of self-relationship and group relationship provided by an embodiment of the present invention;

[0072] Figure 2 A network architecture diagram of self-relationship, group interaction and synchronous Transformer provided by an embodiment of the present invention;

[0073] Figure 3 A schematic diagram of a visualization result of a "playing basketball" action of a 3DPW-SoMoF / RC dataset provided by an embodiment of the present invention;

[0074] Figure 4 A schematic diagram of the structure of a multi-person action prediction device based on the combination of self-relationship and group relationship provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0075] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0076] The following describes a multi-person action prediction method based on the combination of self-relationship and group relationship according to an embodiment of the present invention with reference to the accompanying drawings.

[0077] Figure 1 A flowchart of a multi-person action prediction method based on the combination of self-relationship and group relationship provided by an embodiment of the present invention.

[0078] like Figure 1 As shown, the multi-person action prediction method based on the combination of self-relationship and group relationship includes the following steps:

[0079] S101: Acquire a training data set; the training data set includes a human motion sequence;

[0080] Given a scene with N people (each with J joint points), the present invention defines the observed sequence as Represents the length of the observed action sequence, each Represents the three-dimensional joint point coordinates of each joint point of the action of the nth person at time t. Similarly, the present invention defines the predicted sequence as P represents the predicted sequence length. The true value can be defined as The goal of the present invention is to obtain a predicted action sequence through an observed action sequence and make it close to the true value.

[0081] Figure 2 The network architecture diagram showing the self-relationship and group interaction as well as the synchronous Transformer, Figure 2 The overall framework is described, in which two branches process group interaction representation and self-relationship representation respectively, and finally these two representations are fused through synchronous Transformer to achieve more accurate prediction.

[0082] Furthermore, in one embodiment of the present invention, obtaining a training data set includes:

[0083] The observed human action sequence is defined as Represents the length of the observed action sequence, each The three-dimensional joint coordinates representing each joint point of the action of the nth person at time t;

[0084] Define the velocity coordinates of the nth person at time t It is obtained by position coordinate difference, the formula is as follows:

[0085]

[0086] The position coordinates and the velocity coordinates Stitch together to get a new coordinate Finally, a new sequence is obtained As training data.

[0087] S102: constructing a basic human motion prediction model, wherein the human motion prediction model includes: a group interaction representation learning module, a self-relation representation learning module, and a synchronous Transformer;

[0088] Furthermore, in one embodiment of the present invention, the step of constructing a basic human motion prediction model includes:

[0089] Acquire group interaction representation through the group interaction representation learning module;

[0090] Acquire the self-relation representation through the self-relation representation learning module;

[0091] The interaction between the self-relation representation and the group interaction representation is enhanced by the synchronous Transformer.

[0092] The present invention uses the joint point position coordinates and velocity coordinates as the input of the module, enhancing the information of the joint points instead of just using the position coordinates. Suppose the velocity coordinates of the nth person at time t are It is obtained by position coordinate difference, the formula is as follows:

[0093]

[0094] The present invention converts the position coordinates and velocity coordinates Stitch together to get a new coordinate Finally, a new sequence is obtained as input to the encoder.

[0095] Further, in one embodiment of the present invention, the acquiring the group interaction representation through the group interaction representation learning module includes:

[0096] Use a two-layer multilayer perceptron as the encoder to expand the training dataset to a higher dimension F encoder :

[0097]

[0098] Among them, Z 1 , Z 2It represents the training data generated by the concatenation of position and speed, and the superscripts 1 and 2 represent the first person and the second person, respectively;

[0099] F encoder Input is fed into a Transformer consisting of four stacked layers of cross attention.

[0100]

[0101]

[0102] in, is the output of the Transformer at layer l, PE(·) represents the position encoding operation, MH(·) represents the multi-head attention calculation, Q 1 represents the query of the first person, V 2 With K 2 The key and value representing the second person, represents the scaling factor;

[0103] Get group interaction representation Represents a splicing operation.

[0104] Further, in one embodiment of the present invention, the obtaining the self-relation representation through the self-relation representation learning module includes:

[0105] The encoder representation F encoder Input is a self-relation representation learning module composed of 13 layers of graph convolution blocks:

[0106]

[0107]

[0108] in, represents the output of the l-th graph convolutional layer of the n-th person, Drop(·) represents random dropout, σ(·) represents the activation function, and A l represents the adjacency matrix of the lth graph convolutional layer, BN(·) represents batch normalization, and W l represents the learnable parameter matrix;

[0109] Get self-relationship representation Represents a splicing operation.

[0110] Furthermore, in one embodiment of the present invention, the step of enhancing the interaction between the self-relation representation and the group interaction representation by the synchronous Transformer includes:

[0111] The self-relation representation is projected into a new dimension through a secondary projection:

[0112]

[0113] Where W1, W2, W3 are learnable parameters, and ⊙ represents the Hadamard product;

[0114] Representing their own relationships Interacting with the group representation F so-out Add relative position encoding:

[0115]

[0116]

[0117] in, and Represents the input of the lth block, and Represents the output of the lth block;

[0118] The two representations are fused through the variant attention mechanism, where the variant attention mechanism is expressed as:

[0119]

[0120] Among them, MHA(·) is a variant of the attention mechanism;

[0121] Update the interaction representation and relationship representation through the update module:

[0122]

[0123]

[0124] Among them, f IU With f RU They are all composed of two layers of MLP;

[0125] The representation obtained by synchronous Transformer The decoder obtains the final prediction result on the position vector The final prediction result on the velocity vector

[0126] S103: inputting the training data set into the basic human motion prediction model for training to obtain a human motion prediction model; wherein the loss function used in the training process is a final loss function obtained by weighted combination of the average position error of each joint and the average velocity error of each joint;

[0127] Furthermore, in one embodiment of the present invention, the step of inputting the training data set into the basic human motion prediction model for training to obtain the human motion prediction model further includes:

[0128] The average per-joint position error loss and the average per-joint velocity error loss are combined into the final loss function in a weighted manner; where

[0129] The average per-joint position error loss is defined as:

[0130]

[0131] Where P is the number of frames in the predicted pose sequence, J is the number of joints used in training, represents the true value of the 3D coordinate of the j-th joint in the t-th frame, represents the predicted coordinates, ||||2 represents the l2 norm;

[0132] The average per-joint velocity error loss is defined as:

[0133]

[0134] in, is the velocity information of the jth joint in the true value sequence of the tth frame, Represents the predicted speed information;

[0135] The final loss function is expressed as:

[0136]

[0137] S104: Implement multi-person motion prediction according to the human motion prediction model.

[0138] The visualization results of the "playing basketball" action based on self-relationships, group interactions, and synchronized Transformers in the 3DPW-SoMoF / RC dataset are as follows: Figure 3 shown.

[0139] The multi-person action prediction method based on the combination of self-relationship and group relationship proposed in the embodiment of the present invention, on the one hand, proposes a processing method for explicitly representing individual relationship and group relationship; and proposes to learn group interaction relationship through cross-attention mechanism; explicitly constructs the intrinsic relationship of individual skeletal joints through GCN; on the other hand, a novel synchronous Transformer based on a variant of the attention mechanism is proposed, which can synchronously fuse the extracted intrinsic relationship of individual skeletal joints and the interaction relationship between groups to produce smoother and more accurate actions.

[0140] In order to implement the above embodiment, the present invention also proposes a multi-person action prediction device based on the combination of self-relationship and group relationship.

[0141] Figure 4 A schematic diagram of the structure of a multi-person action prediction device based on the combination of self-relationship and group relationship provided by an embodiment of the present invention.

[0142] like Figure 4 As shown, the multi-person action prediction device based on the combination of self-relationship and group relationship includes: an acquisition module 100, a construction module 200, a training module 300, and a prediction module 400, wherein:

[0143] An acquisition module, used to acquire a training data set; the training data set includes a human body motion sequence;

[0144] A construction module is used to construct a basic human motion prediction model, wherein the human motion prediction model includes: a group interaction representation learning module, a self-relation representation learning module, and a synchronous Transformer;

[0145] A training module, used for inputting the training data set into the basic human motion prediction model for training to obtain a human motion prediction model; wherein the loss function used in the training process is a final loss function obtained by weighted combination of the average position error of each joint and the average velocity error of each joint;

[0146] The prediction module is used to realize multi-person action prediction according to the human action prediction model.

[0147] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the multi-person action prediction method based on the combination of self-relationships and group relationships as described above.

[0148] To achieve the above-mentioned purpose, the fourth aspect of the present invention proposes a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the multi-person action prediction method based on the combination of self-relationships and group relationships as described above.

[0149] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0150] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0151] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A multi-person action prediction method based on the combination of self-relationship and group relationship, characterized in that: The following steps are involved: Get the training dataset; The training data set includes human action sequences; Constructing a basic human motion prediction model, the human motion prediction model includes: a group interaction representation learning module, a self-relation representation learning module, and a synchronous Transformer; The training data set is input into the basic human motion prediction model for training to obtain a human motion prediction model; wherein the loss function used in the training process is a final loss function obtained by weighted combination of the average position error of each joint and the average velocity error of each joint; Implementing multi-person motion prediction according to the human motion prediction model; The constructing of the basic human motion prediction model comprises: Acquire group interaction representation through the group interaction representation learning module; Acquire the self-relation representation through the self-relation representation learning module; Enhance the interaction between the self-relation representation and the group interaction representation by the synchronous Transformer; The acquiring the group interaction representation through the group interaction representation learning module includes: Use a two-layer multilayer perceptron as the encoder to scale the training dataset to a higher dimension : in, , It represents the training data generated by the concatenation of position and speed, and the superscripts 1 and 2 represent the first person and the second person, respectively; Will Input is fed into a Transformer consisting of four stacked layers of cross attention. in, It is The output of the Transformer of the layer, represents the position encoding operation, represents multi-head attention calculation, represents the query of the first person, and The key and value representing the second person, represents the scaling factor; Get group interaction representation , Represents a splicing operation; The obtaining of the self-relation representation through the self-relation representation learning module includes: The encoder characterization Input is a self-relation representation learning module composed of 13 layers of graph convolution blocks: in, Representative The first The output of the graph convolutional layer, stands for random dropout, represents the activation function, Representative The adjacency matrix of the graph convolutional layer, stands for batch normalization, represents the learnable parameter matrix; Get self-relationship representation , Represents a splicing operation; The step of enhancing the interaction between the self-relation representation and the group interaction representation by the synchronous Transformer includes: The self-relation representation is projected into a new dimension through a secondary projection: in is a learnable parameter, It means Adama product; Representing their own relationships Interacting with groups Add relative position encoding: in, and Representative The input of a block, and Representative The output of a block; The two representations are fused through the variant attention mechanism, where the variant attention mechanism is expressed as: in, It is a variant of the attention mechanism; Update the interaction representation and relationship representation through the update module: in, and They are all composed of two layers of MLP; The representation obtained by synchronous Transformer The decoder obtains the final prediction result on the position vector The final prediction result on the velocity vector .

2. The method according to claim 1, characterized in that The step of obtaining a training data set includes: The observed human action sequence is defined as , Represents the length of the observed action sequence, each Representative Individuals in The three-dimensional joint coordinates of each joint point of the action at any moment; Definition Individuals in Velocity coordinates at time It is obtained by position coordinate difference, the formula is as follows: The position coordinates and the velocity coordinates Stitch together to get a new coordinate , and finally obtain a new sequence As training data.

3. The method according to claim 1, characterized in that The step of inputting the training data set into the basic human motion prediction model for training to obtain the human motion prediction model further comprises: The average per-joint position error loss and the average per-joint velocity error loss are combined into the final loss function in a weighted manner; where The average per-joint position error loss is defined as: in, is the number of frames in the predicted pose sequence, is the number of joints used in training, Indicates The 3D coordinates of the joints The true value of the frame, represents the predicted coordinates, express norm; The average per-joint velocity error loss is defined as: in, It is The true value sequence of the frame The velocity information of each joint, Represents the predicted speed information; The final loss function is expressed as: 。 4. A multi-person action prediction device based on the combination of self-relationship and group relationship using the method as claimed in claim 1, characterized in that: Includes the following modules: An acquisition module is used to obtain a training data set; The training data set includes human action sequences; A construction module is used to construct a basic human motion prediction model, wherein the human motion prediction model includes: a group interaction representation learning module, a self-relation representation learning module, and a synchronous Transformer; A training module, used for inputting the training data set into the basic human motion prediction model for training to obtain a human motion prediction model; wherein the loss function used in the training process is a final loss function obtained by weighted combination of the average position error of each joint and the average velocity error of each joint; The prediction module is used to realize multi-person action prediction according to the human action prediction model.

5. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for predicting multi-person actions based on the combination of self-relationship and group relationship as described in any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-person action prediction method based on the combination of self-relationship and group relationship as described in any one of claims 1 to 3 is implemented.

Images