A reaction action generation method in two-person interaction
By using intent prediction and causal graph learning models, the accuracy problem of reaction action generation in existing technologies has been solved, achieving efficient and accurate generation of two-person interactive actions, and improving the intelligence and naturalness of interaction between robots and game NPCs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-06-30
- Publication Date
- 2026-07-31
AI Technical Summary
Existing technologies suffer from inaccurate action types and joint positions/postures when generating reaction actions, especially in two-player interaction scenarios, particularly in the inaccurate generation of NPC actions in fighting games.
An intention prediction model and a causal graph learning model were designed. By predicting the latent variables of the agent's intention, and combining them with the causal graph learning model, the reaction actions of the passive party are generated, thereby improving the accuracy of the type and position/pose of the generated actions.
It significantly improves the accuracy and efficiency of generating reaction actions, making the generation of robot and game NPC actions more natural and flexible, meeting real-time requirements, and adapting to various generation frameworks.
Smart Images
Figure CN122491352A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the interdisciplinary fields of computer vision and computer graphics, specifically to a method for generating reaction actions in two-person interaction. Background Technology
[0002] In real-world interpersonal interactions, one person's action often triggers a response from another. For example, when two people shake hands, one extends their hand, and the other needs to reciprocate with a similar gesture. Therefore, action generation uses one person's action as a condition to generate a reasonable response from the other. This is a key technology in AI interactive intelligence, playing a crucial role in improving the timeliness and realism of interactions.
[0003] Furthermore, automated response generation can replace manual production, lowering the production threshold and improving content production efficiency and diversity. This method has wide applications in fields such as industry, education, healthcare, entertainment, gaming, and animation, such as enabling companion robots and medical robots to have natural responsiveness.
[0004] In the past, the field of reaction generation could be divided into two categories based on algorithm principles: diffusion-based methods and autoregressive generation methods.
[0005] Diffusion model-based methods: These methods, such as ReGenNet, take the input of the actor as a condition, model the interaction relationship between the two through a Transformer, and use a diffusion model architecture to predict the actions of the reactant.
[0006] Autoregressive methods: These methods, such as Ready-to-React, use VQ-VAE to encode the historical actions of themselves and the opponent as hidden sequences, and then use an autoregressive approach to generate the passive party's actions.
[0007] Past methods primarily suffer from inaccurate generation results, typically falling into two categories: First, inaccurate reaction action types. For example, a celebratory gesture like raising a glass might be misinterpreted by the model as the other person throwing the glass, resulting in an incorrect generated passive action. Second, inaccurate joint positions and postures within the reaction action. For instance, in a handshake, past methods often failed to accurately grasp the handshake.
[0008] Therefore, existing technologies still lack a method for generating reaction actions that are accurate in type, position / posture. For example, in fighting games, there is a lack of a method for generating reaction actions of NPCs (non-player characters, passive parties) to player characters (active parties) that are accurate in type, position / posture. Summary of the Invention
[0009] To overcome the problems existing in the prior art, this invention provides a method for generating reaction actions in two-person interaction. An intent prediction model is designed to predict the latent intent variables of the Actor's (initiator's) actions, thus providing strong prior knowledge and constraints when generating Reactor (responder's) reaction actions, thereby improving the accuracy of generated action response types. A causal graph learning model is designed to guide the generation accuracy of Reactor joints, thereby improving the position and pose accuracy of the entire Reactor action generation.
[0010] This invention proposes a method for generating reaction actions in two-person interaction, comprising an active party and a passive party, and including the following steps:
[0011] A two-person interaction model is constructed and trained. The two-person interaction model includes a variational intent prediction model and a causal graph learning model. The trained two-person interaction model includes a trained variational intent prediction model and a trained causal graph learning model.
[0012] Obtain the spatiotemporal characteristics of the active party's actions;
[0013] The spatiotemporal features of the active party's actions are input into the trained variational intent prediction model to obtain the latent intent variables of the active party's actions.
[0014] By inputting the spatiotemporal features of the active party's actions and the latent variables of its intentions into the trained causal graph learning model, a causal graph representing the influence relationship between each joint of the active party and each joint of the passive party is obtained.
[0015] Based on the spatiotemporal features of the active party's actions and the causal graph, the passive party's actions are generated.
[0016] Preferably, for the Online configuration, the steps for obtaining the cause-effect graph include:
[0017] Active party generation When the action occurs at a given moment, it is achieved through aggregation. Historical information is used to obtain the j-th joint feature of the active party at time t. Then, the key matrix of the j-th joint of the active party at time t is obtained through processing by a multilayer perceptron. ;
[0018] definition For the learnable joint embeddings of the passive party, the embeddings of the i-th joint of the passive party are concatenated with the latent variables of the intent, and then processed using a multilayer perceptron to obtain the learnable joint embeddings of the passive party. Query matrix of each joint ;
[0019] based on and Calculate the correlation, and then apply the activation function. Processing to obtain the active party's first time at time t The first joint for the passive party Graph structure of the influence relationship of each joint ;
[0020] Based on graph structure Constructing a cause-and-effect graph .
[0021] As a preferred option, the two-person interaction model also includes a conditional action generation model. The trained two-person interaction model also includes a trained conditional action generation model. The trained conditional action generation model generates the passive party's actions based on the spatiotemporal features of the active party's actions and the causal graph.
[0022] A time mask matrix was designed in the conditional action generation model to ensure that the passive party can only see the current and past actions of the active party.
[0023] As a preferred approach, the Transformer is used to construct the action generation network in the conditional action generation model;
[0024] Time mask matrix used This indicates that the constraint rule for the time mask matrix is: when the historical moment... Less than or equal to At time t, the mask matrix takes the value of 0; when Greater than At time t, the mask matrix takes the value of negative infinity.
[0025] As a preferred definition For the passive party One joint, up to The state characteristics of actions already performed at any given moment;
[0026] Time mask matrix Calculation method:
[0027]
[0028] when hour, This will cause the attention weight at that moment to be forcibly reset to zero after Softmax, thus preventing the leakage of future information of the Actor.
[0029] Cause-effect graph Attention scores are constructed by introducing a Mask bias, which serves as the spatial dimension of attention, into attention computation. ;
[0030]
[0031] Atten_Score is a temporary variable that represents the attention score. , These are two matrices, representing the parameters the model needs to learn. Indicates transpose, D is The feature dimensions. This is a characteristic of the Actor (the active party), meaning that at time t, the j-th joint of the Actor in the past... time( Features of (=1...t). Representation diagram One of the elements (graph structure). It represents a very small value greater than 0.
[0032]
[0033] Softmax is the Softmax function. This is the temporary variable obtained, used for the following calculation:
[0034]
[0035] In the above formula This is a matrix containing the parameters learned by the model. The results are then mapped back to a 6D rotated representation through a linear layer to obtain the Reactor's actions. :
[0036]
[0037]
[0038] Preferably, the variational intent prediction model includes a prior network. The steps of inputting the spatiotemporal features of the agent's actions into the trained variational intent prediction model to obtain the latent variables of the agent's actions include:
[0039] The spatiotemporal features of the active party's actions are pooled and then input into the trained prior network, which outputs the mean and variance estimated by the prior network.
[0040] Based on the mean and variance estimated by the prior network, a normal distribution is constructed to obtain the prior distribution of the initiative's intention, and then the latent variable of the initiative's intention is obtained.
[0041] Preferably, the two-person interaction model includes a spatiotemporal feature encoding model, a variational intent prediction model, a causal graph learning model, and a conditional action generation model. The variational intent prediction model includes a prior network and a posterior network. The training process of the two-person interaction model includes:
[0042] Acquire training data, which includes the action sequence of the active party and the actual reaction action sequence of the passive party;
[0043] The spatiotemporal features of the active party's action sequence and the passive party's actual reaction action sequence are extracted using a spatiotemporal feature coding model to obtain the spatiotemporal features of the active party's action sequence and the passive party's action sequence.
[0044] The spatiotemporal features of the active party's actions and the passive party's actions are input into the posterior network of the variational intent prediction model to obtain the latent intent variables of the active party's actions.
[0045] The spatiotemporal features of the active party's actions and the latent variables of its intentions are input into the causal graph learning model to obtain a causal graph that represents the influence relationship between each joint of the active party and each joint of the passive party.
[0046] The spatiotemporal features of the active party's actions and the causal graph are input into the conditional action generation model to generate the passive party's actions.
[0047] Construct a loss function, update the network parameters of the model based on the loss function, and repeat the training until the model converges.
[0048] Preferably, the spatiotemporal features of the active party's actions and the passive party's actions are concatenated, pooled, and then input into the posterior network, which outputs the mean and variance estimated by the posterior network.
[0049] Based on the mean and variance estimated by the posterior network, a normal distribution is constructed to obtain the prior distribution of the initiator's intention, and then the latent variable of the initiator's intention is obtained.
[0050] Preferably, the steps for obtaining the spatiotemporal features of the active party's actions include:
[0051] Obtain the action sequence of the active party It is represented by 6D continuous rotation; Represents the set of real numbers. This represents the total number of frames in the action sequence. Indicates the number of joints in the human skeleton;
[0052] The velocity of a single joint in a single frame is defined as the difference between the state of the same joint in the current frame and the previous frame. Based on this, the overall velocity of the active party can be expressed as... Introducing velocity makes the modeling of position changes more accurate.
[0053] After concatenating the action sequence and overall velocity of the active party, a lightweight graph convolutional network is used to extract spatial topological features, and then a lightweight Transformer encoder is used to extract temporal features, thus obtaining the spatiotemporal features of the active party's actions. Each joint has Dimensional features.
[0054] Preferably, the loss function includes divergence loss, graph loss, and reconstruction loss. By calculating the corresponding loss value and obtaining the gradient, the weights of the spatiotemporal feature encoding model, variational intent prediction model, causal graph learning model, and conditional action generation model are updated using the gradient. The training process is repeated until the model converges.
[0055] Preferably, the gradient generated by the divergence loss is used to update the parameters of the prior and posterior networks;
[0056] The gradient generated by the graph loss is used to update the parameters of the causal graph learning model;
[0057] The gradient generated by the reconstruction loss is used to update the parameters of the entire model.
[0058] Compared with the prior art, the beneficial effects of the present invention are:
[0059] (1) An intent prediction model was designed to predict the latent variables of the Actor's (active party's) intention, thus providing stronger prior knowledge and constraints when generating Reactor (responder's) reaction actions, thereby improving the accuracy of generated action response types. A causal graph learning model was designed, in which each joint of the passive reactant adaptively perceives the influence of each joint of the active party, guiding the Reactor's joint-level generation accuracy. This improves the position and pose accuracy of the entire Reactor action generation.
[0060] (2) Significantly improves the efficiency of human-to-human interaction animation production. In animation and game production scenarios, a moderately skilled engineer needs about 0.5 days to create a 5-second animation. Our method, however, only requires 0.05~0.1 seconds (depending on the size of the model used). It also enhances the intelligence of robots. After integrating our method, robots can interact with the outside world more naturally and flexibly. Furthermore, it increases the fun of games. In the past, NPC actions in games were often pre-set. After integrating our method, natural, reasonable, and diverse actions can be generated for NPCs online in real time.
[0061] (3) This solution can adapt to a variety of generative frameworks and is highly flexible. The two innovative designs above can be used in the diffusion model framework as well as the autoregressive generative framework. It can be flexibly switched according to the limitations of the deployment environment. Attached Figure Description
[0062] Figure 1 This is a schematic diagram of the parametric representation of the human body;
[0063] Figure 2 A schematic diagram of the reasoning process of the method of the present invention;
[0064] Figure 3 This is a schematic diagram of the training process of the method of the present invention;
[0065] Figure 4 This is a visual demonstration of two people working together to move things in the comparative example of this invention;
[0066] Figure 5 This is a visual representation of one person helping another stand up in the comparative example of this invention;
[0067] Figure 6 This is a visual representation of one person holding another person's arm in the comparative example of this invention;
[0068] Figure 7 This is a visual representation of one person massaging another person in the comparative example of this invention;
[0069] Figure 8 This is a visual representation of one person whispering to another in the comparative example of this invention. Detailed Implementation
[0070] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0071] This invention proposes a method for generating responsive actions in two-person interactions, belonging to the intersection of computer vision and computer graphics. This invention can be applied to fields such as games, animation, embodied intelligence, and digital humans to generate responsive actions in two-person interactions. Specifically, given a person's action (Actor), this method generates a responsive action (Reactor) to interact with that person. For example, when two people shake hands, one extends their right hand, and the other also extends their right hand (the action should be naturally coordinated with the actor's action in time and space) to complete the handshake.
[0072] Some symbol definitions for this method
[0073] : Total number of frames in the action sequence.
[0074] The number of joints in the human skeleton (usually) ),like Figure 1 As shown.
[0075] Feature dimensions of the hidden layer of a neural network.
[0076] The input action sequence of the Actor is represented by 6D continuous rotations. It is the set of real numbers;
[0077] :The ground truth sequence of the Reactor's (passive party) response actions.
[0078] The sequence of Reactor reaction actions generated (predicted) by the model.
[0079] : The latent space's intent vector, used to model the actor's intent for their actions.
[0080] Online Settings: Reactor The action at any given moment can only be seen by the Actor. Actions within a specific time period cannot predict the actor's future actions (for online real-time generation, it's impossible to foresee the actor's future actions). However, this limitation does not apply to offline settings.
[0081] The working principle of each module is described below.
[0082] 1. Spatiotemporal Feature Coding Model
[0083] Design objective: The original 6D rotation matrix consists of independent numerical values in a local coordinate system, lacking spatial kinematic topology and temporal coherence between joints. This module maps it to hidden features containing higher-order dynamic information.
[0084] Definition: The velocity of a single joint in a single frame is Then the overall velocity of Actor(X) can be expressed as
[0085] Spatial topological features are extracted using a lightweight graph convolutional network (GCN, 3 layers):
[0086] Temporal features are extracted using a lightweight Transformer encoder (Encoder, 3 layers):
[0087]
[0088] D represents the feature dimension. In the two formulas above:
[0089] : Concatenation operation along the feature dimension. Standard timing position coding. The final global spatiotemporal feature representation of the Actor.
[0090] During the training phase, we use the same spatiotemporal feature encoding network to extract the spatiotemporal features of the real reaction action Reactor (Y):
[0091] Research and development process description:
[0092] 1) We initially tried to use only 6D rotation representation, without introducing velocity ( The generated results showed a large positional error. Introducing velocity made the modeling of positional changes more accurate.
[0093] 2) Spatial topological features: We previously tried using the Spatial Transformer for extraction, but found it inferior to GCN. The spatial and temporal features described above were extracted using a heterogeneous model, resulting in better robustness and diversity of the features.
[0094] 2. Variational Intent Prediction
[0095] Design Purpose: Faced with the same attack action from an Actor, the Reactor can either "block" or "dodge." Conventional models typically only learn average reactions (leading to unrealistic actions). The intent prediction module outputs a latent variable. This allows for the acquisition of "higher-level semantic strategies" to control responses.
[0096] 2.1 Training Phase
[0097] like Figure 2 As shown, the encoder observes simultaneously and Post-hoc networks Predict the distribution parameters of the actual intention that would produce the actual reaction action.
[0098]
[0099] Represents the posterior network. This represents the set of all parameters of the network. This indicates that average pooling is performed along the time and joint dimensions, compressing the tensor into a one-dimensional global vector. , These represent the mean and variance of the posterior network estimate.
[0100] To enable gradient backpropagation, reparameterization sampling based on a normal distribution is required.
[0101] ;
[0102] This indicates a normal distribution. It is an abbreviation for diagonal matrix. Assuming that the dimensions of z are independent, we get the simplified covariance matrix.
[0103] This represents the estimated posterior distribution.
[0104] 2.2 Reasoning Stage
[0105] like Figure 1 As shown, the prediction network only predicts the actions of the actors. Predict the possible distribution of intentions.
[0106]
[0107]
[0108] In the above formula Indicates the prior network, This represents the set of all parameters of the network.
[0109] This represents the estimated prior distribution.
[0110] 2.3 Loss Function Design—KL Divergence Loss
[0111]
[0112] D represents divergence, and the subscript KL stands for Kullback-Leibler. The specific method for calculating divergence is existing knowledge in the field of machine learning. The || in the middle is read as "relative," indicating how much the distribution after || differs from the distribution before ||.
[0113] The prior and posterior network parameters are updated using this loss function.
[0114] Research and development process description:
[0115] We previously tried to predict an intent directly using models such as Transformer or Mamba. The results showed that the accuracy was not high, and the results varied depending on the actor's different action types. While the results are similar, the final generated actions are not as reasonable as those in this solution.
[0116] 3. Intention-driven causal graph learning
[0117] Design Purpose: Traditional cross-attention mechanisms are often dense, but some correlations are meaningless. This module aims to better model the physical correlations between the various parts of two characters. For example, in an action where the Actor extends their right hand, corresponding to striking the Reactor's head, the correlation between the right hand and the head should be increased. It is the complete spatiotemporal characteristics of an Actor, containing frame, There are 1 joint, and each joint has 1 joint. Dimensional features.
[0118] For Online settings, generate the Reactor's first... Action at frame time, feature of the j-th joint of the Actor Only aggregation Historical information, actor characteristics Perform average pooling over the time dimension:
[0119] In the above formula, a fixed-weighted average or an exponential moving average can be used in the time dimension (the action at the most recent moment has a greater weight).
[0120] definition For learnable joint embeddings of the Reactor, the following operations are performed:
[0121]
[0122] In the above formula, It is an abbreviation for the Query matrix in the attention mechanism (Attention). Represents the Reactor's first Each joint. MLP stands for Multilayer Perceptron, and its subscripts Q and K both represent MLP, but with different parameters. Indicates the obtained The feature dimensions. This is an abbreviation for the Key matrix in the attention mechanism, where the subscript t indicates time t and j indicates the j-th joint of the Actor.
[0123] Then:
[0124]
[0125] Indicates transpose. It is a directed adjacency matrix, representing a graph structure, which indicates the position of Actor at time t. The first joint of the Reactor The influence weight of each joint. It replaces the traditional Softmax and improves the sparsity of the matrix. This forms a graph . This is a temporary variable defined in this invention, which can be called the correlation matrix.
[0126] Both MLP and MLP are learnable parameters.
[0127] Development process description: We previously tried not to use a learnable... Use directly The training process was found to be normal, but the generated actions during inference were very poor. Analysis revealed that the problem stemmed from the lack of accurate... This leads to inconsistencies between the reasoning phase and the training phase, with the reasoning phase yielding... That's inaccurate; the real issue is that the training phase leaked the true labels.
[0128] 4. Conditional Action Generation
[0129] We use the Transformer model as the generative network.
[0130] definition For Reactor Each joint, up to the current moment. So far, the state corresponding to the actions performed by the Reactor (which can be understood as the features of the decoded actions after passing through the Transformer; dec is an abbreviation for decoding).
[0131] The diagram As a spatial dimension attention mask bias injection:
[0132] In the above formula, Atten_Score is a temporary variable that represents the attention score. , These are two matrices, representing the parameters the model needs to learn, and D is... The feature dimensions. This is a characteristic of the Actor (the active party), meaning that at time t, the j-th joint of the Actor in the past... time( Features of (=1...t). Representation diagram One of the elements, It represents a very small value greater than 0. This is a temporary variable defined in this invention, denoted as the time mask matrix, and its calculation method and function are as follows:
[0133]
[0134] when hour, This will cause the attention weight at that moment to be forcibly reset to zero after Softmax, thus preventing the leakage of future information of the Actor.
[0135]
[0136] Softmax is the Softmax function. This is the temporary variable obtained, used for the following calculation:
[0137]
[0138] In the above formula This is a matrix containing the parameters learned by the model. The results are then mapped back to a 6D rotated representation through a linear layer to obtain the Reactor's actions. :
[0139]
[0140]
[0141] Research and development process description:
[0142] If the above is not introduced This can lead to a significant decrease in the real-time performance and accuracy of action responses during actual deployment, because the Actor cannot see the Reactor's future actions (on the training set, since the Actor and Reactor's actions are input together, they can see each other's actions over the entire time period). Therefore, to meet the constraints of real-world scenarios, a new mechanism must be introduced during the training and inference phases. It meets the constraints of actual use cases.
[0143] 5. Loss function during training
[0144] In addition to the KL divergence loss mentioned above, two other loss functions were designed:
[0145] Reconstruction losses ;
[0146] It is a coefficient with a value of 0-1, which controls speed loss.
[0147] For adjacency matrix The regularization loss is reduced by using L1 regularization.
[0148] The final overall loss function is:
[0149] ;
[0150] These are two coefficients, each with a value between 0 and 1, used to control the weight of the corresponding loss. The generated gradient will also guide The update of parameters changes the intent of its extraction. Increasingly accurate.
[0151] Research and development process description: We found that without This will lead to The result is too dense, which causes the generated Reactor actions to frequently jitter and abruptly change. Introducing... , The results are sparser but more accurate.
[0152] 6. Reasoning process ( )
[0153] We use an autoregressive generation method:
[0154] Initial state: In At that time, the input is a special [START] token.
[0155] • Gradual restoration: Generated from left to right, in the generation of the first... Frame time, Includes numbers from 1 to All the information in the frame.
[0156] • Mask Token: We use a Causal Mask (lower triangular matrix) to make... Only see at any time Information at the present moment is displayed, while information about future moments is hidden.
[0157] · As the time axis moves to the right, The coverage time window is getting longer and longer, and more and more actions are being decoded, until... All actions within the specified length are decoded.
[0158] To evaluate the results generated by this invention and previous ReGenNet methods, we adopted a user study approach commonly used in the industry. Specifically, we generated 30 results and had 18 participants evaluate each result from the two methods based on three dimensions: 1) accuracy of generated action type; 2) accuracy of generated action position / pose; and 3) overall stability. The percentage of participants who selected the best result was then tallied (higher is better). The evaluation results are shown in the table below:
[0159] Table 1 Subjective evaluation of the method of this invention and ReGenNet
[0160]
[0161] As can be seen from the above, our method outperforms ReGenNet's generation results across all three evaluation dimensions.
[0162] Some visual examples are shown
[0163] The following are some generated visualizations showing the following action: two people working together to move something. The blue figures represent the Actor's actions (input), and the red figures represent the generated Reactor's actions. The first row shows the results of this approach, and the second row shows the ReGenNet results.
[0164] Figure 4 Action: Two people are moving things together (top: this solution; bottom: ReGenNet). In the first two frames of the image below, the red figures are facing the wrong direction, resulting in incorrect action type and posture.
[0165] Figure 5 Action: One person helps another stand up (top: this solution; bottom: ReGenNet). In the image below, frames 3-5, the position and posture of the red figure are inaccurate, and the two people are intersecting (clipping).
[0166] Figure 6 Action: One person is holding another person's arm (top: this solution; bottom: ReGenNet). In the image below, frames 3 to 5, the position of the red person is inaccurate, causing clipping issues (for example, in frame 4, the arm is inserted into the body).
[0167] Figure 7 Action: One person massages another (Top: This solution; Bottom: ReGenNet). In the image below, frames 1 and 2 show clipping, and frames 3-5 are incorrectly positioned; the Reactor should be in the middle of the hand.
[0168] Figure 8Action: One person whispers to another (top: this solution; bottom: ReGenNet). In the image below, frame 3 shows clipping, and frame 4 shows inaccurate positioning, causing the other person to be a bit far away.
[0169] As can be seen from the subjective evaluation table and the visual examples, the action types and positions / postures generated by this solution are more accurate, demonstrating the advantages of this solution.
[0170] 7. Practical Application Cases
[0171] The reaction action generation method proposed in this invention has been applied to a fighting game. In this game, the player character corresponds to the active side, and NPCs (non-player characters) correspond to the passive side. It can generate natural reactions to 26 common actions, and typical action examples are shown in the table below.
[0172]
[0173] In this game, the inference time of the method of this invention is 11ms, achieving a reaction frame rate of over 90FPS, which meets the real-time requirements of the game scene. This invention is applied to six types of NPCs in this fighting game.
[0174] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.
Claims
1. A method for generating reaction actions in two-person interaction, comprising an active party and a passive party, characterized in that, Includes the following steps: A two-person interaction model is constructed and trained, wherein the two-person interaction model includes a variational intent prediction model and a causal graph learning model; the trained two-person interaction model includes a trained variational intent prediction model and a trained causal graph learning model. Obtain the spatiotemporal characteristics of the active party's actions; The spatiotemporal features of the active party's actions are input into the trained variational intent prediction model to obtain the latent intent variables of the active party's actions. The spatiotemporal features of the active party's actions and the latent variables of the intention are input into a trained causal graph learning model to obtain a causal graph representing the influence relationship between each joint of the active party and each joint of the passive party. Based on the spatiotemporal features of the active party's actions and the causal graph, the passive party's actions are generated.
2. The method for generating reaction actions in a two-person interaction according to claim 1, characterized in that, For the Online configuration, the steps for obtaining the cause-effect graph include: Active party generation When the action occurs at a given moment, it is achieved through aggregation. Historical information is used to obtain the j-th joint feature of the active party at time t. Then, the key matrix of the j-th joint of the active party at time t is obtained through processing by a multilayer perceptron. ; definition For the learnable joint embeddings of the passive party, the embeddings of the i-th joint of the passive party are concatenated with the latent variables of the intent, and then processed using a multilayer perceptron to obtain the learnable joint embeddings of the passive party. Query matrix of each joint ; based on and Calculate the correlation, and then apply the activation function. Processing to obtain the active party's first time at time t The first joint for the passive party Graph structure of the influence relationship of each joint ; Based on graph structure Constructing a cause-and-effect graph .
3. The method for generating reaction actions in two-person interaction according to claim 2, characterized in that, The two-person interaction model also includes a conditional action generation model. The trained two-person interaction model also includes a trained conditional action generation model. The trained conditional action generation model generates the passive party's action based on the spatiotemporal features of the active party's action and the causal graph. A time mask matrix is designed in the conditional action generation model to ensure that the passive party can only see the current and past actions of the active party.
4. The method for generating reaction actions in two-person interaction according to claim 3, characterized in that, The Transformer is used to construct the action generation network in the conditional action generation model; The time mask matrix is used This indicates that the constraint rule for the time mask matrix is: when the historical moment... Less than or equal to At time t, the mask matrix takes the value of 0; when Greater than At time t, the mask matrix takes the value of negative infinity.
5. The method for generating reaction actions in two-person interaction according to claim 1, characterized in that, The variational intent prediction model includes a prior network. The steps of inputting the spatiotemporal features of the active party's actions into the trained variational intent prediction model to obtain the latent variables of the active party's actions include: The spatiotemporal features of the active party's actions are pooled and then input into the trained prior network, which outputs the mean and variance estimated by the prior network. Based on the mean and variance estimated by the prior network, a normal distribution is constructed to obtain the prior distribution of the initiative's intention, and then the latent variable of the initiative's intention is obtained.
6. The method for generating reaction actions in two-person interaction according to claim 1, characterized in that, The two-person interaction model includes a spatiotemporal feature encoding model, a variational intent prediction model, a causal graph learning model, and a conditional action generation model. The variational intent prediction model includes a priori network and a posterior network. The training process of the two-person interaction model includes: Acquire training data, which includes the action sequence of the active party and the actual reaction action sequence of the passive party; The spatiotemporal features of the active party's action sequence and the passive party's actual reaction action sequence are extracted using a spatiotemporal feature coding model to obtain the spatiotemporal features of the active party's action sequence and the passive party's action sequence. The spatiotemporal features of the active party's actions and the spatiotemporal features of the passive party's actions are input into the posterior network of the variational intent prediction model to obtain the latent intent variables of the active party's actions. The spatiotemporal features of the active party's actions and the latent variables of the intention are input into the causal graph learning model to obtain a causal graph that represents the influence relationship between each joint of the active party and each joint of the passive party. The spatiotemporal features of the active party's actions and the causal graph are input into the conditional action generation model to generate the passive party's actions. Construct a loss function, update the network parameters of the model based on the loss function, and repeat the training until the model converges.
7. The method for generating reaction actions in two-person interaction according to claim 6, characterized in that, The spatiotemporal features of the active party's actions and the passive party's actions are concatenated, pooled, and then input into the posterior network. The mean and variance estimated by the posterior network are output. Based on the mean and variance estimated by the posterior network, a normal distribution is constructed to obtain the prior distribution of the initiator's intention, and then the latent variable of the initiator's intention is obtained.
8. The method for generating reaction actions in two-person interaction according to claim 7, characterized in that, The steps for obtaining the spatiotemporal features of the active party's actions include: Obtain the action sequence of the active party It is represented by 6D continuous rotation; Represents the set of real numbers. This represents the total number of frames in the action sequence. Indicates the number of joints in the human skeleton; The velocity of a single joint in a single frame is defined as the difference between the state of the same joint in the current frame and the previous frame. Based on this, the overall velocity of the active party can be expressed as... ; After concatenating the action sequence and overall velocity of the active party, a lightweight graph convolutional network is used to extract spatial topological features, and then a lightweight Transformer encoder is used to extract temporal features, thus obtaining the spatiotemporal features of the active party's actions. Each joint has Dimensional features.
9. The method for generating reaction actions in two-person interaction according to claim 6, characterized in that, The loss function includes divergence loss, graph loss, and reconstruction loss. By calculating the corresponding loss value and obtaining the gradient, the weights of the spatiotemporal feature encoding model, variational intent prediction model, causal graph learning model, and conditional action generation model are updated using the gradient. The training process is repeated until the model converges.
10. The method for generating reaction actions in two-person interaction according to claim 9, characterized in that, The gradient generated by the divergence loss is used to update the parameters of the prior and posterior networks; The gradient generated by the graph loss is used to update the parameters of the causal graph learning model; The gradient generated by the reconstruction loss is used to update the parameters of the entire model.