Three-dimensional person-object interaction element prediction method based on first person video

Abstract

The invention discloses a three-dimensional human-object interaction element prediction method based on a first-person view angle video. The method comprises the following steps: 1, extracting features of a first-person video, point cloud features of an object and head motion features; 2, calculating interaction concept features of the object point cloud; 3, calculating main body interaction intention features; 4, decoding object accessibility, human body contact features and interaction semantics; and 5, constructing and training a loss supervision function. According to the method, the relevance between the subject interaction intention and the object interaction concept in the first person interaction is comprehensively considered, so that the three-dimensional person-object interaction elements under the first person perspective can be accurately predicted.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular is a method for predicting three-dimensional human-object interaction elements based on first-person video. Background Art

[0002] Understanding the relationship of 3D human interaction aims to capture the semantic co-occurrence and geometric compatibility of people and objects interacting in 3D space, while human-object interaction understanding aims to mine the co-occurrence relationship and interaction properties between humans and objects. For first-person interactions, it is not only necessary to capture the interaction semantics, but also to understand where the interaction occurs specifically in space, such as human contact and object affordances, which is also crucial. Accurately partitioning spatial regions plays a key role in many applications, such as embodied AI, graphical interaction modeling, robotic manipulation, and scene interactions in augmented / virtual reality (AR / VR). Most existing methods estimate contact or affordance regions separately from people and objects, capturing one aspect of the interaction region but ignoring the collaborative nature between the interacting parties. These methods partition the region where the object should be manipulated without specifying the region used by the subject to perform the manipulation, and vice versa. This omission limits their effectiveness in shaping the final interaction. Some studies have explored the correlation between the interacting parties to jointly estimate the interaction region of the subject and the object, where the observation of the interacting parties is crucial, whether it is the appearance from the third-person perspective or the compatible structure formed by the geometric structure of the subject and the object. However, the observation of the interacting party in the first-person perspective is incomplete, for example, when sitting on a chair or interacting with both hands with head rotation, the interacting party is only partially visible or even completely invisible. This leads to ambiguity between visual observation and interaction content, which weakens the effectiveness of these methods and creates a gap when directly applied in egocentric scenarios. Summary of the invention

[0003] The present invention aims to address the deficiencies of the above-mentioned prior art and proposes a three-dimensional human-object interaction element prediction method based on first-person video, aiming to solve the problem of inconsistency between observed content and interaction semantics in first-person interaction, and to achieve accurate prediction of three-dimensional human-object interaction elements by establishing the correlation between object interaction concepts and subject interaction intentions.

[0004] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0005] The invention discloses a method for predicting three-dimensional human-object interaction elements based on a first-person perspective video, which comprises the following steps:

[0006] Step 1: Extract the features of the first-person video, the point cloud features of the object, and the head motion features:

[0007] Step 1.1, extract the features of the first-person video;

[0008] Get a batch of first-person videos ,in, represents the bth first-person video, represents the number of first-person videos in a batch, F represents the number of frames of each first-person video, Represents the number of channels per frame in the first-person video, and Represents the width and length of each frame in the first-person video;

[0009] The bth first-person video Each frame in is normalized and input into the HRNet network to obtain the bth feature vector ,in, and are the width and height of the eigenvector, respectively. Represents the number of channels of the feature vector;

[0010] Will After performing a self-attention calculation in the temporal dimension, the bth first-person video is obtained Features ;

[0011] Step 1.2, point cloud features of the object;

[0012] Get point cloud data of objects in the same batch ,in, Represents the point cloud data of the bth object, o is the number of points in the point cloud data of each object, Represents the spatial dimension of the object point cloud data;

[0013] Extraction using DGCNN network Pointwise features of ;

[0014] Step 1.3: Head motion feature extraction

[0015] Get the head movement data of the same batch ,in, represents the b-th head motion data, 12 represents the sum of the 3 dimensions of the translation vector and the 9 dimensions of the rotation vector;

[0016] Using MLP layers Encode and get the bth head motion feature ;

[0017] Step 2: Extract and calculate the point cloud data of the bth object After dividing the interactive concept features according to the spatial dimension, the affordance features of the bth object are obtained The semantic features of the interaction with the bth object ;

[0018] Step 3: Extract the bth first-person video After dividing the interaction intention features of the subject in the spatial dimension, the contact features of the bth subject are obtained. The interaction semantic features with the b-th subject ;

[0019] Step 4: Decode object affordance features, human contact features, and interaction semantic features;

[0020] Step 4.1: and After concatenation on the feature dimension, it is input into the MLP layer for mapping to obtain the bth interactive semantic category , where M represents the number of interactive categories;

[0021] Step 4.2: After mapping through the MLP layer, the bth human contact vertex is obtained ,in, Represents the number of vertices in the human body mesh;

[0022] Step 4.3: As the query vector Query, As a key-value pair, after cross-attention calculation, it is input into the MLP layer for mapping to obtain the affordance of the bth object ;

[0023] Step 5: Construct the total loss function :

[0024] Step 6: Use the gradient descent method to calculate the total loss function Optimize so that the total loss function The optimal model for predicting 3D human-object interaction elements from first-person perspective videos is obtained until convergence, which is used to predict the input first-person perspective video to obtain predicted human contact vertices, object affordances, and interaction categories.

[0025] The method for predicting three-dimensional human-object interaction elements based on first-person perspective video according to the present invention is also characterized in that step 2 includes the following steps:

[0026] Step 2.1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively;

[0027] Step 2.2: Generate 2 gradient adjustment tokens and , and and , and After splicing in the spatial dimension, two cross-attention calculations are performed together with the query Query and the two parallel key-value pairs to obtain the b-th object point cloud data Interactive conceptual features of

[0028] Step 2.3: Point cloud data of object b After the interactive concept features of are divided according to the spatial dimension, the affordance features of the bth object are obtained The semantic features of the interaction with the bth object .

[0029] Further, step 3 includes the following steps:

[0030] Step 3.1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively;

[0031] Step 3.2: Generate a gradient adjustment token , and Splicing in the spatial dimension, and It is also spliced ​​in the spatial dimension, so that the two spliced ​​vectors are cross-attention calculated twice to obtain the bth first-person video The interactive intention characteristics of the subject in the

[0032] Step 3.3: The bth first-person video After the interaction intention features of the subject in are segmented according to the spatial dimension, the contact features of the bth subject are obtained The interaction semantic features with the b-th subject .

[0033] Further, step 5 includes the following steps:

[0034] Step 5.1: Use formula (1) to construct the total loss function :

[0035] (1)

[0036] In formula (1), and They represent the human contact loss and object availability loss respectively, and have:

[0037] (2)

[0038] (3)

[0039] In formula (2) and formula (3), represents the label of the bth human contact vertex, Represents the affordance label of the bth object; Represents a constant;

[0040] In formula (1), represents the semantic loss of interaction intention, and is obtained by formula (4):

[0041] (4)

[0042] In formula (4), Indicates the bth first-person video labeled The interaction category labels of .

[0043] The electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the three-dimensional human-object interaction element prediction method, and the processor is configured to execute the program stored in the memory.

[0044] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program, when executed by a processor, executes the steps of the three-dimensional human-object interaction element prediction method.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. The present invention comprehensively utilizes the association between the object interaction concept and the subject interaction intention in the human-object interaction relationship, solving the problem of inaccurate reasoning caused by the mismatch between the observed content and the interactive content in the first-person interaction.

[0047] 2. The present invention utilizes the designed gradient adjustment mechanism to allow the model to adaptively adapt to the body and hand interactions in the first person perspective, and the various interactive elements complement each other, thereby greatly improving the accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1It is the overall flow chart of the present invention;

[0049] Figure 2 It is a framework diagram for extracting video, point cloud and motion features of the present invention;

[0050] Figure 3 It is a framework diagram for extracting the concept features of object interaction of the present invention;

[0051] Figure 4 This is a structural diagram of a human interaction intention feature extraction module of the present invention;

[0052] Figure 5 This is a structural diagram of the decoder module of the present invention. DETAILED DESCRIPTION

[0053] In this embodiment, a method for predicting three-dimensional human-object interaction elements based on an image is provided. Figure 1 As shown, the steps are as follows:

[0054] Step 1: Extract the features of the first-person video, the point cloud features of the object, and the head motion features, such as Figure 2 As shown:

[0055] Step 1.1, extract the features of the first-person video;

[0056] Get a batch of first-person videos ,in, represents the bth first-person video, represents the number of first-person videos in a batch, F represents the number of frames of each first-person video, Represents the number of channels per frame in the first-person video, and Represent the width and length of each frame in the first-person video respectively. In this example, B is 16, F is 8, C is 3, and W and H are both 256.

[0057] The bth first-person video Each frame in is normalized and input into the HRNet network to obtain the bth feature vector ,in, and are the width and height of the eigenvector, respectively. Represents the number of channels of the feature vector. In this example, and Take 16 for both. Take 1024;

[0058] Will After performing a self-attention calculation in the temporal dimension, the bth first-person video is obtained Features .

[0059] Step 1.2, point cloud features of the object;

[0060] Get point cloud data of objects in the same batch ,in, Represents the point cloud data of the bth object, o is the number of points in the point cloud data of each object, Represents the spatial dimension of the object point cloud data. In this example, o Take 2048.

[0061] Extraction using DGCNN network Pointwise features of ;

[0062] Step 1.3: Head motion feature extraction

[0063] Get the head movement data of the same batch ,in, represents the b-th head motion data, 12 represents the sum of the 3 dimensions of the translation vector and the 9 dimensions of the rotation vector;

[0064] Using MLP layers Encode and get the bth head motion feature .

[0065] Step 2: Extract and calculate the point cloud data of the bth object The interactive concept features of Figure 3 As shown:

[0066] Step 2.1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively;

[0067] Step 2.2: Generate 2 gradient adjustment tokens and , and and , and After splicing in the spatial dimension, two cross-attention calculations are performed together with the query Query and the two parallel key-value pairs to obtain the b-th object point cloud data Interactive conceptual features of

[0068] Step 2.3: Point cloud data of object b After the interactive concept features of are divided according to the spatial dimension, the affordance features of the bth object are obtained The semantic features of the interaction with the bth object .

[0069] Step 3: Extract the bth first-person video The interactive intention characteristics of the subject in Figure 4 As shown:

[0070] Step 3.1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively;

[0071] Step 3.2: Generate a gradient adjustment token , and Splicing in the spatial dimension, and It is also spliced ​​in the spatial dimension, so that the two spliced ​​vectors are cross-attention calculated twice to obtain the bth first-person video The interactive intention characteristics of the subject in the

[0072] Step 3.3: The bth first-person video After the interaction intention features of the subject in are segmented according to the spatial dimension, the contact features of the bth subject are obtained The interaction semantic features with the b-th subject .

[0073] Step 4: Decode object affordances, human contact features, and interaction semantics, such as Figure 5 As shown;

[0074] Step 4.1: and After concatenation on the feature dimension, it is input into the MLP layer for mapping to obtain the bth interactive semantic category , where M represents the number of interactive categories. In this example, M is 24;

[0075] Step 4.2: After mapping through the MLP layer, the bth human contact vertex is obtained ,in, Represents the number of vertices in the human body mesh. In this example, Take 6890;

[0076] Step 4.3: As the query vector Query, As a key-value pair, after cross-attention calculation, it is input into the MLP layer for mapping to obtain the affordance of the bth object .

[0077] Step 5: Construct the total loss function :

[0078] Step 5.1: Use formula (1) to construct the total loss function :

[0079] (1)

[0080] In formula (1), and They represent the human contact loss and object availability loss respectively, and have:

[0081] (2)

[0082] (3)

[0083] In formula (2) and formula (3), represents the label of the bth human contact vertex, Represents the affordance label of the bth object; Represents a constant.

[0084] In formula (1), represents the semantic loss of interaction intention, and is obtained by formula (4):

[0085] (4)

[0086] In formula (4), Indicates the bth first-person video labeled The interaction category labels of .

[0087] Step 6: Use the gradient descent method to calculate the total loss function Optimize so that the total loss function The optimal model for predicting 3D human-object interaction elements from first-person perspective videos is obtained until convergence, which is used to predict the input first-person perspective video to obtain predicted human contact vertices, object affordances, and interaction categories.

[0088] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0089] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

[0090] Example:

[0091] In order to verify the effectiveness of the method of the present invention, this example selects the EgoExo4D and GIMO data sets, and uses the Area under the ROC curve (AUC), Similarity (SIM), and average Intersection over Union (aIOU) metrics as quantitative evaluation criteria for availability prediction, and uses Precision, Recall, and F1 as contact prediction criteria for comparison with existing advanced methods.

[0092] In this embodiment, six methods are selected to compare the effects with the method of the present invention, and the selected methods are Capturing and inferring dense full-body human-scene contact (BSTRO), Denseestimation of 3D human-scene contact in the wild (DECO), Learning 3D Human-Object Interaction Relation from 2D Images (LEMON), Interaction-driven 3D Affordance Grounding Network (IAG), and O2o-afford (O2O). According to the experimental results, the results are shown in Table 1:

[0093] Table 1 Comparison of the quantitative results of the method of the present invention and the five selected comparative methods

[0094]

[0095] The experimental results in Table 1 show that the method of the present invention has better effects than the other five methods, thereby proving the feasibility of the method proposed in the present invention.

Claims

1. A method for predicting three-dimensional human-object interaction elements based on first-person perspective video, characterized in that: The following steps are involved: Step 1: Extract the features of the first-person video, the point cloud features of the object, and the head motion features: Step 1.1, extract the features of the first-person video; Get a batch of first-person videos ,in, represents the bth first-person video, represents the number of first-person videos in a batch, F represents the number of frames of each first-person video, Represents the number of channels per frame in the first-person video, and Represents the width and length of each frame in the first-person video; The bth first-person video Each frame in is normalized and input into the HRNet network to obtain the bth feature vector ,in, and are the width and height of the eigenvector, respectively. Represents the number of channels of the feature vector; Will After performing a self-attention calculation in the temporal dimension, the bth first-person video is obtained Features ; Step 1.2, point cloud features of the object; Get point cloud data of objects in the same batch ,in, Represents the point cloud data of the bth object, o is the number of points in the point cloud data of each object, Represents the spatial dimension of the object point cloud data; Extraction using DGCNN network Pointwise features of ; Step 1.3: Head motion feature extraction Get the head movement data of the same batch ,in, represents the b-th head motion data, 12 represents the sum of the 3 dimensions of the translation vector and the 9 dimensions of the rotation vector; Using MLP layers Encode and get the bth head motion feature ; Step 2: Extract and calculate the point cloud data of the bth object After dividing the interactive concept features according to the spatial dimension, the affordance features of the bth object are obtained The semantic features of the interaction with the bth object ; Step 3: Extract the bth first-person video After dividing the interaction intention features of the subject in the spatial dimension, the contact features of the bth subject are obtained. The interaction semantic features with the b-th subject ; Step 4: Decode object affordance features, human contact features, and interaction semantic features; Step 4.1: and After concatenation on the feature dimension, it is input into the MLP layer for mapping to obtain the bth interactive semantic category , where M represents the number of interactive categories; Step 4.2: After mapping through the MLP layer, the bth human contact vertex is obtained ,in, Represents the number of vertices in the human body mesh; Step 4.3: As the query vector Query, As a key-value pair, after cross-attention calculation, it is input into the MLP layer for mapping to obtain the affordance of the bth object ; Step 5: Construct the total loss function : Step 6: Use the gradient descent method to calculate the total loss function Optimize so that the total loss function The optimal model for predicting 3D human-object interaction elements from first-person perspective videos is obtained until convergence, which is used to predict the input first-person perspective video to obtain predicted human contact vertices, object affordances, and interaction categories.

2. The method for predicting three-dimensional human-object interaction elements based on first-person perspective video according to claim 1, characterized in that: Step 2 includes the following steps: Step 2.

1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively; Step 2.2: Generate 2 gradient adjustment tokens and , and and , and After splicing in the spatial dimension, two cross-attention calculations are performed together with the query Query and the two parallel key-value pairs to obtain the b-th object point cloud data Interactive conceptual features of Step 2.3: Point cloud data of object b After the interactive concept features of are divided according to the spatial dimension, the affordance features of the bth object are obtained The semantic features of the interaction with the bth object .

3. The method for predicting three-dimensional human-object interaction elements based on first-person perspective video according to claim 2, characterized in that: Step 3 includes the following steps: Step 3.

1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively; Step 3.2: Generate a gradient adjustment token , and with Splicing in the spatial dimension, and It is also spliced ​​in the spatial dimension, so that the two spliced ​​vectors are cross-attention calculated twice to obtain the bth first-person video The interactive intention characteristics of the subject in the Step 3.3: The bth first-person video After the interaction intention features of the subject in are segmented according to the spatial dimension, the contact features of the bth subject are obtained The interaction semantic features with the b-th subject .

4. The method for predicting three-dimensional human-object interaction elements based on first-person perspective video according to claim 3, characterized in that: Step 5 includes the following steps: Step 5.1: Use formula (1) to construct the total loss function : (1) In formula (1), and They represent the human contact loss and object availability loss respectively, and have: (2) (3) In formula (2) and formula (3), represents the label of the bth human contact vertex, Represents the affordance label of the bth object; Represents a constant; In formula (1), represents the semantic loss of interaction intention, and is obtained by formula (4): (4) In formula (4), Indicates the bth first-person video labeled The interaction category labels of .

5. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the three-dimensional human-object interaction element prediction method described in any one of claims 1-4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional human-object interaction element prediction method according to any one of claims 1 to 4 are executed.

Images