A method for predicting 3D human-object interaction elements based on first-person video

Abstract

This invention discloses a method for predicting three-dimensional human-object interaction elements based on first-person perspective videos. The method comprises the following steps: 1. extracting first-person video features, object point cloud features, and head motion features; 2. calculating interaction concept features from the object point cloud; 3. calculating subject interaction intention features; 4. decoding object affordances, human contact features, and interaction semantics; and 5. constructing and training a loss supervision function. This method comprehensively considers the correlation between subject interaction intention and object interaction concepts in first-person interactions, thereby accurately predicting three-dimensional human-object interaction elements from a first-person perspective.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular to a method for predicting three-dimensional human-object interaction elements based on first-person video. Background Art

[0002] Understanding the relationships of 3D human interactions aims to capture the semantic co-occurrence and geometric compatibility of interacting people and objects in 3D space. Human-object interaction understanding aims to exploit the co-occurrence relationships and interaction properties between people and objects. For first-person interactions, capturing not only the semantics of the interaction but also understanding the specific spatial locations where the interaction occurs, such as human contact and object affordances, is crucial. Accurately demarcating spatial regions plays a key role in many applications, such as embodied AI, graphical interaction modeling, robotic manipulation, and scene interactions in augmented / virtual reality (AR / VR). Most existing methods estimate contact or affordance regions separately for people and objects, capturing one aspect of the interaction region but neglecting the collaborative nature of the interacting parties. These methods demarcate the region where the object should be manipulated without specifying the region used by the subject to perform the manipulation, or vice versa. This omission limits their effectiveness in shaping the final interaction. Some studies have explored the correlation between the interacting parties to jointly estimate the interaction region of the subject and object. Observations of the interacting parties are crucial, whether in terms of third-person perspective appearance or the compatibility structure formed by the subject and object geometry. However, the observation of the interacting party from the first-person perspective is incomplete. For example, when sitting on a chair or interacting with both hands with head rotation, the interacting party is only partially visible or even completely invisible. This leads to ambiguity between visual observations and the interaction content, which weakens the effectiveness of these methods and creates a gap when directly applied in egocentric scenarios. Summary of the Invention

[0003] In order to address the shortcomings of the above-mentioned existing technologies, the present invention proposes a three-dimensional human-object interaction element prediction method based on first-person video, aiming to solve the problem of inconsistency between observation content and interaction semantics in first-person interaction. By establishing the correlation between object interaction concepts and subject interaction intentions, accurate prediction of three-dimensional human-object interaction elements can be achieved.

[0004] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0005] The present invention provides a method for predicting three-dimensional human-object interaction elements based on first-person perspective video, comprising the following steps:

[0006] Step 1: Extract first-person video features, object point cloud features, and head motion features:

[0007] Step 1.1, extract the features of the first-person video;

[0008] Get a batch of first-person videos ,in, represents the bth first-person video, represents the number of first-person videos in a batch, F represents the number of frames of each first-person video, Represents the number of channels per frame in the first-person video, and Represents the width and length of each frame in the first-person video;

[0009] The bth first-person video Each frame in is normalized and input into the HRNet network to obtain the bth feature vector ,in, and are the width and height of the eigenvector, Represents the number of channels of the feature vector;

[0010] Will After performing a self-attention calculation in the temporal dimension, the bth first-person video is obtained Features ;

[0011] Step 1.2, point cloud features of the object;

[0012] Get point cloud data of objects in the same batch ,in, Represents the point cloud data of the b-th object, o is the number of points in the point cloud data of each object, Represents the spatial dimension of the object point cloud data;

[0013] Extraction using DGCNN network Pointwise features of ;

[0014] Step 1.3: Head motion feature extraction

[0015] Get head movement data of the same batch ,in, represents the b-th head motion data, 12 represents the sum of the 3 dimensions of the translation vector and the 9 dimensions of the rotation vector;

[0016] Using MLP layers Encode and get the b-th head motion feature ;

[0017] Step 2: Extract and calculate the point cloud data of the bth object After dividing the interactive concept features of the b-th object according to the spatial dimension, the affordance features of the b-th object are obtained. Interaction semantic features with the b-th object ;

[0018] Step 3: Extract the bth first-person video After dividing the interaction intention features of the subject in the middle and segmenting them according to the spatial dimension, the contact features of the bth subject are obtained Interaction semantic features with the bth subject ;

[0019] Step 4: Decode object affordance features, human contact features, and interaction semantic features;

[0020] Step 4.1, and After concatenation on the feature dimension, the input is mapped into the MLP layer to obtain the bth interaction semantic category. , where M represents the number of interacting categories;

[0021] Step 4.2, After mapping through the MLP layer, the bth human contact vertex is obtained ,in, Represents the number of vertices in the human body mesh;

[0022] Step 4.3, As the query vector Query, As a key-value pair, after cross-attention calculation, it is input into the MLP layer for mapping to obtain the affordance of the bth object ;

[0023] Step 5: Construct the total loss function :

[0024] Step 6: Use the gradient descent method to calculate the total loss function Optimize so that the total loss function Until convergence, the optimal model for predicting three-dimensional human-object interaction elements from first-person perspective video is obtained, which is used to predict the input first-person perspective video to obtain predicted human contact vertices, object affordances and interaction categories.

[0025] The method for predicting three-dimensional human-object interaction elements based on first-person perspective video according to the present invention is also characterized in that step 2 includes the following steps:

[0026] Step 2.1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively;

[0027] Step 2.2: Generate 2 gradient adjustment tokens and , and and , and After splicing in the spatial dimension, two cross-attention calculations are performed together with the query and the two parallel key-value pairs to obtain the b-th object point cloud data interactive conceptual features of

[0028] Step 2.3: Point cloud data of object b After the interactive concept features of are divided according to the spatial dimension, the affordance features of the bth object are obtained Interaction semantic features with the b-th object .

[0029] Furthermore, step 3 includes the following steps:

[0030] Step 3.1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively;

[0031] Step 3.2: Generate a gradient adjustment token , and with Splicing in the spatial dimension, and It is also spliced ​​in the spatial dimension, so that the two spliced ​​vectors are cross-attention calculated twice to obtain the bth first-person video The interactive intention characteristics of the subject in the

[0032] Step 3.3, the bth first-person video After the interaction intention features of the subject in the middle are segmented according to the spatial dimension, the contact features of the bth subject are obtained Interaction semantic features with the bth subject .

[0033] Furthermore, step 5 includes the following steps:

[0034] Step 5.1: Use formula (1) to construct the total loss function :

[0035] (1)

[0036] In formula (1), and They represent the human contact loss and object availability loss respectively, and have:

[0037] (2)

[0038] (3)

[0039] In formula (2) and formula (3), Represents the label of the b-th human contact vertex, Represents the affordance label of the bth object; Indicates a constant;

[0040] In formula (1), represents the semantic loss of interaction intention, and is obtained by formula (4):

[0041] (4)

[0042] In formula (4), Represents the bth first-person video labeled The interaction category labels.

[0043] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the three-dimensional human-object interaction element prediction method, and the processor is configured to execute the program stored in the memory.

[0044] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of the three-dimensional human-object interaction element prediction method when the computer program is executed by a processor.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. The present invention comprehensively utilizes the association between the object interaction concept and the subject interaction intention in the human-object interaction relationship, solving the problem of inaccurate reasoning caused by the mismatch between the observation content and the interaction content in first-person interaction.

[0047] 2. The present invention utilizes a designed gradient adjustment mechanism to allow the model to adaptively adapt to the body and hand interactions in the first person perspective. The various interaction elements complement each other, thereby greatly improving the accuracy of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1It is the overall flow chart of the present invention;

[0049] Figure 2 This is a video, point cloud, and motion feature extraction framework diagram of the present invention;

[0050] Figure 3 This is a framework diagram for extracting conceptual features of object interaction in the present invention;

[0051] Figure 4 This is a structural diagram of the human body interaction intention feature extraction module of the present invention;

[0052] Figure 5 This is a structural diagram of the decoder module of the present invention. DETAILED DESCRIPTION

[0053] In this embodiment, a method for predicting three-dimensional human-object interaction elements based on images is provided. Figure 1 As shown, the steps are as follows:

[0054] Step 1: Extract the features of the first-person video, the point cloud features of the object, and the head motion features, such as Figure 2 As shown:

[0055] Step 1.1, extract the features of the first-person video;

[0056] Get a batch of first-person videos ,in, represents the bth first-person video, represents the number of first-person videos in a batch, F represents the number of frames of each first-person video, Represents the number of channels per frame in the first-person video, and Represents the width and height of each frame in the first-person video. In this example, B is 16, F is 8, C is 3, and W and H are both 256.

[0057] The bth first-person video Each frame in is normalized and input into the HRNet network to obtain the bth feature vector ,in, and are the width and height of the eigenvector, Represents the number of channels of the feature vector. In this example, and Take 16 for both. Take 1024;

[0058] Will After performing a self-attention calculation in the temporal dimension, the bth first-person video is obtained Features .

[0059] Step 1.2, point cloud features of the object;

[0060] Get point cloud data of objects in the same batch ,in, Represents the point cloud data of the b-th object, o is the number of points in the point cloud data of each object, Represents the spatial dimension of the object point cloud data. In this example, o Take 2048.

[0061] Extraction using DGCNN network Pointwise features of ;

[0062] Step 1.3: Head motion feature extraction

[0063] Get head movement data of the same batch ,in, represents the b-th head motion data, 12 represents the sum of the 3 dimensions of the translation vector and the 9 dimensions of the rotation vector;

[0064] Using MLP layers Encode and get the b-th head motion feature .

[0065] Step 2: Extract and calculate the point cloud data of the bth object The interactive concept features of Figure 3 As shown:

[0066] Step 2.1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively;

[0067] Step 2.2: Generate 2 gradient adjustment tokens and , and and , and After splicing in the spatial dimension, two cross-attention calculations are performed together with the query and the two parallel key-value pairs to obtain the b-th object point cloud data interactive conceptual features of

[0068] Step 2.3: Point cloud data of object b After the interactive concept features of are divided according to the spatial dimension, the affordance features of the bth object are obtained Interaction semantic features with the b-th object .

[0069] Step 3: Extract the bth first-person video The interaction intention characteristics of the subject in Figure 4 As shown:

[0070] Step 3.1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively;

[0071] Step 3.2: Generate a gradient adjustment token , and with Splicing in the spatial dimension, and It is also spliced ​​in the spatial dimension, so that the two spliced ​​vectors are cross-attention calculated twice to obtain the bth first-person video The interactive intention characteristics of the subject in the

[0072] Step 3.3, the bth first-person video After the interaction intention features of the subject in the middle are segmented according to the spatial dimension, the contact features of the bth subject are obtained Interaction semantic features with the bth subject .

[0073] Step 4: Decode object affordances, human contact features, and interaction semantics, such as Figure 5 As shown;

[0074] Step 4.1, and After concatenation on the feature dimension, the input is mapped into the MLP layer to obtain the bth interaction semantic category. , where M represents the number of interaction categories. In this example, M is 24;

[0075] Step 4.2, After mapping through the MLP layer, the bth human contact vertex is obtained ,in, Represents the number of vertices in the human body mesh. In this example, Take 6890;

[0076] Step 4.3, As the query vector Query, As a key-value pair, after cross-attention calculation, it is input into the MLP layer for mapping to obtain the affordance of the bth object .

[0077] Step 5: Construct the total loss function :

[0078] Step 5.1: Use formula (1) to construct the total loss function :

[0079] (1)

[0080] In formula (1), and They represent the human contact loss and object availability loss respectively, and have:

[0081] (2)

[0082] (3)

[0083] In formula (2) and formula (3), Represents the label of the b-th human contact vertex, Represents the affordance label of the bth object; Represents a constant.

[0084] In formula (1), represents the semantic loss of interaction intention, and is obtained by formula (4):

[0085] (4)

[0086] In formula (4), Represents the bth first-person video labeled The interaction category labels.

[0087] Step 6: Use the gradient descent method to calculate the total loss function Optimize so that the total loss function Until convergence, the optimal model for predicting three-dimensional human-object interaction elements from first-person perspective video is obtained, which is used to predict the input first-person perspective video to obtain predicted human contact vertices, object affordances and interaction categories.

[0088] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0089] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0090] Example:

[0091] In order to verify the effectiveness of the method of the present invention, this example selects the EgoExo4D and GIMO datasets, and adopts the Area under the ROC curve (AUC), Similarity (SIM), and Average Intersection over Union (aIOU) metrics as quantitative evaluation criteria for availability prediction, and uses Precision, Recall, and F1 as contact prediction criteria for comparison with existing advanced methods.

[0092] In this example, six methods are selected to compare their effects with the method of the present invention. The selected methods are Capturing and inferring dense full-body human-scene contact (BSTRO), Denseestimation of 3D human-scene contact in the wild (DECO), Learning 3D Human-Object Interaction Relation from 2D Images (LEMON), Interaction-driven 3D Affordance Grounding Network (IAG), and O2o-afford (O2O). The experimental results are shown in Table 1:

[0093] Table 1 Comparison of quantitative results between the method of the present invention and the five selected comparative methods

[0094]

[0095] The experimental results in Table 1 show that the method of the present invention is better than the other five methods, thereby proving the feasibility of the method proposed in the present invention.

Claims

1. A method for predicting three-dimensional human-object interaction elements based on first-person perspective video, characterized in that: The following steps are involved: Step 1: Extract first-person video features, object point cloud features, and head motion features: Step 1.1, extract the features of the first-person video; Get a batch of first-person videos ,in, represents the bth first-person video, represents the number of first-person videos in a batch, F represents the number of frames of each first-person video, Represents the number of channels per frame in the first-person video, and Represents the width and length of each frame in the first-person video; The bth first-person video Each frame in is normalized and input into the HRNet network to obtain the bth feature vector ,in, and are the width and height of the eigenvector, Represents the number of channels of the feature vector; Will After performing a self-attention calculation in the temporal dimension, the bth first-person video is obtained Features ; Step 1.2, point cloud features of the object; Get point cloud data of objects in the same batch ,in, Represents the point cloud data of the b-th object, o is the number of points in the point cloud data of each object, Represents the spatial dimension of the object point cloud data; Extraction using DGCNN network Pointwise features of ; Step 1.3: Head motion feature extraction Get head movement data of the same batch ,in, represents the b-th head motion data, 12 represents the sum of the 3 dimensions of the translation vector and the 9 dimensions of the rotation vector; Using MLP layers Encode and get the b-th head motion feature ; Step 2: Extract and calculate the point cloud data of the bth object After dividing the interactive concept features of the b-th object according to the spatial dimension, the affordance features of the b-th object are obtained. Interaction semantic features with the b-th object ; Step 3: Extract the bth first-person video After dividing the interaction intention features of the subject in the middle and segmenting them according to the spatial dimension, the contact features of the bth subject are obtained Interaction semantic features with the bth subject ; Step 4: Decode object affordance features, human contact features, and interaction semantic features; Step 4.1, and After concatenation on the feature dimension, the input is mapped into the MLP layer to obtain the bth interaction semantic category. , where M represents the number of interacting categories; Step 4.2, After mapping through the MLP layer, the bth human contact vertex is obtained ,in, Represents the number of vertices in the human body mesh; Step 4.3, As the query vector Query, As a key-value pair, after cross-attention calculation, it is input into the MLP layer for mapping to obtain the affordance of the bth object ; Step 5: Construct the total loss function : Step 6: Use the gradient descent method to calculate the total loss function Optimize so that the total loss function Until convergence, the optimal model for predicting three-dimensional human-object interaction elements from first-person perspective videos is obtained, which is used to predict the input first-person perspective video to obtain predicted human contact vertices, object affordances, and interaction categories.

2. The method for predicting three-dimensional human-object interaction elements based on first-person perspective video according to claim 1, characterized in that: Step 2 includes the following steps: Step 2.

1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively; Step 2.2: Generate 2 gradient adjustment tokens and , and and , and After splicing in the spatial dimension, two cross-attention calculations are performed together with the query and the two parallel key-value pairs to obtain the b-th object point cloud data interactive conceptual features of Step 2.3: Point cloud data of object b After the interactive concept features of are divided according to the spatial dimension, the affordance features of the bth object are obtained Interaction semantic features with the b-th object .

3. The method for predicting three-dimensional human-object interaction elements based on first-person perspective video according to claim 2, characterized in that: Step 3 includes the following steps: Step 3.

1. Generate a semantic token ,Will and After splicing in the spatial dimension as the query, and As two parallel key-value pairs respectively; Step 3.2: Generate a gradient adjustment token , and with Splicing in the spatial dimension, and It is also spliced ​​in the spatial dimension, so that the two spliced ​​vectors are cross-attention calculated twice to obtain the bth first-person video The interactive intention characteristics of the subject in the Step 3.3, the bth first-person video After the interaction intention features of the subject in the middle are segmented according to the spatial dimension, the contact features of the bth subject are obtained Interaction semantic features with the bth subject .

4. The method for predicting three-dimensional human-object interaction elements based on first-person perspective video according to claim 3, characterized in that: Step 5 includes the following steps: Step 5.1: Use formula (1) to construct the total loss function : (1) In formula (1), and They represent the human contact loss and object availability loss respectively, and have: (2) (3) In formula (2) and formula (3), Represents the label of the b-th human contact vertex, Represents the affordance label of the bth object; Represents a constant; In formula (1), represents the semantic loss of interaction intention, and is obtained by formula (4): (4) In formula (4), Represents the bth first-person video labeled The interaction category labels.

5. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the three-dimensional human-object interaction element prediction method according to any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional human-object interaction element prediction method according to any one of claims 1 to 4 are executed.

Images