A Multimodal Feature Fusion Method for Human Interaction Detection Based on Bipartite Graph Structure

Abstract

This invention belongs to the field of human-object interaction detection in computer vision. It discloses a multimodal feature fusion method for human interaction detection based on a bipartite graph structure. The method first acquires a batch of images to generate an initial detection set. Then, the initial detection set is input into a backbone feature extraction network to extract visual and spatial information from the images. Next, pose estimation is used to extract key points of the person in the image, which are then input into a keypoint feature processing network module to obtain pose information. The extracted visual and spatial features are then fused and input into an inference network to obtain classification logic outputs, and further processed by the keypoint feature processing network to obtain classification logic outputs. Finally, the two logic outputs are fused to obtain the final classification result. This invention fully utilizes the various information present in the images, enabling a more comprehensive feature representation and effectively improving the accuracy and robustness of human-object interaction detection.

Description

Technical Field

[0001] This invention belongs to the field of human-object interaction detection in computer vision, specifically involving a multimodal feature fusion method for human-object interaction detection based on a bipartite graph structure. Background Technology

[0002] Human-object interaction (HOI) detection is an important area of ​​computer vision research, providing crucial clues for related high-level vision tasks and finding wide applications in image understanding and robotics. The goal of this task is to locate people and objects in a scene and identify the interaction relationships between them. Specifically, given an input image, it identifies all "person-verb-object" tripartite relationships.

[0003] Because its output representation is essentially similar to that of object detection, HOI detection is often treated as a downstream task of object detection. However, HOI recognition differs from image recognition. Image recognition mainly distinguishes scene types and prominent object types, focusing on the overall understanding of the image. HOI recognition, on the other hand, identifies interactions between people and objects at a more refined level. It involves not only people and object instances but also obtaining contextual clues about these instances in the image. By reasoning about the instances and contextual clues, it achieves a more accurate recognition of the interaction relationships between people and objects.

[0004] The main differences between various HOI detection methods lie in the different strategies employed for instance objects during the interaction recognition stage. Currently, there are three implementation methods based on global instances: fusing spatial location information, fusing appearance information, and fusing human pose information. Methods based on spatial location information tend to quickly locate objects associated with specific actions; methods based on appearance features often use the appearance of people or objects as a condition to narrow down the search space for target objects; and methods based on human pose information emphasize the different levels of importance of human body parts near the interaction area during detection.

[0005] However, all three implementation methods mentioned above affect the final recognition result to varying degrees, resulting in poor accuracy and robustness, thus impacting the accuracy of the recognition. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a multimodal feature fusion method for human-object interaction detection based on a bipartite graph structure. This method extracts visual information, spatial information, and human posture information from images to construct corresponding features. The features, after message passing, are fused to achieve the final interaction inference, thereby improving the accuracy and robustness of human interaction detection.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0008] This invention is a multimodal feature fusion method for detecting human-object interactions based on a bipartite graph structure. The method includes the following steps:

[0009] Step 1) Input a batch of images;

[0010] Step 2) Use the Faster R-CNN object detection algorithm on each image from Step 1) to detect people and object instances in the image, and generate an initial detection set;

[0011] Step 3) Input the initial detection set obtained in Step 2) into the ResNet50-FPN backbone network to extract visual and spatial features;

[0012] Step 4) Construct the person and object instances obtained in Step 2) into heterogeneous nodes in the bipartite graph, and match them one by one to form person pairs;

[0013] Step 5) Pass the features of the person pairs obtained in Step 4) in the bipartite graph to eliminate negative person pairs, and then perform interactive reasoning to obtain the classification logic output class logits1.

[0014] Step 6) Extract key points from the person pair constructed in Step 4), input the key point information into the key point feature processing network module, and obtain human body key point feature information and classification logic output class logits2.

[0015] Step 7) The classification logic outputs obtained in Step 5) and Step 6) are fused to obtain the final classification result, thereby realizing the detection of the interaction relationship between people and objects.

[0016] A further improvement of the present invention is that step 4) includes the following steps:

[0017] Step 41) Project the candidate boxes corresponding to the people and object instances onto the Features map to generate the corresponding features;

[0018] Step 42) Represent the features of people and objects in the bipartite graph as nodes;

[0019] Step 43) For the generated n nodes, place the nodes classified as human at the top and obtain the features of the first n_h nodes, which are the features of human nodes;

[0020] Step 44) The remaining n-n_h nodes are the object nodes. Then, match the human nodes and object nodes one by one.

[0021] A further improvement of the present invention is that step 5) includes the following steps:

[0022] Step 51) The message passing function between people and object nodes is represented as follows;

[0023] Human node updates:

[0024]

[0025]

[0026] Updates to object nodes:

[0027]

[0028]

[0029] Where LN represents the LayerNorm operation, σ is the ReLU activation function, and α represents the adjacency weights between nodes in the adjacency matrix. It is obtained by performing a softmax operation on values ​​with the same index i during message passing; similarly It is obtained by performing softmax on values ​​with the same index j, representing row normalization and column normalization respectively. M is the message passing function, which has different parameters for different directions. This represents the merged object node. And after analyzing the edge features of the person node and object node at the current iteration number, the features of the person node are iteratively updated, similar to... This updates the object nodes. ij For edge features, spatial features are mapped to 1024 dimensions using an MLP; x i ,y j These represent the visual features of people and objects, respectively, and are also processed by MLP to achieve 1024 dimensions.

[0030] Step 52) After fusing the visual and spatial features of people and objects, MLP processing is used to obtain a classification logic output class logits1.

[0031] A further improvement of the present invention is that: the human body key point feature extraction in step 6) includes the following steps:

[0032] Step 61) Obtain the position coordinates (x, y, y) of 17 key points of the human body in the image. i ,y i );

[0033] Step 62) Represent the feature of the i-th keypoint using the following formula:

[0034]

[0035] Where w and H represent the width and height of the image, respectively; (x, y) are the center coordinates of the detection box.

[0036] Step 63) The human keypoint feature processing network module consists of a Batch Norm layer, a ReLU layer, a Dropout layer, and a fully connected layer. First, the dimensions of the four fully connected layers are set to 128, 64, and 256, respectively, along with the number of interaction categories K in the dataset. The (N, 17, 2)-dimensional vector is passed through the fully connected layer and then a fixed layer to finally obtain an (N, K)-dimensional classification logic output, class logits2. The fixed layer is composed of a combination of a Batch Norm layer, a ReLU layer, and a Dropout layer.

[0037] A further improvement of the present invention is that the feature fusion described in step 7) includes the following steps:

[0038] Step 71) Merge the two logic outputs obtained in Step 5 and Step 6);

[0039] Step 72) The training process introduces focal loss as a binary classification loss function to reduce the weight of a large number of negative samples during training. The formula for the focal loss function is:

[0040]

[0041] in It is the probability score of a certain action category, y∈{0,1} is the binary label, β∈[0,1] and These are hyperparameters. β is used to balance the ratio of positive to negative samples, and γ is used to adjust the rate at which the weights of simple samples decrease.

[0042] The beneficial effects of this invention are:

[0043] This invention first acquires a batch of images, extracting visual, spatial, and pose information from them. Then, it initializes the feature representations of human and object nodes using a bipartite graph to construct human-object pairs. Subsequently, it performs bidirectional message passing between humans and objects to eliminate negative human-object pairs in complex interactions. The extracted 17 keypoints are processed by a keypoint feature processing network module to obtain pose features. Finally, the classification results of the processed visual, spatial, and pose features are fused and focal loss is introduced for training, thereby achieving more accurate detection of human-object interaction relationships.

[0044] Specifically:

[0045] (1) This invention uses a bipartite graph structure to initialize human and object nodes and obtain the feature representation of human and object nodes.

[0046] (2) This invention introduces a two-way message passing algorithm to update human and object nodes, thereby eliminating negative human-object pairs in complex interactions.

[0047] (3) This invention uses a multimodal feature fusion method to fuse visual features, spatial features and pose features in an image to obtain a more refined feature representation, which effectively improves the accuracy and reliability of detecting human-object interaction relationships. Attached Figure Description

[0048] Figure 1 This is a flowchart of the multimodal feature synthesis method of the present invention.

[0049] Figure 2 This is a diagram of the multimodal feature fusion architecture based on a bipartite graph structure, which is the basis of this invention.

[0050] Figure 3 This is a diagram of the key feature processing network module of the present invention. Detailed Implementation

[0051] The embodiments of the present invention will be disclosed below with reference to the drawings. For clarity, many practical details will be described in the following description. However, it should be understood that these practical details are not intended to limit the invention. That is, in some embodiments of the invention, these practical details are not essential.

[0052] like Figure 1-3 As shown, this invention is a multimodal feature fusion method for detecting human-object interactions based on a bipartite graph structure, comprising the following steps:

[0053] Step 1: Input a batch of images;

[0054] Step 2: Use the Faster R-CNN object detection algorithm on each image from Step 1 to detect people and object instances in the image, and generate an initial detection set;

[0055] Step 3: Input the initial detection set obtained in Step 2 into the ResNet50-FPN backbone network to extract visual and spatial features;

[0056] Step 4: Construct the person and object instances obtained in Step 2 into heterogeneous nodes in a bipartite graph, and match them one by one to form person pairs. This includes the following steps:

[0057] Step 41: Project the candidate boxes corresponding to people and object instances onto the Features map to generate the corresponding features;

[0058] Step 42: Represent the features of people and objects in the form of nodes in the bipartite graph;

[0059] Step 43: For the generated n nodes, place the nodes classified as human at the top and obtain the features of the first n_h nodes, which are the features of human nodes;

[0060] Step 44: The remaining n-n_h nodes are the object nodes. Then, match the human nodes and object nodes one by one.

[0061] Step 5: Pass the features of the person pairs obtained in Step 4 through message passing in the bipartite graph to eliminate negative person pairs, and then perform interactive reasoning to obtain the classification logic output class logits1.

[0062] Includes the following steps:

[0063] Step 51: The message passing function between people and object nodes is represented as follows;

[0064] Human node updates:

[0065]

[0066]

[0067] Updates to object nodes:

[0068]

[0069]

[0070] Where LN represents the LayerNorm operation, σ is the ReLU activation function, and α represents the adjacency weights between nodes in the adjacency matrix. It is obtained by performing a softmax operation on values ​​with the same index i during message passing; similarly It is obtained by performing softmax on values ​​with the same index j, representing row normalization and column normalization respectively. M is the message passing function, which has different parameters for different directions. This represents the merged object node. And after analyzing the edge features of the person node and object node at the current iteration number, the features of the person node are iteratively updated, similar to... This updates the object nodes. ij For edge features, spatial features are mapped to 1024 dimensions using an MLP; x i ,y j These represent the visual features of people and objects, respectively, and are also processed by MLP to achieve 1024 dimensions.

[0071] Step 52: After fusing the visual and spatial features of people and objects, the data is processed by MLP to obtain a classification logic output, class logits1.

[0072] Step 6: Extract key points from the person pair constructed in Step 4, input the key point information into the key point feature processing network module, and obtain human body key point feature information and classification logic output class logits2.

[0073] The extraction of key human features includes the following steps:

[0074] Step 61) Obtain the position coordinates (x, y, y) of 17 key points of the human body in the image. i ,y i );

[0075] Step 62) Represent the feature of the i-th keypoint using the following formula:

[0076]

[0077] Where W and H represent the width and height of the image, respectively; (x, y) are the center coordinates of the detection box.

[0078] Step 63) The human keypoint feature processing network module consists of a Batch Norm layer, a ReLU layer, a Dropout layer, and a fully connected layer. First, the dimensions of the four fully connected layers are set to 128, 64, and 256, respectively, along with the number of interaction categories K in the dataset. The (N, 17, 2)-dimensional vector is passed through the fully connected layer and then a fixed layer to finally obtain an (N, K)-dimensional classification logic output, class logits2. The fixed layer is composed of a combination of a Batch Norm layer, a ReLU layer, and a Dropout layer.

[0079] Step 7: The classification logic outputs obtained in Step 5 and Step 6 are fused to obtain the final classification result, thereby realizing the detection of the interaction relationship between people and objects.

[0080] Feature fusion includes the following steps:

[0081] Step 71: Combine the two logic outputs obtained in Step 5 and Step 6;

[0082] Step 72: During the training process, focal loss is introduced as a binary classification loss function to reduce the weight of a large number of negative samples during training. The formula for the focal loss function is:

[0083]

[0084] in It is the probability score of a certain action category, y∈{0,1} is the binary label, β∈[0,1] and These are hyperparameters. β is used to balance the ratio of positive to negative samples, and γ is used to adjust the rate at which the weights of simple samples decrease.

[0085] This invention primarily utilizes visual and spatial information from images to construct person-object pairs. It iterative message passing updates the information of both people and objects, thereby eliminating negative person-object pair instances in complex interactions. Human pose information is constructed from these person-object pairs and input into a keypoint processing network module to obtain human pose features, which are then used to improve the model's classification performance. Finally, the visual, spatial, and pose information are fused and input into a behavior classification network for inference of person-object interaction relationships.

Claims

1. A multi-modal feature fusion method for person interaction detection based on a bipartite graph structure, characterized in that: The multi-modal feature fusion method comprises the following steps: Step 1, input a batch of pictures; Step 2, use the Faster R-CNN target detection algorithm to detect the people and object instances in each picture in step 1, and generate an initial detection set; Step 3, input the initial detection set obtained in step 2 into the ResNet50-FPN backbone network to extract visual features and spatial features; Step 4, construct the people and object instances obtained in step 2 into heterogeneous nodes in a bipartite graph, and match them into pairs one by one to build a pair of people; Step 5, perform message passing on the features of the pair of people obtained in step 4 in the bipartite graph to eliminate negative pairs of people, and then perform interaction reasoning to obtain classification logic output class logits1; Step 6, extract key points from the pair of people constructed in step 4, and input the key point information into a key point feature processing network module to obtain human key point feature information and classification logic output class logits2; Step 7, fuse the classification logic output class logits1 obtained in step 5 and the classification logic output class logits2 obtained in step 6 to obtain the final classification result, thereby realizing the interaction relationship detection between people and objects, wherein the human key point feature extraction in step 6 comprises the following steps: Step 6-1: Obtain the position coordinates of the key points of the human body in the picture ; Step 6-2: The first keypoint feature is expressed by the following equation: ​ ， wherein , respectively represent the width and height of the picture, is the center coordinate of the detection frame; The human key point feature extraction in step 6 is completed through a human key point feature processing network module, which is composed of a Batch Norm layer, a ReLU layer, a Dropout layer and a fully connected layer. First, set the dimensions of the four fully connected layers to be 128, 64, 256 and the number of interaction categories K of the data set, respectively. A vector of dimension (N, 17, 2) is passed through the fully connected layer and then a fixed layer to finally obtain a classification logic output class logits2 of dimension (N, K), wherein the fixed layer is composed of a Batch Norm layer, a ReLU layer and a Dropout layer.

2. The multi-modal feature fusion method for person interaction detection based on a bipartite graph structure according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4-1: Project the candidate boxes corresponding to the people and object instances onto the Features map to generate corresponding features; Step 4-2: Represent the features of people and objects in the form of nodes in the bipartite graph; Step 4-3: For the generated n nodes, place the nodes with the class of people at the top, and obtain the features of the first n_h nodes, i.e. the features of the person nodes; Step 4-4: The remaining n - n_h nodes are object nodes, and then the person nodes and object nodes are matched into pairs one by one.

3. The multi-modal feature fusion method for person interaction detection based on a bipartite graph structure according to claim 1, characterized in that: The step 5 performs message passing on the features of the pair of people to eliminate negative pairs of people, and then performs interaction reasoning to obtain classification logic output class logits1, comprising the following steps: Step 5-1: The message passing function between the person and object nodes is represented as follows: Update of person node: ， ， Update of object node: ， ， wherein represents a LayerNorm operation, is a ReLU activation function, represents the adjacency weight between nodes in the adjacency matrix, is obtained by performing softmax on values with the same index i, is obtained by performing softmax on values with the same index , respectively represents row normalization and column normalization, and M is a message passing function with different parameters for different directions, represents the fusion of the object node and the edge feature of the person node and the object node at the current iteration number, and iteratively updates the feature of the person node, is the updated object node, is the edge feature, which is obtained by mapping the spatial feature through an MLP to a dimension of 1024, respectively represents the visual feature of the person and the object, which is also processed through an MLP to a dimension of 1024; Step 5-2: The visual features and spatial features of the human and object are fused and processed by an MLP to obtain a classification logic output class_logits1.

4. The multi-modal feature fusion method for person interaction detection based on a bipartite graph structure according to claim 1, characterized in that: In step 7, the classification logic output class_logits1 obtained in step 5 and the classification logic output class_logits2 obtained in step 6 are fused, specifically including the following steps: Step 7-1: The two logic outputs obtained in steps 5 and 6 are fused; Step 7-2: A focal loss is introduced as a binary classification loss function in the training process, which reduces the weight of a large number of negative samples in the training, and the formula of the focal loss function is as follows: ， wherein is a probability score for a certain action class, is a binary label, and is a hyperparameter, used to balance the ratio of positive and negative samples, used to regulate the rate of simple sample weight reduction.

Images