A person interaction detection method, system, device and medium based on contrast learning and graph convolutional neural network
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIDIAN UNIV
- Filing Date
- 2024-05-11
- Publication Date
- 2026-07-21
Smart Images
Figure CN118397364B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of human interaction detection technology, and specifically relates to a human interaction detection method, system, device and medium based on contrastive learning and graph convolutional neural networks. Background Technology
[0002] Human-object interaction (HOI) detection aims to solve the problem of interactions between people and objects. By studying HOI detection, we can better understand how people interact with the world, enabling machines to observe and understand their surroundings and make rapid decisions like humans. This can promote the development of technologies such as video surveillance and service robots. Unlike common tasks such as object detection and action recognition, HOI detection requires a higher level of visual understanding. HOI detection aims to locate pairs of human and object instances and identify the interactions between them, typically generating HOI predictions in the form of <human, object, interaction> triples.
[0003] Patent application number [202211512924.0] discloses a human interaction detection method based on a bipartite graph structure. This method integrates features from multiple modalities, extracts key points of people in the image through pose estimation, and inputs them into a key point feature processing network module to obtain pose information. Then, the extracted visual features and spatial features are fused and input into an inference network to obtain classification logic output, and a classification logic output is obtained through the key point feature processing network. Finally, the logic outputs of the two are fused to obtain the final classification result.
[0004] While the existing technical solutions described above can make full use of the various information present in the image to obtain a more comprehensive feature representation and make the final feature information richer, they do not solve the long-tail problem of data. For a small number of instances, simply relying on expanding feature information is still insufficient to enable them to be adequately trained.
[0005] Patent application number [202311084633.0] discloses a cascaded constraint decoding method for character interaction detection based on suggestive guided queries. This method fuses visual query vectors, spatial query vectors, and semantic query vectors to obtain a final fused query vector. It then uses a main branch decoder and a suppression branch decoder to decode the fused query vector, performs interaction perception fusion on the outputs of the two branches, and finally passes it through a multilayer perceptron to obtain the interaction detection result. This method can uncover deeper levels of interactive representation and improve the model's generalization ability and performance.
[0006] As mentioned above, although the existing technical solutions integrate vectors of multiple dimensions and expand the vector features used for classification prediction, adding additional feature information is still insufficient for the model to fully learn the features of the category for those few interaction categories. It also increases the computational cost of the entire network and affects the computational performance of the model.
[0007] In summary, the existing technology currently has the following technical problems:
[0008] 1. Existing technologies do not take into account the long-tail distribution of data when training models, resulting in insufficient learning of rare instances and thus affecting model performance;
[0009] 2. Existing technologies attempt to improve the performance of classification methods from various angles, but they generally involve relatively complex designs, resulting in low computational efficiency and relatively long training times. Summary of the Invention
[0010] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device and medium for detecting human interaction based on contrastive learning and graph convolutional neural networks. By obtaining similar instances of the current image instance from contrastive learning, the number of instance sets during the training process is expanded. At the same time, the classification prediction task is transformed into an edge classification task in a graph structure. The present invention has the characteristics of simplifying the task difficulty and improving the performance of interactive behavior prediction.
[0011] A method for detecting human interaction based on contrastive learning and graph convolutional neural networks includes the following steps:
[0012] Step 1: Construct a deep neural network;
[0013] Step 2: Initialize the deep neural network constructed in Step 1;
[0014] Step 3: Merge the human interaction detection image dataset with the bounding box information and category of the instances predicted by the object detector to generate a new image set, which includes a new training set and a test set;
[0015] Step 4: Use the new training set obtained in Step 3 to train the deep neural network initialized in Step 2.
[0016] Step 5: Input the new test set obtained in Step 3 into the deep neural network trained in Step 4 to perform classification prediction and obtain the test accuracy of the test set.
[0017] Step 1 specifically includes:
[0018] Step 1.1: Construct a six-layer feature extraction network to extract features from instances. Its structure is as follows: input layer → first convolutional layer → first residual convolutional layer → second residual convolutional layer → third residual convolutional layer → fourth residual convolutional layer.
[0019] Step 1.2: Construct a four-layer projection head network to perform dimensionality reduction on the instance features extracted in Step 1.1, and obtain the dimensionality-reduced instance features. Its structure is as follows: 1st fully connected layer → 2nd fully connected layer → 3rd fully connected layer → 4th fully connected layer.
[0020] Step 1.3: Construct a three-layer spatial information pairing network to obtain pairing information between human instances and object instances in the image. The first two layers are parallel structures, with the structure as follows: parallel first and second fully connected layers → third fully connected layer.
[0021] Step 1.4: Construct a five-layer edge classification graph convolutional neural network. Input the instance features after dimensionality reduction obtained in Step 1.2 and the pairing information of human instances and object instances in the image obtained in Step 1.3 into the constructed edge classification graph convolutional neural network for edge classification tasks to obtain the final interactive prediction results. The structure of the edge classification graph convolutional neural network is as follows: 1st residual graph convolutional layer → 2nd residual graph convolutional layer → 3rd residual graph convolutional layer → 4th residual graph convolutional layer → 1st fully connected layer.
[0022] The specific method for step 2 is as follows:
[0023] Step 2.1: Randomly select a number in the range (0, 1) and use the selected number as the initial weight of each node in the deep neural network;
[0024] Step 2.2: Use the initial weight of each node as the weight of each node in the deep neural network during the first iteration.
[0025] The character interaction detection image dataset in step 3 includes a training set and a test set. For all instances in the training set and the test set, there are labeled bounding box information and their corresponding classification labels. The images in the training set and the test set of the character interaction detection image dataset are processed by the object detector to predict the bounding box information and their categories of the instances. The detected instance bounding boxes and their categories are merged with the dataset to generate new training sets and test sets.
[0026] The specific method for step 4 is as follows:
[0027] Step 4.1: Randomly select R images from the new training set generated in step 3, where R is the batch size of the data. Input the randomly selected R images into the feature extraction network in step 1.1 to obtain the backbone features.
[0028] Step 4.2: Based on the size of the R images randomly selected in Step 4.1 and the size of the instance features extracted by the feature extraction network in Step 1.1, the bounding box information of the instance is scaled to obtain the scaled-down instance features;
[0029] Step 4.3: The reduced instance features obtained in Step 4.2 are compared and learned by the projection head network in Step 1.2 to calculate similarity and obtain similar instances;
[0030] Step 4.4: Based on the similar instances obtained in Step 4.3, calculate the contrast loss value for correctly identifying similar instances in the other images besides the current image in the R images in Step 4.1;
[0031] Step 4.5: Input the instance features and spatial features of the current image obtained in Step 4.2 into the first and second fully connected layers of the spatial information pairing network in Step 1.3, respectively, to obtain the paired information of the current image instance;
[0032] Step 4.6: Construct an adjacency matrix of the graph structure based on the instances in the current image, the similar instances obtained in step 4.3, and the pairwise information of the current image instances obtained in step 4.5.
[0033] Step 4.7: The adjacency matrix obtained in step 4.6 and the instance features obtained in step 4.2 are then passed through the third fully connected layer in step 1.3. The resulting features are then fed into the first four residual graph convolutional layers of the edge classification graph convolutional neural network in step 1.4 to obtain the final human body features and object features used for classification prediction.
[0034] Step 4.8: The human body features and object features used for classification prediction in Step 4.7 are concatenated pairwise and then fed into the first fully connected layer in Step 1.4 for edge classification prediction. Finally, the edge classification prediction labels of all edges in R images are output.
[0035] Step 4.9: Calculate the cross-entropy loss value for each instance of the R images in Step 4.1 that is correctly classified based on the edge classification prediction labels obtained in Step 4.8.
[0036] Step 4.10: Using the contrast loss value calculated in Step 4.4 and the cross-entropy loss value calculated in Step 4.9, take the partial derivative with respect to the weights of each node in the deep neural network after initialization in Step 2 to obtain the gradient value of the weights of each node in the deep neural network.
[0037] Step 4.11: Using the backpropagation algorithm, update all parameters in the deep neural network initialized in Step 2 based on the gradient values of each node calculated in Step 4.10, and obtain the updated deep neural network.
[0038] Step 5 specifically includes:
[0039] Step 5.1: Input the new test set generated in step 3 into the deep neural network updated in step 4.11, output the predicted classification label corresponding to each image, and calculate the accuracy of the deep neural network on the test set.
[0040] Step 5.2: Determine whether the accuracy in Step 5.1 is still increasing. If it is no longer increasing, the trained deep neural network is obtained; otherwise, proceed to Step 4.1.
[0041] The specific method for step 4.3 is as follows:
[0042] Calculate the Euclidean distance between the current image and instances in other images from the same batch, and determine whether the two instances are similar based on the value of the Euclidean distance:
[0043]
[0044] Where X and Y represent instances in the current image and other images in the same batch, respectively, x i y i Let X and Y represent the i-th feature values of samples X and Y respectively, and N represent the total number of features;
[0045] The specific method for step 4.6 is as follows:
[0046] Let the interaction pair matrix of instances in the current image be . N c Human instances in the current image and object instances The sum, that is The similarity matrix obtained by contrastive learning is Where N o The adjacency matrix is calculated as the total number of instances in the same batch of images other than the one currently predicting the interaction behavior. Where N is the sum of the number of all instances of all images in this batch, i.e., N = N c +N o Generate pairwise relation matrices between instances in the currently predicted image and instances in other images based on pairs_matrix and sim_matrix.
[0047] PairsCtoO=pairs_matrix×sim_matrix
[0048] Finally, the pair_matrix and PairsCtoO are combined to generate the adjacency matrix.
[0049] This invention also provides a character interaction detection system based on contrastive learning and graph convolutional neural networks, comprising:
[0050] The model building module is used to build deep neural networks and generate the entire deep neural network.
[0051] The deep neural network initialization module is used to initialize the deep neural network.
[0052] The data acquisition module is used to merge the human interaction detection image dataset with the bounding box information and category of the instances predicted by the object detector to generate a new image set, which includes a new training set and a test set.
[0053] The deep neural network training module is used to train the initialized deep neural network using a new training set.
[0054] The deep neural network classification and prediction module is used to input new test sets into the trained deep neural network, perform classification and prediction, and obtain the test accuracy of the test set.
[0055] This invention also provides a human interaction detection device based on contrastive learning and graph convolutional neural networks, comprising:
[0056] Memory: A computer program that stores the above-mentioned method for detecting human interaction based on contrastive learning and graph convolutional neural networks, and is a computer-readable device;
[0057] Processor: Used to implement the human interaction detection method based on contrastive learning and graph convolutional neural networks when executing the computer program.
[0058] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned method for detecting human interaction based on contrastive learning and graph convolutional neural networks.
[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0060] 1. This invention utilizes the idea of contrastive learning. In step 4.3, the contrastive learning design is incorporated into the model. Based on human and object instances in other images, the similarity between them and instances in the current image is calculated to determine whether the instance can be used as an expanded instance, thereby greatly increasing the number of instances participating in training.
[0061] 2. This invention utilizes the loss function designed for the contrastive learning phase in step 4.4. For each instance, it compares it with other instances of the same category and with instances of all other categories. This ensures that instances of each category are sufficiently compared with samples from other categories, thereby better learning category-specific feature representations and ensuring that each class contributes approximately to the model's learning during training.
[0062] 3. In step 4.6 of this invention, a graph structure is constructed based on the human body and objects in the image and whether there is interaction between them. A graph convolutional neural network is used as the classification prediction model, which transforms the classification task into an edge classification task in a graph neural network, thus simplifying the difficulty of the task.
[0063] 4. In steps 4.5 and 4.6, the present invention uses a spatial feature pairing algorithm to construct the adjacency matrix. First, people and object instances in the current image are paired, and then a graph-structured adjacency matrix is constructed, which enhances the accuracy of the adjacency matrix and provides a guarantee for the subsequent learning of the graph convolutional neural network.
[0064] In summary, this invention utilizes contrastive learning to obtain similar instances in other images, transforming the interactive behavior classification task into an edge classification task in a graph structure. This approach has the advantages of expanding the number of instances, enabling the network model to learn more effectively, simplifying the task difficulty, and shortening the training time. Attached Figure Description
[0065] Figure 1 This is a flowchart of the present invention.
[0066] Figure 2 This is the complete network model diagram of the present invention.
[0067] Figure 3 This is a model diagram of the projection head network of the present invention.
[0068] Figure 4 This is a flowchart of the deep neural network training stage of the present invention.
[0069] Figure 5 This is a model diagram of the edge classification graph convolutional neural network of the present invention.
[0070] Figure 6 This is a residual graph convolution structure diagram of the present invention. Detailed Implementation
[0071] The invention will now be further described with reference to the accompanying drawings.
[0072] The approach to achieving the objective of this invention is as follows: First, the image dataset is fed into the object detector to obtain predictions of the bounding boxes for each instance, preparing for subsequent inputs. Then, a deep neural network model is constructed for contrastive learning and graph convolution calculation. The model's input consists of two parts: one is the image currently being identified for interactive behavior recognition, and the other is other images from the same batch. First, contrastive learning is performed to calculate the similarity of instances in the two input images, obtaining similar instances in the current image and generating an adjacency matrix. Then, the instances are transformed into a graph structure and fed into the downstream graph neural network classification module for interactive behavior classification, resulting in a trained deep learning network. During prediction, the image is directly fed into the trained network model. The final model will then outline the people and objects with interactive behaviors and list the interactive behaviors between them.
[0073] Reference Figure 1 The implementation steps of the present invention will be described in detail below.
[0074] A method for detecting human interaction based on contrastive learning and graph convolutional neural networks includes the following steps:
[0075] Step 1: Construct a deep neural network, which consists of a feature extraction network, a projection head network, a spatial information pairing network, and an edge classification graph convolutional neural network.
[0076] Reference Figure 2 The deep neural network constructed in this invention will be described in further detail below.
[0077] Step 1.1: Construct a six-layer feature extraction network to extract the backbone features of the image. Its structure is as follows: input layer → first convolutional layer → first residual convolutional layer → second residual convolutional layer → third residual convolutional layer → fourth residual convolutional layer.
[0078] In step 1.1, the parameters of each layer of the feature extraction network are set as follows: the total number of feature maps in the input layer is set to 3; the total number of feature maps in the first convolutional layer is set to 64, the kernel size is set to 7×7 pixels, and the stride size is set to 2 pixels; the total number of feature maps in the first residual convolutional layer is set to 256, the kernel size is set to 3×3 pixels, and the stride size is set to 2 pixels; the total number of feature maps in the second residual convolutional layer is set to 512, the kernel size is set to 3×3 pixels, and the stride size is set to 2 pixels; the total number of feature maps in the third residual convolutional layer is set to 1024, the kernel size is set to 3×3 pixels, and the stride size is set to 2 pixels; the total number of feature maps in the fourth residual convolutional layer is set to 2048, the kernel size is set to 3×3 pixels, and the stride size is set to 2 pixels.
[0079] Step 1.2: Construct a four-layer projection head network, with the following structure: 1st fully connected layer → 2nd fully connected layer → 3rd fully connected layer → 4th fully connected layer;
[0080] Reference Figure 3 The projection head network will be further described.
[0081] In step 1.2, the projection head network parameters are set as follows: the total number of feature maps for the first fully connected layer is set to 1024; the total number of feature maps for the second fully connected layer is set to 2048; the total number of feature maps for the third fully connected layer is set to 2048; and the total number of feature maps for the fourth fully connected layer is set to 1024.
[0082] Step 1.3: Construct a three-layer spatial information pairing network, where the first two layers are parallel structures, and their structures are as follows: parallel first and second fully connected layers → third fully connected layer;
[0083] In step 1.3, the spatial information pairing network parameters are set as follows: the output nodes of the first fully connected layer are set to 1024, the output nodes of the second fully connected layer are set to 1024, and the number of nodes in the third fully connected layer is set to N. 2 , where N is the total number of people and object instances in the current image.
[0084] Step 1.4: Construct a five-layer edge classification graph convolutional neural network, whose structure is as follows: 1st residual graph convolutional layer → 2nd residual graph convolutional layer → 3rd residual graph convolutional layer → 4th residual graph convolutional layer → 1st fully connected layer;
[0085] Reference Figure 5 We will further describe the edge classification graph convolutional neural network.
[0086] In step 1.4, the parameters of the edge classification graph convolutional neural network are set as follows: the total number of feature maps in the first residual graph convolutional layer is set to 512; the total number of feature maps in the second residual graph convolutional layer is set to 1024; the total number of feature maps in the third residual graph convolutional layer is set to 2048; the total number of feature maps in the fourth residual graph convolutional layer is set to 1024; the output node of the first fully connected layer is set to the number of categories; and the residual graph convolutional neural layer structure is as follows. Figure 6 .
[0087] Step 2: Initialize the deep neural network constructed in Step 1;
[0088] The specific method for step 2 is as follows:
[0089] Step 2.1: Randomly select a number in the range (0, 1) and use the selected number as the initial weight of each node in the deep neural network;
[0090] Step 2.2: Use the initial weight of each node as the weight of each node in the deep neural network during the first iteration.
[0091] Step 3: Merge the HICO-DET dataset with the bounding box information and category of each instance predicted by the object detector to generate a new image set, which includes a new training set and a test set;
[0092] The HICO-DET dataset includes a training set and a test set. For all instances in the training set and the test set, there are labeled bounding box information and their corresponding classification labels. The images in the training set and the test set of the HICO-DET dataset are processed by the object detector to predict the bounding box information and their categories of the instances. The predicted instance bounding boxes and their categories are then merged with the HICO-DET dataset to generate new training sets and test sets.
[0093] Step 4: Use the new training set obtained in Step 3 to train the deep neural network initialized in Step 2.
[0094] Reference Figure 4 The training process of the deep neural network of the present invention will be described in further detail below.
[0095] Step 4.1: Randomly select 4 images from the new training set generated in step 3. Input the 4 randomly selected images into the feature extraction network in step 1.1 to obtain backbone features. Use the deep neural network initialized in step 2 as the deep neural network for subsequent weight updates.
[0096] Step 4.2: Scale the person bounding box and the object bounding box according to the size of the four images randomly selected in Step 4.1 and the size of the backbone features extracted by the feature extraction network in Step 1.1.
[0097] The size of the bounding box is proportionally reduced based on the size of each of the four images and the size of the skeletal features, according to the following formula:
[0098]
[0099]
[0100] Among them, image i Represents the i-th image, feature i Ori_boses represents the backbone features corresponding to the i-th image. iLet X represent the initial bounding box size in the i-th image, where X∈{human, object, interaction} indicates that the bounding box belongs to a person, an object, or an interaction region, respectively. The subscripts x and y of the image and backbone features represent the width and height, respectively. Finally, new bounding box information with proportional reduction is generated, and then the backbone network is cropped based on this bounding box information.
[0101] Step 4.3: The reduced features obtained in Step 4.2 are compared and learned by the projection head network in Step 1.2 to calculate similarity and obtain similar instances;
[0102] Calculate the Euclidean distance between the current image and three other images in the same batch, and determine whether the two instances are similar based on the value of the Euclidean distance:
[0103]
[0104] Where X and Y represent instances in the current image and the other three images in the same batch, respectively, x i y i Let X and Y represent the i-th feature values of samples X and Y respectively, and N represent the total number of features, which is 1024 in this case.
[0105] Step 4.4: Based on the similar instances obtained in Step 4.3, calculate the contrast loss value for correctly identifying similar instances in the other 3 images (excluding the current image) out of the 4 images in Step 4.1.
[0106] The specific formula for the contrast loss function is shown below:
[0107]
[0108] Where B represents all categories, B y B represents all categories contained in this batch of images. j Let y represent the j-th category, |·| represent the quantity, and y j Let z represent all samples in the j-th category. i Let A represent the feature matrix of the i-th sample, A\B represent the difference set, i.e., the elements that belong to set A but not to set B. Here, A\B represents all instances excluding the instance currently being calculated, and D(x,y) represents the Euclidean distance between feature vectors x and y.
[0109] Step 4.5: Input the instance features and spatial features of the current image obtained in Step 4.2 into the first and second fully connected layers of the spatial information pairing network in Step 1.3 respectively to obtain the pairing information of the current image instance;
[0110] The spatial information used in the spatial pairing algorithm mainly includes: the center coordinates of the bounding boxes, width, height, aspect ratio, and region. This information is generated by normalizing the corresponding dimensions of the image. In addition, to represent the pairwise relationship between the human body and objects, this invention also uses the IoU size formed by the bounding boxes of the human body and objects, and a directional encoding based on [(ReLU(dx)),(ReLU(-dx)),(ReLU(dy),(ReLU(-dx))], where dx and dy are the differences between the center coordinates of the human body bounding box and the object bounding box. Furthermore, the spatial information and its logarithm are concatenated, and a minimal constant ε is added to ensure numerical stability, generating the final spatial features:
[0111]
[0112] Where p represents the spatial characteristics used, and ε represents a minimal constant to ensure numerical stability. This indicates a splicing operation.
[0113] Based on instance features and spatial features, the pairing information of instances in the current image is finally generated as pairs_matrix;
[0114] Step 4.6: Based on the instances in the current image, the similar instances obtained in step 4.3, and the pairwise information of the current image instances obtained in step 4.5, construct the adjacency matrix of the graph structure as the necessary input information for the subsequent graph convolutional neural network.
[0115] Let the interaction pair matrix of instances in the current image be . N c Human instances in the current image and object instances The sum, that is The similarity matrix obtained by contrastive learning is Where N o Given the total number of instances in the same batch of images other than the currently predicted interaction, the adjacency matrix to be calculated is adj∈R. N×N Where N is the sum of the number of all instances of all images in this batch, i.e., N = N c +N o Meanwhile, since an undirected graph structure is used, the adjacency matrix adj is a symmetric matrix. Now, the adjacency matrix is generated based on the existing conditions.
[0116] First, based on pairs_matrix and sim_matrix, generate a pairwise relation matrix between instances in the currently being predicted image (hereinafter referred to as the current image) and instances in other images.
[0117] The calculation formula is as follows:
[0118] PairsCtoO=pairs_matrix×sim_matrix
[0119] The currently obtained matrices pairs_matrix and pairsCtoO are components of the final adjacency matrix adj. adj can be divided into four parts, with the two obtained matrices occupying two of them. Specifically, pairs_matrix is the upper left part, representing the adjacency information in the current image, and pairsCtoO is the upper right part, representing the adjacency information between the current image and other images. Since adj is a symmetric matrix, the lower left and upper right parts have the same meaning and can be mapped from the symmetrical position of the upper right corner. The lower right part represents the adjacency information of each instance in other images. Here, the pairing conditions in other images are not considered, and they are treated as all unpaired, that is, there is no disconnection between instances, and zero matrix filling is used.
[0120] Step 4.7: The adjacency matrix obtained in step 4.6 and the instance features obtained in step 4.2 are then passed through the third fully connected layer in step 1.3. The resulting features are then fed into the first four residual graph convolutional layers of the edge classification graph convolutional neural network in step 1.4 to obtain the final human body features and object features used for classification prediction.
[0121] Step 4.8: The human body features and object features used for classification prediction in Step 4.7 are concatenated pairwise and then fed into the first fully connected layer in Step 1.4 for edge classification prediction. Finally, the edge classification prediction labels of all edges in the four images are output.
[0122] Step 4.9: Calculate the cross-entropy loss value for each instance correctly classified in the four images in Step 4.1 based on the edge classification prediction labels obtained in Step 4.8.
[0123] The calculation formula is as follows:
[0124]
[0125] Among them, y mn This indicates whether the m-th sample belongs to the n-th category. This represents the model's predicted probability that the m-th sample belongs to the n-th category.
[0126] Step 4.10: Using the contrast loss value calculated in Step 4.4 and the cross-entropy loss value calculated in Step 4.9, take the partial derivative with respect to the weights of each node in the deep neural network after initialization in Step 2 to obtain the gradient value of the weights of each node in the deep neural network.
[0127] Step 4.11: Using the backpropagation algorithm, update all parameters in the deep neural network initialized in Step 2 based on the gradient values of each node calculated in Step 4.10, and obtain the updated deep neural network.
[0128] The backpropagation algorithm process includes:
[0129] Update the weights of each node in the deep neural network initialized in step 2 according to the following formula:
[0130]
[0131] in, Let g represent the updated weights of the c-th node in the deep neural network after initialization in step 2. c Let represent the weights of the c-th node in the deep neural network before the update in step 2, and β represent the learning rate of the deep neural network after step 2. This represents the gradient value of the weight of the c-th node in the deep neural network after initialization in step 2, calculated in step 4.10.
[0132] Step 5: Input the new test set obtained in Step 3 into the deep neural network trained in Step 4 to perform classification prediction and obtain the test accuracy of the test set.
[0133] Step 5.1: Input the new test set generated in step 3 into the deep neural network updated in step 4.11, output the predicted classification label corresponding to each image, and calculate the accuracy of the deep neural network on the test set.
[0134] The calculation formula is as follows:
[0135] First, calculate the accuracy (AP) for each class in the new test set generated in step 3:
[0136]
[0137]
[0138] Among them, Tp k FP represents the number of correctly predicted class labels in the k-th class of the new test set generated in step 3, as output by the predicted class labels in this step. k FN represents the number of mispredicted class labels in the k-th class of the new test set generated in step 3, which were incorrectly predicted by the predicted class labels output in this step. k This represents the number of unpredicted true labels of class k in the new test set generated in step 3. Precision here... k and Recall kBoth are sequences of length equal to the number of predicted class labels for the k-th class, and each Precision corresponds to one Recall, because Precision and Recall are calculated once for each predicted class label determined to be TP / FP / FN.
[0139] Subsequently, an 11-point sampling method was used, dividing the recall into 11 points: {0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}. The accuracy P of the 11-point recall was obtained by taking the maximum precision within the interval from these points to 1 using the following formula:
[0140]
[0141] Where i∈{0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1}, and These represent the j-th Recall and Percision in the k-th class label of the new test set generated in step 3, respectively.
[0142]
[0143] in, Let represent the accuracy of the i-th predicted classification label in the k-th class of the new test set generated in step 3, where i∈{0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1}.
[0144] Finally, the average of the accuracy scores for all classes in the new test set generated in step 3 is calculated to obtain mAP:
[0145]
[0146] Where m represents the total number of categories in the new test set generated in step 3, AP i This represents the accuracy of the i-th category.
[0147] Step 5.2: Determine whether the accuracy in Step 5.1 is still increasing. If it is no longer increasing, the trained deep neural network is obtained; otherwise, proceed to Step 4.1.
[0148] This invention also provides a character interaction detection system based on contrastive learning and graph convolutional neural networks, comprising:
[0149] The model building module is used to build the deep neural network in step 1 and generate the entire deep neural network.
[0150] The deep neural network initialization module is used to initialize the deep neural network constructed in step 1 in step 2.
[0151] The data acquisition module is used to merge the HICO-DET dataset with the bounding box information and category of the instances predicted by the object detector in step 3 to generate a new image set, which includes a new training set and a test set.
[0152] The deep neural network training module is used in step 4 to train the deep neural network initialized in step 2 using the new training set obtained in step 3.
[0153] The deep neural network classification and prediction module is used to input the new test set obtained in step 3 into the deep neural network trained in step 4 in step 5, perform classification and prediction, and obtain the test accuracy of the test set.
[0154] This invention also provides a human interaction detection device based on contrastive learning and graph convolutional neural networks, comprising:
[0155] Memory: A computer program that stores the above-mentioned method for detecting human interaction based on contrastive learning and graph convolutional neural networks, and is a computer-readable device;
[0156] Processor: Used to implement the human interaction detection method based on contrastive learning and graph convolutional neural networks when executing the computer program.
[0157] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the aforementioned method for detecting human interaction based on contrastive learning and graph convolutional neural networks.
[0158] The effects of this invention are further illustrated by the following simulation experiments.
[0159] 1. Simulation experimental conditions:
[0160] The hardware platform for the simulation experiment of this invention is: Intel(R) Core(TM) i7-8700K CPU, 3.70GHz*12, NVIDIA GeForce GTX 3090Ti, and 64G of memory.
[0161] The software platform for the simulation experiments of this invention is PyTorch.
[0162] 2. Simulation content and result analysis:
[0163] The simulation experiment of this invention uses a labeled human interaction detection dataset to train a deep neural network, and then inputs the original image into the deep learning network for category judgment.
[0164] The simulation experiment of this invention uses the HICO-DET dataset for character interaction detection. This dataset contains 47,179 interaction images, with 37,633 images in the training set and 9,546 images in the test set. Furthermore, HICO-DET divides the dataset into three types: "Full," "Rare," and "None Rare." "Full" includes all 600 HOI categories; "Rare" contains only 138 categories with fewer than 10 training instances each; and "None Rare" contains the remaining 462 categories with 10 or more training instances each. The deep neural network is trained iteratively 20 times using the training set to obtain the trained deep neural network. Then, the test sample set is used to test the trained deep neural network to obtain the recognition accuracy of this invention.
[0165] The recognition accuracy of the method of this invention is compared with that of existing methods on the same dataset. The recognition accuracy results of the several methods after training are shown in Table 1:
[0166] Table 1 Comparison of Character Interaction Detection Classification Results
[0167] PastaNet 22.65 21.17 23.09 ATL 23.67 17.64 25.47 SKGHOI 26.95 21.28 28.56 QPIC 29.07 21.85 31.23 SSRT 30.35 25.42 31.83 SCG 31.33 24.72 33.31 MSTR 31.17 25.31 32.92 UPT <![CDATA[ 31.66 ]]> <![CDATA[ 25.94 ]]> <![CDATA[ 33.36 ]]> Ours 32.63 27.89 33.42
[0168] As shown in Table 1, the character interaction detection method based on contrastive learning and graph convolutional neural networks proposed in this invention achieves relatively good results. Specifically, when using the ResNet-50 backbone network pre-trained on MS COCO, it can be observed that the model of this invention exhibits good performance in all three modes. In the "Full" mode, it outperforms the second-best method by 0.97%, and in the "Rare" mode, it outperforms by 1.95%, showing significant results. In the "None-Rare" mode, it is also comparable to the current state-of-the-art model. Since the number of instances for each category is less than 10 images in the "Rare" mode, it indicates that the model proposed in this invention has achieved certain success in solving the long-tail problem and has important practical significance.
[0169] The names of the various comparison methods are as follows:
[0170] PastaNet:[Yong-Lu Li,Liang Xu,Xinpeng Liu,Xijie Huang,Yue Xu,ShiyiWang,Hao-Shu Fang,Ze Ma,Mingyang Chen,and Cewu Lu.Pastanet:Toward humanactivity knowledge engine.In Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition,pages 382–391,2020.];
[0171] ATL:[Hou Z,Yu B,Qiao Y,et al.Affordance transfer learning for human-object interaction detection[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2021:495-504.];
[0172] SKGHOI:[Zhu L,Lan Q,Velasquez A,et al.SKGHOI:Spatial-SemanticKnowledge Graph for Human-Object Interaction Detection[J].arXiv preprintarXiv:2303.04253,2023.];
[0173] QPIC:[Tamura M,Ohashi H,Yoshinaga T.QPIC:Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information:,10.48550 / arXiv.2103.05399[P].2021.];
[0174] SSRT:[Zhong
[0175] SCG:[Zhang FZ,Campbell D,Gould S.Spatially Conditioned GraphsforDetecting Human-Object Interactions[C] / / InternationalConference onComputerVision.2021.];
[0176] MSTR:[Kim B, Mun J, On KW, et al.Mstr:Multi-scale transformer for end-to-end human-object interaction detection[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2022:19578-19587.];
[0177] UPT: [Zhang FZ, Campbell D, Gould S. Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition.2022:20104-20112.];
[0178] like Figure 1 The diagram shown is a flowchart of the present invention. The present invention consists of five steps: constructing a deep neural network, initializing the deep neural network, generating an image set, training the deep neural network, and performing classification prediction on a new test set.
[0179] like Figure 2 The diagram shown is a complete network model of the present invention. It can be observed that the model is divided into three main parts: the backbone feature extraction part, the projection head part, and the graph convolutional neural network classification processing part.
[0180] like Figure 3 The diagram shown is a model of the projection head network of the present invention. It can be observed that this part is mainly composed of a fully connected layer and a multilayer perceptron model. The two fully connected networks are used for dimensionality reduction to reduce the number of parameters and computational cost. The MLP contains two layers, both of which calculate the values of all nodes in the previous layer.
[0181] like Figure 4 The diagram shows a flowchart of the neural network training stage of this invention. It can be observed that the network training stage of this invention consists of ten steps: acquiring backbone features, further feature extraction using the projection head, calculating contrastive loss, generating an adjacency matrix, graph convolutional neural network classification and prediction, calculating cross-entropy loss, calculating test set accuracy, determining whether the test set accuracy is still increasing, and if it is still increasing, calculating the gradient of network nodes and updating the weights of network nodes; if it is no longer increasing, the trained network model is obtained.
[0182] like Figure 5 The diagram shown is a model of the edge classification graph convolutional neural network of the present invention. It can be observed that this part consists of two fully connected layers and four residual graph convolutional layers. The first fully connected layer is used for dimensionality reduction, and the second fully connected layer acts as a classifier to output the final prediction result.
[0183] like Figure 6 The diagram shown is a model diagram of the residual graph convolutional neural module of the present invention. It can be observed that this part consists of three graph convolutional layers and ReLU and BN operations. The first fully connected layer is used for dimensionality reduction, the second fully connected layer is used as a classifier to output the final prediction result, and the corresponding elements of the outputs of the two graph convolutional layers are added together. GraphConv(C1C2) represents a graph convolutional layer with input dimension C1 and output dimension C2.
[0184] Application prospects of this invention:
[0185] First, human interaction detection is a problem of great practical significance. It is the key to automatically understanding the theme of an image and automatically describing the main content of the image. It can be applied to the field of video surveillance to classify and supervise human behavior.
[0186] This invention addresses a key issue in human interaction detection: the long-tail distribution of data. This represents an innovative and practically applicable step forward in both the current problem and methodological prospects. Its innovation lies in its pioneering use of contrastive learning to calculate the similarity between instances, obtaining similar instances and constructing an adjacency matrix. Then, it employs an edge-classification graph convolutional neural network to transform the classification prediction task into an edge classification task within a graph structure—a method not used in previous approaches. Its practical applicability is demonstrated by the fact that the proposed method can effectively predict objects that interact with humans and the interactions between them. The method avoids overly complex design, allowing for easy deployment of training and preparation. Furthermore, its high computational efficiency and short training time enable rapid training and subsequent deployment in real-world applications.
Claims
1. A method for detecting human interaction based on contrastive learning and graph convolutional neural networks, characterized in that, Includes the following steps: Step 1: Construct a deep neural network; Step 2: Initialize the deep neural network constructed in Step 1; Step 3: Merge the human interaction detection image dataset with the bounding box information and category of the instances predicted by the object detector to generate a new image set, which includes a new training set and a test set; Step 4: Use the new training set obtained in Step 3 to train the deep neural network initialized in Step 2. Step 5: Input the new test set obtained in Step 3 into the deep neural network trained in Step 4 to perform classification prediction and obtain the test accuracy of the test set. Step 1 specifically includes: Step 1.1: Construct a six-layer feature extraction network to extract features from instances. Its structure is as follows: input layer → first convolutional layer → first residual convolutional layer → second residual convolutional layer → third residual convolutional layer → fourth residual convolutional layer. Step 1.2: Construct a four-layer projection head network to perform dimensionality reduction on the instance features extracted in Step 1.1, and obtain the dimensionality-reduced instance features. Its structure is as follows: 1st fully connected layer → 2nd fully connected layer → 3rd fully connected layer → 4th fully connected layer. Step 1.3: Construct a three-layer spatial information pairing network to obtain pairing information between human instances and object instances in the image. The first two layers are parallel structures, with the structure as follows: parallel first and second fully connected layers → third fully connected layer. Step 1.4: Construct a five-layer edge classification graph convolutional neural network. Input the instance features after dimensionality reduction obtained in Step 1.2 and the pairing information of human instances and object instances in the image obtained in Step 1.3 into the constructed edge classification graph convolutional neural network for edge classification tasks to obtain the final interactive prediction results. The edge classification graph convolutional neural network structure is as follows: 1st residual graph convolutional layer → 2nd residual graph convolutional layer → 3rd residual graph convolutional layer → 4th residual graph convolutional layer → 1st fully connected layer; The specific method for step 4 is as follows: Step 4.1: Randomly select from the new training set generated in step 3. Zhang images, among which This refers to the batch size of the data, which will be randomly selected. After the image is input into the feature extraction network in step 1.1, the backbone features are obtained; Step 4.2: Based on the size of the R images randomly selected in Step 4.1 and the size of the instance features extracted by the feature extraction network in Step 1.1, the bounding box information of the instance is scaled to obtain the scaled-down instance features; Step 4.3: The reduced instance features obtained in Step 4.2 are compared and learned by the projection head network in Step 1.2 to calculate similarity and obtain similar instances; Step 4.4: Based on the similar instances obtained in Step 4.3, calculate the value obtained in Step 4.
1. The contrast loss value for correctly identifying similar instances in images other than the current image in a given image; Step 4.5: Input the instance features and spatial features of the current image obtained in Step 4.2 into the first and second fully connected layers of the spatial information pairing network in Step 1.3, respectively, to obtain the paired information of the current image instance; Step 4.6: Construct an adjacency matrix of the graph structure based on the instances in the current image, the similar instances obtained in step 4.3, and the pairwise information of the current image instances obtained in step 4.
5. The specific method for step 4.6 is as follows: Let the interaction pair matrix of instances in the current image be . , Human instances in the current image and object instances The sum, that is The similarity matrix obtained through contrastive learning is ,in The adjacency matrix is calculated as the total number of instances in the same batch of images other than the one currently predicting the interaction behavior. ,in It is the sum of the number of all instances of all images in this batch, i.e. ;according to and Generate a pairwise relation matrix between instances in the currently predicted image and instances in other images. : Ultimately and The combination generates an adjacency matrix; Step 4.7: The adjacency matrix obtained in step 4.6 and the instance features obtained in step 4.2 are then passed through the third fully connected layer in step 1.
3. The resulting features are then fed into the first four residual graph convolutional layers of the edge classification graph convolutional neural network in step 1.4 to obtain the final human body features and object features used for classification prediction. Step 4.8: The human features and object features used for classification prediction in Step 4.7 are concatenated pairwise and then fed into the first fully connected layer in Step 1.4 for edge classification prediction. Finally, the edge classification prediction labels of all edges in R images are output. Step 4.9: Calculate the values from step 4.1 based on the edge classification prediction labels obtained in step 4.
8. The cross-entropy loss value for each instance of an image correctly classified; Step 4.10: Using the contrast loss value calculated in Step 4.4 and the cross-entropy loss value calculated in Step 4.9, take the partial derivative with respect to the weights of each node in the deep neural network after initialization in Step 2 to obtain the gradient value of the weights of each node in the deep neural network. Step 4.11: Using the backpropagation algorithm, update all parameters in the deep neural network initialized in Step 2 based on the gradient values of each node calculated in Step 4.10, and obtain the updated deep neural network.
2. The method for detecting human interaction based on contrastive learning and graph convolutional neural networks according to claim 1, characterized in that, The specific method for step 2 is as follows: Step 2.1: Randomly select a number in the range (0, 1) and use the selected number as the initial weight of each node in the deep neural network; Step 2.2: Use the initial weight of each node as the weight of each node in the deep neural network during the first iteration.
3. The method for detecting human interaction based on contrastive learning and graph convolutional neural networks according to claim 1, characterized in that, The character interaction detection image dataset in step 3 includes a training set and a test set. For all instances in the training set and the test set, there are labeled bounding box information and their corresponding classification labels. The images in the training set and the test set of the character interaction detection image dataset are processed by the object detector to predict the bounding box information and their categories of the instances. The detected instance bounding boxes and their categories are merged with the dataset to generate new training sets and test sets.
4. The method for detecting human interaction based on contrastive learning and graph convolutional neural networks according to claim 1, characterized in that, Step 5 specifically includes: Step 5.1: Input the new test set generated in step 3 into the deep neural network updated in step 4.11, output the predicted classification label corresponding to each image, and calculate the accuracy of the deep neural network on the test set. Step 5.2: Determine whether the accuracy in step 5.1 is still increasing. If it is no longer increasing, the trained deep neural network is obtained; otherwise, proceed to step 4.
1.
5. The method for detecting human interaction based on contrastive learning and graph convolutional neural networks according to claim 1, characterized in that, The specific method for step 4.3 is as follows: Calculate the Euclidean distance between the current image and instances in other images from the same batch, and determine whether the two instances are similar based on the value of the Euclidean distance: in, These represent instances in the current image and other images in the same batch, respectively. Representing samples respectively The i-th eigenvalue, This represents the total number of features.
6. A character interaction detection system based on contrastive learning and graph convolutional neural networks, based on the method of any one of claims 1 to 5, characterized in that, include: The model building module is used to build deep neural networks and generate the entire deep neural network. The deep neural network initialization module is used to initialize the deep neural network. The data acquisition module is used to merge the human interaction detection image dataset with the bounding box information and category of the instances predicted by the object detector to generate a new image set, which includes a new training set and a test set. The deep neural network training module is used to train the initialized deep neural network using a new training set. The deep neural network classification and prediction module is used to input new test sets into the trained deep neural network, perform classification and prediction, and obtain the test accuracy of the test set.
7. A human interaction detection device based on contrastive learning and graph convolutional neural networks, characterized in that, include: Memory: A computer program for a human interaction detection method based on contrastive learning and graph convolutional neural networks as described in any one of claims 1-5, which is a computer-readable device; Processor: Used to implement the human interaction detection method based on contrastive learning and graph convolutional neural network as described in any one of claims 1-5 when executing the computer program.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the human interaction detection method based on contrastive learning and graph convolutional neural networks as described in any one of claims 1-5.